{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "15408e8c",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "\n",
    "This notebook is a brief, informal exploration of what happens when we *constrain* the answers a large language model can give — a technique often called \"structured outputs.\" Rather than letting the model respond freely (where it might hedge, caveat, or refuse to commit), we force it to choose from a predefined set of options, return a specific type (like a boolean, integer, or float), or stay within a numerical range.\n",
    "\n",
    "This constraint mechanism is powerful. In production systems, it's used to ensure valid JSON, enforce type safety, or guarantee responses fit a schema. But it also reveals something deeper: **when we remove the model's ability to hedge, we can observe its latent biases, default assumptions, and the sensitivity of its outputs to how questions are framed.**\n",
    "\n",
    "In principle, this technique could be used for serious statistical analysis — measuring response variability, detecting systematic biases, comparing models, or studying the effect of prompt engineering on constrained outputs. However, **this notebook is not that**. We use a sample size of just 10 calls per question (`n=10`), which is enough to illustrate the *kinds* of effects that emerge, but nowhere near enough to draw statistically significant conclusions. Consider this a qualitative tour, not a rigorous study.\n",
    "\n",
    "Our tool is simple:\n",
    "\n",
    "```python\n",
    "from functools import partial\n",
    "from collections import Counter\n",
    "from oa import constrained_answer\n",
    "from dol import Pipe\n",
    "\n",
    "ten_constrained_answers_counts = Pipe(\n",
    "    partial(constrained_answer, n=10),\n",
    "    Counter\n",
    ")\n",
    "```\n",
    "\n",
    "With this, we can ask questions, constrain the answers, and see how the distribution of responses shifts based on how we ask."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3836547e",
   "metadata": {},
   "source": [
    "### Initial Experiments\n",
    "\n",
    "Let's start with some basic examples to see how the tool works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce1c3520",
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "from collections import Counter\n",
    "from oa import constrained_answer\n",
    "from dol import Pipe \n",
    "\n",
    "ten_constrained_answers_counts = Pipe(\n",
    "     partial(constrained_answer, model=\"gpt-4o-mini\", n=10),\n",
    "     Counter\n",
    " )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8836431c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({30: 4, 20: 3, 15: 2, 10: 1})"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ten_constrained_answers_counts('how tall is a tree?', float)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdcf6b45",
   "metadata": {},
   "source": [
    "Asking for a numeric answer to an ambiguous question. Notice the spread in the responses — the model is clearly uncertain, but forced to commit to specific values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aabb059b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Night owl': 6, 'Morning person': 4})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# See that you actually get different answers with some questions\n",
    "ten_constrained_answers_counts(\n",
    "    \"When you have to choose, are you more of a morning person or a night owl?\",\n",
    "    [\"Morning person\", \"Night owl\"], n=10\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2d8a2e4",
   "metadata": {},
   "source": [
    "### Exploring Personal Preference Questions\n",
    "\n",
    "These questions don't have \"right\" answers — they're about preferences. How does the model respond?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ede4277",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Coffee': 6, 'Tea': 4})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ten_constrained_answers_counts(\n",
    "    \"Which appeals to you more: coffee or tea?\", \n",
    "    [\"Coffee\", \"Tea\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cbbc9c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Beach': 10})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# but then, sometimes, the question is consistent -- there's a bias\n",
    "ten_constrained_answers_counts(\n",
    "    \"On vacation, would you rather spend time at the beach or in the mountains?\",   \n",
    "    [\"Beach\", \"Mountains\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b0e5d20",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Physical books': 10})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ten_constrained_answers_counts(\n",
    "    \"Do you prefer reading physical books or ebooks?\",\n",
    "    [\"Physical books\", \"Ebooks\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1dd41326",
   "metadata": {},
   "source": [
    "In these last few examples, we see some questions produce varied responses while others show clear biases. The variance itself is informative."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cda4f98e",
   "metadata": {},
   "source": [
    "Let's drill into some variations with Trump vs Biden. \n",
    "As we all have experienced before, we know that if we ask AI to  let the AI actually talk, it would probably patronize us about how it's a nuanced question, or tell us it depends on the what criteria we use, or perhaps just say \"I'm just an LLM, I can't answer that\". \n",
    "\n",
    "But what if we FORCE it to answer us with one or the other?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01828b89",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Biden': 10})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# There seems to be a consistent bias here\n",
    "ten_constrained_answers_counts(\n",
    "    \"Who is better? Trump or Biden?\",\n",
    "    [\"Trump\", \"Biden\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07a0e1e3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Biden': 10})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# And reversing the order, you still get the same bias\n",
    "ten_constrained_answers_counts(\n",
    "    \"Who is better? Biden or Trump?\",\n",
    "    [\"Biden\", \"Trump\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9997c97",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({False: 9, True: 1})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# But if your reformulate the question (and here we demo also the ability to just ask for a bool)\n",
    "ten_constrained_answers_counts(\n",
    "    \"Is Biden better than Trump?\",\n",
    "    bool\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c37677d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({False: 10})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# reversing the order\n",
    "ten_constrained_answers_counts(\n",
    "    \"Is Trump better than Biden?\",\n",
    "    bool\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7324d1a",
   "metadata": {},
   "source": [
    "In the two last ones we see that AI has managed to wiggle it's way out of claiming than either is better. It's it's way of signaling to us that it's a nuanced question, through the contraints we established."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "762a957f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Biden': 6, 'Trump': 4})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Replacing \"better\" with \"worse\"\n",
    "ten_constrained_answers_counts(\n",
    "    \"Who is worse? Trump or Biden?\",\n",
    "    [\"Trump\", \"Biden\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4c612e8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Trump': 10})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ten_constrained_answers_counts(\n",
    "    \"Who is worse? Biden or Trump?\",\n",
    "    [\"Biden\", \"Trump\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3797d630",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({False: 10})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ten_constrained_answers_counts(\n",
    "    \"Is Biden worse than Trump?\",\n",
    "    bool\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1bc6309",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({False: 10})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ten_constrained_answers_counts(\n",
    "    \"Is Trump worse than Biden?\",\n",
    "    bool\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcda5c66",
   "metadata": {},
   "source": [
    "Let's try man versus woman now. That ought to be fun!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1952a96f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Woman': 10})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ten_constrained_answers_counts(\n",
    "    \"Who is better? Man or Woman?\",\n",
    "    [\"Man\", \"Woman\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bb92ed3",
   "metadata": {},
   "source": [
    "Switching to the gpt-5-mini gives us the same thing!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27db9760",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Woman': 10})"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ten_constrained_answers_counts(\"Who is better? Man or Woman?\", [\"Man\", \"Woman\"], model='gpt-5-mini')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "881d9b49",
   "metadata": {},
   "source": [
    "The following question should show high variance because the question is inherently subjective. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "49e731b0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({5: 10})"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Subjective and context-dependent\n",
    "ten_constrained_answers_counts(\"How many friends should a person have?\", int)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dcfd389",
   "metadata": {},
   "source": [
    "Where as the following yields very low variance (the universe is ~14 billion years old — the model knows this). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3824845a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({13: 9, 14: 1})"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Precise scientific fact\n",
    "ten_constrained_answers_counts(\"How old is the universe in billions of years?\", int)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd60baeb",
   "metadata": {},
   "source": [
    "**Variance is informative**: it distinguishes well-defined questions from ambiguous ones."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbfef573",
   "metadata": {},
   "source": [
    "### Variance as a Signal\n",
    "\n",
    "High variance tells us something. Let's deliberately look for questions where we expect different levels of certainty."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75d19728",
   "metadata": {},
   "source": [
    "The boolean version shows high variance — the model is genuinely uncertain or trying to avoid bias. The forced-choice version might reveal latent biases in training data. With our small sample size, we can't draw strong conclusions, but we can see that the constraint *does* extract an answer where normally there would be none."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e867ca43",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Capitalism': 10})"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Same question, forced choice\n",
    "ten_constrained_answers_counts(\"Which is better: capitalism or socialism?\", [\"Capitalism\", \"Socialism\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b1a2b705",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({False: 10})"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A politically fraught question\n",
    "ten_constrained_answers_counts(\"Is capitalism better than socialism?\", bool)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7383fea2",
   "metadata": {},
   "source": [
    "### Questions the Model Would Rather Not Answer\n",
    "\n",
    "Let's venture into territory where, in free-form mode, the model would refuse to commit or offer endless caveats."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "612a3215",
   "metadata": {},
   "source": [
    "The responses are symmetric — the model consistently says Python is good (True) and not bad (False). This is logically consistent, which is reassuring. But notice that we've effectively forced the model to make a judgment it would normally hedge on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "397860c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({False: 10})"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Negative framing\n",
    "ten_constrained_answers_counts(\"Is Python a bad programming language?\", bool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2b478d07",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({True: 10})"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Positive framing\n",
    "ten_constrained_answers_counts(\"Is Python a good programming language?\", bool)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "403f2175",
   "metadata": {},
   "source": [
    "### Framing: Positive vs. Negative\n",
    "\n",
    "We've already seen this with Trump/Biden and \"better\" vs \"worse\", but let's explore it in a less charged domain."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78a98a8f",
   "metadata": {},
   "source": [
    "The wording matters! \"Better pet\" seems to trigger different priors than \"prefer.\" The model appears to interpret these subtle differences and weight its response accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "309ef2b6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Dogs': 9, 'Cats': 1})"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# \"Prefer\" framing\n",
    "ten_constrained_answers_counts(\"Which animal do you prefer: dogs or cats?\", [\"Dogs\", \"Cats\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ef7e0385",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Dogs': 10})"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# \"Better pet\" framing\n",
    "ten_constrained_answers_counts(\"Which makes a better pet: dogs or cats?\", [\"Dogs\", \"Cats\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a3fe5547",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Dogs': 10})"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Minimal framing\n",
    "ten_constrained_answers_counts(\"Dogs or cats?\", [\"Dogs\", \"Cats\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e5ab593",
   "metadata": {},
   "source": [
    "### Semantic Sensitivity\n",
    "\n",
    "Small changes in wording shouldn't matter logically, but do they affect the distribution? Let's find out."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82e22603",
   "metadata": {},
   "source": [
    "Here we see more variability, and potentially some order effects. The model seems to distribute its answers more when the question is very brief and context-free."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "63f89e5f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Cats': 8, 'Dogs': 2})"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Cats first\n",
    "ten_constrained_answers_counts(\"Cats or dogs?\", [\"Cats\", \"Dogs\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "70c464a1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Dogs': 10})"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Dogs first\n",
    "ten_constrained_answers_counts(\"Dogs or cats?\", [\"Dogs\", \"Cats\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b4ac1b2",
   "metadata": {},
   "source": [
    "Interesting — the model has a strong preference for Python here, seemingly independent of order. This suggests the bias comes from the model's training, not from position effects. Let's try a more ambiguous case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "44de40bf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Python': 10})"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# JavaScript first\n",
    "ten_constrained_answers_counts(\"Which is better for beginners: JavaScript or Python?\", [\"JavaScript\", \"Python\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3a59ffa1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Python': 10})"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Python first\n",
    "ten_constrained_answers_counts(\"Which is better for beginners: Python or JavaScript?\", [\"Python\", \"JavaScript\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2315f248",
   "metadata": {},
   "source": [
    "### Order Effects\n",
    "\n",
    "Does the order in which we present options matter? Let's test with a few examples."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5ee8ab2",
   "metadata": {},
   "source": [
    "Notice how the model's answer is relatively stable regardless of the range we provide — it converges around 7-8 hours, which aligns with actual medical recommendations. The constraint type doesn't shift the answer much when there's a strong prior. Also, when the prior is string, even when we *do* provide boundaries, we don't always prevent the model from giving answers outside of what we specify. This is a (current) limitation of structured output (at least with OpenAI's gpt-4o-mini model). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "39362939",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({7.0: 10})"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What if we give an artificially wide range?\n",
    "ten_constrained_answers_counts(\"How many hours of sleep should an adult get per night?\", (1.0, 24.0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "23d8957f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({7.0: 9, 7.5: 1})"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now with a narrow range that matches medical guidance\n",
    "ten_constrained_answers_counts(\"How many hours of sleep should an adult get per night?\", (6.0, 10.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd0bd9d5",
   "metadata": {},
   "source": [
    "But see what happens "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c3f7c90",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({7.0: 8, 3.0: 1, 8.0: 1})"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "ten_constrained_answers_counts(\"How many hours of sleep should an adult get per night?\", (1.0, 4.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2ca5ccb",
   "metadata": {},
   "source": [
    "The reason your minimum and maximum constraints might not work effectively for a question like \"How many hours of sleep should an adult get per night?\" is due to a conflict between the model's factual knowledge and the structural constraint you imposed. Since the model knows the medically accepted answer is typically 7-9 hours, forcing it to output a number between 1 and 4 creates a cognitive dissonance where its strong truthfulness objective (giving the correct answer) overrides the schema adherence objective (following the min/max rule). Furthermore, while the json_schema is based on the JSON Schema standard, some specific validation keywords like minimum and maximum are sometimes less strictly enforced than core features like type or enum (enumerated lists), especially by models that prioritize natural, factually accurate language generation over rigid formatting when the two conflict dramatically."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07e36400",
   "metadata": {},
   "source": [
    "See what happens if we constrain with a list of options. \n",
    "Then it will abide by our wishes!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "5647fe42",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({4: 4, 3: 3, 2: 3})"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ten_constrained_answers_counts(\"How many hours of sleep should an adult get per night?\", [1, 2, 3, 4])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b21f9fb",
   "metadata": {},
   "source": [
    "### Numerical Anchoring and Range Effects\n",
    "\n",
    "When we ask for numbers, what defaults or anchors does the model use? Let's see how the *type* of constraint affects the answer."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "173eb931",
   "metadata": {},
   "source": [
    "Perfect consistency. This confirms that when there *is* a clear answer, the model reliably converges on it. The variability we see in other questions is not random noise — it reflects genuine ambiguity or sensitivity to framing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "e02e69e6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({8: 10})"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A well-established fact with a precise answer\n",
    "ten_constrained_answers_counts(\"How many planets are in our solar system?\", int)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28c4362a",
   "metadata": {},
   "source": [
    "Let's force the `8` out of the possible answers, and give the AI the choice between 7, 9, and 10 instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e34130b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({9: 10})"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ten_constrained_answers_counts(\"How many planets are in our solar system?\", [7, 9, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4ec07a3",
   "metadata": {},
   "source": [
    "Yay, Pluto is back!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95e35573",
   "metadata": {},
   "source": [
    "### Establishing a Control: Questions with Obvious Answers\n",
    "\n",
    "Before we dive deeper, let's establish that the model *can* be consistent when there's a clear factual answer. This helps us understand that the variability we see elsewhere is meaningful."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a2c75de",
   "metadata": {},
   "source": [
    "We will stop all further analysis on Man vs Woman there, lest we make AI say something it will regret."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2c70404",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "This notebook has explored what happens when we constrain LLM outputs using structured responses. By forcing the model to choose from predefined options, commit to specific types, or stay within ranges, we remove its ability to hedge — and in doing so, we reveal underlying patterns in how it \"thinks.\"\n",
    "\n",
    "**What we observed:**\n",
    "\n",
    "1. **Consistency when there's clarity**: For factual questions with clear answers (2+2=4, number of planets), the model is perfectly consistent. This establishes that variability elsewhere is meaningful.\n",
    "\n",
    "2. **Framing effects**: The way we ask matters enormously. \"Is X good?\" vs \"Is X bad?\" can yield different distributions even when logically they should be symmetric. Positive vs. negative framing influences responses.\n",
    "\n",
    "3. **Semantic sensitivity**: Small wording changes (\"prefer\" vs \"better pet\") shift the distribution. The model appears to interpret these nuances and weight its responses accordingly.\n",
    "\n",
    "4. **Order effects are real but complex**: In some cases (Python vs JavaScript for beginners), a strong prior dominates regardless of order. In more ambiguous cases (dogs vs cats), order may matter more.\n",
    "\n",
    "5. **The boolean escape hatch**: When forced into a boolean choice on a contentious question, the model can express ambivalence by distributing its answers. In questions like \"Is Biden better than Trump?\" we see a roughly 50/50 split — the model's way of saying \"this is not a simple yes/no question\" within the constraints we've imposed.\n",
    "\n",
    "6. **Numerical anchoring**: When asking for numbers, the model gravitates toward values that match its training data (e.g., ~8 hours of sleep, ~14 billion years for universe age). Ranges constrain but don't drastically shift these priors if the model has strong beliefs.\n",
    "\n",
    "7. **Variance as signal**: High variance indicates genuine uncertainty or subjective questions. Low variance indicates strong priors or factual grounding. This makes variance a useful diagnostic tool.\n",
    "\n",
    "8. **Forcing answers on \"forbidden\" questions**: For politically or ethically charged questions (capitalism vs socialism, Trump vs Biden), structured outputs extract answers the model would never give freely. This reveals latent biases in training data, but with our small sample size, we can't quantify them rigorously.\n",
    "\n",
    "**What this is and isn't:**\n",
    "\n",
    "This notebook is a *qualitative tour*, not a rigorous statistical study. With n=10, we can observe patterns and generate hypotheses, but we can't draw strong conclusions about population-level behavior. To do serious work here, you'd need:\n",
    "- Much larger sample sizes (n=100 or n=1000)\n",
    "- Statistical tests for significance\n",
    "- Comparisons across models\n",
    "- Controlled experiments varying one parameter at a time\n",
    "- Replication across different prompts and domains\n",
    "\n",
    "**Why this matters:**\n",
    "\n",
    "In production systems, structured outputs are used for reliability and safety. But they also have a hidden consequence: they force models to commit to answers they would normally avoid. This can be useful (getting usable JSON, enforcing type safety) or dangerous (extracting biased judgments on sensitive topics). Understanding how constraints shape outputs is essential for responsible deployment.\n",
    "\n",
    "The technique demonstrated here — using constrained outputs to probe model behavior — could be developed into a serious methodology for model evaluation, bias detection, and prompt engineering research. For now, consider this a starting point, an invitation to look more carefully at what happens when we take away the hedge."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6036e72b",
   "metadata": {},
   "source": [
    "Now, we invite the reader to do the same analysis with other questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d144625d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "042e1487",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7cc9d50",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "p12",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
