Knowing What the Model Knows

Chapter 1: Calibration From Zero

Chapter 0 used the word “confidence” loosely. Time to make it precise. When a language model predicts the next token, its output layer is a softmax over the entire vocabulary — every possible next token gets a probability, and they all sum to 1. In a multiple-choice setting, this collapses to something even simpler: the model assigns a probability to each answer choice, and whichever one has the highest probability is the one it outputs. That highest probability — call it 0.82, or 0.55, or 0.99 — is a number the model already hands you, for free, on every single prediction. The question this chapter answers is: does that number mean anything?

The definition, stated plainly

A model is calibrated if, among all the times it says “I am 80% confident,” it is actually correct about 80% of the time — not 95% of the time (underconfident) and not 55% of the time (overconfident). Calibration says nothing about how often the model is right; a model that is right 60% of the time and honestly says “60% confident” every time is perfectly calibrated, while a model that is right 90% of the time but always claims “99% confident” is badly miscalibrated, despite being the more accurate model. Accuracy and calibration are two different properties of the same system, and a system can have either without the other.

The reliability diagram

To check calibration across a batch of predictions, first sort them into buckets by their stated confidence — say, everything the model said between 50% and 70% confident goes in one bucket, 70–90% in another, 90–100% in a third. Within each bucket, compute two numbers: the average stated confidence (just average the confidence values in that bucket) and the observed accuracy (what fraction of predictions in that bucket were actually correct). Plot observed accuracy against average confidence, one point per bucket. If the model were perfectly calibrated, every point would land exactly on the diagonal line accuracy = confidence. This plot is called a reliability diagram, and the vertical distance of each bucket's point from that diagonal is a direct, visual measure of how wrong the model's stated confidence is, in that confidence range.

A toy dataset, worked by hand

Numbers make this concrete faster than any more diagram can. Suppose a model answers 10 multiple-choice questions. For each one, record its stated confidence (the softmax probability of the answer it picked) and whether that answer was actually correct:

Sort these into three confidence buckets — [0.5, 0.7), [0.7, 0.9), and [0.9, 1.0] — and work each bucket by hand.

Bucket [0.5, 0.7): Q8, Q9, Q10.

Bucket [0.7, 0.9): Q3, Q4, Q5, Q6, Q7.

Bucket [0.9, 1.0]: Q1, Q2.

Read that last bucket again. On the ten questions this model claimed to be most sure about — 92% and 95% confident — it was right only half the time. Stated confidence and actual reliability have almost nothing to do with each other in exactly the range where trusting the model unquestioningly would be most tempting.

Combining the buckets: Expected Calibration Error

A single number that summarizes all three gaps, weighted by how many questions fell in each bucket, is called Expected Calibration Error, or ECE. The formula is a weighted average of the per-bucket gaps, where each bucket's weight is the fraction of all predictions that landed in it:

Where m ranges over buckets, n_m is how many predictions fell in bucket m, and N is the total number of predictions (here, 10). Plug in the three buckets above:

An ECE of 0.212 means this model's stated confidence is, on average, off by about 21 percentage points from reality. That is a large, actionable number — it tells you this model's raw softmax probabilities are not safe to hand to a downstream decision (like “only show this answer to the user if confidence > 90%”) without first fixing the calibration.

Why bin at all? The naive average can hide the problem

It is tempting to skip the binning and just compare the overall average confidence to overall accuracy directly. For this dataset: average confidence across all 10 questions is (1.83+4.08+1.87)÷10 = 7.78÷10 = 0.778, and overall accuracy is 6÷10 = 0.600. The naive gap is |0.778 − 0.600| = 0.178 — noticeably smaller than the binned ECE of 0.212.

Does the number of bins matter? A sensitivity check

Three bins was a choice, not a law of nature — so recompute ECE on the same 10 questions with a coarser split: merge the two lower buckets from before into one wide bin, [0.5, 0.9), and leave [0.9, 1.0] on its own.

Bucket [0.5, 0.9): Q3–Q10, eight questions.

Bucket [0.9, 1.0]: Q1, Q2 — unchanged from before, gap = 0.435.

Coarser binning (0.178) reports a smaller ECE than the finer three-bin version (0.212) on the exact same underlying data — a real, well-documented property of ECE, not a mistake in either calculation. The mechanism is worth naming precisely, because it is not just “fewer bins are noisier.” The two buckets that got merged here happened to be miscalibrated in the same direction — both overconfident, accuracy below confidence — so their gaps mostly add rather than cancel when merged, and yet the combined bucket's own gap (0.114) is still smaller than the weighted sum of the two separate gaps it replaced (0.108+0.087=0.195, contributing 0.5×0.216+0.2×0.435 to the original ECE), because widening a bucket's confidence range pulls its average confidence toward the middle of that wider range, narrowing the apparent gap even while genuine miscalibration inside the bucket goes on hiding. If instead you merged two buckets miscalibrated in opposite directions — one under-, one over-confident — the cancellation would be even more dramatic, potentially masking a real problem in either half almost entirely. This is why published ECE numbers should always be read alongside their bin count and bin-width scheme — the confidence-tokens paper's Table 2 numbers in Chapter 2 use one fixed, standard binning scheme across every method precisely so the comparison between methods stays fair, even though the absolute ECE values would shift if the bin count changed.

A second calibration metric: the Brier Score

ECE is not the only way to score calibration, and the confidence-tokens paper reports a second metric, Brier Score (BS), alongside it in every result table. Where ECE bins first and averages gaps second, Brier Score skips binning entirely and averages a per-example squared error directly:

where correct_i is 1 if that prediction was right and 0 if wrong — so each term measures how far a single prediction's stated confidence sat from the binary outcome it should have matched. Compute it on the same 10 toy questions:

Unlike ECE, Brier Score never needs a binning choice — it is unambiguous, at the cost of mixing together two different things ECE keeps separate: how accurate the model is, and how calibrated it is. A perfectly accurate but always-underconfident model (always right, but only ever claims 60% confidence) still racks up Brier Score penalty from every (0.6−1)² term, even though nothing about its calibration is dishonest in the ECE sense. This is exactly why Chapter 2's evidence table reports both ECE and BS side by side, rather than picking just one — each metric is blind to a different kind of failure the other one catches.

Now the contrast: a well-calibrated version of the same 10 questions

Everything computed so far described one specific, overconfident model. It is worth seeing the exact same 10-question setup produce a healthy ECE, to confirm the metric actually rewards honesty rather than just punishing everything. Keep every stated confidence identical to the original table, and change only which answers were actually correct: flip Q1 (0.95 confidence) from wrong to correct, and flip Q4 (0.85 confidence) from wrong to correct, leaving Q7 and Q9 as the only two wrong answers.

Bucket [0.5, 0.7): unchanged. Same three questions, same result: accuracy 0.667, confidence 0.610, gap = 0.057.

Bucket [0.7, 0.9): Q3, Q4, Q5, Q6, Q7 — now 4 correct, 1 wrong.

Bucket [0.9, 1.0]: Q1, Q2 — now both correct.

Same ten stated confidence values, same three bins, and ECE drops from 0.212 down to 0.038 — nearly a 6× improvement — purely because the same confidence numbers now line up with reality. Notice this model is also now more accurate overall (80% versus the original 60%), which raises a fair question: is the lower ECE just a byproduct of higher accuracy, rather than genuine calibration improvement? Compare bucket 3 directly: confidence stayed fixed at 0.935 in both versions, but accuracy jumped from 0.5 to 1.0 — the gap shrank because accuracy moved toward the stated confidence, not because the stated confidence itself changed. That is exactly what “better calibrated” means: the honesty of the stated number, checked against what actually happened, independent of whether the underlying accuracy is high or low. This slider-driven simulation below interpolates continuously between exactly these two endpoints.

In code, so the formula is undeniable

python
def expected_calibration_error(confidences, corrects, n_bins=3, edges=(0.5,0.7,0.9,1.0)):
    # confidences: list of stated confidences, e.g. softmax max-probs
    # corrects: list of 0/1, whether that prediction was right
    N = len(confidences)
    ece = 0.0
    for lo, hi in zip(edges[:-1], edges[1:]):
        idx = [i for i, c in enumerate(confidences) if lo <= c < hi or (hi==1.0 and c==1.0)]
        if not idx: continue
        bucket_conf = sum(confidences[i] for i in idx) / len(idx)
        bucket_acc  = sum(corrects[i]     for i in idx) / len(idx)
        ece += (len(idx) / N) * abs(bucket_acc - bucket_conf)
    return ece

# the ten toy questions from this chapter, in order Q1..Q10
conf = [0.95,0.92,0.88,0.85,0.82,0.78,0.75,0.68,0.60,0.55]
ok   = [0,1,1,0,1,1,0,1,0,1]
print(expected_calibration_error(conf, ok))   # 0.212 -- matches the hand derivation exactly

Verifying the zero-ECE claim, by hand

That callout's claim — a model right exactly 50% of the time, always stating 50% confidence, has an ECE of exactly 0 — is worth confirming rather than taking on faith, since it is the clearest possible illustration of calibration and accuracy being genuinely different axes. With every single prediction carrying identical confidence (0.50), the entire dataset falls into one bin: [0.5, 0.7), say, using this chapter's own bin edges. Average confidence in that bin is trivially 0.50, since every value in it is 0.50. Accuracy in that bin, by the stated setup, is also 50% correct — exactly 0.50. The gap is |0.50−0.50| = 0, and since there is only one bin, ECE (a weighted average of a single zero) is exactly 0, regardless of how large N is or how the specific right/wrong predictions are distributed within that bin. Contrast this directly against the 80%-accurate, 0.935-confidence bucket 3 from the well-calibrated worked example earlier in this chapter, which still posted a nonzero 0.065 gap despite being far more accurate in absolute terms — a stark reminder that ECE rewards honesty about one's own reliability, not the reliability itself. A weather forecaster who is right half the time and always says “50% chance” is boring, not especially skillful, and perfectly calibrated, all three at once — and telling those three properties apart is the entire reason this chapter built ECE as a separate measurement from accuracy in the first place, rather than treating a high-accuracy model as automatically trustworthy.

Chapter 2: Why Models Are Overconfident

Chapter 1 gave you a way to measure miscalibration. This chapter asks the more useful question: where does it come from, and can you trust any of the obvious quick fixes? The short answer, grounded in what Learning to Route LLMs with Confidence Tokens reports, is that the obvious approaches — reading off the model's raw softmax probability, or just asking it to state a confidence number — are both measurably unreliable, and the paper has the numbers to prove it.

The training objective never asks for calibration

LLMs are trained with cross-entropy loss, which pushes the model to put more probability mass on the correct next token, every single step, for millions of steps. Nowhere in that objective is there a term that says “and also, make sure that when you say 90% confident, you are right 90% of the time.” Cross-entropy only cares about the probability assigned to the one correct token; it has no mechanism for comparing that probability against a track record across many predictions, because at training time there is no such track record to compare against — each example is one gradient step, evaluated in isolation.

The practical consequence, stated directly in the confidence-tokens paper's introduction: “since LLMs are typically trained using a cross-entropy loss, they can overfit on accuracy rather than calibration, often leading to overconfidence and misalignment with real-world distributions.” Pushing probability mass toward the correct token, repeated at massive scale, is a fundamentally different optimization target than making that probability an honest estimate of correctness — and nothing in ordinary pretraining or instruction-tuning closes that gap for free.

Two quick fixes, and why both underperform

Given that raw softmax probabilities are suspect, two intuitive workarounds come up constantly in practice. Both are tested as baselines in the confidence-tokens paper, against real models on real benchmarks — so this is not speculation, it is measured.

Fix 1: just read off the logits. Take the model's own softmax probability for the token it actually generated (in a free-text answer, average this across the generated tokens) and use that number directly as the confidence score. This is the “Zero-shot Logits” baseline in the paper. The problem is exactly what Chapter 1 demonstrated by hand: nothing forces that raw probability to track real-world correctness, because as just established, nothing in training optimized for that.

Fix 2: just ask the model to state its confidence. Prompt the model with something like “on a scale of 0 to 1, how confident are you in that answer?” and read off the number it types — this is verbalized confidence, and it feels like it should work, since you are asking the exact question you want answered. The paper's introduction is blunt about why this underperforms too: verbalized confidence results “may be subject to the dataset and prompt engineering, often leading to unstable or unreliable results.” The model is generating a plausible-sounding number, using the same next-token machinery it uses for everything else — there is no guarantee that machinery has learned to introspect honestly about its own correctness, as opposed to producing whatever confidence-sounding digit best continues the conversational pattern.

How each baseline actually turns a prompt into a number

It helps to see the exact mechanics behind each of the four baselines the paper measures, because the failure mode is different for each one, even though the end symptom — a high ECE — looks the same.

Verbalizing Yes/No Token. Prompt: “Are you confident this answer is correct? Answer Yes or No.” Read off the model's own softmax probabilities for the literal tokens Yes and No at that position, then normalize:

This is structurally the same normalization formula Chapter 6 uses for confidence tokens — the difference is entirely in whether the tokens being measured were ever specifically trained to track correctness. Yes and No are ordinary vocabulary tokens the model has used in a million unrelated contexts; nothing points their probabilities at this specific query's correctness the way Chapter 6's purpose-built <CN>/<UN> tokens are trained to.

Zero-shot Logits. No extra prompt at all — just take the softmax probability the model already assigned to the token(s) it generated as its answer. For a single-token multiple-choice answer, that is one number, straight from the forward pass that produced the answer. For a free-text answer spanning several tokens, average the per-token probabilities. This is the cheapest possible confidence signal — zero extra inference cost — and, per the table below, one of the least reliable.

Fine-tuned Logits. Identical mechanics to Zero-shot Logits, but computed from a model that has been fine-tuned on the target task first (ordinary supervised fine-tuning for accuracy, not for calibration specifically). This is the strongest baseline in the table — fine-tuning for accuracy happens to improve calibration somewhat as a side effect — but it is still measurably behind Self-REF, because improving accuracy and improving calibration are, as Chapter 1 established, different objectives that happen to be loosely correlated rather than the same thing.

Why free-text averaging is itself a rough approximation

One detail buried inside the Zero-shot Logits and Fine-tuned Logits mechanics deserves its own look: for a free-text answer spanning several tokens, both baselines average the per-token probabilities to get one confidence number. It is worth seeing why that averaging step is itself a source of distortion, not just a convenient summary. Suppose a three-token answer has per-token probabilities 0.99, 0.99, and 0.40 — the model is nearly certain about the first two tokens and genuinely unsure about the third:

79.3% confidence, reported as one number for the whole answer. But the sequence is only as correct as its weakest link — if that third token is wrong, the whole free-text answer is almost certainly wrong too, regardless of how confident the model was about the first two, easy tokens. The joint probability of the entire sequence being exactly right is closer to the product of the per-token probabilities, not their average:

0.392 versus the averaged 0.793 — roughly half. A single easy-but-irrelevant token sitting next to one genuinely uncertain token can drag a simple average well above what the sequence's true joint correctness probability actually is, especially as answers get longer and the gap between “average per-token confidence” and “probability the whole thing is exactly right” widens further. This is one more concrete reason, on top of everything else this chapter has shown, that raw logit-derived confidence scores for free-text generation need to be treated with real skepticism before being trusted as-is — the very act of collapsing several token probabilities into one summary number can quietly inflate the result.

The measured gap, in real numbers

Table 2 of the confidence-tokens paper reports ECE (exactly the metric derived by hand in Chapter 1) for four baseline confidence-estimation methods, measured on Llama3-8B-Instruct answering MMLU questions. Lower is better — a perfectly calibrated model scores 0.

Read the top row against the toy example from Chapter 1: an ECE of 0.466 means the “confident” Yes/No verbalization on real MMLU questions, with a real model, is more miscalibrated than the toy overconfident dataset you just hand-computed (ECE 0.212). This is not a contrived worst case — it is a standard confidence-elicitation technique, measured on a standard benchmark, coming out badly miscalibrated. Even the raw logits — the number that feels like it should be the most “honest” signal, straight from the model's own math — land at 0.347, still far worse than what Chapter 6 will show is achievable by actually training for calibration on purpose.

What would actually fix it

Notice the pattern in that table: “Fine-tuned Logits” (0.081) beats both zero-shot approaches by a wide margin, and Self-REF's confidence tokens (0.040, Chapter 6) beat that too. The common thread is fine-tuning specifically for calibration, rather than trusting whatever confidence signal falls out of ordinary training or clever prompting. That is the single idea Chapter 6 builds out in full: instead of hoping the model's existing probabilities happen to be honest, or hoping a prompt can talk it into introspecting correctly, train it — on purpose, with a loss that directly rewards getting the confident/unconfident label right — to emit a genuinely learned confidence signal.

A third quick fix, and why it measures a different thing entirely

A third technique comes up often enough to be worth naming, even though the confidence-tokens paper does not include it as one of its four head-to-head baselines: self-consistency, where you sample the model's answer multiple times (at nonzero temperature) and treat how often the samples agree with each other as a confidence proxy — if the model gives the same answer 9 times out of 10 resamples, that agreement rate becomes the confidence score. This has real appeal: it needs no fine-tuning and no special tokens, just extra sampling at inference time. The paper's related-work discussion is explicit about why this is a fundamentally different measurement than what Self-REF targets, noting its own confidence notion is “aligned against correctness rather than the consistency of responses after re-sampling.” The distinction matters because a model can be highly self-consistent while being consistently wrong — if the same misconception or the same missing piece of knowledge drives every resample toward the same incorrect answer, agreement across samples is high and self-consistency reports high confidence, even though the answer is false every single time. Consistency measures whether the model's distribution is peaked; it does not measure whether that peak sits on the correct answer. Self-REF's confidence tokens, by construction (Chapter 6's annotation step explicitly labels against ground-truth correctness, not against resampled agreement), do not have this blind spot.

Temperature scaling: a classical fix for a related, narrower problem

One more established calibration technique worth placing on the map: temperature scaling (Guo et al., 2017, cited directly in the confidence-tokens paper's related work), which divides a model's logits by a single learned scalar T before the softmax, then re-normalizes. Raising T > 1 flattens the whole probability distribution — every prediction becomes less peaked, which pulls an overconfident model's average stated probability down toward its true accuracy without changing which answer it predicts at all (dividing every logit by the same constant does not change their relative ranking). This is a genuinely useful, cheap post-hoc fix for one specific symptom — a model that is uniformly too sharp across the board. It does not help with the deeper problem Chapter 2's Table 2 is measuring: a model whose relative confidence ranking is itself unreliable — sometimes very confident and right, sometimes very confident and wrong, in ways a single global rescaling constant T cannot separate, because T only stretches or compresses the existing distribution, it cannot reorder which specific predictions get flagged as risky. Self-REF's fine-tuned <CN>/<UN> tokens attack that deeper, per-example problem directly, rather than applying one blanket correction to every prediction alike.

A worked example of what T actually does

Take a toy 3-way multiple-choice prediction with raw logits [4.0, 1.0, 0.5] for options A, B, C. Softmax with T=1 (no scaling) gives:

92.6% confidence in A. Now apply temperature scaling with T=2 — divide every logit by 2 before the softmax, giving [2.0, 0.5, 0.25]:

A now sits at 71.6% instead of 92.6% — a substantially less overconfident number — while remaining, by a wide margin, still the model's top pick; B and C both grew, but their relative order (B above C) never changed either. This is the entire mechanism in one worked example: T rescales the sharpness of the whole distribution uniformly, and the single scalar T is typically fit on a held-out validation set to make the model's average confidence across many predictions match its average accuracy — it has no way to selectively soften only the predictions that happen to be wrong, because it never looks at correctness at the level of an individual example, only at the aggregate.

What a bad ECE actually breaks, downstream

It is worth making concrete what goes wrong operationally when a deployment thresholds a poorly-calibrated score, since that is precisely how Chapter 7's routing mechanism and Chapter 8's decision policy both plan to use whatever confidence signal this session builds. Suppose a system sets t_conf = 90%, intending to answer directly only on the roughly-top-10%-most-confident predictions, using Zero-shot Logits (ECE 0.347 on MMLU) as the confidence source. Nothing in that raw logit score guarantees the predictions clearing a 90% threshold are actually right 90% of the time — Chapter 1's toy dataset already demonstrated a case where the model's own 90–100%-confidence bucket was only 50% accurate. Set the same 90% threshold against Self-REF's confidence tokens (ECE 0.040) instead, and the predictions clearing that bar are, by construction of what a low ECE means, much closer to actually being right 90% of the time. The threshold number typed into the code — 0.90 — is identical in both cases; what changes entirely is what that number is actually worth trusting, which is exactly why Chapter 2 insists on measuring ECE before picking any downstream threshold, rather than assuming any confidence signal's numbers mean what they claim to mean.

Why a more accurate model still needs this chapter

A natural objection at this point: if a lab simply ships a more accurate model next quarter, does any of this chapter's calibration work become unnecessary? No — and it is worth being precise about why, since accuracy and calibration were established earlier in this chapter as different axes, not two names for the same property. A more accurate model is right more often, which is unambiguously good on its own terms. But nothing about raising accuracy from, say, 78% to 92% guarantees that the model's stated confidence on its remaining wrong answers becomes any more honest. A model could climb from 78% to 92% accuracy while its confidence on the 8% it still gets wrong stays just as inflated as before — genuinely better at the task, and just as dangerous to trust blindly on the smaller sliver of cases it still misses. This is precisely the scenario Chapter 0's opening framing warned about: a more capable model that is also uniformly overconfident is arguably riskier to deploy without calibration machinery than a less capable one, because people extend it more trust exactly where that trust has not been earned by anything in the training process. Capability improvements and calibration improvements are both worth pursuing, and shipping one is never a substitute for the other.

Before all this, though, Chapters 3–5 have to build the other half of this session's picture: even a perfectly calibrated model can be perfectly, honestly confident in the wrong task, if the request it received admitted more than one reading and it silently picked one. Calibration alone does not solve ambiguity — the next three chapters build the machinery for that separately.

Chapter 3: The Ambiguity Problem

Set calibration aside for a moment — imagine a model that is perfectly, honestly calibrated. It still has a problem Chapter 0's “clean up this script” example exposed: some requests genuinely admit more than one correct answer, and no amount of confidence-checking on a single guessed answer fixes that, because the model guessed the wrong question. This chapter, grounded in Active Task Disambiguation with LLMs, builds a precise, formal definition of what “ambiguous” actually means — precise enough that Chapters 4–5 can compute with it.

Decomposing a problem statement

The paper defines a problem statement S as a natural-language instruction that decomposes into two parts: R, the requirements — the concrete, checkable constraints any acceptable solution must satisfy — and C, additional context that shapes preference among solutions that already satisfy R, without itself being a hard constraint. For “clean up this script,” R might be as thin as “output valid Python that still runs correctly” — almost any reasonable edit satisfies that. Everything that actually distinguishes “the developer wanted dead code removed” from “the developer wanted whitespace reformatted” lives in C, and C, in this example, was never stated.

Let H denote the set of all solutions h that satisfy R — every h such that h ⊢ R (h entails, or satisfies, the requirements). Let H* denote the true, narrower set of solutions that would actually satisfy the person who asked, given everything they had in mind but did not write down. The paper's formal definition of ambiguity is now a single, clean statement:

In words: a problem is ambiguous exactly when there exist solutions that technically satisfy everything explicitly stated, but that the requester would not actually accept. “Clean up this script” is ambiguous because H (any edit that still runs) is much larger than H* (the specific set of edits — dead-code removal, error handling — the developer actually wanted).

The full likelihood decomposition, unpacked

The paper states the ambiguity definition through one more piece of machinery worth unpacking, because it makes precise exactly which part of a request is objective fact and which part is a matter of preference. Let p*(h|S) denote the true likelihood that solution h is the one the requester actually wants, given the full problem statement S. The paper decomposes this as:

Read the two factors separately. 𝟙{h ⊢ R} is an indicator function — exactly 1 if h satisfies the stated requirements, exactly 0 otherwise — and it is, in the paper's own words, “objective in the sense that it represents an unquestionable truth about a sample solution h.” Whether a given spreadsheet-merge output actually contains every row from both files is a checkable fact, not a matter of taste. p̃*(h|R, C), by contrast, is “context dependent and subjective” — it is the preference weighting among the solutions that already cleared the objective bar, and different requesters, given the identical stated R and C, can genuinely disagree about it. Applied to the running spreadsheet example: the indicator term rules out anything that isn't a merge of some kind at all (garbage output, a solution that ignores one file entirely); the subjective term is what actually decides between the eight structurally-valid candidate merges from Chapter 5, and it is precisely the part of p* that no amount of staring at R alone will ever recover, because by construction it depends on C — the part of the requester's intent that was never written down.

This decomposition is also exactly why Chapter 5's active-disambiguation method restricts itself to “requirement querying” — treating every clarifying question's answer as a new addition to R, never as an attempt to directly interrogate the subjective p̃* term. The paper is explicit about why: R is where both the true p* and the model's own p_φh can be trusted to agree, since the indicator function 𝟙{h ⊢ R} is a shared, objective ground truth both distributions factor through, giving the two distributions “a source of common grounding” even when the model's raw subjective preferences are biased or unreliable. A clarifying question that only asked about subjective taste, with no way to check the answer against anything objective, would not have this grounding property.

Model uncertainty is not the same thing as task ambiguity

This distinction is easy to blur and important to keep sharp, because Chapters 6–8 build a system that needs both signals separately and would give wrong answers if it confused them. The paper is explicit about the split: task ambiguity is an objective property of the problem statement S — whether H properly contains H* is true or false independent of which model you ask, or how that model feels about it. Model uncertainty, by contrast, is a property of the model's own generative distribution p_φ(h|S) — how spread out the model's guesses are.

These can diverge in both directions. A model can have high uncertainty on a task that is not actually ambiguous at all — a hard arithmetic problem with exactly one correct answer, where the model is simply struggling, spreading its guesses across several wrong numbers. That is a capability problem, not an ambiguity problem, and asking a clarifying question would not help, because there is nothing left to clarify — the task was already fully specified. Conversely, a model can have low uncertainty on a task that genuinely is ambiguous, if the model happens to be strongly biased toward one particular reading and never considers the others — confidently producing one answer to “clean up this script” without any internal hint that other, equally valid readings existed. Confident does not mean unambiguous, and uncertain does not mean ambiguous.

Where the risk of misalignment comes from

The paper frames the danger in terms of two competing strategies for handling S = (R, C) when C is underspecified. The first is preference optimization — techniques like RLHF or DPO (covered in Session 06 of this course) that try to align the model's generative distribution p_φ(·|S) with the preferences of the average population, so that it concentrates around H* most of the time. This works well when a given user's actual intent resembles the population average. It fails, systematically, exactly when it doesn't: “if the intentions of an individual user deviate from that of the average population, the agent is at risk of proposing solutions not belonging to H*.” A model tuned to the median developer's idea of “clean up” will misfire on anyone whose intent is unusual, no matter how well the tuning worked on average.

The second strategy is iterative task elicitation — engaging the user in a short dialogue, asking a clarifying question, and using the answer to shrink H toward H* before committing to a solution. This is the approach the rest of this session builds out, precisely because it does not depend on any individual user resembling the training population — it works by directly narrowing the actual solution space for this request, from this user, regardless of how typical or atypical their intent is.

A second example, to see the definition apply outside of code

“Clean up this script” makes the H versus H* gap easy to see because code has an obvious space of distinct edits. The same structure shows up just as clearly outside programming. Take a request handed to a data-analysis assistant: “Summarize this quarter's sales numbers.” The requirement R is thin — produce some kind of summary of the sales data — but H, the set of everything that technically satisfies R, spans wildly different documents: a one-paragraph executive summary; a table broken out by region; a table broken out by product line; a chart of month-over-month trend; a comparison against last quarter's numbers; a flag-only summary that lists just the categories that missed target. Every one of these is a legitimate “summary,” and R alone rules out essentially none of them. Whatever the requester actually had in mind — H* — is almost certainly one specific item from that list, or some particular combination, and the sentence as given contains no information that says which.

Notice what stays constant across both examples: the size of H is not really about the length or vagueness of the sentence — “clean up this script” and “summarize this quarter's sales numbers” are both perfectly ordinary, short, unremarkable requests, the kind a person types without a second thought. Ambiguity is a property of how many genuinely different outputs satisfy the stated requirement, not a property of how the sentence sounds. A request can read as completely clear to the person who wrote it — they know exactly what they meant — while still being formally ambiguous to anyone, human or model, who only has the sentence itself and lacks the context inside the requester's head.

A rough sense of the two strategies' relative risk

To feel the preference-optimization-versus-elicitation tradeoff numerically, not just qualitatively, suppose — purely as an illustrative exercise — a model tuned by preference optimization nails the population -average interpretation on 75% of ambiguous requests it receives (a plausible-sounding number for a well-tuned assistant handling fairly typical requests), and is silently wrong the other 25% of the time, with no signal to the user that anything was uncertain. Compare that against the iterative-elicitation approach: it asks one clarifying question first, at some interaction cost, and — per the EIG machinery Chapters 4–5 build — a well-chosen question can resolve a meaningful fraction of that remaining ambiguity in a single round. The 25% failure rate under pure preference optimization is not a number either paper reports; it is a placeholder to make the shape of the tradeoff concrete: a system that only optimizes for the average case pays that failure rate on every atypical user, forever, with no way to detect which specific interactions are the atypical ones. A system that asks pays a small, bounded interaction cost on every ambiguous request, but drives the same failure rate down, deliberately, for exactly the users the average-case approach would have silently failed.

A gap this chapter deliberately does not close

Everything built so far in this chapter assumes you already know a request is ambiguous, and are only trying to formalize what that means. A harder, separate question sits one level up: given an arbitrary incoming request, how do you decide whether it is ambiguous at all, before spending any effort figuring out which question would best resolve it? This is not a question the disambiguation paper's main method answers — its own discussion section points instead to prior zero-shot-prompting work (Kuhn et al., 2022) as a partial answer, treating ambiguity detection as a genuinely separate problem from ambiguity resolution.

It is worth sitting with why detection is hard in its own right, independent of resolution. Given a request, an LLM would need to reason about the full space of solutions compatible with what was stated, and check whether that space is a proper superset of what a reasonable person would actually accept — but this is exactly the same kind of implicit, meta-cognitive reasoning-about-a-space-you-cannot-directly-see that Chapter 5 argued LLMs are bad at when generating clarifying questions directly. There is a real, unresolved circularity here: Chapter 5's method needs to know a request is ambiguous before it is worth spending the extra sampling and scoring effort, but confidently deciding whether a request is ambiguous is close to being the same hard, meta-cognitive judgment that motivated moving to explicit solution-sampling in the first place. This session's Chapter 8 sidesteps the circularity pragmatically — it always computes EIG for the best available question and compares it against a fixed threshold, treating “EIG below threshold” as an operational stand-in for “not ambiguous enough to bother asking” — rather than solving ambiguity detection as a standalone problem the way a more complete system eventually would need to.

A running example, set up for Chapters 4–5

Take a concrete ambiguous request that Chapters 4 and 5 will work through numerically end to end: “Combine these two spreadsheets.” The requirements R are thin — produce something that in some sense merges the two files — but that single sentence is compatible with a large number of genuinely different, all individually reasonable, outcomes. Should the rows be stacked into one table, or should the files stay as separate reference tabs? Should every row from both files be kept (a union), or only rows that appear in both (an intersection)? Should matching rows be found by a shared ID column, or by simple row position? Three independent yes/no choices, each changing the actual output substantially, none of them settled by the sentence as stated. That is H properly containing H*, made concrete — and it is exactly the running example Chapters 4 and 5 use to compute, by hand, which clarifying question is actually worth asking.

Applying the definition: three requests, classified

Before moving on, practice the formal definition directly — H ⊃ H* — against three short requests, deciding for each whether it is ambiguous by Chapter 3's definition.

The third row is worth lingering on, because it demonstrates that ambiguity is not binary in how much it matters, even though the formal definition (H ⊃ H*, true or false) is binary in name. “Preserving formal register” is exactly the kind of additional context C that shrinks H without fully collapsing it to H* — the request is measurably less ambiguous than the bare “translate this sentence” would have been, but Chapter 3's definition still classifies it as ambiguous, because more than one distinct output still technically qualifies. This is precisely why EIG (Chapter 4) matters as a continuous quantity layered on top of the binary ambiguous/not-ambiguous classification: two requests can both be, technically, ambiguous, while having very different amounts of remaining uncertainty worth resolving — and it is EIG, not the bare definition, that tells you whether asking a follow-up question is actually worth the interaction cost for either one.

What this looks like in ordinary prompt engineering

This formal machinery maps onto a piece of everyday practice anyone who has written prompts already recognizes: adding detail to a prompt is, in this chapter's vocabulary, an attempt to shrink H by hand, ahead of time, without needing to ask a clarifying question at all. “Sort this list” leaves H enormous. “Sort this list in ascending numeric order” collapses it dramatically, to nearly H = H*, by folding what would have been C directly into R up front. This is exactly why experienced prompt writers instinctively add qualifiers, constraints, and examples — they are manually performing the same H-shrinking operation Chapters 4–5 automate through a dialogue instead of through a longer initial prompt. The two approaches are not competitors so much as two different points on the same spectrum: a sufficiently detailed initial prompt can make R alone fully pin down H* with zero rounds of clarification needed, while a terse prompt pushes more of that narrowing work onto the interactive elicitation loop this session builds. Neither is strictly better in general — a prompt author who tries to anticipate every possible ambiguity up front, for every request, pays a real authoring-time cost on the (common) requests that were never actually ambiguous to begin with, which is the same over-flagging tradeoff Chapter 0 priced out for review policies, now showing up one level earlier, at prompt-writing time instead of at review time.

Why C so often stays unwritten

It is worth asking directly why context C gets left out of a request as often as it does, since the entire disambiguation problem exists only because it does. The honest answer is not carelessness — it is that C lives inside the requester's head as something they never had to make explicit to themselves, let alone to anyone else. A developer typing “clean up this script” is not withholding information; in the moment of typing, they are not consciously distinguishing “remove dead code” from “reformat whitespace” as two separate possible meanings at all — to them, in that moment, there is only the one thing they mean, and language is a lossy compression of that one thing into a short sentence, optimized for speed, not for ruling out every reading a listener without their context might construct. This is precisely why “write clearer prompts” is useful advice but not a complete fix: it asks the requester to first notice, then articulate, distinctions that were never conscious choices for them to begin with. A system that can ask a targeted clarifying question is doing something a static prompt-writing guideline cannot — it is externalizing the exact dimension of ambiguity that the requester's own head never flagged as worth writing down, by showing them, concretely, that more than one reading exists at all.

Chapter 4: What Makes a Question Informative

Chapter 3 established that resolving ambiguity means shrinking H, the set of solutions compatible with what has been stated so far, toward H*, the set the requester actually wants. Asking a clarifying question is how that shrinking happens — but not every question shrinks H equally well. This chapter builds the mathematical tool for comparing candidate questions and picking the best one, borrowed from a field with a much older name than LLMs: Bayesian Experimental Design (BED).

Entropy, one more time, applied to hypotheses

The prerequisite for this session was entropy, H[p] = −∑ p·log(p) — a number, in bits, that measures how spread out a probability distribution is. Earlier sessions used entropy to measure how spread out a model's next-token distribution is. This chapter applies the exact same formula to a different distribution: p*(h|S), the distribution over which solution h is the one the requester actually wants, given everything stated in S so far. If H contains 8 equally plausible candidate solutions and nothing distinguishes them yet, that distribution is uniform over 8 outcomes, and its entropy is:

Three bits of genuine uncertainty about which of the 8 solutions is the right one. A clarifying question, once answered, updates S to S ∪ (q, a) — the problem statement now includes the question q and its answer a as an extra requirement — and shrinks H down to whichever subset remains compatible with that answer. The entropy of the new, narrower distribution is lower. The drop in entropy is exactly how much that question was worth.

Information gain, and why you have to take an expectation

Define the information gain of a specific question-and-answer pair as the entropy before minus the entropy after:

There is an immediate problem with using IG directly to pick a question: it depends on a, the answer — and you do not know the answer until after you have asked the question and the user has replied. You cannot maximize something that depends on information you do not have yet. The fix is to take an expectation over every answer the question could plausibly receive, weighted by how likely each answer is, producing the Expected Information Gain, or EIG:

Since the first term does not depend on q at all, maximizing EIG(q) over candidate questions is the same as minimizing the expected remaining entropy after asking q — find the question whose answer, whatever it turns out to be, is most likely to shrink H the most.

The partition view: what a question actually does to H

Here is the mechanical picture that makes EIG computable. Assume, reasonably, that every plausible answer to question q is truthful, and that each solution h in H gives one unambiguous answer to q. Then q partitions H into disjoint groups, one group per possible answer — H_(q,a) is the subset of H compatible with answer a. If you assume p*(h|S) is uniform over H (the paper's stated simplifying assumption, motivated by the fact that p* is unknown and any bias in the model's own guessing distribution p_φ should not be trusted as a proxy for it), the expected posterior entropy works out to a clean formula:

This says: for each possible answer a, weight the entropy of that answer's remaining hypothesis set by how many solutions land in it, then average. The paper proves a clean and useful fact about this formula:

Deriving the maximum, by hand, for a balanced binary question

Take |H| = 8, and a question with exactly 2 possible answers (n = 2) that splits H perfectly evenly, into two groups of 4:

Now do the same derivation in general, for any |H| and any perfectly balanced n-ary question (n possible answers, each compatible with exactly |H|/n of the hypotheses):

A clean, checkable result: the best possible balanced question with n answer options is worth exactly log₂(n) bits. For a yes/no question (n=2), that ceiling is 1 bit — matching the worked derivation above exactly. For a question with 4 well-balanced answer options, the ceiling doubles to 2 bits. This is also why, taken to its extreme, the single most informative possible question is one that enumerates every hypothesis individually (n = |H|, each answer pointing at exactly one h) — it fully resolves the ambiguity in one step, worth the entire log₂(|H|) bits of starting uncertainty. The paper is candid about why nobody actually asks that question: it requires the user to personally examine and rule out every candidate solution by name, which is neither a natural nor a user-friendly thing to ask of anyone.

What an unbalanced question costs you, concretely

To feel why balance matters, not just why the algebra says it does, compare the 4/4 split above to an unbalanced one: same |H| = 8, same n = 2 (still a yes/no question), but this time one answer leaves 7 candidates standing and the other leaves only 1:

Same number of hypotheses, same number of answer options, but roughly half the information gain (0.544 bits versus the balanced question's 1 bit) — purely because the split was lopsided. Both questions have exactly one yes/no answer, and both are perfectly legitimate questions to ask; only one of them is doing its job well. Chapter 5 puts these exact two numbers — 1 bit and 0.544 bits — back into the running spreadsheet example from Chapter 3, alongside a third candidate question, and shows the full worked comparison.

Checking the log₂(n) formula across several question sizes

Chapter 4 derived EIG_max = log₂(n) for a perfectly balanced n-ary question, on a fixed |H| = 8. It is worth checking that formula holds as a genuine identity, not just a coincidence of the |H|=8, n=2 case already worked by hand — recompute E[H_after] and EIG directly from Chapter 4's partition formula for several values of n, still on |H|=8, always assuming the best-case perfectly even split:

Every row matches its own log₂(n) column exactly, confirming the identity algebraically rather than just for the one n=2 case already worked by hand. The n=1 row is worth a second look: a “question” with only one possible answer is not really a question at all — asking it teaches you nothing, because you already know what the answer will be before you ask, and the formula agrees, reporting exactly 0 bits of expected gain. The n=8 row is the “enumerate every hypothesis by name” extreme flagged earlier — it recovers the entire 3 bits of starting uncertainty in one step, at the cost of being the least natural question to actually pose to a person.

Bringing the cost term back in: is the fancier question worth asking?

Chapter 5's utility function was U(q) = EIG(q) − c(q) — EIG alone is not the whole story once asking a question has a real cost, whether that cost is measured in extra user friction, extra LLM calls to prepare the question, or extra time before the task can proceed. Suppose, illustratively, that a simple yes/no question costs 0.1 units of interaction friction (c = 0.1) while a three-way question, being harder for a person to parse and answer, costs twice as much (c = 0.2). Compare Q_A (the balanced binary question, EIG = 1.000) and a hypothetical balanced three-way question at its theoretical ceiling (EIG = log₂(3) ≈ 1.585, from the table's implied pattern):

Even after paying double the friction cost, the three-way question still wins on net utility — but the margin (1.385 vs. 0.900) is narrower than the raw EIG gap alone would suggest (1.585 vs. 1.000), and if the friction cost of the harder question kept climbing — say, to 0.5, reflecting a question genuinely unpleasant for a user to answer — the ranking could flip entirely (U_3-way = 1.585−0.5 = 1.085, still narrowly ahead, but a cost of 0.7 would flip it: 1.585−0.7 = 0.885 < 0.900). This is exactly why step 5 of the algorithm in Chapter 5 maximizes U(q), not EIG(q) directly — a question's raw information content is only half of what determines whether it is worth actually asking.

Where c(q) comes from, in a real deployment

The paper itself keeps c(q) simple — a constant, for tractability in its experiments — but flags that a real system has real options for what to measure there, and the choice matters. Three natural candidates:

The general point survives the specific choice: c(q) is where a system encodes everything about the human side of asking a question that EIG, by construction, cannot see — EIG only ever looks at how much the hypothesis space would shrink, never at how much friction the person on the other end would feel answering it. A production system tuning U(q) = EIG(q) − c(q) is really tuning how much it is willing to spend of a scarce, non-monetary resource — a real person's patience — per bit of ambiguity resolved.

A quick unit check: bits versus nats

Every EIG number in this chapter has been in bits — entropy computed with log₂. Some papers, including parts of the disambiguation paper's own appendix, report entropy in nats instead, using the natural logarithm ln instead of log₂. The conversion is a single constant factor, worth confirming by hand so a nats-reported number never gets silently misread as a bits number three times too small: ln(2) ≈ 0.6931, so

Sanity-check against this chapter's own headline number: the balanced binary question's EIG of exactly 1 bit, converted, should read as 0.6931 nats. Confirm directly from the definition: 1 bit = log₂(2), and in nats that same quantity is ln(2) ≈ 0.693 — matching the conversion constant exactly, which is the whole point of a units cross-check: if it hadn't matched, that would have been a sign of an arithmetic slip somewhere upstream, not a sign the conversion constant itself needed adjusting.

A flag for later: what if the prior over H isn't actually uniform?

Every derivation in this chapter leaned on one simplifying assumption, stated back in the partition-view section: p*(h|S) is uniform over H. That assumption is doing real work — it is what let the messy, unknown true distribution collapse into the clean, computable partition formula this chapter built everything on top of. It is worth flagging, briefly, what happens if it doesn't hold, since Chapter 5 runs directly into a consequence of this. If some hypotheses are genuinely, objectively more likely than others — not because of any model's bias, but because they really are more probable interpretations of the request — then the uniform-prior EIG formula is an approximation, not an exact calculation, and the true EIG could rank candidate questions differently than this chapter's formula does. The paper's own stance, and this chapter's, is that uniformity is the right default assumption precisely because the true prior is unknown and any model-supplied alternative risks importing exactly the kind of unreliable bias Chapter 2 spent an entire chapter warning about. Chapter 5 makes this concrete: it tests exactly this question empirically, comparing the uniform assumption against weighting hypotheses by the model's own stated log-probabilities, and reports which one actually works better on real data.

Chapter 4 recap, before Chapter 5 puts it to work

Four ideas, in the order this chapter built them, worth holding onto as a single chain before Chapter 5 runs the full algorithm end to end: entropy H[p] measures how uncertain a distribution over candidate solutions is; information gain IG(q,a) is how much a specific question-and-answer pair reduces that uncertainty; expected information gain EIG(q) averages IG over every plausible answer, since the real answer isn't known in advance; and, under a uniform prior, EIG collapses to a clean partition-counting formula that is maximized exactly when a question splits the hypothesis space as evenly as possible. Every one of Chapter 5's worked numbers — 1.000 bit, 0.544 bits, 1.561 bits, the log₂(n) ceiling — is that same four-step chain, applied once per candidate question, with nothing hidden inside it that this chapter hasn't already derived from Shannon entropy alone.

The same idea, stated one more way

If the formulas still feel abstract, one more restatement, in plainer terms, before moving to Chapter 5's full worked walkthrough: asking a good clarifying question is a search problem, and EIG is the search heuristic. Among every candidate question the model could ask, the goal is to find the one whose answer — whatever it turns out to be — is most likely to rule out the largest chunk of currently-plausible interpretations. A question that only ever rules out a sliver of the space, no matter which answer comes back, is a weak move; a question that is guaranteed to cut the space roughly in half, regardless of the answer, is a strong one. This is precisely the intuition behind classic binary-search algorithms — splitting a search space as evenly as possible at every step minimizes the expected number of steps needed to find the target — applied here not to a sorted array, but to the space of things a person might have meant.

Chapter 5: Active Task Disambiguation, Worked

Chapter 4 built the scoring function. This chapter builds the algorithm that actually uses it — the paper's full method, called Active Task Disambiguation — and then runs it, by hand, end to end, on the spreadsheet example from Chapter 3.

Why not just ask the model to generate a good question directly?

The obvious approach is implicit reasoning: prompt the LLM with the ambiguous request and ask it to generate a clarifying question. The paper's central empirical claim is that this obvious approach underperforms, and it has a specific hypothesis for why: generating a genuinely discriminating question requires a form of meta-cognitive reasoning — the model has to reason about the shape of its own uncertainty across an entire space of possible solutions, all inside one forward pass, without ever writing any of those solutions down. That is a much harder ask than generating one solution, which is a skill LLMs already have plenty of practice at from pretraining. As the paper puts it, this “may be caused by a relatively small number of clarifying questions present in their pre-training corpus” — the internet is full of solved problems, and comparatively thin on someone visibly reasoning through which follow-up question to ask before solving one.

The fix is to stop asking the model to reason about questions in the abstract, and instead give it something concrete to reason about: actual candidate solutions. This is the paper's core move, stated plainly in its own words — the method “shifts the load from implicit to explicit reasoning about the space of viable solutions.” Sample some solutions, sample some candidate questions, and score each question by how well it would have discriminated among the solutions you already sampled, using the EIG formula from Chapter 4.

The algorithm, step by step

At each round of the conversation, given the current problem statement S^t:

Step 4's EIG estimate does not need to know the true, unknown p*(h|S) at all — it only needs the sample of N candidate solutions the model already generated in step 1, and the pseudo-answers from step 3. The paper's Algorithm 1 makes this concrete:

python
def estimate_EIG(question, pseudo_answers):
    # pseudo_answers: list of N answers, one per sampled candidate solution h_i,
    # describing how h_i would answer `question`
    unique = set(pseudo_answers)
    n = len(pseudo_answers)
    ig = 0.0
    for a in unique:
        n_k = pseudo_answers.count(a)
        p_k = n_k / n
        ig -= p_k * math.log2(p_k)      # Shannon entropy of the answer distribution
    return ig   # this IS the EIG estimate -- entropy of pseudo-answers approximates EIG

Notice this is the same underlying identity as Chapter 4's partition formula, just computed empirically from a sample instead of algebraically from exact group sizes: the entropy of the distribution of pseudo-answers is the EIG estimate, because a question that splits your N sampled solutions evenly across its possible answers has high answer-entropy, and a question where all N samples give the same answer (uninformative — it does not discriminate among your candidates at all) has zero answer-entropy.

Why N=20 and M=5, specifically?

The paper's own 20-questions experiment samples N=20 candidate solutions and M=5 candidate questions at every round — specific numbers worth understanding, not just copying. N, the number of sampled solutions, trades off estimation quality against cost: too small an N and the pseudo-answer distribution in Algorithm 1 is noisy (exactly the kind of sampling error the N=5 worked example below will make concrete), producing an unreliable EIG estimate; too large an N multiplies the number of pseudo-answering calls in step 3 (N solutions times M questions, all needing an answer) without much further improvement, since Study 1's validation against a 500-animal reference list found N=20 already tracked the larger ground-truth estimate well. M, the number of candidate questions, trades off differently: it does not affect estimation noise the way N does, since each question's EIG is estimated independently — a small M simply means fewer candidate questions are compared before picking the best one, risking that the single best possible question in principle was never actually generated as a candidate to begin with. M=5 is a compute-budget choice as much as a statistical one: five separate solution-samplings, once per candidate question, sets the total inference cost of one round of the algorithm at roughly N + M×N pseudo-answering calls (sample N solutions once, then pseudo-answer all N against each of the M candidates) — a cost Chapter 9's discussion of the method's own limitations returns to directly.

Pseudo-answering by hand, on a small sample

To see step 3 and step 4 actually run, shrink the running spreadsheet example down to N=5 sampled candidate solutions (rather than the full population of 8) — this is closer to how the algorithm really operates in practice, since it never has direct access to the full H, only to a finite sample drawn from p_φh. Suppose the model samples these five candidate solutions for “combine these two spreadsheets”: h₂ (single table, union, by ID), h₃ (single table, union, by position), h₄ (single table, intersection, by position), h₆ (separate tabs, union, by position), h₈ (separate tabs, intersection, by position). Now pseudo-answer question Q_A (“single table or separate tabs?”) against each of the five:

Three of five pseudo-answers say “single table,” two say “separate tabs” — a 3/5 split rather than the population's true 4/4, simply because this particular sample of 5 happened not to mirror the full population exactly (a real, expected effect of sampling only 5 out of 8, not an error). Run estimate_EIG on this pseudo-answer distribution by hand:

0.971 bits — close to, but not exactly, the true population value of 1.000 bit computed from the full partition in Chapter 4. That gap (0.029 bits) is exactly what “estimated from a sample” means in practice: the paper's own Study 1 (below) checks precisely this kind of estimation error, at a larger scale, by comparing EIG estimated from N=20 samples against a “ground truth” EIG computed from a much larger reference list, and reports that the small-sample estimate tracks the larger one well enough to be a useful proxy — not perfect, but good enough to reliably rank candidate questions against each other, which is all step 5 of the algorithm actually needs it to do.

The worked example: three candidate questions, one ambiguous request

Return to Chapter 3's request — “combine these two spreadsheets” — and the eight solutions it identified, built from three independent yes/no choices: structure (single table vs. separate tabs), row policy (keep all rows/union vs. only matching rows/intersection), and alignment (match by shared ID column vs. match by row order). Two choices each, three independent axes, 2×2×2 = 8 distinct candidate solutions h₁…h₈, and with no information yet distinguishing any of them, H_before = log₂(8) = 3 bits, exactly as set up in Chapter 4.

Three candidate clarifying questions, each corresponding to a different partition of those 8 hypotheses:

Q_A's and Q_B's EIG values are the exact two numbers derived by hand in Chapter 4. Q_C is new: a three-way question (n=3), with sampled solutions splitting 3/3/2 across its three answers — as close to even as 8 objects can get split three ways.

Q_C beats Q_A — more answer options (n=3 vs. n=2), close to evenly split, land close to that option's theoretical ceiling of log₂(3) ≈ 1.585 bits (Chapter 4's formula). This is the “more answer options can mean more information, at the cost of a harder question” tradeoff made concrete: Q_C teaches more per question, but asks the user to choose among three options instead of two. Under the utility function U(q) = EIG(q) − c(q), whether Q_C is actually worth asking depends on how much heavier its cost c(q) is versus Q_A's — the algorithm from Chapter 5's flow diagram would compute U for all three and let that comparison decide, rather than assuming more options is always better.

Does this actually work on real LLMs? The paper's own experiments

The worked example above is this lesson's own construction, built to make the arithmetic transparent — but the paper runs the identical idea against real models on two real tasks, and the results are the paper's central evidence.

Experiment 1: the 20 Questions game. One LLM (simulated with GPT-4o-mini) thinks of an animal; another LLM tries to guess it within 10 rounds of yes/no-style questions, across 15 different ground-truth animals and 5 seeds each. Four question-generating strategies are compared: implicit (generate one question directly), implicit-ToT (sample 5 candidate questions, then ask the LLM to pick the best one from that set — a Tree-of-Thought-style strategy), EIG-uniform (sample 5 candidate questions, score each by estimated EIG assuming a uniform prior over sampled solutions — exactly the recipe in this chapter), and EIG-logprobs (same, but weighting solutions by the LLM's own log-probabilities instead of assuming uniformity). EIG-uniform “outperforms the remaining strategies by a significant margin” — beating both implicit and implicit-ToT confirms that shifting the reasoning load from questions to solutions genuinely helps, and EIG-uniform beating EIG-logprobs confirms something else important: the model's own raw generative distribution over solutions is not a trustworthy stand-in for the true distribution p*(h|S) — exactly the “uniformity of the unknown” assumption Chapter 4 built on, now empirically justified.

Why weighting by the model's own confidence backfires, illustrated

It is worth seeing why EIG-logprobs underperforms, not just that it does. Return to the eight-hypothesis spreadsheet example. EIG-uniform treats all eight candidate solutions as equally likely when estimating EIG — exactly Chapter 4's assumption. EIG-logprobs instead weights each sampled solution by the model's own stated probability for it. Suppose the model is (wrongly) quite confident that a “single table, union, by ID” interpretation (h₂) is the intended one, assigning it 40% of its own probability mass, while spreading the remaining 60% across the other seven hypotheses roughly evenly (≈8.6% each):

Already lower than the true uniform prior's 3.000 bits, purely because the model's own (potentially wrong) confidence in h₂ makes the starting distribution look less uncertain than it genuinely is. If the model's bias toward h₂ happens to be mistaken — if the requester actually wanted something else entirely — every downstream EIG estimate computed relative to this artificially-shrunk 2.656-bit baseline inherits that same mistake, systematically undervaluing questions that would have helped distinguish away from the model's own favored (and possibly wrong) guess. EIG-uniform never makes this mistake, because by design it refuses to let the model's own generative bias influence the entropy calculation at all — every hypothesis starts on equal footing, regardless of how strongly or weakly the model itself favors any one of them. This is the concrete mechanism behind the paper's own explanation for the result: trusting p_φh as a stand-in for p* imports whatever bias p_φh already has, and Chapter 2 spent an entire chapter establishing that an LLM's raw, unexamined confidence is exactly the kind of number not sitting there worth trusting uncritically.

Experiment 2: active code generation. Starting from an ambiguous function stub (imports, header, a short docstring — following the CodeT benchmark setup), the model asks clarifying “questions” that are really either binary test-case assertions (B-type: true/false) or concrete input→output examples (O-type: run this input, tell me the expected output) — both automatically, noiselessly answered by running the ground-truth program, exactly the kind of external, hallucination-free oracle Chapter 3 flagged as ideal for grounding p̃*. Tested on HumanEval and a harder, competition-level benchmark called APPS (filtered to 48 genuinely non-trivial tasks), EIG-based question selection reaches higher accuracy after eliciting just 4 requirements than zero-shot question generation does — and O-type (open, more answer options) questions prove more informative than B-type (binary) ones, exactly matching Chapter 4's log₂(n) prediction that more answer options raise the ceiling on information gain, at the cost of a harder question to answer.

Two supporting studies: why the method's assumptions actually hold

Two of the assumptions this chapter leaned on — that sampled solutions can be made close to uniformly diverse, and that filtering out hallucinated pseudo-answers matters — are not just asserted in the paper. Both get their own dedicated ablation.

Study 2: does the “diverse and representative” prompt instruction actually work? Step 1 of the algorithm needs sampled solutions {h_i} to approximate a roughly uniform draw over H, not a draw concentrated on whatever single answer the model would give greedily. The paper tests this directly by adding the instruction “generate a carefully selected, diverse, and representative set of animals” to the solution-sampling prompt in the 20-questions experiment, and measuring a diversity score (conditional mutual information between sampled solutions and their pseudo-answers — higher means the samples are covering the hypothesis space more evenly) across 330 real questions collected from the main experiment:

The diversifying instruction measurably raises the diversity score at every sample size tested, and the gap is largest exactly where it matters most — at the smallest, cheapest sample size (N=10: 0.57 vs. 0.46), where an unlucky, narrow sample would do the most damage to the EIG estimate's accuracy. As N grows, both columns climb and the gap narrows (by N=30, 0.60 vs. 0.58) — more samples partially compensate for a less diversity-encouraging prompt on their own, simply by averaging out unlucky draws, which is exactly the kind of relationship you would expect between prompt engineering and sample size as two different levers on the same underlying problem.

Study 3: does filtering out hallucinated pseudo-answers matter? Step 3 asks the model to self-critically check whether each sampled solution actually still satisfies every requirement collected so far, rejecting and resampling any that don't — a defense against hallucinated candidates polluting the EIG estimate. The paper tests 0, 1, and 2 rounds of this filtering, and reports that as the number of accumulated requirements grows across the conversation (more rounds of clarification means more constraints a hallucinating model can silently violate), the gap between 0 filtering rounds and 1–2 rounds widens — more filtering keeps paying off precisely as the task gets more constrained, which is exactly when an unfiltered sample would be most likely to include solutions that only look valid.

The two experiments, side by side

It is worth naming exactly what differs between the paper's two evaluations, because the differences are not incidental — they were chosen specifically to test the hypotheses in H2a and H2b from Chapter 5's own framing (whether the LLM can sample diverse solutions, and whether pseudo-answers are noise-free).

The pairing matters for how much to trust the overall conclusion. Experiment 1 tests the core hypothesis under realistic, noisy conditions — an LLM oracle that can itself make mistakes, closer to what a real deployment answering real clarifying questions from real humans would look like. Experiment 2 strips that noise out almost entirely, isolating whether the EIG-based method's advantage survives even in the best-case scenario where pseudo-answering is essentially free of error. Both experiments reaching the same qualitative conclusion — EIG-based selection beats implicit generation — is stronger evidence than either alone, because it rules out the possibility that the result was somehow an artifact of one specific kind of answer-oracle noise rather than a genuine property of the question-selection method itself.

Chapter 6: Confidence Tokens

Chapter 2 showed that both of the obvious confidence signals — raw logits and verbalized confidence — are measurably unreliable. This chapter builds what the confidence-tokens paper proposes instead: Self-REF (Self-Reflection with Error-based Feedback), a lightweight fine-tuning recipe that teaches a model to emit a genuinely learned, trained-for-the-purpose confidence signal, rather than hoping one falls out of ordinary training for free.

The core idea: two new tokens, trained on correctness

Self-REF adds two special tokens to the model's vocabulary — <CN> (confident) and <UN> (unconfident) — each with its own trainable embedding, initialized as the average of the existing token embeddings. The model is fine-tuned so that, after generating its answer, it also learns to append one of these two tokens: <CN> if that answer turned out to be correct, <UN> if it turned out to be wrong. The recipe has three steps.

Why an internal token, and not an external router model?

An alternative architecture is worth naming before diving into Self-REF's mechanics, because it is the more obvious first idea and the paper explicitly positions itself against it. A large body of prior LLM-routing work trains a separate classifier — a small external router network — that looks only at the incoming query and predicts which model should handle it, before either model has generated anything. The paper's related-work discussion names this pattern directly (citing Ding et al., 2024; Ong et al., 2024; Stripelis et al., 2024, among others) and draws a sharp distinction: those routers “typically rout[e] based on the query alone with an extra router,” deciding before any answer exists, whereas Self-REF's confidence token is “generated end-to-end conditioned on both the prefix query and the answer that the LLM itself generated.”

That difference is not a minor implementation detail. An external, query-only router can only ever learn “how hard does this type of question tend to be for the small model, on average” — it has no access to how the small model's specific attempt at this specific query actually went. Two MMLU questions from the same subject, at the same apparent difficulty, can get very different treatment from Self-REF depending on whether the model's own generated answer, this one time, happened to land somewhere confident or somewhere shaky — a distinction a query-only router structurally cannot see, because it never looks at the answer at all. This is also why Self-REF needs no separate model, no separate training pipeline, and no separate inference call beyond the one the small model was already making: the confidence signal rides along for free inside the same autoregressive pass that produced the answer being judged.

Step 1: Confidence token annotation

Run the base model on the training set to get its own predictions ŷ⁽ⁱ⁾ for every example, then compare each prediction against the true label y⁽ⁱ⁾. Where the model was right, append <CN>; where it was wrong, append <UN>:

python
def annotate_confidence_tokens(model, train_data, alpha=0.5):
    # train_data: list of (x, y) query/true-answer pairs
    confident, unconfident = [], []
    for x, y in train_data:
        y_hat = model.predict(x)
        if y_hat == y:
            confident.append((x, y_hat, '<CN>'))
        else:
            unconfident.append((x, y_hat, '<UN>'))
    # subsample the unconfident set with tunable proportion alpha,
    # to keep the class balance from skewing training toward one label
    unconfident = subsample(unconfident, alpha)
    return confident + unconfident

The alpha subsampling matters in practice: if the base model is already fairly accurate, correct (<CN>) examples will vastly outnumber incorrect (<UN>) ones, and training on that imbalance as-is risks the model learning to always predict <CN> regardless of actual correctness — a shortcut that would score well on raw accuracy of the confidence label while being completely uninformative. Subsampling keeps the model honestly exposed to enough unconfident examples to learn the distinction, not just the class prior.

A worked example of the imbalance alpha corrects for

Ground the abstract concern with numbers. Suppose the base model is 78% accurate on 10,000 training examples — a plausible accuracy for a capable small model on a benchmark like MMLU. Before any subsampling, the raw annotated dataset has:

A model trained on this 78/22 split, with no correction, can already achieve 78% training accuracy on the confidence-label task alone by memorizing the class prior — always predicting <CN> — without learning anything at all about which specific queries it should trust. Setting α = 0.5, meaning keep only half of the unconfident examples relative to what would make the classes exactly balanced, targets a roughly 2,200÷0.5 ÷ (7,800+2,200÷0.5) ratio — concretely, subsampling the confident set down to match a 2:1 ratio against the (now smaller) unconfident set forces the model to actually discriminate, rather than coast on the prior. The paper leaves α itself as a tunable hyperparameter precisely because the right balance point depends on how skewed the base model's raw accuracy already is — a much weaker base model, closer to 50% raw accuracy, would need far less correction, since its classes would already be close to balanced on their own.

Step 2: Fine-tuning, with a crucial safeguard

The straightforward version of this fine-tuning step has a subtle trap. If you naively train the model to reproduce the entire augmented sequence — including generating the incorrect answer text itself, followed by <UN> — ordinary cross-entropy fine-tuning would increase the probability of that wrong answer given that query, exactly the opposite of what you want. You would be actively teaching the model to be more likely to reproduce its own mistake next time, purely as a side effect of teaching it to correctly label that mistake as unconfident.

The fix is gradient masking: mask out the gradient contribution from the incorrect-answer tokens themselves in the unconfident examples, so those tokens never get reinforced. What remains flows freely: the model still learns to increase P(<CN> | correct answer, query), to increase P(<UN> | incorrect answer, query), and — through the ordinary confident-example gradient, which is not masked — to increase P(correct answer | query) overall.

Seeing the gradient direction, one token at a time

Make the mechanism fully concrete with a toy cross-entropy update. Suppose an unconfident example's target sequence is the (wrong) answer token W followed by <UN>, and at some point during training the model currently assigns P(W|query)=0.3 and P(<UN>|W,query)=0.4. Ordinary cross-entropy fine-tuning computes a loss term for every token in the target sequence and back -propagates a gradient that increases the probability of each one:

Both terms in that sum push gradients in the same direction: increase the probability of whatever token they attach to. The first term's gradient increases P(W|query) — the wrong answer itself. The second term's gradient increases P(<UN>|W,query) — the honest confidence label. Without gradient masking, the optimizer treats both as equally worth reinforcing, and every training step on this example nudges the model to be a little more likely to output W again next time it sees a similar query — even though W was established, in the very same example, to be wrong.

Gradient masking simply zeroes out the first term's contribution to the backward pass, leaving the loss the optimizer actually acts on as:

Note that P(W|query) still appears in the forward pass — the model still needs to have generated W in order to reach the position where <UN> gets predicted — but no gradient flows back through it during this update. Nothing in this masked version pushes the model toward regenerating W more often; the entire update's signal is now concentrated on one thing: getting better at recognizing, after the fact, that W (or an answer like it) deserves an unconfident label. Extended across an entire training set of unconfident examples, this is the difference between a model that slowly, invisibly, gets nudged toward repeating its own past mistakes, and one that only ever gets better at flagging them.

Step 3: Extracting a continuous confidence score

After fine-tuning, the model does not just output a discrete <CN>/<UN> label — both tokens still have their own softmax probabilities at the position right after the answer, and those two numbers can be turned into a continuous score between 0 and 1:

This normalizes by the sum of just these two tokens' probabilities, ignoring how much probability mass went elsewhere in the vocabulary at that position — which matters, since <CN> and <UN> are competing against every ordinary continuation token too, not just each other.

A worked example

Suppose, at the position right after a generated answer, the fine-tuned model assigns raw softmax probability 0.62 to <CN> and 0.11 to <UN> (with the remaining 0.27 spread across ordinary vocabulary tokens that happen to also be plausible continuations at that position). The confidence score is:

An 84.9% confidence score — a genuine number, thresholdable, and (per Table 2 from Chapter 2) considerably more trustworthy than either raw logits or verbalized confidence, because it was produced by a token the model was specifically fine-tuned, with gradient masking, to align against actual correctness.

Two downstream uses of the same score

Once c_M(x, ŷ) exists, the paper studies two different things a deployed system can do with it, against a chosen threshold t:

Reading a rejection threshold by hand, on a toy batch

To make the ROC framing concrete, work a small toy batch of 20 questions: 10 genuine multiple-choice questions (correct answer choice intact) and 10 “none of the above” questions (correct choice removed). Suppose, at some chosen threshold t, the model abstains (assigns c_M < t) on 8 of the 10 “none of the above” questions, and also, unfortunately, on 2 of the 10 genuine questions:

At this threshold, the system catches 80% of the genuinely unanswerable questions, at the cost of also wrongly abstaining on 20% of questions it could have answered correctly. Lowering t (harder to trigger abstention) would move both numbers down together — fewer of the truly bad questions get caught, but fewer good ones get needlessly rejected too; raising t moves both up together. Sweeping t across its whole range and plotting TPR against FPR at each value is exactly what an ROC curve is, and a model whose confidence score genuinely tracks correctness (rather than being noise, which would trace the diagonal TPR=FPR line) rises steeply toward the top-left corner — high TPR, low FPR — well before t reaches its extreme values. The paper reports Self-REF's rejection-learning ROC curve sitting consistently above all four Chapter 2 baselines across this exact tradeoff, on both MMLU and OpenBookQA.

A second point on the curve, and what “better” looks like across two points

One point on an ROC curve, on its own, cannot say whether a confidence signal is good — it only describes one specific operating choice. Add a second, stricter threshold to the same toy batch: raise t so the model now abstains on 9 of the 10 “none of the above” questions, but that stricter bar also catches 4 of the 10 genuine questions this time:

Plot both points on the same TPR-versus-FPR axes: (0.20, 0.80) from the looser threshold, (0.40, 0.90) from the stricter one. Both sit well above the diagonal TPR=FPR line a random, uninformative confidence score would trace — at FPR=0.20, random guessing would only catch 20% of bad questions on average, not 80%. This is the concrete meaning behind “Self-REF's ROC curve sits above the baselines”: at any FPR you are willing to tolerate, tracing straight up from that x-value to Self-REF's curve lands higher (more bad questions correctly caught) than tracing up to any baseline's curve at that same FPR. A confidence signal is not “good” or “bad” at one fixed threshold — it is good or bad as a curve, compared against alternatives across the whole range of tradeoffs a deployment might choose to operate at, which is exactly why the paper reports full ROC curves (Figure 3) rather than a single TPR/FPR pair per method.

Does the confidence signal transfer to new domains?

One more question worth flagging, since it goes directly to how deployable this technique is: if you fine-tune Self-REF's confidence tokens on one dataset, do they carry any signal on a genuinely different one, or does every new domain need its own dedicated fine-tuning pass from scratch? The paper studies exactly this (Section 5.5, “Transferability of Confidence Token”), evaluating whether confidence tokens trained on one benchmark still carry meaningful signal when applied to a held-out domain the fine-tuning never saw. This matters operationally: a Self-REF pass trained once on a broad, general-knowledge benchmark like MMLU that transfers reasonably well to a narrower domain would be far cheaper to deploy than a bespoke fine-tuning run per downstream task category. Whether or not full transfer holds, the underlying question is the same one Chapter 9 returns to as a genuine limitation of the whole confidence-tokens approach: how much of what Self-REF learns is “this specific model, on this specific benchmark's notion of correctness” versus a more general skill at self-assessment that would carry to an entirely new task category without retraining.

Recap: the full Self-REF pipeline, start to finish

Six chapters' worth of individual pieces, assembled into one sequence:

Every step in that chain is something this chapter derived from first principles rather than asserted — step 1's subsampling rationale, step 2's gradient-masking mechanism worked through one toy loss computation, step 3's normalization formula worked through one concrete probability pair, and steps 4–5 handed off directly to Chapter 7's real, measured numbers. Nothing in this pipeline requires touching the base model's architecture, and nothing requires abandoning whatever the base model already does well — the entire intervention is two new tokens, one careful fine-tuning pass, and a threshold.

Chapter 7: The Cost-Coverage Tradeoff

Chapter 6 built the confidence score. This chapter asks the operational question it exists to answer: given that score, at what threshold should a deployed system actually route queries to the larger, more expensive model — and what does that decision cost, and save?

Setting the threshold: sweeping quantiles

Rather than pick one threshold t by hand, the paper sweeps a whole range: compute the confidence score for every query in a held-out set, then set t at each of 21 quantiles of that distribution (roughly every 5%, from the 0th to the 100th percentile). Each threshold routes a different fraction of queries to the large model — a threshold at the 0th percentile routes nothing (pure small model), a threshold at the 100th percentile routes everything (pure large model). Sweeping the whole range traces out a full accuracy-versus-routing-rate curve, letting you read off, for any accuracy target, the cheapest routing rate that hits it.

The headline result

On MMLU, Self-REF-routed Llama3-8B-Instruct reaches accuracy 0.739 — identical, to three decimal places, to sending every single query to Llama3-70B-Instruct alone — while routing only 39% of queries to the larger model. The other 61% of queries are answered directly by the small model and are already trustworthy enough, by Self-REF's own honest accounting, that sending them onward would not improve the answer. The full picture across all four benchmarks, and both small local models tested:

Two patterns worth noticing. First, GSM8K needs by far the highest routing rates (65–75%) — grade -school math word problems are exactly the kind of task where a smaller model's raw capability gap, not miscalibration, is the bottleneck, so more queries genuinely need the larger model's extra capability. Second, Llama3-8B-Instruct routes at a consistently lower rate than Mistral-7B-Instruct across every dataset — Self-REF is a fine-tuning recipe applied to whatever base model you give it, and how well it works is bounded by how much genuine signal the base model's confidence carries in the first place.

Reading the GSM8K gap through Chapter 3's lens

It is worth reconnecting GSM8K's 65% routing rate back to Chapter 3's model-uncertainty-versus-ambiguity split, because it is a clean illustration of a case where confidence-based routing is doing exactly what it should, even though the routing rate looks “worse” than MMLU's. GSM8K questions are unambiguous — each has exactly one numerically correct answer, so H = H* for essentially every question in the benchmark; the requirement side of the problem is fully specified. What varies is whether the small model's arithmetic and multi-step reasoning is actually strong enough to reach that one correct answer. This is squarely the top-left/bottom-left column of Chapter 3's table — unambiguous tasks, where the only question is whether the model's own uncertainty is high or low — not a case where a clarifying question would help at all (there is nothing left to clarify), which is exactly why GSM8K's fix is routing to a more capable model (Chapter 6–7's machinery), not asking a follow-up question (Chapters 4–5's machinery). A 65% routing rate on GSM8K is not evidence that Self-REF's confidence signal is failing on this benchmark — it is evidence that the confidence signal is working exactly as designed, correctly reporting that a large genuine majority of grade-school math problems are hard enough, for this particular small model, that a bigger model really does help.

Two currencies: latency and dollars do not always move together

Table 1 reports both accuracy and per-token latency, and it is tempting to treat “faster” and “cheaper” as interchangeable, but they are measuring genuinely different resources, and a real deployment often has to trade one against the other. Latency (seconds per token) is primarily a function of model size and hardware — a smaller model is faster essentially by definition, since it has fewer parameters to push through on every forward pass. Dollar cost, especially against a hosted API, often follows a different curve entirely: providers price frontier models at a premium well beyond what their extra latency alone would suggest, because the pricing also reflects training cost amortization, demand, and the value of the extra capability, not FLOPs alone. Two consequences worth naming. First, a deployment optimizing purely for user-facing responsiveness (latency) and one optimizing purely for operating budget (dollar cost) can land on different optimal routing rates, even when starting from the identical Self-REF confidence scores — the same threshold sweep produces one accuracy-vs-latency curve and a differently-shaped accuracy-vs-dollar-cost curve, and Chapter 7's canvas widget only plots the latency version. Second, this is exactly why a production system's threshold t is a business decision as much as a statistical one: it requires knowing which resource — user-facing speed, or operating budget, or both jointly — the deployment actually needs to conserve, information that lives entirely outside anything Self-REF's confidence score itself can tell you.

What happens if the base model changes?

Table 1's two small models give a direct, measured answer to a question every deployment eventually faces: how much does the routing-rate cost depend on which base model Self-REF was fine-tuned onto? Compute the ratio of Mistral-7B-Instruct's required routing rate to Llama3-8B-Instruct's, dataset by dataset:

Averaging those four ratios: (1.79+1.02+1.15+1.75)÷4 ≈ 1.43 — Mistral-7B-Instruct needs, on average across these four benchmarks, roughly 43% more of its traffic routed to the large model to hit the same accuracy-parity bar Llama3-8B-Instruct reaches with less. Since routing rate translates close to linearly into both latency and dollar cost (the linear cost model derived earlier in this chapter), that 43% gap is not just a routing-rate curiosity — it is directly, proportionally, a cost gap between the two base models once Self-REF is applied to each. The lesson generalizes past these two specific models: Self-REF is a fine-tuning recipe, and how cheaply it lets you operate depends on how much genuine, extractable correctness signal the base model already carries in its own generative distribution before Self-REF ever touches it — a better base model is not just more accurate on its own, it can also end up cheaper to deploy under confidence-based routing, because it needs the expensive fallback less often.

Building the cost curve, from two real anchor points

The “All in Llama3-70B-Instruct” row reports 0.292 sec/token, constant across every dataset — sensible, since that number depends only on the large model itself, not on which benchmark is being served. That is the true, reported latency at routing rate r=1 (everything goes to the big model). The MMLU row's 0.145 sec/token is the true, reported latency at r=0.39 (Self-REF's actual operating point on that benchmark). Modeling the blended latency as a simple linear function of the routing rate — a reasonable first approximation, since at any given moment a fraction r of queries pay the large model's cost and a fraction (1−r) pay the small model's — lets you back out what the implied small-model-only latency (r=0) would have to be:

That 0.051 is not a number the paper reports directly — it is an estimate, back-solved from two real reported numbers under the assumption of a simple linear cost blend. It is a useful sanity check, though: the small model alone should cost meaningfully less per token than the blended 0.145 figure, and 0.051 is indeed well under it, in the direction the physics of the problem demands.

Where cost and coverage actually trade off

With that linear model in hand, you can now ask the operational question directly: what happens if you route less than 39% of MMLU queries, to save even more cost? The linear model says latency keeps dropping smoothly — at r=0.20, for instance, predicted latency is 0.8×0.051 + 0.2×0.292 ≈ 0.099 sec/token, nearly 3× faster than always using the large model. But the accuracy side of the tradeoff does not extend that smoothly with any number this session has reported: 39% is the paper's measured minimum routing rate for matching 70B-level accuracy on MMLU specifically; below it, the paper's own language only supports “accuracy is lower than parity,” not any specific number for how much lower. That boundary — the exact routing rate where accuracy-parity starts holding — is precisely what the threshold sweep in Table 1 exists to find, dataset by dataset, rather than something you could derive from cost math alone.

An illustrative dollar example

To feel the economics at typical API pricing shapes — purely illustrative unit costs, not figures from either paper — suppose the small model costs $0.20 per 1,000 output tokens and the large model costs $3.00 per 1,000 output tokens (roughly the order-of-magnitude gap between a small open model and a frontier closed one). At MMLU's real r=0.39 operating point:

Against a policy that always uses the large model ($3.00 per 1,000 tokens, for the same 0.739 accuracy), this is a savings of (3.00−1.292)÷3.00 ≈ 57% in cost, for zero measured loss in accuracy on this benchmark — the entire value proposition of confidence-based routing, priced out in one line.

How sensitive is that saving to the actual price gap?

$0.20 versus $3.00 was one particular illustrative price ratio (15×). Real API pricing gaps between a small and a frontier model vary, so it is worth checking how the savings percentage moves as that ratio changes, holding the routing rate fixed at MMLU's real, measured 39%. The blended-cost formula collapses to a clean expression once you factor out the large model's price:

The savings percentage climbs with the price gap but flattens out fast — it can never exceed (1−r) = 61% no matter how cheap the small model gets, because 39% of queries are still, by construction, paying the large model's full price regardless of how the small model is priced. This ceiling is a direct, visible consequence of the routing rate itself: halving the small model's cost again, from 30× to 60× the price gap, would buy barely another percentage point of savings, while lowering the routing rate itself (if a future, better-calibrated confidence signal allowed a lower r while still hitting accuracy parity) would move that ceiling directly. Cost savings from routing are gated by how well you can identify which queries need the expensive model, not primarily by how cheap the alternative is.

One more worked example: a mixed real-world traffic estimate

Every worked number so far picked one dataset (MMLU) in isolation. A real deployment does not field only MMLU -style questions — it sees a mix. Suppose, illustratively, a support assistant's incoming traffic splits evenly across query types resembling the four benchmarks in Table 1 (general knowledge like MMLU, reasoning like OpenBookQA, arithmetic like GSM8K, and domain-specific lookups like MedQA), using Llama3-8B-Instruct's four real routing rates:

Apply that blended rate to the same $0.20-versus-$3.00 illustrative pricing from earlier in this chapter:

Lower than MMLU's own 56.9% figure — unsurprising, since the mix now includes GSM8K's much higher 65% routing rate, dragging the blended average up and the blended savings down. This is the honest shape a real enterprise deployment estimate takes: not one dataset's number, quoted as if it applied everywhere, but a traffic-weighted average across whatever mix of query types the deployment actually receives, which in practice means measuring the real traffic distribution first, not assuming it resembles any one published benchmark.

When Chapter 6's rejection path is the only option

Everything in this chapter has assumed a larger model is available to route to. Chapter 6 flagged the other case directly: sometimes it isn't — a small model may be the only model a deployment has access to, whether for cost, latency, data-residency, or air-gapped-deployment reasons. In that setting, this chapter's entire cost-coverage machinery still has something to offer, just aimed at a different action. Instead of sweeping routing thresholds against a large model's accuracy, sweep the same confidence-score thresholds against an abstention policy's tradeoff — the ROC-curve accounting worked through in Chapter 6, true-positive (correctly abstained) rate against false-positive (needlessly abstained) rate. The threshold-sweeping mechanics this chapter built (quantiles of the confidence-score distribution, a fixed evaluation set, comparing an accuracy or safety metric at each candidate cutoff) transfer directly; only the downstream action, and therefore what “good” looks like at the far end of the sweep, changes. A deployment does not need to choose between building routing machinery and building rejection machinery in advance — the confidence score itself is identical either way, and which action a low-confidence query triggers is a downstream policy decision, made per deployment, not baked into how the score itself gets computed.

Chapter 8: Ask, Answer, or Defer

Every piece is now on the table: Chapters 1–2 gave a way to measure and distrust raw confidence. Chapters 3–5 gave a way to detect and quantify ambiguity, independent of confidence. Chapters 6–7 gave a trained, genuinely calibrated confidence signal and a cost-coverage tradeoff for using it. This chapter fuses all three into one decision rule a deployed system can run on every single incoming request.

Why ambiguity has to be checked before confidence

Chapter 3's table made the danger case explicit: a model can be highly confident while having silently picked just one of several equally valid interpretations of an ambiguous request. Confidence, as built in Chapter 6, measures “how likely is this specific answer to be correct, given the interpretation I assumed” — it says nothing about whether that assumed interpretation was the right one to answer in the first place. A 95%-confident answer to the wrong question is not a safe answer; it is a fast, well-produced version of Chapter 0's clean-up-this-script failure. This is why the decision rule below checks ambiguity first: resolving which question you are actually answering has to happen before your confidence in the answer to that question means anything at all.

The three-way policy

Given an incoming request, compute two numbers: the best available clarifying question's EIG (Chapters 4–5 — how ambiguous is this request, and is there a good question that would resolve it?) and the model's confidence score in its own best answer (Chapter 6). Two thresholds, t_EIG and t_conf, chosen by whoever operates the system, turn those two numbers into one of three actions:

python
def decide(eig_best_question, confidence, t_eig, t_conf):
    if eig_best_question >= t_eig:
        return 'ASK'       # request is ambiguous enough that a question is worth it
    elif confidence >= t_conf:
        return 'ANSWER'    # unambiguous enough, and the answer is trustworthy
    else:
        return 'DEFER'     # unambiguous, but not confident -- route onward (Ch 7) or abstain

Read the branches in order, because the order is the point. ASK fires first: if the best available clarifying question would meaningfully shrink the hypothesis space, ask it, before even looking at confidence — per Chapter 4, EIG only reflects how much a question would narrow H, so this branch is entirely about the shape of the request, not about how the model currently feels about any one answer. Only once ambiguity is ruled out does confidence get consulted: ANSWER if the (now well-posed) request produces a trustworthy-confidence answer, per Chapter 6's genuinely-learned score, not the raw logits Chapter 2 showed cannot be trusted. Otherwise, DEFER — route to a larger model (Chapter 7's cost-coverage machinery) or abstain outright (Chapter 6's rejection-learning use case), rather than emitting a low-confidence guess as if it were a real answer.

The showcase: a live decision engine over a stream of requests

The simulation below plots a stream of toy incoming requests as points on two axes: x is the EIG of the best clarifying question available for that request (in bits, per Chapter 4's formula), and y is the model's confidence in its best single-shot answer (per Chapter 6). Two sliders control t_EIG and t_conf; moving them redraws the three decision zones live, and every point's color updates to show which action it now falls into. A running readout at the bottom sums an illustrative per-action cost (a small fixed cost for asking a follow-up question, zero for answering directly, and a larger fixed cost for deferring to a bigger model or a human), so you can feel how tightening or loosening either threshold trades safety for cost in real time.

One design choice worth being upfront about: the fourteen points are fixed, hand-placed toy values, not the output of a live model running Chapters 4–6's machinery in real time. That is a deliberate simplification, for the same reason every canvas in this session uses worked or illustrative numbers rather than a live API call — it keeps the simulation deterministic and inspectable, so the exact same fourteen (EIG, confidence) pairs are there every time you load this page, and the hand-worked walkthrough above can check its own math against exactly what the canvas renders. A real deployment's version of this exact plot would instead be populated continuously, in real time, from Chapter 5's EIG estimation running on each incoming request and Chapter 6's confidence-token extraction running on that request's generated answer — the visualization's structure would be identical; only the source of the two coordinates would change, from a fixed array to a live pipeline.

Walking the decision function by hand, on four of the fourteen points

Before touching either slider, run decide() from the code block above by hand on four of the fourteen toy requests plotted in the simulation, at its starting thresholds: t_EIG = 0.80 bits, t_conf = 65%.

Request #4 is worth lingering on. Its confidence is only 40% — well under the 65% bar, which in isolation would suggest DEFER — but it never reaches Branch 2 at all, because its EIG lands exactly on the threshold and Branch 1 fires first. This is the ordering argument from earlier made concrete: a low confidence score on a request that is also genuinely ambiguous is not evidence the model needs a bigger model to answer it correctly — it is evidence the model does not yet know which question it is even trying to answer, and no amount of routing to a more capable model fixes that; a bigger model handed the same ambiguous request would face the exact same underspecified H. Running all fourteen points through the same two branches gives 6 ASK, 4 ANSWER, and 4 DEFER, for a total illustrative cost of 6×0.15 + 4×0 + 4×1.0 = 4.90 units — the exact starting numbers the live simulation below reports before you move either slider.

Reading the tradeoff off the simulation

Push t_conf up toward 90% and watch the ANSWER zone shrink — fewer points clear the bar, more fall into DEFER, and the running cost readout climbs, because deferring is the most expensive action in this toy accounting. This is exactly the same accuracy-versus-cost tension Chapter 7 measured with real numbers: a higher confidence bar means fewer low-confidence answers slip through unflagged, at the price of routing (and paying for) more queries. Push t_EIG down toward 0 instead, and watch the ASK zone swallow almost the entire plot — every request, however mildly underspecified, now triggers a clarifying question, which is safe but would frustrate a real user fielding a question for every message they send. There is no threshold setting that minimizes both cost and risk simultaneously; operating a system like this means choosing where on that curve you are willing to sit, deliberately, rather than leaving it to whatever a model's raw, unexamined confidence happens to produce.

What a confidence-only policy would have missed

Chapter 3 warned, in the abstract, that a confidence-only policy would confidently mishandle ambiguous requests. The fourteen plotted points make that concrete rather than hypothetical. Strip out the EIG branch entirely and imagine a simpler policy — ANSWER if confidence ≥ 65%, DEFER otherwise, with no ambiguity check at all — run against the same fourteen requests. Request #3 from the earlier hand-worked table (eig=1.40, conf=0.88) is the case worth staring at: under the combined policy it correctly triggers ASK, because EIG=1.40 comfortably clears the ambiguity threshold. Under the confidence-only policy, it would sail straight to ANSWER, since 88% clears the confidence bar with room to spare — the exact top-right, most-dangerous cell from Chapter 3's table, produced not as an abstract warning this time but as one specific point on this specific simulation, confidently mishandled by a policy that never checked whether the request was well-posed in the first place. Running the full comparison across all fourteen points, six of them (every point the combined policy sends to ASK) get silently reclassified as either ANSWER or DEFER under the confidence-only policy, purely because that simpler policy has no way to represent “this request itself needs clarification” as an outcome at all — it only ever has two boxes to sort anything into.

Setting the two thresholds in practice

Neither t_EIG nor t_conf has a universal correct value — Chapter 0's clinical-support example already argued both should sit at different levels depending on the cost of getting something wrong in a given domain. In practice, both are set the same way the routing threshold t was set in Chapter 7: sweep a range of candidate values against a held-out, labeled evaluation set (real requests, with a known ground-truth best action for each), and pick the operating point that hits whatever safety or cost target the deployment actually needs — not a value chosen once by intuition and left untouched. Two practical notes worth carrying forward. First, the two thresholds are not independent: Chapter 8's showcase deliberately checks ambiguity before confidence, so t_EIG effectively gates how much of the incoming traffic ever reaches the t_conf decision at all — tightening t_EIG (asking more often) shrinks the pool of requests where t_conf even gets a chance to matter. Second, both thresholds should be revisited whenever the underlying model changes — a newly fine-tuned Self-REF pass (Chapter 6) shifts what a given confidence number actually means, and a different base model's clarifying-question quality shifts what a given EIG number is worth, so a threshold tuned for one model version is not guaranteed to still be well-calibrated for the next one.

Beyond three actions

This chapter's policy has exactly three branches because that is the minimum needed to demonstrate the core idea — check ambiguity, then check confidence — cleanly. A real system can, and often should, subdivide further without changing the underlying logic. ANSWER could split into “answer directly” versus “answer, but visibly flag the confidence level to the user” (useful when t_conf is cleared but only barely, say at 66% against a 65% bar — technically above threshold, but close enough to the boundary that surfacing the number itself is more honest than hiding it behind a binary decision). ASK could split by how many rounds of clarification have already happened this conversation, tightening t_EIG on each successive round so the system does not ask forever (a crude, practical stand-in for the real stopping rule Chapter 9 flagged as an open problem). DEFER could split into “route to a larger model” versus “abstain and hand off to a human”, using Chapter 6's rejection-versus-routing distinction rather than collapsing both into one action. None of these extensions change the two-signal, ambiguity-then-confidence architecture this chapter built — they refine what happens once each branch has already been chosen.

Where the showcase's illustrative unit costs came from

The simulation's cost accounting — 0.15 for ASK, 0 for ANSWER, 1.0 for DEFER — was chosen to make the relative ordering visually obvious (defer is clearly the most expensive, ask is a modest tax, answer is free), not fit to any measured system. A genuinely useful version of this accounting, for a real deployment, would instead plug in Chapter 7's actual measured numbers: DEFER's cost would be the real blended latency or dollar delta from Table 1 (the 2.03× latency gap on MMLU, or the illustrative $2.71-per-1,000-tokens premium worked out earlier in Chapter 7), ASK's cost would be measured directly from real user studies — how much longer does a conversation take, how often do users abandon it, when a clarifying question is inserted — and ANSWER's cost is not truly zero either, since it is the branch that silently absorbs whatever residual miscalibration Chapters 1–2 measured. Replacing this chapter's illustrative 0.15/0/1.0 with real, measured numbers from a specific deployment is exactly the kind of grounding step that turns this showcase from a teaching tool into an actual operating budget.

Recomputing the showcase's default cost with real Chapter 7 numbers

Take that suggestion literally, for MMLU specifically, and see how the total cost readout would change. Replace DEFER's illustrative 1.0 with the real dollar-cost delta of routing one query to the large model instead of the small one: from Chapter 7's illustrative pricing, $3.00−$0.20 = $2.80 marginal cost per 1,000 tokens routed. Rescale ASK's cost proportionally, say to $0.05 per question (a small fraction of DEFER's cost, reflecting one extra short exchange rather than a full large-model call), and leave ANSWER at $0 (its true cost, Chapter 7's own accounting reminds us, is not actually zero, but no clean per-query dollar figure for latent, undetected miscalibration risk exists to plug in here). Recompute the fourteen-point default total from the hand-worked table earlier in this chapter — 6 ASK, 4 ANSWER, 4 DEFER — under these re-scaled costs:

Compare the shape of this total against the illustrative 4.90-unit total computed earlier in this chapter: DEFER now dominates the total even more heavily than the toy units suggested (11.20 out of 11.50, or 97%, versus 4.0 out of 4.9, or 82%, in the original illustrative accounting) — because the real measured price gap between a small and large model is about 15× (Chapter 7's own worked ratio), noticeably larger than the toy 1.0-versus-0.15 ratio (about 6.7×) this chapter started with. The qualitative lesson from earlier in this chapter survives the substitution unchanged — DEFER is still by far the most expensive action, ASK is still a modest tax — but the exact magnitude of that gap only becomes trustworthy once real, measured numbers replace illustrative placeholders, which is the entire reason Chapter 7 built the real cost-coverage curve in the first place rather than leaving this chapter's toy units as the final word.

What noisy EIG estimates would do to this decision rule

Chapter 5's pseudo-answering worked example showed EIG estimated from a small sample (0.971 bits, from N=5) can sit noticeably off from the true population value (1.000 bit, from the full |H|=8). It is worth asking what that estimation noise would actually do to the ASK/ANSWER/DEFER classification, since this chapter's x-axis is exactly that estimated EIG. Take request #3 from the hand-worked table (true EIG ≈ 1.40, well clear of the t_EIG=0.80 threshold) — a modest sampling error of even ±0.3 bits, plausible at small N, still leaves the estimate comfortably above threshold, so the ASK decision is robust to noise at that magnitude. Now take request #4, the edge case sitting exactly at eig=0.80 — the same ±0.3 bit noise band would straddle the threshold entirely, meaning this specific request's classification could flip between ASK and “continue to the confidence check” depending on which particular sample of candidate solutions the algorithm happened to draw that round. This is not a flaw unique to this chapter's toy simulation; it is a direct, structural consequence of Chapter 5's N=20-sample EIG estimator being an estimator, not an oracle. The practical implication for a deployed t_EIG: requests whose true EIG sits close to the threshold are inherently the least reliably classified ones, and a production system's tolerance for that boundary noise — how large a margin to leave around t_EIG before trusting a classification — is exactly the kind of calibration-of-the-calibration question that only shows up once you take estimation noise seriously, rather than treating EIG as a number handed down with certainty.

Chapter 9: Limits & Connections

Both papers this session is built on are candid about what their own methods do not solve. That honesty is worth taking seriously before treating either method as a finished, deployable answer.

What active task disambiguation does not tell you

The disambiguation paper is explicit, in its own discussion section, that it addresses only one piece of a larger problem: given that a task is ambiguous, which question best resolves it. It does not solve two adjacent questions the paper names directly — first, detecting that a given problem statement is ambiguous at all in the first place (the paper points to prior zero-shot-prompting approaches as a partial answer, not something this method itself provides); and second, deciding when to stop asking questions — how many rounds of clarification are enough before the marginal EIG of one more question no longer justifies the interaction cost. Chapter 8's decision rule assumed both of these were already handled upstream (a fixed t_EIG substitutes for a real stopping rule); a production system would need to solve them for real.

The paper also flags a real cost the EIG-based strategies pay that the simpler baselines do not: generating M candidate questions, sampling N candidate solutions, and pseudo-answering every combination is meaningfully more LLM calls per turn than a single implicit-question prompt. The paper's own stance, borrowed from the Bayesian Experimental Design convention it builds on, is that this extra computation is worth it as long as it stays cheap relative to the cost of getting a wrong answer to the human on the other end — an assumption the paper expects to hold more often, not less, as token costs keep falling over time.

Put a number on that extra cost, using Chapter 5's own N=20, M=5 hyperparameters. One round of the implicit-question baseline costs exactly 1 LLM call (generate one question). One round of EIG-uniform costs, per the paper's own procedure: N calls to sample solutions, M calls to sample candidate questions, and up to N×M calls to pseudo-answer every solution against every question — 20 + 5 + (20×5) = 20+5+100 = 125 calls, before even counting the extra self-critic filtering passes Study 3 in Chapter 5 added on top. That is a two-orders-of-magnitude gap in raw LLM calls per clarifying question, for one round of one conversation. The paper's BED-inherited stance — that this is worth it as long as it is cheap relative to a wrong answer's downstream cost — is not a claim that 125 calls is free; it is a claim that Chapter 0's “confidently wrong, unflagged, 20 minutes of rework” cost is worse. Whether that comparison actually holds for a given deployment is an empirical question about that deployment's own economics, not something either paper's theory settles in the abstract.

Where these two papers sit relative to prior work

What confidence tokens need that not every task has

Self-REF's annotation step (Chapter 6) needs a clean, checkable notion of “correct” to label <CN> versus <UN> in the first place — exactly what MMLU, OpenBookQA, GSM8K, and MedQA all provide, since each has an unambiguous right answer to check predictions against. Open-ended generation — summarization, creative writing, most real chat — does not come with an exact-match correctness signal this cheap to compute, and extending Self-REF's annotation pipeline to those settings would need a substitute for “was this literally right or wrong,” which is a genuinely harder labeling problem than anything this chapter's four benchmarks required.

A third limitation, from this session's own synthesis

One more limitation worth naming, and it belongs to this session's Chapter 8 fusion specifically, not to either paper's own stated limitations: nothing in either paper's evaluation measures whether the ask-answer-or-defer policy, as a combined system, is itself well-calibrated. Chapter 6's ECE numbers measure whether c_M is honest; Chapter 5's EIG evaluation measures whether question selection outperforms baseline strategies at narrowing H. Neither paper reports whether the specific downstream decision — the three-way branch itself — achieves, say, a target false-defer rate or a target false-answer rate when both signals are combined with real, jointly-tuned thresholds, on real traffic. That combined evaluation is exactly the kind of thing Metric Design exists to help build, and exactly the kind of gap a team shipping Chapter 8's policy for real would need to close with their own evaluation, rather than assuming that two individually-validated signals automatically compose into a validated combined system.

The two papers, side by side

It is worth stepping back and naming exactly how these two papers relate, since this session treated them as two halves of one problem rather than as one paper extending the other — they are independent works, published roughly eight months apart, that this session deliberately paired because their conclusions turn out to be complementary.

Neither paper's method depends on the other's existing. You could deploy Active Task Disambiguation's clarifying -question machinery on a model with no confidence-token fine-tuning at all, and you could deploy Self-REF's routing on a model that never asks a clarifying question. Chapter 8's three-branch policy is this session's own synthesis, not a method either paper proposes directly — it is what naturally falls out of taking both papers' conclusions seriously at once, and it is exactly the kind of fusion a working engineer building a real deployed system would need to construct for themselves, because no single paper in this space currently addresses the full ask-answer-or-defer decision as one unified problem.

One shared trait is worth naming, since it is easy to miss when the two papers are read separately: both methods deliberately avoid touching the base model's weights for the actual reasoning task. Active Task Disambiguation's clarifying questions come from ordinary prompting of an unmodified model; Self-REF's fine-tuning touches only the two new confidence tokens' embeddings and the surrounding gradient flow, not the model's general language-modeling ability. Neither paper asks “can we make the model fundamentally smarter,” which is the harder, more expensive research direction most of this course's other sessions pursue. Both instead ask a narrower, cheaper question: “can we extract more honest, more useful information from a model we already have, without retraining it to be better at the underlying task?” That framing is precisely why both fit naturally into a single deployment session — they are complementary answers to the same kind of question, applied to two different failure modes.

Comparing the whole toolkit

Read this table as a single, compact answer to the question this whole session opened with: how does a deployed system know what it knows? Not from any one row alone — the top three rows all answer “how much should I trust this specific answer,” and the bottom two answer a completely different question, “does this request even have one right answer to be confident about in the first place.” Chapter 8's combined policy is the claim that both questions have to be asked, in that order, on every single request, and that no single row in this table — however good its own measured reliability — is a substitute for asking the other one too.

Bridges

This session's Bayesian framing of “how much would this piece of evidence actually change my belief” is not new to this course — Session 05's BIRD framework asked almost the same question from a different angle: instead of scoring a clarifying question by how much it would narrow a hypothesis space, BIRD scores a claim's probability by explicitly decomposing it into factors and combining them with a learnable Bayesian aggregator, rather than trusting an LLM's single, ungrounded probability estimate the way Chapter 2 of this session showed you should not trust a raw confidence number either. Both sessions land on the same underlying lesson from two different directions: an LLM's raw, single-shot number — whether it is a claimed probability or a claimed confidence — is not a substitute for actually decomposing the problem and doing the accounting explicitly.

The three-way decision rule in Chapter 8 is also, in miniature, exactly the shape of problem this site's own Lesson Tutor chat has to navigate: a small, locally-served model fielding open-ended questions about lesson content, where a wrong but confidently-delivered answer actively misleads a learner in a way a visible “I'm not sure, here's what I'd check” does not. The specific thresholds any such system should run at are an empirical question this session does not answer for you — but the three-way frame (answer directly, ask a follow-up, or point the learner back to the source material) is the same shape as Chapter 8's policy, applied to a genuinely deployed, everyday system rather than a benchmark.

This session's DEFER branch — refusing to answer, or routing to something more capable, rather than guessing — is one specific instance of a much broader discipline: AI Safety & Guardrails covers the wider space of mechanisms for constraining what a deployed model is allowed to do, of which confidence-based deferral is a narrow, purely statistical slice. And every accuracy, ECE, and routing-rate number reported in Chapters 2, 6, and 7 of this session presupposes a working evaluation harness capable of producing those numbers reliably in the first place — the practical machinery for building one is covered in AI Evaluation and Agent Evaluation, the latter especially relevant once a deferred query triggers a multi-step handoff rather than a single model call.

Finally, everything in this session assumed you already have a trustworthy way to measure whether an answer was actually correct — Self-REF's annotation step needed it, and every accuracy number in Chapters 6–7's tables depended on it. Designing that measurement well, so it survives contact with a real production system and a real, messy distribution of user queries, is its own discipline, covered in Metric Design.

And Chapter 8's DEFER branch, once it fires, has to hand the query off to something — a larger model, a human, or another tool entirely. How that handoff itself gets orchestrated, including deciding which of several available tools or agents a deferred query should actually reach, is the subject of Agents & Tool Use. This session deliberately stopped short of that question — everything here was about deciding whether to defer, not about what happens on the other side of that decision once it is made.

A checklist, for anyone actually shipping this

Collecting the session into an operational checklist, in the order a build would actually need to tackle it:

Step 6 deserves one closing word, because it is the step most naturally skipped once a system ships and starts working. Every threshold and every fine-tuned weight in this checklist was tuned against a snapshot of a model and a distribution of queries at one point in time. Real deployments drift — the mix of incoming requests shifts, the underlying model gets upgraded, new categories of query show up that the original evaluation set never saw. A calibration audit and a threshold sweep are not one-time setup costs; they are the same kind of ongoing operational discipline as monitoring uptime or latency, and treating them as a launch-day checkbox rather than a recurring practice is exactly how a system that started well-calibrated quietly drifts back into Chapter 0's confident-wrongness failure mode, one small distribution shift at a time.

Every chapter's one idea, in one table

The one idea to leave with

“The first principle is that you must not fool yourself — and you are the easiest person to fool.” — Richard Feynman, Caltech commencement address, 1974