Embedding Benchmarks

Chapter 1: What a Benchmark Actually Measures

Chapter 0 left you with an uncomfortable fact: 73% of the number you sorted on was measuring something other than search. To use a benchmark well you have to know what each part of it is measuring — not by its name, but by what it does to your vectors.

That is the framing for this chapter. Every MTEB task family is a small program that takes your model's embeddings and produces a score. If you know the program, you know the property. And once you know the property, you can predict which families will and will not correlate with your job.

The eight families, and the one concrete thing each one does

MTEB defines eight task types. Seven appear in the English benchmark; the eighth, bitext mining, is inherently multilingual. Here is each one as a procedure, with a real dataset and the metric that gets reported.

1. Classification — can a linear probe find the label?

The procedure. Embed a labelled training split and a labelled test split. Freeze those vectors. Fit a logistic regression — a single linear layer followed by a softmax — on the training vectors. Report accuracy on the test vectors.

A concrete example. Banking77: 13,083 customer-service utterances from a banking chatbot, labelled with 77 fine-grained intents (“card_arrival”, “lost_or_stolen_card”, “exchange_rate”). The question being asked is: are the embeddings of “my card hasn't arrived” and “where is my new card” on the same side of some hyperplane that separates them from the other 76 intents?

python
# the classification probe, in five lines — this is essentially all of it
X_train = model.encode(train_texts)          # (10_003, 768) float32
X_test  = model.encode(test_texts)           # ( 3_080, 768) float32
clf = LogisticRegression(max_iter=1000).fit(X_train, y_train)
acc = (clf.predict(X_test) == y_test).mean()   # the reported number

What property this rewards. Linear separability, and nothing else. Notice what is absent: no distance is ever computed. The probe never asks whether two same-label points are close, only whether a hyperplane can be drawn.

2. Clustering — does unsupervised structure match human labels?

The procedure. Embed a set of documents that carry hidden gold labels. Run mini-batch k-means with k set to the number of gold classes. Compare the resulting partition to the gold partition with V-measure, a score from 0 to 1 that averages how pure each cluster is (homogeneity) and how little each gold class is split apart (completeness).

A concrete example. ArxivClusteringP2P: arXiv paper title-plus-abstract strings, gold label = the arXiv category. Does k-means on the embeddings recover “cs.CL” versus “math.PR” versus “q-bio.NC”?

What property this rewards. Global isotropic topical structure. k-means uses Euclidean distance to centroids, which means it wants clusters that are roughly spherical and roughly equal in scale. An embedding space where all vectors crowd into a narrow cone — a real and well-documented pathology called anisotropy — can still be linearly separable (so classification is fine) while being terrible for k-means, because every pairwise distance is nearly the same.

3. Pair classification — is one threshold enough?

The procedure. You get pairs of texts and a binary label (duplicate / not duplicate). Compute the cosine similarity of each pair. Sweep every possible threshold and report the best achievable score — MTEB's main metric here is average precision computed from the cosine scores.

A concrete example. SprintDuplicateQuestions: pairs of questions from a technical support forum, labelled as asking the same thing or not.

What property this rewards. A globally calibrated similarity scale. Because a single threshold must work for the whole dataset, a model that is locally correct but whose “duplicate” cosine is 0.9 for one topic and 0.6 for another will score poorly even though it never confuses a pair within a topic. Retrieval, by contrast, only ever compares scores within one query — so retrieval does not care about global calibration at all. Same embeddings, opposite demands.

4. Reranking — order a short candidate list

The procedure. For each query you are handed a small list of candidate documents, some marked relevant and some not. Sort the list by cosine similarity to the query. Report mean average precision (and MRR) over queries. There is no corpus-wide search; the candidates are given.

A concrete example. AskUbuntuDupQuestions: for a given Ubuntu forum question, rank twenty candidate questions so that the true duplicates come first.

What property this rewards. Local ordering among plausible candidates. This is the hardest kind of discrimination — every candidate is already topically related, so the easy wins are gone. It is also the closest family to what a reranker does in a production RAG stack, which makes it a better proxy for that stage than the retrieval family is.

5. Retrieval — find the needle in the whole corpus

The procedure. Embed an entire corpus (thousands to millions of passages). Embed each query. Compute similarity between every query and every document, take the top k, and score with nDCG@10. In MTEB, this family is BEIR — the retrieval datasets are drawn from that benchmark, which Chapter 2 covers in detail.

python
# the retrieval probe — note the shapes, they are the whole story
D = model.encode(corpus)            # (5_183_000, 768) — this is 15 GB in float32
Q = model.encode(queries)           # (    6_980, 768)
D = D / np.linalg.norm(D, axis=1, keepdims=True)   # normalize -> dot == cosine
Q = Q / np.linalg.norm(Q, axis=1, keepdims=True)
for chunk in batches(Q, 64):
    S = chunk @ D.T                    # (64, 5_183_000) — 331M floats per batch
    top = np.argpartition(-S, 10, axis=1)[:, :10]

What property this rewards. Asymmetric query-to-document matching at scale. Two things make this different from every other family. First, the query and the document are not the same kind of text: a seven-word question must land near a two-hundred-word passage, so the geometry is asymmetric by design. Second, the competition is enormous — a correct document must beat five million distractors, so the metric is exquisitely sensitive to the tail of the similarity distribution in a way that a twenty-candidate reranking task never is.

6. STS — agree with human similarity judgments

The procedure. Each item is a sentence pair with a human similarity score, traditionally on a 0–5 scale. Compute cosine similarity for each pair. Report the Spearman rank correlation between your cosines and the human scores.

A concrete example. STSBenchmark: “A man is playing a guitar” / “A man plays a guitar” scores 4.8; “A man is playing a guitar” / “A woman is slicing an onion” scores 0.2.

What property this rewards. Monotone agreement with human paraphrase judgment in the symmetric, short-text, near-neighbour regime. Every pair is two sentences of comparable length and register. Nothing about this predicts behaviour on a seven-word query against a two-hundred-word passage — and that is precisely the gap that bit you in Chapter 0.

7. Summarization — the odd one out

The procedure. For each source document you have several machine-written summaries, each with a human quality rating, and several human-written reference summaries. Score each machine summary by its similarity to the references, then correlate those scores with the human ratings using Spearman.

A concrete example. SummEval, and in the English benchmark it is the only dataset in its family — one dataset carrying 1.8% of the headline average all by itself, with correspondingly enormous variance. Treat this column as noise unless you have a specific reason not to.

8. Bitext mining — the multilingual family

The procedure. Given a set of sentences in language A and a set in language B, find for each A-sentence its true translation in B by nearest neighbour. Report F1.

A concrete example. Tatoeba, across 100-plus language pairs. What it rewards is a shared cross-lingual space — the property that “the cat sat on the mat” and “le chat s'est assis sur le tapis” occupy nearly the same point. Purely English benchmarks cannot see this property at all, which is why an English-only average tells you nothing about whether a model will survive a multilingual corpus.

The average, computed by hand

The widget claims two models with different shapes land on nearly the same headline number. Do not take its word for it — compute one. Model A's per-family means are: classification 76.0, clustering 46.0, pair classification 86.0, reranking 58.0, retrieval 52.0, STS 84.0, summarization 30.0. The headline average is a mean over datasets, so each family contributes with its dataset count as weight:

Now Model B: classification 73.0, clustering 44.0, pair classification 84.0, reranking 60.0, retrieval 56.0, STS 81.0, summarization 29.0.

So the leaderboard reads A = 63.54, B = 63.05, and A sits above B. But look at the retrieval column: B is at 56.0 and A is at 52.0. If you are building search, B is four full nDCG points better at the only thing you need, and the leaderboard has told you to pick A.

Concept → realization: which family should you actually read?

Map your product to columns, not to averages. This table is the practical output of this chapter.

Notice that no row says “read the average.” The average exists to give a single sortable column to a leaderboard, which is a user-interface requirement, not a scientific one. Every serious use of a benchmark begins by throwing that column away and reading the two or three that correspond to the probe your product actually runs.

And once you have picked the right column, you still have to know what its number means — which is why Chapters 3 and 4 build every one of these metrics from scratch. You cannot reason about a 4-point nDCG gap until you have computed nDCG with a pencil at least once.

Concept → realization: one vector, seven probes

Here is the same 768-dimensional vector going through all seven English probes, with real shapes. Reading this once removes most of the mystery about why the families disagree.

python
v = model.encode("my card hasn't arrived yet")      # (768,) float32 — ONE vector, seven fates

# 1 CLASSIFICATION — the vector is a feature row; distance is never computed
X = np.stack([...])                # (10_003, 768)  ->  LogisticRegression  ->  accuracy

# 2 CLUSTERING — Euclidean distance to k centroids; scale and shape matter
km = MiniBatchKMeans(n_clusters=20).fit(X)      # centroids (20, 768)  ->  v_measure

# 3 PAIR CLASSIFICATION — one global threshold over all pairs
sim = cos(A, B)                  # A,B both (n, 768)  ->  (n,)  ->  average precision

# 4 RERANKING — scores compared only WITHIN one query's candidate list
sc = q @ C.T                       # q (768,), C (20, 768)  ->  (20,)  ->  MAP over queries

# 5 RETRIEVAL — the same dot product against millions of competitors
sc = q @ D.T                       # D (5_183_000, 768)  ->  (5183000,)  ->  nDCG@10

# 6 STS — cosine correlated against a human 0-5 rubric
rho = spearman(cos(A, B), gold)   # (n,) vs (n,)  ->  one number

# 7 SUMMARIZATION — machine summaries scored by similarity to human references
rho = spearman([max(cos(m, r) for r in refs) for m in machine], human_ratings)

Now read the three quantities that actually differ. Probe 1 never computes a distance. Probes 3 and 6 compare similarities across different pairs, which requires a globally consistent scale. Probes 4 and 5 compare similarities only within one query, which does not. And probe 5 does it against five million competitors while probe 4 does it against twenty.

The one number the leaderboard does show you, and how to read it

Alongside the score columns, most embedding leaderboards print the model's parameter count, embedding dimension and maximum sequence length. Those three are not decoration; they are the columns that determine whether you can deploy the thing at all, and Chapter 5 is about reading them. For now, one habit: never look at a score column without looking at the dimension column in the same glance, because a 4,096-dimensional model and a 384-dimensional one are not competing in the same category, however adjacent their rows.

Chapter 2: BEIR and the Zero-Shot Cliff

Chapter 1 told you to read the retrieval column. This chapter tells you what is inside it, and delivers the single most useful empirical fact in embedding engineering: a retriever's in-domain victory does not transfer, and a keyword algorithm from the 1990s is still, on a large fraction of real corpora, the thing to beat.

The setup that made everyone uncomfortable

Around 2020 the dense retrieval story looked finished. DPR — a pair of BERT encoders, one for questions and one for passages, trained contrastively on question-answer data — had beaten BM25 convincingly on the benchmarks it was trained for. The narrative was clean: keyword search is lexical and brittle, embeddings are semantic and robust, the future is dense.

Then BEIR asked a question nobody had asked systematically: what happens if you take these models and, without any further training, point them at eighteen different retrieval corpora they have never seen?

That word — zero-shot, meaning the model gets no training examples from the target dataset, not even its training split — is the entire design of BEIR. It is a heterogeneity benchmark. It does not ask “how good is this retriever?” It asks “how well does whatever you learned somewhere else survive contact with somewhere new?”

What is actually in BEIR

Eighteen public datasets across nine domains, deliberately chosen to disagree with each other about almost everything: how long the queries are, how long the documents are, how many relevant documents exist per query, and — the subtle one — what “relevant” even means.

Stop on the ArguAna row and read it again. The query is a paragraph-length argument, and the correct answer is a paragraph that disagrees with it. A model trained to place semantically similar texts close together has been trained to do the precise opposite of what this dataset rewards — a good counter-argument shares topic and vocabulary but opposes the stance, so the naive nearest neighbour is a paraphrase, which is wrong here.

The metric BEIR chose, and why

BEIR reports nDCG@10 as its headline metric for every dataset. This is a real choice with real consequences, and the reasoning is worth spelling out.

Recall-oriented metrics like recall@100 need something close to complete relevance judgments — if a system finds a genuinely relevant document that nobody judged, recall punishes it. Precision-oriented metrics at a fixed cutoff throw away the ordering inside the cutoff. nDCG@10 sits in between: it is graded (it can use “highly relevant” versus “somewhat relevant”, which several BEIR datasets provide), it is rank-sensitive (moving the right answer from position 5 to position 1 improves it), and it only looks at ten documents, which is roughly what a real user or a real language-model context window sees.

Chapter 3 builds nDCG from nothing. For now, take it as “a number between 0 and 1 where 1 means the perfect ordering.”

The result: the cliff

Here is the shape of what BEIR found. The values below are rounded approximations of the study's reported nDCG@10 scores — use them for the shape of the story and consult the paper's tables for exact figures.

Three readings, in increasing order of usefulness.

Reading one: DPR — the model that had convincingly beaten BM25 in its own setting — scores about fourteen nDCG points below an untrained keyword algorithm when moved to unfamiliar corpora. That is not a small regression. That is the difference between a working product and a broken one.

Reading two: the in-domain column does not predict the zero-shot column. TAS-B beats ANCE in-domain and they are nearly tied out of domain; BM25 is nearly last in-domain and near the top out of domain. If you rank models by MS MARCO performance and deploy on your own corpus, the ranking you used was uninformative.

Reading three — the one to act on: the best system is not a model, it is a pipeline. BM25 retrieves a hundred candidates cheaply and robustly; a cross-encoder — a model that reads the query and document together and outputs one relevance score — reorders them. That combination was the top of the table and it is still the default architecture of a serious search stack.

BM25, derived far enough to see why it refuses to die

You cannot respect the baseline until you know what it does. BM25 scores a document for a query by summing, over the query's terms, three ideas multiplied together.

where f(t,d) is how many times term t occurs in document d, |d| is the document's length, avgdl is the average document length in the corpus, and k₁ ≈ 1.2 and b ≈ 0.75 are the two tuning constants. Take the three ideas one at a time, with numbers.

Idea 1 — rare terms are worth vastly more. The inverse document frequency is

with N the number of documents and df(t) the number containing t. Run it on your docs site, N = 40,000. The error code ERR_TLS_CERT_ALTNAME_INVALID appears in df = 3 documents:

Now the word “the”, appearing in df = 39,000 documents:

The ratio is 9.34 ÷ 0.025 ≈ 369. BM25 considers one occurrence of your error code worth about three hundred and sixty-nine occurrences of “the”, and it knows this from counting your corpus, with no training, no gradient, and no chance of having missed the token during pretraining.

Idea 2 — repetition saturates. The middle factor grows with f(t,d) but flattens. Fix a document of average length, so the bracket reduces to k₁ = 1.2, and compute the term factor for several counts:

The first occurrence buys 1.000. The second buys 0.375 more. The tenth buys about 0.02 more. Keyword stuffing does not work, because the function was designed with a ceiling.

Idea 3 — long documents are discounted. The b term in the denominator scales with |d| / avgdl. For a document twice the average length:

against 1.375 in an average-length document. Two mentions in a short focused page beat two mentions in a long rambling one, which is the correct instinct and is why b exists.

Three ideas, two constants, zero parameters learned by gradient descent. That is the thing your seven-billion-parameter embedding model has to beat on your corpus, and on corpora full of rare identifiers it frequently does not.

Where dense wins and where it loses — and why

The zero-shot average hides a wild per-dataset spread. Explore it, then read the mechanism table below.

And the flip side, because dense retrieval is genuinely better at some things: it wins wherever the match is paraphrastic and the vocabulary is ordinary. Quora duplicate questions, FiQA forum answers, and anything where the user says “how do I stop my card from being charged twice” and the document says “preventing duplicate transactions” with not one shared content word. BM25 scores that pair zero. The dense model scores it high. Both behaviours are correct and neither is sufficient.

So ship both: reciprocal rank fusion, by hand

If one system is right about rare tokens and the other is right about paraphrase, the engineering answer is not to choose. It is to run both and merge the ranked lists. The standard merge is reciprocal rank fusion, which is remarkable in that it uses only ranks, never scores — so you never have to make a BM25 score commensurable with a cosine, which is impossible anyway.

Work an example. Three documents, two lists:

Final order: Z, X, Y. Document Z was never first in either list, and it wins — because two independent systems both put it near the top, and that agreement is stronger evidence than one system's enthusiasm. That is the entire philosophy of fusion in one arithmetic example.

Concept → realization: what this changes on Monday

Three concrete actions come out of this chapter and they cost almost nothing.

1. Measure BM25 on your corpus first. It takes an afternoon with an off-the-shelf index and it gives you the number every dense model must beat. If your candidate embedding model cannot beat BM25 on your own evaluation set, you have learned something enormous for one day of work.

2. Ship the hybrid. Rank fusion of BM25 and dense costs one extra index and about twenty lines of code, and it is robust in the exact place pure-dense is fragile: rare identifiers.

3. Put a reranker at the end. The BEIR table's winner was a pipeline, not a model. Retrieve a hundred with cheap methods, rerank with a cross-encoder, keep ten. You pay one extra model call per query on a hundred short pairs, and you buy back most of the zero-shot cliff.

Score one document, end to end

The three ideas above become a single number when you actually run them. Do it once by hand, because the result is surprising and it explains a class of production bug.

Query: “webhook retry policy”. Corpus: N = 40,000 chunks, average length avgdl = 220 tokens. Document frequencies from your index: webhook in 320 documents, retry in 140, policy in 900.

Candidate 1: the Webhook Retries reference page. Length 180 tokens; term counts 6, 9, 2. First the length factor, which appears in every term's denominator:

Candidate 2: the long Webhooks overview. Length 900 tokens; term counts 8, 1, 3. It mentions webhook more times than the reference page did.

25.70 against 13.17. The focused page wins by nearly two to one despite mentioning the head term less often, and you can point at exactly why: the length factor K went from 1.04 to 3.98, which crushed every term's contribution, and the rarest term retry — the one carrying the most IDF — appears nine times in the reference and once in the overview.

Chapter 3: nDCG From Scratch

Two chapters have now used nDCG@10 as if it were obvious. It is not obvious. It is a designed object with four separate decisions baked into it, and every one of those decisions changes what the number rewards. In this chapter you build it from nothing, and then compute one by hand, digit by digit.

Do not skip the arithmetic. The difference between an engineer who can argue about a retrieval regression and one who cannot is entirely whether they have ever computed this by hand.

The problem: compress a ranking into one number

A retrieval system hands you an ordered list of ten documents for a query. Some are perfect, some are useful, most are junk. You need a single number so you can average over queries and compare two systems. What should that number do?

Write down the requirements first, because the formula falls out of them almost mechanically.

Step 1: grades, not booleans

Start with graded relevance. Each judged document gets an integer grade, conventionally

Some datasets only have 0/1. Some use 0–2. The machinery does not care. What matters is that you have committed to an ordering of usefulness — and that commitment is a human decision made by whoever wrote the annotation guidelines, not a property of the documents.

Step 2: the naive sum, and why it fails

The simplest score is: add up the relevance of the top ten.

This satisfies requirement one and fails requirement two spectacularly. Consider two rankings of exactly the same ten documents:

System A puts the perfect document first. System B buries it at position ten. The metric says they are identical. Any metric that cannot distinguish these is useless for ranking, because ranking is the ordering. So we need a position-dependent weight.

Step 3: the discount — derive it from a user model

What weight? The honest way to choose is to ask what the weight means. Interpret the weight at position i as the probability that the user actually examines position i. Then

is a real quantity with a real interpretation, and any decaying function of i is a candidate model of attention. Two obvious candidates:

Compute both for ranks 1 through 10 and look at the difference:

The harmonic discount says rank 10 is worth one tenth of rank 1. The logarithmic discount says it is worth 0.2891 — nearly three times as much. nDCG uses the logarithmic one, and the reason is a deliberate design tradeoff: with a harmonic discount the metric becomes overwhelmingly a measure of “did you get position one right”, and it stops being able to distinguish two systems that both missed the top slot. The gentle log discount keeps the whole top-ten window informative.

Step 4: the gain — why 2^rel − 1

Now the other half: how much is a grade-3 document worth compared to a grade-1 document? The obvious answer is “three times as much” — the linear gain, gain = rel. The standard used by the retrieval-evaluation tools that MTEB and BEIR run is the exponential gain:

Why? Because it encodes a specific editorial opinion: one perfect answer is worth more than a pile of mediocre ones. Test that claim with arithmetic. Put one grade-3 document at rank 1:

Now put three grade-1 documents at ranks 1, 2 and 3 instead:

Under exponential gain the single perfect document is worth more than three times the trio. Under linear gain the trio would score 2.131 against the single document's 3.000 — a far closer call. Which is right depends entirely on your product. For a question-answering front end where one correct answer ends the session, exponential is right. For a research tool where a user wants breadth, linear may be right.

Step 5: assemble DCG

Multiply gain by discount and sum over the cutoff:

This satisfies requirements one, two and four. It fails requirement three: it is unbounded, and a query with many relevant documents produces a bigger DCG than a query with one, no matter how well the second query was served.

Step 6: normalize — divide by the best possible

The fix is to ask: what is the largest DCG this query could possibly achieve? Sort all judged documents for the query by grade, descending, take the top k, and compute DCG of that. Call it IDCG@k, the ideal DCG. Then

Now every query is on the same 0-to-1 scale, 1.0 means “you produced an ordering as good as any ordering could be”, and averaging across queries is meaningful. That is the whole construction.

The full worked example — nDCG@10 with a pencil

Query: “how do I reset my password”. Your judges have graded six documents in the collection: one at grade 3, two at grade 2, three at grade 1. Everything else is grade 0.

System A returns these ten documents, in this order, with these grades:

Step A1 — convert grades to gains using 2^rel − 1:

Step A2 — multiply each by its discount and keep only the non-zero rows (a zero gain contributes nothing, which is why sparse relevance makes this tractable by hand):

Step A3 — add them up, one at a time so you can check each partial sum:

Step A4 — build the ideal ranking. Sort all six judged documents by grade: one 3, two 2s, three 1s. The ideal order of grades is [3, 2, 2, 1, 1, 1] followed by zeros, so the ideal gains are [7, 3, 3, 1, 1, 1, 0, 0, 0, 0]:

Step A5 — divide.

Check the division the slow way if you like: 11.56653 × 0.74 = 8.55923, leaving 0.04460, and 0.04460 ÷ 11.56653 = 0.00386, so 0.74 + 0.00386 = 0.74386. Round to 0.744.

Now move one document and watch

System B returns exactly the same six judged documents, in exactly the same positions — ranks 1, 3, 4, 6, 8, 10 — but assigns the grades differently: it puts the grade-3 document first and pushes the grade-2 documents to ranks 3 and 4.

The ideal ranking is unchanged — it depends only on the judgments, not on the system — so IDCG@10 = 11.56653 again:

0.744 versus 0.930. An eighteen-point gap, produced entirely by which of the relevant documents sits at rank one. Both systems returned the same six documents in the same six slots. This is what “rank sensitive” means, quantitatively.

The two things nDCG cannot see

Every metric has a blind spot and honesty requires naming them.

1. Unjudged documents count as zero. nDCG has no category for “nobody looked at this.” If your new model surfaces a genuinely excellent document that no annotator ever graded, the metric treats it as junk and your score goes down for being better. This is pooling bias, and it is not hypothetical: benchmark judgments were pooled from the systems that existed when the dataset was built, so a system that behaves differently from all of them is systematically punished.

The correction term grows with how novel your system is. This is one concrete mechanism by which an old benchmark quietly favours models that resemble old models.

2. The cutoff hides everything below it. nDCG@10 cannot distinguish a system that puts the answer at rank 11 from one that puts it at rank 900. If your pipeline reranks the top 100, that distinction matters enormously — a document at rank 11 will be recovered by the reranker and one at rank 900 will not. Report recall@100 alongside nDCG@10 whenever a reranker exists downstream. Chapter 4 builds recall.

Concept → realization: the twelve-line implementation

python
import math

def dcg(grades, k=10):
    # grades[i] is the relevance of the doc your system put at rank i+1
    return sum((2**g - 1) / math.log2(i + 2)      # i is 0-based -> rank i+1 -> log2(i+2)
               for i, g in enumerate(grades[:k]))

def ndcg_at_k(ranked_grades, all_grades, k=10):
    # all_grades = every judged grade for this query, system-independent
    ideal = sorted(all_grades, reverse=True)
    idcg  = dcg(ideal, k)
    if idcg == 0: return 0.0      # no relevant docs at all -> undefined, report 0 or skip
    return dcg(ranked_grades, k) / idcg

# our worked example, to the digit
A = [2,0,3,1,0,2,0,1,0,1]; judged = [3,2,2,1,1,1]
print(ndcg_at_k(A, judged))    # 0.7438...

Two details in that code are where real bugs live. First, math.log2(i + 2) — the off-by-one that turns rank 1 into log₂(2) = 1. Get it wrong and rank 1 becomes log₂(1) = 0 and you divide by zero. Second, idcg == 0: queries with no relevant documents in the collection have an undefined nDCG. Scoring them as 0 drags your average down by however many such queries you have; skipping them changes the denominator. Decide explicitly and write the decision down, because two teams making different choices will report different numbers for the same system.

Chapter 4: Recall, MRR, MAP and Spearman

nDCG is the headline, but it is not the only number on a benchmark row, and it is often not the right number for your product. Four more metrics do the rest of the work. All four are three lines of arithmetic, all four are computed here by hand, and each one has exactly one situation where it is the correct choice.

Recall@k — did the answer make it into the window?

Recall@k is the fraction of all relevant documents that appear anywhere in the top k. Order inside the window is invisible to it.

Take system A from Chapter 3: A = [2, 0, 3, 1, 0, 2, 0, 1, 0, 1], with six relevant documents in the collection. Treat any grade ≥ 1 as relevant.

And system B = [3, 0, 2, 2, 0, 1, 0, 1, 0, 1]? Its relevant documents also sit at ranks 1, 3, 4, 6, 8, 10.

Identical. Recall cannot see the eighteen-point nDCG gap, because recall was never designed to look at order.

Precision@k, and why retrieval papers mostly skip it

Precision@k is the fraction of the top k that is relevant: for system A, precision@10 = 6 ÷ 10 = 0.600. The problem is the denominator. If a query has only one relevant document in the whole collection, the best achievable precision@10 is 1 ÷ 10 = 0.100 — a perfect system scores 0.1 and looks broken. Precision@k is not comparable across queries with different numbers of relevant documents, which is exactly the normalization problem nDCG solved with IDCG.

MRR — how fast to the first good thing

Mean reciprocal rank finds the position of the first relevant document and takes one over it, then averages across queries.

Three queries, worked:

Notice how violently nonlinear MRR is. Moving a query's first hit from rank 2 to rank 1 gains 1 − 0.5 = 0.500. Moving it from rank 10 to rank 9 gains 1/9 − 1/10 = 0.1111 − 0.1000 = 0.0111 — forty-five times less. MRR is essentially a measure of “is the answer at the top” with a small consolation prize for everything else.

That makes it the correct metric for navigational queries, where exactly one document is the answer and the user is done as soon as they see it: “reset password page”, “invoice #88213”, “the docs for fetchWithRetry”. It is the wrong metric for research-style queries where a user wants five good sources.

For systems A and B: both have a relevant document at rank 1, so RR = 1.0 for both, and MRR cannot see the gap either.

MAP — precision measured at every hit

Average precision walks down the ranking, and every time it steps onto a relevant document it records the precision at that exact position. Then it averages those recorded values over the total number of relevant documents.

where R is the number of relevant documents for the query. MAP is the mean of AP over queries. Work it for system A, whose relevant documents sit at ranks 1, 3, 4, 6, 8, 10 with R = 6:

Now system B. Its relevant documents are at ranks 1, 3, 4, 6, 8, 10 — the same positions. Every precision value is the same. AP(B) = 0.7181. Identical again.

Which one, when

Report at least two. A single number is a summary of a summary, and the pair (recall@k, nDCG@k) tells you whether a regression was a retrieval failure (the document fell out of the window) or an ordering failure (it is still there but buried). Those two failures have completely different fixes.

Spearman — the metric for the STS family

Retrieval metrics all score a ranked list. The STS family scores something different: a set of sentence pairs, each with a human similarity rating, against your model's cosine similarities. The reported number is the Spearman rank correlation, and understanding why rank correlation is a small but important piece of judgment.

Definition. Spearman's ρ is simply the ordinary Pearson correlation computed on the ranks rather than the values. When there are no ties it collapses to a formula you can evaluate in your head:

where d_i is the difference between the two ranks of item i.

Where that formula comes from, in four lines. Let x and y be the two rank vectors, each a permutation of 1..n. Their means are equal, both (n+1)/2, and their sums of squared deviations are equal, both V = n(n²−1)/12. Write S for the cross term Σ(x−x̄)(y−ȳ). Then

and since Pearson on equal-variance vectors is ρ = S ÷ V, substituting S = V − Σd²/2 gives

No magic. It is Pearson with the rank structure substituted in.

Spearman, worked on six sentence pairs

Six pairs, human gold similarity on the 0–5 scale, and your model's cosine:

The model swapped exactly one adjacent pair — it thought pair 4 was more similar than pair 3 — and was otherwise in perfect agreement, and Spearman reports 0.943.

Why not Pearson? Compute it and see

Pearson on the raw numbers measures linear agreement. The gold scores live on 0–5; the cosines live, in practice, in a compressed band around 0.4–0.95. Compute Pearson on the same six pairs.

Deviations, products and squares:

Pearson 0.910, Spearman 0.943. Pearson is lower, and it is lower for a reason that has nothing to do with the model's ordering quality: cosine similarity is a squashed, nonlinear function of human similarity. A model whose cosines run 0.41 to 0.91 is penalized by Pearson for not spreading its scores linearly across the gold range, even if it orders every pair correctly.

Concept → realization: the metric contract

Whenever you report a retrieval number, four facts have to travel with it or the number is not reproducible. Write them next to the score in your dashboard, not in a document nobody opens.

Every one of these has silently caused a team to believe a model was better or worse than it was. They are free to fix — four strings in a log line.

MAP across queries, and why it is an area under a curve

The single-ranking AP above is one query. MAP averages AP across the query set, and the multi-query version is worth computing once because the treatment of missing relevant documents is where people get it wrong.

Look at q2 carefully. Four relevant documents exist and only three were found, so the sum has three terms but the denominator is 4. The missing document contributes a silent zero. That is deliberate: AP is a recall-aware precision, and a system that finds three of four is not allowed to score as if four of four were the target.

Why AP is the area under the precision-recall curve. Walk down the ranking and plot precision against recall. Recall only changes when you hit a relevant document, and each such hit increases recall by exactly 1/R. So the area under that step curve is

which is exactly the AP formula. The average precision is not an average of anything arbitrary — it is a Riemann sum, and the “average” in the name is the 1/R factor.

Hit rate: the metric your product manager will actually ask for

One more, because it is the number people quote in RAG discussions under a dozen different names. Hit rate@k (also success@k, or answer-in-context rate) is the fraction of queries with at least one relevant document in the top k.

On 300 evaluation queries, if 213 have at least one relevant document in the top 5:

Two properties make it useful and one makes it dangerous. It is interpretable — 71% of the time the evidence was in the window, so 29% of the time the generator could not possibly have been right. And when every query has exactly one relevant document (R = 1), hit rate@k equals recall@k, which is why the two are so often confused. The danger: with R > 1 it saturates almost immediately, because finding one document out of six scores the same as finding all six. Use it as a ceiling report, never as the thing you optimize.

Chapter 5: Reading a Leaderboard Honestly

You now know what the tasks measure and how the metrics are computed. This chapter is about the leaderboard itself as an artifact: a sorted table of numbers, each of which is an estimate with an error bar that is never printed, produced by models that differ in ways the table does not show.

There are five things to check before you believe a row. Four are confounds and one is a noise floor. Every one of them has cost somebody a quarter.

Confound 1: the average hides the variance

Two models can share a headline average and have completely different risk profiles. Consider two models both averaging 63.5 across 56 datasets:

Model V has seven datasets where it collapses. If one of those seven happens to resemble your corpus, you have adopted a model that scores 21 on your job while the leaderboard promised 63.5. The mean gave you no warning, because a mean is a first moment and catastrophe lives in the tail.

What to do instead. Read the per-dataset columns for the families you care about and look at the minimum, not the mean. A model's worst score in your family is a much better predictor of your downside than its average across all families. This is the same reasoning that makes p99 latency more useful than mean latency — and for the same reason: you experience the tail, not the average.

Confound 2: model size

The leaderboard sorts by score. It does not sort by cost, and the cost differences between adjacent rows are often two orders of magnitude. Make it concrete for a five-million-chunk corpus.

Index memory. A dense index stores one vector per chunk:

The 4,096-dimensional model may be two points better on the leaderboard. It also does not fit in the memory of the machine you were planning to use, which converts a two-point gain into a re-architecture. (Quantizing to int8 divides each of those by four — 81.92 GB becomes 20.48 GB — at a small and measurable accuracy cost that you should measure rather than assume.)

Encoding throughput. A 110-million-parameter encoder and a 7-billion-parameter encoder differ by roughly 64× in parameters and, in practice, by a similar factor in tokens per second per GPU. Re-embedding five million chunks is an afternoon with one and a multi-day job with the other — and you will re-embed every time you change the model, so this cost recurs.

Query latency. The document side is amortized; the query side is not. Every user query runs the encoder in the request path. A large encoder can add tens of milliseconds before the vector search even starts.

Confound 3: sequence length

Every encoder has a maximum input length in tokens — commonly 512 for the BERT-derived family, 8,192 for the modern long-context ones. Text beyond the limit is silently truncated. No exception, no warning, just a shorter vector than you thought.

The benchmark cannot catch this, because benchmark passages are short. MS MARCO passages average roughly 55 words; almost nothing in the retrieval family approaches 512 tokens. So a 512-token model and an 8,192-token model are evaluated on identical inputs and the limit never binds.

Now your corpus. Suppose you chunk at 1,000 characters with 200 characters of overlap, and your text averages about 4 characters per token:

But your API reference pages are code-heavy, and code tokenizes far worse — often 2 to 2.5 characters per token because identifiers split into pieces:

The truncated portion is invisible: the chunk still embeds, still gets indexed, still gets retrieved occasionally. It simply cannot be found by anything that was said in its second half. This produces exactly the kind of bug that takes three weeks to diagnose, because the symptom is “search is a bit worse for some pages” and there is no error anywhere.

Confound 4: the model was evaluated with a prompt you are not using

Several strong embedding families are asymmetric by convention: they were trained with a short instruction prefix distinguishing queries from documents, and their benchmark numbers were produced with those prefixes in place. The E5 family uses "query: " and "passage: "; the BGE family uses an instruction on the query side; instruction-tuned embedders take a task description as an argument.

python
# what the benchmark ran
q = model.encode("query: how do I reset my password")
d = model.encode("passage: To reset your password, open Settings ...")

# what you shipped, because the prefixes felt like boilerplate
q = model.encode("how do I reset my password")
d = model.encode("To reset your password, open Settings ...")
# -> several nDCG points gone, no error, no warning, no log line

This is the single most common way a team fails to reproduce a published number. The model is not broken and the benchmark is not lying — you are running a different function than the one that was measured. Read the model card, and reproduce one benchmark dataset locally before you trust your integration.

The same applies to pooling (mean-pooling the token vectors versus taking the CLS position) and to normalization (whether vectors are L2-normalized before the dot product). Every one of these is a silent behavioural switch that changes your scores while producing no error.

Confound 5 — and the important one: the noise floor

Every score on a leaderboard is a sample mean over a finite set of queries. It therefore has a standard error, and differences smaller than that error are noise. The leaderboard does not print the error, so you must estimate it.

The arithmetic. Per-query nDCG@10 values are spread widely — many queries score 0 or 1, few land in the middle — so a standard deviation around σ = 0.35 is typical. On a dataset with n = 500 queries:

Comparing two models by their independent means, the difference has a larger error:

Read that again: on a 500-query dataset, a four-point gap between two models is inside the noise. Most of the gaps you are agonizing over on a leaderboard are smaller than four points.

The fix: pair your comparisons

There is an enormous amount of signal being thrown away above. The two models are being run on the same queries, so the query-to-query difficulty variation — which is most of that σ = 0.35 — is common to both and should cancel. Compute the per-query difference and take the mean of that instead.

Because the two systems agree on most queries, most d_i are exactly zero and the spread collapses. A typical paired standard deviation is σ_d = 0.15:

The same 500 queries went from resolving 4.3 points to resolving 1.3 points — a 3.3× improvement in sensitivity, for zero additional data, purely by not discarding the pairing. This is the highest-return change you can make to an evaluation harness.

How many queries do you need?

Turn it around. You want to reliably detect a 3-point difference (δ = 0.03). For a paired two-sided test at 95% confidence with 80% power, the required sample size is

Two hundred to three hundred and fifty queries. That is the entire answer to “how big does my evaluation set need to be”, and it is a number you can reach in a day of annotation. Chapter 7 is about spending that day well.

One more trap: 56 chances to be surprised

If you compare two models across 56 datasets and test each one for significance at the 5% level, then even if the models were identical you would expect

So “model X beats model Y on three datasets” is precisely what pure noise looks like. If you want to make per-dataset claims, correct for multiple comparisons or, more practically, ignore per-dataset differences entirely and only make claims about the family means you care about.

The five-question checklist

When the normal approximation is a lie: bootstrap instead

Everything above assumed the mean of per-query differences is approximately normal. With 300 queries that is usually fine, but per-query nDCG is a lumpy distribution — a spike at 0, a spike at 1, and a scatter in between — so it is worth knowing the assumption-free alternative.

The bootstrap replaces the formula with resampling. You have 300 paired differences. Draw 300 of them with replacement, take the mean, and write it down. Do that ten thousand times. You now have a distribution of ten thousand plausible means, and the 2.5th and 97.5th percentiles of that distribution are your 95% interval — no normality, no standard-error formula, no degrees of freedom.

python
import random, statistics
d = [ndcg_new[q] - ndcg_old[q] for q in queries]     # 300 paired differences
means = []
for _ in range(10_000):
    sample = [random.choice(d) for _ in d]           # resample WITH replacement
    means.append(sum(sample) / len(sample))
means.sort()
lo, hi = means[250], means[9750]                     # the 2.5th and 97.5th percentiles

Two details that matter. Resample queries, not individual judgments — the query is your unit of independence, and resampling within a query pretends you have more data than you do. And resample the differences, not each system separately, for exactly the pairing reason above.

The sign test: the comparison that assumes almost nothing

Sometimes you do not even want to average, because a single catastrophic query can drag a mean around. Then count instead. On each query, does the new system win, lose, or tie?

Ties carry no information about direction, so discard them and ask: out of the 100 queries where something changed, is 62 wins more than a coin flip would give? Under the null hypothesis the count of wins is a binomial with n = 100 and p = 0.5:

So the new system wins significantly more often than it loses. Notice what this test does not need: no distributional assumption, no variance estimate, not even the size of each win. It answers a narrower question — “does it win more often?” rather than “is it better on average?” — and those two questions genuinely can disagree.

Reporting a comparison so nobody can misread it

One template, five lines, and it prevents most of the arguments this chapter is about. Every retrieval comparison you publish internally should look like this.

text
candidate  bge-style-v2      vs incumbent  e5-style-v1
eval       golden-v4 · 312 queries · pooled top-10 from 3 systems · judged 2026-07
metric     nDCG@10 · gain 2^rel-1 · rel>=1 for binary · 6 queries with no relevant docs skipped
result     paired mean +0.021  (95% CI +0.008 to +0.034)   ·  wins 71 / losses 44 / ties 197
secondary  recall@100 +0.004 (CI -0.006 to +0.014)  ·  hit rate@5 +0.018  ·  worst query -0.44

Read what each line prevents. Line 2 stops anyone comparing against a different judgment version. Line 3 stops the gain-scheme and threshold ambiguity from Chapter 4. Line 4 gives a paired interval and the win/loss counts, so a reader can see both the average and the shape. Line 5 shows that recall barely moved — so this is an ordering win, not a retrieval win — and names the single worst regression, which is the query somebody will inevitably ask about.

That last field matters more than it looks. A mean improvement with a −0.44 worst case is a system that broke something badly for somebody. Find that query, look at it, and decide deliberately whether you are willing to trade it. A report that shows only the mean lets you ship that trade without noticing you made it.

Chapter 6: Contamination and Leaderboard Overfitting

Everything so far assumed the benchmark was measuring honestly and you were reading it carelessly. This chapter reverses that. Even read perfectly, a public benchmark degrades over time — not because anyone cheats, but because of two mechanisms that operate automatically the moment a leaderboard becomes something people optimize.

The two mechanisms are different, they compound, and both are quantifiable.

Mechanism 1: contamination — the test distribution is in the training set

Contamination is when material from the evaluation appears, in some form, in the training data. For embedding models it comes in three strengths, and only the strongest one feels like cheating.

Strength 1: training on the official training split. Almost every retrieval dataset ships a train split and a test split. MS MARCO, Natural Questions, HotpotQA, FEVER, Quora — these are the workhorses of open embedding training. It is entirely legitimate to train on a train split. But BEIR's whole premise is zero-shot evaluation, and a model trained on MS MARCO triples is not zero-shot on MS MARCO's test set, nor really on anything derived from the same annotation process.

The consequence is not that the numbers are fake. It is that the numbers on those datasets measure in-domain performance while presenting themselves as zero-shot, and in-domain performance is exactly the quantity Chapter 2 showed does not transfer.

Strength 2: synthetic data shaped like the benchmark. A now-standard recipe is to prompt a large language model to generate query-passage training pairs. If you generate them in the style of the benchmark's tasks — short web-style questions over Wikipedia passages, duplicate-question pairs, claim-evidence pairs — you have not touched a single test item, and you have still trained specifically on the benchmark's distribution. This is the most common form today and it is nearly undetectable from the outside.

Strength 3: the test items themselves. Benchmark corpora are public documents on the internet. Pretraining corpora are scraped from the internet. The intersection is not empty. Nobody intends this and everybody has some of it.

Put a number on the inflation

Model the effect simply. Let c be the fraction of evaluation items that are contaminated, m the accuracy on those (near-memorized, so high), and g the true generalization accuracy on clean items. The observed score is the mixture:

Take a model whose true generalization is g = 0.60, whose memorized items score m = 0.97, with c = 25% contamination:

The leaderboard says 69.25. Your corpus, which contains none of the contaminated items, delivers 60.00. A 9.25-point gap, and every number in that calculation is plausible.

Invert it to check: given observed 0.6925 with known c = 0.25 and m = 0.97,

The fingerprint you can actually look for

You cannot audit a model's training data. You can look at the shape of its gains. Suppose a new model improves the retrieval family by 4.5 points over the previous generation. Split the 15 retrieval datasets into those with public training splits (say 9 of them) and those without (6):

The headline says “+4.5 nDCG on retrieval”. Your corpus has no training split and never will, so you are on the +0.4 branch. This is not a hypothetical — it is precisely the shape of the disappointment in Chapter 0, and you can check for it yourself by reading the per-dataset columns and sorting them by whether a training split exists.

Mechanism 2: leaderboard overfitting — the winner's curse

The second mechanism requires no contamination at all. It happens even if every model is trained honestly on clean data, and it is pure statistics.

Suppose a thousand teams each build a model. Every model's true quality is identical — call it 60.0. The benchmark measures each one with independent noise of standard deviation 1.0 point. Now sort by measured score and look at the top row.

The top row is not the best model. It is the model whose noise draw was luckiest. The expected maximum of n independent standard normal draws grows like

That leading term overshoots slightly; the standard refinement subtracts a correction term:

So with a 1-point standard error, the model at the top of a thousand-entry leaderboard is expected to be about 3.1 points above its own true quality — entirely from selection, with no misconduct anywhere. For a hundred entries the inflation is about 2.4 points; for ten, about 1.5.

And this analysis was for a single round. Real leaderboards run continuously: teams submit, see the score, adjust, resubmit. That feedback loop is adaptive overfitting, and it inflates faster than the one-shot calculation above, because information about the test set leaks into every subsequent design decision — architecture choices, data mixtures, pooling strategies, which of five checkpoints to publish.

Goodhart, stated precisely for benchmarks

“When a measure becomes a target, it ceases to be a good measure.” The precise version for embedding benchmarks is worth writing out, because it tells you exactly which claims survive.

Note the first row. Benchmark scores remain perfectly good measurements. What they have lost is their predictive validity — the property that made them useful for choosing. Those are different things and conflating them is the entire error.

What the community does about it, and what you should do

The field's responses are structural and worth knowing, because they tell you which benchmark generation you are reading.

The last row is the only one you control, and it is the only one guaranteed to be uncontaminated, because you made it out of your own logs and never published it. Everything in Chapter 7 is about building it.

The benchmark half-life

Both mechanisms grow with time, which means a benchmark's usefulness has a shape you can reason about.

This is why benchmark generations exist at all. When the top twenty entries sit within two points of each other — well inside the four-point noise band from Chapter 5 — the instrument has stopped resolving differences and someone builds a harder one. Knowing which generation you are reading is part of reading it.

The canary set: an eval that cannot have been trained on

There is one construction that defeats pretraining leakage completely, and it costs nothing beyond discipline: build your evaluation from documents that did not exist when the model was trained.

Suppose a candidate model scores 0.68 nDCG@10 on your general evaluation and 0.59 on the post-cutoff canary slice, while your incumbent scores 0.65 and 0.64. Read those four numbers carefully:

The candidate looked three points better and is five points worse on documents it cannot have seen. The canary slice found in one afternoon what a quarter of production would have found the slow way.

How far can adaptive rounds push it?

The winner's-curse calculation assumed a single round of a thousand submissions. Real development is iterative: try ten variants, keep the best, try ten more informed by what you learned. How much can that compound?

An upper bound is easy and useful. Ten rounds of ten variants, in the worst case where every variant is an independent draw, is no worse than selecting the best of one hundred:

So the compounding is bounded, and bounded by a number smaller than the thousand-submission case. The real danger is not the arithmetic — it is that the variants are not independent. Each round's design choices are informed by the previous round's test-set feedback, which is precisely the information leak that makes a held-out set stop being held out.

Auditing a model card in ten minutes

You cannot inspect training data, but model cards and papers usually list dataset names, and that is enough for the audit that matters. Here is the checklist, in order of how much it tells you per minute spent.

The workhorse datasets to watch for — the ones whose training splits appear in nearly every open embedding recipe — are MS MARCO, Natural Questions, HotpotQA, FEVER, Quora duplicate questions, and various StackExchange dumps. When a model's training list contains those and its benchmark gains are concentrated on the corresponding test sets, you have found the fingerprint from the previous section without needing any privileged information.

What a benchmark still buys you

Six sections of caveats can read as “benchmarks are worthless.” They are not, and being precise about their remaining value is part of using them well.

So the correct posture is neither trust nor dismissal. A benchmark is a coarse filter with a resolution of maybe five points and a systematic bias toward whatever it has been optimized against. Use it at that resolution: to cut twenty candidates down to three. Then switch instruments.

Chapter 7: Building Your Own Evaluation

Everything so far has been diagnosis. This chapter is the cure, and it is smaller than you fear: roughly three hundred queries, a written definition of relevance, one week of one person's time, and a script. After that you never have to guess about an embedding model again, because you can measure.

We will build it in six steps, with the arithmetic for each so you can plan the week honestly.

Step 1: choose the queries — and stratify them

Your queries come from your logs. That part is easy. The mistake is sampling them uniformly at random, because query traffic is brutally skewed: a small number of head queries account for a large share of volume, while the tail — the long list of queries asked once ever — is where search actually fails.

Suppose your traffic splits like this and you sample 300 queries proportionally to volume:

Proportional sampling puts 90 of your 300 slots on a set of only 100 distinct queries, most of which already work — they are head queries precisely because people found what they wanted and kept asking. You have spent 30% of your budget measuring the part of the system that is not broken.

Stratify instead: fix a quota per segment, over-weight the tail, and add a fourth bucket that pure sampling will never give you — the known failures. Zero-result queries, queries followed immediately by a reformulation, sessions that ended without a click. Those are your product's actual bug reports, and thirty of them in an evaluation set is worth three hundred head queries.

Step 2: pool the candidates — you cannot judge the whole corpus

For each query, relevance is defined over the entire corpus — forty thousand chunks. Judging all of them is 12 million judgments and is not happening. The standard technique is pooling: run every system you are considering, take each one's top k, and judge the union.

Concretely, with three systems (BM25, your current model, the candidate model) and k = 10:

Add a 20% double-judged subset for measuring agreement (Step 4) and budget the time:

That is the whole price. One person-week buys you a permanent, private, uncontaminated instrument that answers every future model question in an hour.

Step 3: write the judgment protocol before anyone judges anything

This is the step teams skip and it is the step that determines whether the numbers mean anything. “Is this document relevant?” is not a question — it is four different questions that different annotators will answer differently. Write down the grades, with anchor examples from your own corpus.

Then answer the edge cases in writing, because they will come up on the first afternoon and every annotator will decide differently in the moment:

This document is your relevance relation — the same object that Chapter 2 showed differs between BEIR datasets and is the deepest reason scores do not transfer. Writing it down converts an unspoken assumption into a testable artifact.

Step 4: measure agreement, because your labels are not ground truth

Two annotators will disagree. You need to know how much, because disagreement is the resolution limit of your entire evaluation: no system difference smaller than the label noise is measurable.

Raw agreement is not enough, because two annotators who both say “not relevant” 90% of the time will agree 82% of the time by pure chance. The correction is Cohen's kappa:

where p_o is the observed agreement and p_e is the agreement expected if both annotators labelled independently at their own base rates. Kappa is 1 for perfect agreement, 0 for chance-level, and negative for systematic disagreement.

Worked example one. Two annotators double-judge 100 query-document pairs as relevant / not:

On the conventional scale, 0.61–0.80 is “substantial”. This is a usable golden set.

Worked example two — the trap. Now a different pair of annotators, on a pool where almost nothing is relevant:

Ninety-five percent raw agreement, and a kappa of 0.52 — worse than the pair who agreed only 85% of the time. Almost all of that 95% is two people independently saying “no” to obviously irrelevant documents, which is not evidence that they share a definition of relevance. Whenever you hear “our annotators agree 95% of the time”, ask for the kappa.

Step 5: scale the labelling with a judge you have calibrated

Five thousand judgments is one person-week. Fifty thousand is not. The modern answer is a language model as a relevance judge — and the honest version of that answer requires exactly the machinery you just built.

Have the model judge the same double-judged subset your humans did, and compute the same kappa. Suppose you get:

The model agrees with a human nearly as well as a second human does. That is the standard that licenses using it at scale — not “the outputs looked reasonable.” Below that bar, use the model only to pre-filter: let it grade everything, have humans review only the items it grades 2 or 3, which is where the metric's mass lives.

And keep a human-only slice forever. If the judge model changes — a version bump, a prompt edit — your entire history of scores shifts under you, and only the frozen human slice will tell you.

Step 6: the minimum viable RAG evaluation

For a retrieval-augmented generation stack, evaluate two layers separately. This is the single most useful structural decision, because it tells you which half is broken.

The reason for the ordering is arithmetic. If recall@5 is 0.72, then 28% of queries cannot be answered correctly no matter how good the generator is — the evidence was never in the context. Tuning prompts against that ceiling is wasted work. Fix the retriever first; the retrieval layer is also a hundred times cheaper to evaluate, so you can run it on every commit.

Step 7: turn it into a gate

An evaluation you run when you remember is not protection. Wire it into the change process for anything that touches retrieval — the embedding model, the chunking strategy, the index parameters, the query rewriting, the reranker.

python
# the regression gate — paired, because Chapter 5
d = [ndcg(new, q) - ndcg(old, q) for q in golden_queries]   # len 300
mean_d = sum(d) / len(d)
se     = stdev(d) / len(d)**0.5
lo, hi = mean_d - 1.96*se, mean_d + 1.96*se

if hi < 0:      fail("significant regression")          # whole interval below zero
elif lo > 0:    pass_("significant improvement")
else:          warn(f"inconclusive: {mean_d:+.3f} +/- {1.96*se:.3f}")

# and always, separately, the cases you are not allowed to break
assert all(rank_of_gold(new, q) <= 3 for q in must_never_break)

Note the third branch. Most changes will be inconclusive, and that is the correct and honest output — it means the change is smaller than your instrument can resolve, so ship or do not ship on other grounds (cost, latency, simplicity) rather than pretending the number decided.

And note the last line. The must_never_break list is a handful of queries whose answers are non-negotiable — your top support articles, your pricing page, the error codes that generate tickets. They get a hard assertion, not a statistical test. An average can absorb a catastrophe on one query; a business cannot.

Growing the set without corrupting it

A golden set is not a one-time artifact; it decays. Your corpus changes, your product adds features, and your query mix drifts. Three rules keep it honest as it grows.

Rule 1 — version everything, and re-score history when you change it. The judgments are the denominator of every nDCG you have ever recorded (that is what IDCG is). Adding documents or upgrading grades silently invalidates your history. Tag the judgment set with a version, store it, and when you bump the version, re-run every archived system against the new one before comparing.

Rule 2 — add queries on a schedule, not on a whim. Adding a query because a model failed it makes the set an adversarial collection against that model, and your gate stops representing your traffic. Add quarterly, sampling by the same stratification you used originally, plus any new must-never-break cases.

Rule 3 — never publish it. The instant your evaluation set is public, it is training data for somebody, and the contamination story of Chapter 6 becomes your story. Share the aggregate numbers; keep the queries and judgments internal.

The anti-patterns, named

The click-log row deserves a second sentence because it is the most tempting shortcut in the list. Click data is abundant, free and continuously refreshed — and it is generated entirely from documents your current system chose to show. A document your retriever never surfaces gets zero clicks, so it looks irrelevant, so any model trained or evaluated on clicks learns to reproduce the current system. It is a feedback loop with your own ranker, which is the one thing an evaluation must not be.

Chapter 8: The nDCG Playground

This is the payoff chapter. Everything you derived in Chapters 3 and 4 is now a thing you can grab with your fingers. Ten documents, graded 0 to 3 by an imaginary annotator, in a ranking you control. Drag any document to any position and every metric recomputes live.

The goal is not to play. The goal is to build the intuition that lets you look at a two-point nDCG difference in a pull request and know, without computing anything, roughly what changed in the ranking.

Six experiments, and what each one proves

Experiment 1 — the ideal order. Press “ideal order.” nDCG reads exactly 1.000, because DCG now equals IDCG by construction. This is the definition made visible: nDCG is not “how good is this ranking” in the abstract, it is “how close is this ranking to the best one possible for this query's judgments.” A query where only one marginal document exists can still score 1.000.

Experiment 2 — move only the grade-3 document. Put it at rank 1, note the number, then drag it to rank 2, 3, 4, and so on, changing nothing else. Watch the drops:

Most of the damage happens in the first two steps. Moving from rank 1 to rank 2 costs 2.58 of the 4.98 total possible loss — 52% of the whole penalty is incurred by one position. That is the practical meaning of the logarithmic discount, and it is why reranking the top three is worth so much more engineering than reranking positions 7 through 10.

Experiment 3 — A versus B, the chapter-3 result, live. Press “system A”, read every number, then press “system B.” nDCG@10 jumps from 0.744 to 0.930. Now look at the other four readouts: recall@5, recall@10, MRR and MAP do not move at all. You proved this with arithmetic in Chapter 4; watching it happen is what makes it stick.

Experiment 4 — shrink the cutoff. Set the ranking to something mediocre, then slide k from 10 down to 3. Two things change at once and it is worth separating them. DCG shrinks because you are summing fewer terms. IDCG also shrinks, because the ideal ranking is also truncated at k. So nDCG can move in either direction: if your good documents are early it goes up as k shrinks, and if they are late it collapses.

Experiment 5 — the reverse order. Press “worst order.” The number is not 0. It is well above zero, because all the relevant documents are still in the window, just at the bottom, and the logarithmic discount at rank 10 is still 0.289 rather than something near zero. nDCG@10 near 0.4 does not mean “40% correct”; on a window that contains every relevant document it can mean “perfectly backwards.” Metrics are not percentages, and the floor of a metric is a number you should know for your own data before you interpret any score.

Experiment 6 — swap two zeros. Drag a grade-0 document from rank 6 to rank 7 and back. Nothing moves. Every metric is completely blind to the ordering of irrelevant documents — their gain is 2⁰ − 1 = 0, so their discount is multiplied by nothing. Two systems can produce visibly different result pages and score identically if they only disagree about junk. When you are debugging a “why did the score not change” mystery, this is usually the answer.

What to take to work

Three calibrations that this widget installs, which are hard to get any other way.

That third row is not a joke. The most common cause of a spectacular overnight metric jump is an identifier mismatch that silently turned every judgment into a miss, or a fresh pooling pass that added judgments the old run never had. When a number moves more than you can explain mechanically, suspect the harness before the model.

Experiment 7 — where the ideal ranking comes from

One thing the widget quietly demonstrates every time you press a button: the IDCG readout never changes. Drag documents all day and it sits at 11.567 (for k = 10). That is not a bug and it is the most commonly misunderstood part of the metric.

IDCG is computed from the judgments, not from your ranking. It is the DCG of the best ordering that any system could possibly produce for this query given what the annotators graded. It changes only when the judgments change or the cutoff changes. Two consequences follow immediately.

First: if you re-judge your golden set — add documents, upgrade a grade from 1 to 2 — every historical nDCG number you have recorded becomes incomparable, because the denominator moved. Version your judgments and re-score old systems when you change them.

Second: a query with a single grade-1 document has an IDCG of exactly 1.000, so putting that one marginal document at rank 1 scores a perfect 1.000. nDCG is graded on a curve, per query. A dataset full of easy queries will report high nDCG for every system and discriminate between none of them, which is one more reason to look at per-query distributions rather than only the mean.

What the playground is lying about

Every simulation simplifies, and being explicit about how is part of teaching honestly. Three ways this one is kinder than reality.

None of these make the metric useless. They make it a measurement with an error bar and a bias, which is what every measurement is. The mistake is treating a number computed to four decimal places as though it were exact.

Read the diff: three real scenarios

Practice the diagnosis. Each row is a change you shipped and the metric movement it produced. Cover the right column and work out the cause before reading it.

Experiment 8 — where the reranker earns its keep

Set the ranking to something realistic — press “shuffle” until the good documents are spread around ranks 3 to 9 — and then read the contribution column down the list. Those numbers are the exact budget a reranker is competing for.

A reranker's job is to permute the ten documents already in the window. The most it can possibly win is the difference between your current DCG and the best arrangement of those same ten:

Press “ideal order” to see the first term, then your arrangement to see the second. If the gap is small, a reranker cannot help you and your problem is retrieval — the right documents are not in the window at all. If the gap is large, a reranker is the cheapest available improvement, because it changes no index and re-embeds nothing.

Predict before you drag

The fastest way to install the intuition is to guess first. Set the widget to “system A” and answer these three before touching anything; the numbers are all computable from the discount table in Chapter 3.

The third row is the one worth internalizing. Removing an irrelevant document from a high position is not neutral — it promotes everything beneath it. That is why deduplication and junk filtering often buy more nDCG than a better retriever: they are not adding good documents, they are removing the ones that were pushing good documents down.

Experiment 9 — collapse the grades and watch nDCG go blind too

Chapter 4 established that recall, MRR and MAP cannot separate systems A and B because they are binary. It is worth proving that this is a property of binary judgments, not of those particular formulas — because nDCG on binary judgments goes blind in exactly the same way.

Take system A and flatten every grade to relevant / not relevant, so a grade-3 and a grade-1 both become 1:

Now every gain is 2¹ − 1 = 1, so the DCG is just the sum of the discounts at the relevant positions — ranks 1, 3, 4, 6, 8, 10:

The ideal binary ranking puts all six relevant documents in the first six positions:

And system B? Its relevant documents sit at the same six ranks, so its binary grade vector is identical and its nDCG is 0.8749 as well. Under binary judgments, nDCG joins the other four metrics in reporting that two systems eighteen points apart are the same system.

This also explains a benchmark detail you will now notice. Several BEIR datasets ship only binary judgments, and their nDCG@10 columns are therefore doing less work than the graded ones — they are, in effect, a discounted recall. When you compare two models on such a dataset, you are measuring where the relevant documents landed, never which relevant document landed first.

Choosing k for a RAG stack, concretely

Before the table, one framing sentence that saves arguments. The cutoff is not a property of the metric and it is not a matter of taste — it is a description of your product. Whoever or whatever consumes the ranking reads a fixed number of documents, and k is that number. Everything below is just that rule applied to the four shapes a retrieval consumer usually takes.

The cutoff slider is not a toy setting; it is a decision you have to make once and then live with. Match it to what your generator actually reads.

The last row is worth a sentence because it is increasingly the common case and it breaks the framing of this whole lesson in an interesting way. When a system can retry, a bad first retrieval costs latency rather than correctness, and the metric you care about shifts from “was it right” to “how many attempts did it take.” Everything derived here still applies per call — you still need graded judgments, paired comparisons and a private eval — but the number you report to the business becomes a cost, not an accuracy.

Chapter 9: Connections and What to Do Monday

We started with a regression you shipped on a Monday and could not explain. You can explain it completely now, and more usefully, you can prevent the next one. This chapter assembles the pieces into a procedure, marks the limits of what any benchmark can do, and points at where to go next.

The decision procedure

Every embedding model choice, in order. Each step is cheap and each one can end the process.

Benchmark or your eval? Both, for different jobs

The benchmark is a screening instrument and your eval is the decision instrument. Using either one for the other's job is the error this whole lesson has been about.

The limits of a single vector

One honest caveat about the entire dense-embedding paradigm, because it bounds what any benchmark can promise.

When you compress a document into one fixed-length vector, you have committed to a fixed information budget. A document that answers many different questions must place all of those answers in one point. As the corpus grows, more documents crowd into any given direction, and the margin between the right answer and the distractors shrinks — this is the mechanism behind the observed difficulty of dense retrieval at very large index sizes. It is not a training problem you can fix with more data; it is a capacity constraint.

The architectural answers are worth knowing because they show up on the same leaderboards:

Every one of these is a statement that a single 768-number summary of a document is not always enough. Whenever your evaluation says a strong model is failing on your corpus and you cannot fix it with prompts or chunking, one of these is usually the structural answer.

Six things to remember when everything else fades

1. A leaderboard reports an average over its task distribution. Your task is one point, and it need not lie anywhere near that distribution.

2. The task families demand different geometry — classification only needs linear separability, which is invariant to transforms that destroy cosine similarity. So the average is not a quantity.

3. In-domain performance does not predict zero-shot performance, and BM25 — whose term weights are computed from your corpus — is still the baseline to beat on anything full of rare identifiers.

4. nDCG is the only common retrieval metric that reads grades and position together. Recall, MRR and MAP are binary and will happily report that two visibly different rankings are identical.

5. Pair your comparisons. The same 500 queries resolve 4.3 points unpaired and 1.3 points paired, for no extra data.

6. The top of a crowded leaderboard is inflated by selection even when nobody cheats, and by contamination when the benchmark's training splits are in everybody's training mixture. Your private eval is the only instrument immune to both.

Keep exploring

← Vector Embeddings — what these vectors are and how text becomes one
← Similarity Metrics — cosine, dot product, Euclidean, and when each is right
← Text Chunking — the decision upstream of every retrieval score in this lesson
← Embedding Layers — where a token vector comes from inside the model
→ Vector Databases — indexes, approximate search, and the recall you lose to speed
→ RAG — the pipeline these metrics are protecting
→ Multimodal RAG — the same evaluation problem when documents are images and audio
→ Evaluation Statistics — confidence intervals, bootstrap and significance in depth
→ Evaluation Plots — how to show a metric difference so it cannot be misread
→ Experiment Design — power, sample size and stratification beyond the sketch in Chapter 5
→ The Metrics Ladder — connecting a retrieval metric to a product number an executive cares about
→ GenAI Evaluation — judging the generation layer once retrieval is fixed
→ AI Evaluation — the broader evaluation toolkit for model-backed products
→ Regression Testing for ML — turning the gate in Chapter 7 into infrastructure
→ Agent Evaluation — the same discipline when the system takes actions
→ Contrastive Learning & CLIP — the training objective that produced most of these embedders
→ BERT — the encoder nearly every text embedding model descends from

References

1. Muennighoff, Tazi, Magne, Reimers. “MTEB: Massive Text Embedding Benchmark.” EACL, 2023. arXiv:2210.07316
2. Thakur, Reimers, Rücklé, Srivastava, Gurevych. “BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.” NeurIPS Datasets and Benchmarks, 2021. arXiv:2104.08663
3. Enevoldsen et al. “MMTEB: Massive Multilingual Text Embedding Benchmark.” 2025. arXiv:2502.13595
4. Järvelin, Kekäläinen. “Cumulated Gain-based Evaluation of IR Techniques.” ACM Transactions on Information Systems 20(4), 2002. The paper that introduced DCG and nDCG.
5. Robertson, Zaragoza. “The Probabilistic Relevance Framework: BM25 and Beyond.” Foundations and Trends in Information Retrieval 3(4), 2009.
6. Reimers, Gurevych. “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.” EMNLP, 2019. arXiv:1908.10084
7. Karpukhin et al. “Dense Passage Retrieval for Open-Domain Question Answering.” EMNLP, 2020. arXiv:2004.04906
8. Khattab, Zaharia. “ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.” SIGIR, 2020. arXiv:2004.12832
9. Santhanam et al. “ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction.” NAACL, 2022. arXiv:2112.01488
10. Formal et al. “SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval.” 2021. arXiv:2109.10086
11. Nogueira, Lin. “Document Expansion by Query Prediction.” 2019. arXiv:1904.08375
12. Hofstätter et al. “Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling.” SIGIR, 2021. arXiv:2104.06967
13. Wang et al. “Text Embeddings by Weakly-Supervised Contrastive Pre-training” (the E5 family). 2022. arXiv:2212.03533
14. Xiao et al. “C-Pack: Packed Resources For General Chinese Embeddings” (the BGE family). 2023. arXiv:2309.07597
15. Ni et al. “Large Dual Encoders Are Generalizable Retrievers” (GTR). 2021. arXiv:2112.07899
16. Su et al. “One Embedder, Any Task: Instruction-Finetuned Text Embeddings” (INSTRUCTOR). 2022. arXiv:2212.09741
17. Kusupati et al. “Matryoshka Representation Learning.” NeurIPS, 2022. arXiv:2205.13147
18. Reimers, Gurevych. “The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes.” ACL, 2021. arXiv:2012.14210
19. Sciavolino et al. “Simple Entity-Centric Questions Challenge Dense Retrievers.” EMNLP, 2021. arXiv:2109.08535
20. Bajaj et al. “MS MARCO: A Human Generated MAchine Reading COmprehension Dataset.” 2016. arXiv:1611.09268
21. Conneau, Kiela. “SentEval: An Evaluation Toolkit for Universal Sentence Representations.” LREC, 2018. arXiv:1803.05449
22. Cohen, J. “A Coefficient of Agreement for Nominal Scales.” Educational and Psychological Measurement 20(1), 1960. The source of the kappa in Chapter 7.

The one-page runbook

Print this. It is the whole lesson compressed to the things you do rather than the things you know.

The three sentences to say in a meeting

Most of this lesson is technique. This part is politics, and it is what actually stops the bad decision.

When somebody says “this model is number one on the leaderboard”: “That average is about a quarter retrieval and three quarters things we do not do. What is its score on the retrieval family, and what is its worst dataset in that family?”

When somebody shows a two-point improvement: “Is that paired, and how many queries? On five hundred queries an unpaired two-point gap is inside the noise, and a paired one is not — so which is it?”

When somebody wants to ship on a public number: “We can measure it on our own three hundred queries this afternoon. If it agrees with the leaderboard we have lost an afternoon; if it disagrees we have saved a quarter.”

None of the three requires you to win an argument about architecture. Each one replaces an opinion with a measurement that somebody can go and take, which is the only kind of argument that ends.

Glossary of everything this lesson defined

Where embedding evaluation is heading

Four shifts are visible right now, and each one changes what a benchmark can tell you.

Instructions become part of the input. Instruction-tuned embedders take a task description alongside the text — “represent this document for retrieval of counter-arguments” — which means the relevance relation stops being baked into the weights and becomes an argument at call time. That is exactly the fix for the ArguAna problem in Chapter 2. It also means the benchmark must specify the instruction, and two evaluations of the same model with different instructions are measuring different functions.

Context windows grow. When an encoder accepts eight thousand tokens rather than five hundred, the chunking decision that dominated your retrieval quality starts to dissolve. It does not disappear — a single vector summarizing eight thousand tokens is a harder compression problem, not an easier one — but the silent-truncation failure of Chapter 5 becomes rarer, and the capacity argument of the previous section becomes sharper.

Benchmarks get bigger and rank-aggregated. Hundreds of tasks across hundreds of languages, aggregated by rank rather than by mean, with per-family views promoted over the headline column. This dilutes contamination and stops wide-spread tasks from dominating — and it does not change a single thing about Chapter 0, because your task is still one point.

The judge becomes a model. Relevance labelling at scale increasingly runs through a language model, which makes the calibration step of Chapter 7 — measure the judge against humans with the same kappa you would use for a second annotator — the load-bearing part of the whole pipeline. A cheap judge with an unmeasured agreement rate is not an evaluation; it is a second model's opinion wearing an evaluation's clothes.