Embeddings I: LLMs as Embedding Models

Chapter 1: Causal Attention Meets Pooling

Session 01 established the core fact about a decoder-only transformer: every token position produces a hidden vector, and thanks to causal attention — the triangular mask that blocks each position from looking at anything after it — that hidden vector can only depend on the tokens at or before its own position. Position 5 has read tokens 0 through 5. It has never seen token 6. That was exactly the right property for next-token prediction, where seeing the future would be cheating.

Now we need something different from that same machine. An embedder does not want one vector per token. It wants one vector for the whole input. Given N hidden vectors, one per token, how do you collapse them into a single d-dimensional summary? This operation is called pooling, and the choice of pooling strategy turns out to matter more than it has any right to.

The bidirectional-era default: mean pooling

For a BERT-style bidirectional encoder, the standard answer (used by Sentence-BERT, SimCSE, and most embedders before 2024) is mean pooling: average all N token vectors together, position by position. The logic is clean. Bidirectional attention means every token position has already seen the entire sequence — token 2 attended to token 8 just as freely as token 8 attended to token 2. So every position is, in a sense, already a full-sentence summary from its own local point of view, and averaging N independent full-sentence summaries is a reasonable way to reduce noise, the same instinct behind averaging N independent measurements of anything.

Why that logic breaks under a causal mask

Run the same argument for a decoder. Token 2’s hidden state has seen tokens 0, 1, and 2 — three tokens out of, say, eight. Token 8’s hidden state has seen all eight. Mean pooling averages these together with equal weight, which means it is averaging a vector that saw 37.5% of the sentence with a vector that saw 100% of it, as if they carried the same amount of information. They do not.

Make this concrete with a hand count. For a sequence of N tokens under a causal mask, the position at index i (0-indexed) has attended to i+1 tokens out of N. Its coverage — the fraction of the sequence it has actually seen — is (i+1)/N. Sum that coverage across all N positions and divide by N to get the average coverage that a naive mean-pool is built from:

For an 8-token input: (8+1) ÷ 16 = 9/16 = 56.25%. Mean pooling over a causal decoder’s hidden states is, on average, built from vectors that have seen barely more than half the sentence. One position, and only one, is guaranteed full coverage — the last one, which has attended to all N tokens because there is nothing after it to be masked from.

The causal mask, counted by hand

Before trusting the coverage formula, verify it on a sequence small enough to count on your fingers. Take N = 4 tokens: “The cat sat down.” A causal attention mask is a 4×4 grid of (query position, key position) pairs, and position i is allowed to attend to key position j only when j ≤ i. Count the allowed pairs directly: position 0 sees 1 key (itself), position 1 sees 2, position 2 sees 3, position 3 sees 4.

Put another way: a causal decoder throws away 37.5% of the connections a bidirectional encoder of the same length would use, and it throws them away unevenly — all of the loss falls on the early positions, none of it on the last one. That asymmetry is exactly why last-token pooling is not an arbitrary convention; it is the only position that paid none of that 37.5% cost.

The asymptotic case: why this matters more for longer inputs

Re-run the average-coverage formula from a moment ago, (N+1)/(2N), for a document-length sequence instead of a one-sentence one. At N = 32 tokens:

Compare it to the N = 8 case computed above (56.25%). As N grows, average coverage keeps sliding toward the algebraic limit:

So for long passages — the long-document retrieval tasks Chapter 2’s taxonomy explicitly designs for — mean pooling under a causal mask asymptotically wastes half its information budget on average, no matter how long the document gets. Last-token pooling stays at 100% coverage regardless of length, because its one guaranteed-full-context position does not depend on N at all. The gap the ablation measured in Chapter 1’s table (−0.4 for mean pooling) was measured on MTEB’s mostly short-to-medium inputs; the architectural argument predicts that gap would widen, not shrink, on genuinely long documents.

One more implementation subtlety: which side do you pad?

The last_token_pool code below already handles both padding conventions, but it is worth knowing why production embedding checkpoints (including the public E5-Mistral release) typically instruct callers to set tokenizer.padding_side = "left" rather than the more common right-padding default. With left-padding, every sequence’s real final token lands in the exact same column — the last one — regardless of how much padding precedes it, so a single hidden_states[:, -1] slice is correct for the entire batch with no per-example index lookup at all. Right-padding is fine too (the sequence_lengths branch above handles it correctly), but it is one more place a shape mismatch can hide, and left-padding removes the failure mode entirely by construction. When in doubt, prefer whichever convention the checkpoint’s own model card specifies — this is exactly the kind of detail that is invisible in a loss curve and only shows up as quietly wrong embeddings downstream.

Instructing only the query, never the document

E5-Mistral does one more thing before pooling: it wraps the query side of a pair in a short natural language instruction, but leaves the document side untouched.

where {task_definition} is a one-sentence description of what this particular retrieval task is asking for — “Given a web search query, retrieve relevant passages that answer the query,” for instance. The document gets no such wrapper; it is embedded as-is.

This asymmetry is a deliberate engineering decision, not an oversight. If you had to re-instruct the document side too, you would have to re-embed your entire corpus every time you wanted to serve a different kind of query against it. Leaving documents instruction-free means you can build the document index once, store it, and reuse it for every task — classification, retrieval, clustering — simply by changing the instruction on the query side at request time. The cost of a new capability drops from “re-embed a ten-million-document corpus” to “change one sentence in the prompt.”

Make the “one sentence changes at request time” claim concrete with two actual query strings against the exact same untouched document index. A web-search request against Chapter 0’s documentation corpus:

And a classification request — a completely different task family, run against the very same documents, with no re-embedding step in between:

Both strings are embedded by the identical model, through the identical last_token_pool readout, and land wherever that instruction happens to point the last-token vector in the model’s representation space. Nothing about the 40,000-chunk document index built in Chapter 0 has to change between these two requests — only the seven-or-so words after Instruct: do. That is the entire mechanism behind the “change one sentence in the prompt” claim, made as literal as it gets.

The forward pass, in code, including the bug that silently ignores your instruction

Here is the whole thing as runnable-shaped PyTorch, including the left-padding subtlety that trips up almost every first implementation.

python
import torch

def last_token_pool(hidden_states, attention_mask):
    # hidden_states: (B, L, d_llm)  -- one vector per token, per example
    # attention_mask: (B, L)        -- 1 for real tokens, 0 for padding
    left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0])
    if left_padding:
        # every sequence's real last token IS the final column -- easy case
        return hidden_states[:, -1]
    # right-padded batch: the real last token is at a DIFFERENT index per example
    sequence_lengths = attention_mask.sum(dim=1) - 1          # (B,) -- index of each real [EOS]
    batch_size = hidden_states.shape[0]
    return hidden_states[torch.arange(batch_size), sequence_lengths]  # (B, d_llm)

Read the bug this guards against. If you right-pad a batch (short sequences padded with zeros on the right, the common default) and then naively slice hidden_states[:, -1], you get the hidden state of a padding token for every sequence shorter than the batch’s longest one — not the real [EOS]. The model still runs, the loss still decreases somewhat because the longest sequence in every batch happens to be correct, and the resulting embeddings for every shorter sequence are quietly built from garbage. This is precisely the kind of “looks fine, trains, ships broken” failure the Concept → Realization habit exists to catch: print sequence_lengths for a batch with mixed-length inputs and confirm it matches what you expect before you trust a single downstream number.

The pooling choice is not just more correct — it is also free

There is a second, purely computational reason last-token pooling is the right default that has nothing to do with coverage. Count the actual floating-point work each strategy does once the transformer’s forward pass has already produced all N hidden vectors of dimension d. Mean pooling has to touch every one of them: sum N vectors together element-by-element, then divide each of the d dimensions by N.

Last-token pooling does none of that. The last hidden vector already exists, sitting in memory as hidden_states[:, -1] — reading it out is a slice, not a computation. Zero additions, zero divisions, for any sequence length at all:

Two million extra floating-point additions per 512-token input sounds small next to the billions of operations the transformer itself performs to produce those hidden states in the first place — and it is small, which is exactly why mean pooling’s real cost was never its compute budget. The point of this comparison is not that mean pooling is expensive; it is that last-token pooling wins on every axis at once. It is architecturally more correct (100% coverage regardless of length, from earlier in this chapter), and it is also strictly cheaper to compute, and the second property gets more pronounced, not less, exactly as documents get longer and mean pooling’s summation grows with them while last-token pooling’s cost stays at zero. A design choice that wins on correctness and on cost simultaneously, with no tradeoff between them, is rare enough in engineering to be worth noticing when it happens.

What the ablation actually shows

The paper ran the controlled comparison — same model, same data, same everything except pooling strategy — and reports the result on the MTEB average (56 English datasets):

The gap is real but small — not the wide margin the coverage argument above might lead you to predict. Why? Because Mistral-7B was pretrained on trillions of tokens of next-token prediction across many transformer layers, and by the deeper layers, information from far-back tokens has already propagated forward through repeated rounds of self-attention, even into positions that never directly attended to it in a single layer. Mean pooling is architecturally the wrong tool for a causal model — the Chapter 1 argument above is correct — but a sufficiently deep, sufficiently well-pretrained decoder partially compensates for the wrong tool being used. Small architectural mistakes on top of a very strong base model tend to cost points, not correctness.

Chapter 2: Brainstorm, Then Generate

Chapter 1 answered the mechanical question: given a query-document pair, how does a decoder turn each side into one vector. This chapter answers the harder question the entire paper actually hinges on: where do the training pairs come from, if you have just refused the field’s standard answer — billions of weakly-supervised web pairs plus a curated labeled set — from Chapter 0?

E5-Mistral’s answer is blunt: don’t curate. Generate. Use a strong proprietary LLM (the paper uses GPT-3.5-Turbo and GPT-4 via Azure OpenAI) to write the training pairs itself, across a deliberately designed taxonomy of task types and a long list of languages.

The naive version, and why it fails

The obvious first attempt is a single prompt: “write a query and a matching document.” The paper tried exactly this and reports that data diversity “was not as satisfactory” as the two-step approach below. The failure mode is predictable if you have ever asked an LLM to “be creative” in one shot: it reaches for the same handful of obvious, high-probability task shapes every time — product reviews, weather queries, movie summaries — because those are the most probable completions of a vague creative-writing instruction. You get thousands of examples that are really five task types wearing different words.

The fix: separate “what tasks exist” from “give me one example”

The two-step prompt splits the job. Step 1, brainstorm: ask the LLM to list about twenty distinct task definitions within one category, as plain sentences, with no actual query-document content yet. Step 2, generate: hand the model exactly one of those twenty task definitions, plus a few randomly sampled placeholders, and ask for a single concrete (query, positive document, hard-negative document) triple as JSON.

Why this works where the one-shot version failed: brainstorming twenty task definitions forces the model to spread across the space of what a category could mean before committing to any single example, the same way asking a person to list twenty ideas produces more variety than asking for one idea twenty separate times. Step 2 then only has to be creative about the surface details of one already-specific task, which is a much easier and much more diverse thing to vary.

The taxonomy: four asymmetric shapes, two symmetric ones

Not every embedding task has the same query-document geometry. The paper organizes tasks into two families.

Symmetric tasks skip the brainstorming step entirely — the task definition is already obvious (“find the translation”), so there is nothing to diversify at that level; the placeholders alone provide enough variation.

Reading the actual prompt

Here is a condensed version of the paper’s real short–long generation prompt (Appendix C), with the randomly-sampled placeholders left visible so you can see exactly what gets varied at generation time:

prompt (step 2, short–long match, condensed)
You have been assigned a retrieval task: {task}

Write one text retrieval example for this task in JSON format, with keys:
  "user_query": a string, {query_type}, {query_length}, {clarity}, diverse in topic
  "positive_document": a string, relevant to the query, at least {num_words} words
  "hard_negative_document": relevant-looking but less useful than the positive

Both query and documents should be in {language}.
Both require {difficulty} level education to understand.
Output a JSON object only. Be creative!

where {query_type} ∈ {extremely long-tail, long-tail, common}, {difficulty} ∈ {high school, college, PhD}, {clarity} ∈ {clear, understandable with effort, ambiguous}, and {language} is sampled from the language list used by XLM-R, weighted toward higher-resource languages. Every one of these placeholders is re-rolled independently every time the prompt runs — the same task definition from Step 1 can therefore produce dozens of genuinely different examples.

A second prompt, to see the pattern repeat

Compare the short–long template above against the paper’s short–short matching template (word ↔ definition, entity ↔ fact — the shape used for tasks like “match a scientific paper title to a title that cites it” or “match a notable person’s name to their occupation”), condensed the same way:

prompt (step 2, short–short match, condensed)
You have been assigned a text matching task: {task}

Write one example for this task in JSON format, with keys:
  "input": a string, a random input specified by the task
  "positive_document": a string, relevant to "input" according to the task

Both should be very short (a sentence or phrase) — avoid substantial
word overlap, or the task becomes too easy. "input" and "positive_document"
should be independent of each other. Values in {language}.
Output a JSON object only. Be creative!

Notice what is deliberately absent from this template compared to the short–long one: no hard_negative_document field at all. The paper’s own reasoning is that short–short matching tasks are already difficult enough — two very short, non-overlapping strings that are still supposed to relate — that generating an explicit hard negative adds cost without adding much training signal. This is a small but telling design choice: the taxonomy is not one rigid template stamped out four times, it is four templates each tuned to what actually makes that task shape hard to learn.

Concept → realization: the two-step pipeline as actual code

The flow diagram above is the idea. Here is the same pipeline as a function, with the actual data shape flowing through each step made explicit — this is the level of detail that lets you reproduce the recipe rather than just recognize it.

python
def generate_synthetic_examples(category, n_task_defs=20, examples_per_def=3):
    # Step 1 -- brainstorm: ONE call produces a list of task strings, no query/doc content yet
    task_defs = llm.complete(
        f"Brainstorm {n_task_defs} distinct retrieval task definitions for: {category}"
    )                                            # return type: list[str], length ~20

    examples = []
    for task in task_defs:                       # ~20 iterations
        for _ in range(examples_per_def):        # re-roll placeholders each time
            placeholders = {
                "query_type":  random.choice(["long-tail", "common"]),
                "difficulty":   random.choice(["high school", "college", "PhD"]),
                "language":     sample_language(),   # weighted toward high-resource
            }
            # Step 2 -- generate: ONE call per (task, placeholder-roll) produces one JSON triple
            raw = llm.complete(build_step2_prompt(task, placeholders))
            try:
                parsed = json.loads(raw)              # dict with 3 keys, or raises on malformed output
                if parsed not in seen:            # exact-string dedup
                    examples.append(parsed); seen.add(parsed)
            except json.JSONDecodeError:
                continue                             # discard, matching Ch.0's "filter, don't perfect" principle
    return examples                                  # list[dict], each {"user_query", "positive_document", "hard_negative_document"}

Trace the shapes through this function once, because they are exactly what Chapter 3’s training loop expects on the other end. task_defs is a flat list of strings — roughly 20 per category, no structure beyond plain sentences. Each pass through the inner loop calls the LLM again with one task definition plus a fresh draw of placeholders, and expects back a JSON object with exactly three string fields. Anything that fails to parse as that exact shape is silently dropped, not repaired — there is no attempt to salvage a malformed response, because at this scale (hundreds of thousands of generations) a simple discard-and-move-on policy is cheaper and more robust than trying to fix broken JSON one response at a time. The seen set is doing Chapter 0’s “filter noisy data at scale” principle at the level of a single Python data structure: no model call, no heuristic, just an exact-match guard against the cheapest failure mode (the same example generated twice).

Worked numbers: what 500k examples actually looked like

The generation run produced 500,000 synthetic examples drawn from 150,000 unique brainstormed task instructions, using Azure OpenAI — about 25% from GPT-3.5-Turbo and the rest from GPT-4, consuming roughly 180 million tokens in total, across 93 languages. Split the model usage into raw counts:

Three GPT-4 examples for every one GPT-3.5-Turbo example. That ratio is itself an engineering decision worth noticing: GPT-4 is the more expensive, higher-quality generator, but the paper does not use it exclusively — it blends in a substantial minority of cheaper GPT-3.5-Turbo output and reports the overall quality as still acceptable after filtering. That is a direct, load-bearing example of the “filter noisy data at scale rather than pay for perfect data” principle the misconception callout above states in the abstract.

Do the division the paper leaves implicit: 500,000 examples ÷ 150,000 unique instructions ≈ 3.33 examples generated per task instruction, on average. That is exactly what the two-step design predicts — each brainstormed task definition gets reused a handful of times with freshly re-rolled placeholders, rather than being used only once.

Now look at where the language coverage actually went. The paper reports that the bottom 75 of the 93 languages (the lower-resource end) received about 1,000 examples each on average:

Which means the remaining 18 higher-resource languages (English chief among them) absorbed the other 85% — roughly 425,000 examples — between them. This is not a flaw introduced by carelessness; it is what you get when the placeholder sampler is deliberately weighted toward high-resource languages, which is itself a reasonable choice if your priority is maximizing average English/European performance. Hold this number. Chapter 5 shows exactly what this imbalance costs on languages outside that top 18, and Chapter 6 shows the same effect measured across the entire field’s embedding models, not just this one.

Quantifying what the synthetic data alone is worth

Chapter 3 will introduce the full training mixture, but one comparison belongs here because it isolates exactly this chapter’s contribution. Train the same model two ways: on MS-MARCO alone (a single large, high-quality, purely human-curated dataset, with no synthetic data at all), versus on MS-MARCO plus this chapter’s synthetic data.

Adding this chapter’s two-step synthetic recipe on top of a single strong human-curated dataset is worth 1.8 points on its own — a controlled, apples-to-apples measurement of exactly what “brainstorm, then generate” buys, holding everything else in the training setup fixed. Combined with Chapter 5’s later result that the full mixture (synthetic plus thirteen public datasets) reaches 66.6, the synthetic recipe is doing real, measurable work at every stage of the mixture, not just as a stand-in when better data is unavailable.

Concept → realization: what a “new capability” costs now

Compare the two recipes on the one metric that actually determines whether a team ships a new capability or shelves it.

Chapter 3: InfoNCE and LoRA, By Hand

We now have millions of (query, positive document) pairs — the 500k synthetic examples from Chapter 2 mixed with a sample from thirteen public datasets: MS-MARCO, Natural Questions, HotpotQA, FEVER, NLI, SQuAD, TriviaQA, Quora Duplicate Questions, MIRACL, MrTyDi, DuReader, T2Ranking, and ELI5. Several of these are sampled at a fraction of their full size per epoch (ELI5 at 10%, MS-MARCO document ranking at 20%, MS-MARCO passage ranking and T2Ranking at 50%) so that no single very large dataset dominates the mixture. After sampling, the total lands at roughly 1.8 million examples.

None of these thirteen datasets was chosen arbitrarily — each contributes a task shape the synthetic generator either cannot produce as reliably or that the field already trusts as a strong, human-verified signal:

This chapter derives the two remaining pieces: the loss function that turns pairs into a trained embedder, and the fine-tuning trick that makes training a 7-billion-parameter model on a modest number of GPUs affordable at all.

InfoNCE: pull the right answer close, push the wrong ones away

Each training example is a query q, a positive document d⁺, and a set of negative documents. Following Chapter 1’s pooling, encode everything to unit-normalized vectors and define a similarity score:

where τ is a temperature hyperparameter, fixed at 0.02 in the paper. The training loss, InfoNCE, is:

Read this as a softmax classification loss in disguise. The “classes” are the positive document plus every negative document n in the set N; the “correct class” is the positive. Minimizing ℒ is exactly maximizing the softmax probability the model assigns to the positive relative to all the wrong candidates. Get the positive ranked far above every negative and the loss goes to zero; rank a negative above the positive and the loss explodes.

Where the negatives come from: free ones and expensive ones

N has two sources. In-batch negatives cost nothing extra to encode: every other query’s positive document in the same training batch automatically serves as a negative for this query, because in a batch of unrelated pairs, query A’s document is (almost always) irrelevant to query B. A batch size of 2048 therefore hands every query up to 2047 negatives for free. Hard negatives cost more: one is added explicitly per pair, mined by running a weaker existing embedder (multilingual-E5-base) and taking documents that rank in its top 100 but are not the true positive — documents that look relevant on the surface (same keywords, same topic) but are not the right answer. Hard negatives are what teach genuinely fine-grained discrimination; a random in-batch negative is almost always trivially unrelated and teaches very little once the model is past its first few hundred steps.

The loss, by hand, on numbers small enough to check

Build a toy example in a 2D unit-vector space where cosine similarity is just the dot product. Say a query has cosine similarity 0.9 with its true positive document, 0.0 with one negative, and −0.5 with a second, harder-to-distinguish negative. To keep the arithmetic legible, use a friendlier τ = 0.5 first (the real τ = 0.02 comes right after).

Now weaken the positive to a more middling cosine similarity of 0.3, everything else unchanged, and watch the loss respond:

A worse-ranked positive costs nearly three times the loss — exactly the behavior a good contrastive loss should have. Now consider why the real system uses τ = 0.02, not 0.5. Cosine similarity is bounded to [−1, 1], a narrow range. Divide a small gap between two cosine scores by a small τ and that gap gets stretched enormously before it hits the exponential — a tiny cosine difference of, say, 0.02 becomes a full unit of difference inside the exponent once divided by τ = 0.02, producing a softmax so sharp it is nearly one-hot. That sharpness is deliberate: with embeddings squeezed into a bounded similarity range, a small τ is what converts a subtle ranking difference into a gradient strong enough to actually move the weights.

Checking a paper number by re-deriving it

The paper states training took “fewer than 1k steps” for one epoch over the ~1.8M-example mixture, with batch size 2048. Verify it:

This is a useful habit for reading any paper: when two numbers are both reported (dataset size, batch size), the step count is not a separate fact to memorize — it is arithmetic you can check yourself, and if it does not come out close, something in your understanding of the setup is wrong.

One more quantity is worth deriving the same way, because it puts a rough ceiling on how much text this entire fine-tuning run actually reads. Training caps sequence length at 512 tokens; if every one of the 1.8M examples in the mixture happened to hit that cap (an upper bound, since most training pairs are shorter than 512 tokens):

Compare that to the trillions of tokens Mistral-7B saw during its own pretraining, referenced repeatedly across this session as the reason the decoder already has rich representations to expose (Chapter 4 makes this argument in full). Fine-tuning reads, at the very most, under a billion tokens — a rounding error next to pretraining’s trillions. That size mismatch is itself a compact restatement of Chapter 0’s “light fine-tuning, not retraining” claim: the fine-tuning stage is not attempting to teach the model language from anywhere near the scale that would require. It is reorganizing a representation that took three to four orders of magnitude more data to build in the first place.

One more number falls out of the same setup for free. Training ran on 32 GPUs (the next section derives why so many), and a global batch size of 2048 has to be split evenly across all of them for standard data-parallel training:

That 64-per-GPU figure matters for a reason beyond bookkeeping: it is the effective local batch size each GPU's forward and backward pass actually has to fit in memory at once, before gradients are synchronized back across the other 31. A batch size of 2048 sounds large in the abstract; 64 examples of up to 512 tokens each, run through a 7-billion-parameter model, is the number that actually determines whether a single GPU runs out of memory.

LoRA: updating 0.6% of a 7-billion-parameter model

Full fine-tuning of all 7 billion Mistral-7B parameters on a comparatively small instruction-tuning mixture risks catastrophic forgetting — the model overwrites the broad language ability it spent trillions of pretraining tokens acquiring, in exchange for fitting a much smaller, narrower dataset. The fix is LoRA (Low-Rank Adaptation): freeze every original weight matrix, and next to each one, attach a pair of small trainable matrices whose product is added back in as a low-rank correction. The frozen base weights never move; only the small correction is learned.

The paper attaches LoRA adapters, rank 16, to all linear layers in the model, and reports the resulting trainable parameter count: 42 million. Against Mistral-7B’s roughly 7 billion total parameters:

The other 99.4% — everything Mistral-7B learned about grammar, world knowledge, and how to read an instruction — stays exactly as it was. Fine-tuning only nudges 0.6% of the geometry into a shape where cosine similarity between the right last-token vectors happens to line up with semantic relevance.

The LoRA rank ablation is worth sitting with, because the result is almost anticlimactic:

Doubling or halving the adapter’s capacity barely moves the score — the paper even notes rank 8 is marginally better and sticks with 16 anyway, because the difference is noise-level. The lesson: adapter capacity is not the bottleneck here. Chapter 2’s data and, as Chapter 4 shows next, the instruction wording are what actually decide how good the embedder gets. A bigger adapter cannot fix data or prompt design problems.

What it actually took to run this: 32 GPUs, 18 hours

Put the whole training job on the table. Fine-tuning ran for roughly 18 hours on 32 V100 GPUs, batch size 2048, sequence length capped at 512 tokens, learning rate 1×10⁻⁴ with a 100-step linear warmup followed by linear decay, and weight decay 0.1. Three engineering techniques made this affordable on hardware from 2018 rather than requiring the newest accelerators:

Notice that even though only 42M parameters are trainable (Chapter 3’s LoRA math), all 7B parameters still have to be present on the GPUs to run the forward pass that produces the embeddings those 42M parameters are being trained against. LoRA shrinks the optimizer’s job dramatically — far less gradient and momentum state to track — but it does not shrink the frozen model itself. That is exactly why the sharding and memory-saving techniques above are still necessary even for a technique whose entire selling point is efficiency.

Coda: how far can a last-token vector actually see?

Chapter 1 argued that last-token pooling has, in principle, 100% coverage of the input no matter how long that input is. The paper stress-tests whether that principle survives in practice, with a synthetic passkey retrieval task: bury a unique person’s name and a random passkey somewhere inside a long, repetitive filler document, generate 100 such documents, and ask the model to retrieve the one document containing a given person’s passkey — a direct test of whether one pooled vector really has “seen” something buried deep inside a long context, not merely attended near the end of it.

With the model’s default 4k-token sliding attention window, accuracy is a perfect 100% for documents up to 4k tokens, then degrades quickly beyond that — unsurprising, since content outside the sliding window is structurally unreachable. Naively widening the window to 32k tokens, keeping everything else fixed, actually made results worse, not better. Only after also changing the rotary positional encoding’s base frequency (from the default to 10⁵) did the model recover to over 90% accuracy within a 32k-token context, at a small cost to short-context performance. The lesson generalizes past this one experiment: pooling strategy (Chapter 1) determines where you read the summary from, but how much of a long input actually reaches that summary position with useful resolution is a separate property of the attention mechanism’s positional scheme — last-token pooling is a necessary condition for full coverage, not a sufficient one.

python
import torch, torch.nn.functional as F

def info_nce(h_q, h_pos, h_negs, tau=0.02):
    # h_q:    (B, d)      one query vector per example in the batch
    # h_pos:  (B, d)      its matching positive document vector
    # h_negs: (B, K, d)   K hard negatives per example (mined offline)
    h_q, h_pos, h_negs = F.normalize(h_q, dim=-1), F.normalize(h_pos, dim=-1), F.normalize(h_negs, dim=-1)

    # in-batch: every row's query against every OTHER row's positive doc too
    sim_pos_all = (h_q @ h_pos.T) / tau              # (B, B) -- diagonal is the true positive
    sim_hard    = torch.einsum('bd,bkd->bk', h_q, h_negs) / tau   # (B, K) -- explicit hard negatives

    logits = torch.cat([sim_pos_all, sim_hard], dim=1)   # (B, B+K)
    labels = torch.arange(h_q.shape[0])              # true positive is column i for row i
    return F.cross_entropy(logits, labels)   # exactly the InfoNCE loss above

Notice this code needs no explicit exponential or division-and-log — cross_entropy over the concatenated similarity logits is InfoNCE, because a softmax followed by negative log-likelihood on the correct class is, symbol for symbol, the formula derived above. This is why InfoNCE is sometimes just called “the standard contrastive loss” without further explanation in papers — it is literally multi-class cross-entropy where the classes are “which document is the real match.”

Chapter 4: Does It Even Need to Learn?

Every strong embedding model before this one included a stage the paper calls contrastive pretraining: before any labeled fine-tuning, run a preliminary training pass on billions of weakly related text pairs (page titles with page bodies, forum questions with forum answers) just to teach the model the basic notion of “these two texts are about the same thing.” E5’s own predecessor used this; BGE used a related RetroMAE pretraining stage. It was treated as load-bearing — skip it, and quality drops badly.

Section 5.1 of the paper runs a direct test of whether a decoder-only LLM still needs that stage, and the result reframes the entire method.

The ablation

For the bidirectional encoder, the extra pretraining stage is doing real, necessary work. For the decoder, it is almost dead weight.

“Negligible” is worth putting exact numbers to rather than taking on faith, and the paper’s detailed results table (Appendix, Table 7) gives them directly. The +8.2-point figure above is specifically the Retrieval category score for XLM-R-large: 42.0 without contrastive pretraining, 50.2 with it. Read the same two rows for Mistral-7B and the contrast is stark — the Retrieval score is 56.9 both ways, identical to one decimal place, and the overall MTEB average moves from 66.6 to 66.7, a difference of +0.1. That is not “a small positive effect rounded down to negligible” for the write-up; it is a retrieval score that quite literally did not move.

Now weigh that non-result against what the extra stage cost to run. The paper pre-trains this contrastive-pretraining variant of Mistral-7B “following the mE5 recipe for 10k steps” — an entire additional training run, on top of the light fine-tune Chapter 3 already covers in full:

Spend more than eleven times the compute of the entire recipe’s actual fine-tuning stage, and move the headline number by a tenth of a point while leaving the metric this session cares most about — retrieval — completely unchanged. That is what “negligible impact” means in exact terms, and it is the concrete evidence behind the “why help one architecture and barely touch the other” explanation the rest of this chapter builds out mechanistically below.

Why the same trick helps one architecture and not the other

A bidirectional encoder trained with masked-language modeling only ever practices one skill: look at a single span of text with some tokens hidden, and predict the missing tokens from the surrounding context. It never practices comparing two different pieces of text against each other — that notion of “closeness between two texts” is simply absent from its pretraining objective. Contrastive pretraining exists specifically to teach that missing skill before fine-tuning tries to make it precise.

A decoder-only LLM pretrained on trillions of tokens of next-token prediction across the open internet has, as an unavoidable side effect of getting good at completion, already built an enormous implicit map of which pieces of text are semantically related. Predicting “what word comes next” accurately across billions of documents forces the model to represent topic, style, and meaning richly at every layer, for essentially every genre of text that exists online — because a model that could not distinguish a legal contract from a recipe would produce terrible next-token predictions for both. The representation the fine-tuning stage needs was already built. Fine-tuning does not teach new semantics; it teaches the model to expose semantics it already has, at the one specific readout position (last-token, Chapter 1) and in the one specific geometry (cosine-comparable, Chapter 3) that a downstream nearest-neighbor search needs.

Make the two pretraining objectives concrete

Put the two objectives side by side on the same toy sentence: “The chef seasoned the [MASK] before searing it.” A masked-language-model, BERT’s pretraining task, sees the whole sentence including everything after the blank, and predicts only the missing word (“steak,” say) from that full bidirectional context. It never has to represent “what is this entire passage about, as a whole, in a form comparable to some other passage” — its unit of work is always one span, judged against its immediate surroundings.

A causal decoder’s pretraining task on the same sentence is different in kind, not just direction: given “The chef seasoned the,” predict the very next word, then given “The chef seasoned the steak,” predict the next one after that, and so on — a few hundred separate predictions across the sentence, each one conditioned on everything that came before it and nothing that comes after. Doing this well, across billions of documents of every genre, forces the model to maintain a running, continuously-updated summary of “what has this text been about so far” at every single position, because that running summary is exactly what makes the next word predictable. By construction, the decoder has spent its entire pretraining life practicing something adjacent to Chapter 1’s embedding readout — compress everything-so-far into a vector good enough to act on. A masked-LM never practiced that at all.

Then what is fine-tuning actually doing, if not teaching new semantics?

This is worth stating precisely, because “it doesn’t need to learn” is easy to overread as “fine-tuning is unnecessary.” It is not unnecessary — Chapter 1’s ablation table showed a real, if modest, gap between pooling choices, and skipping fine-tuning entirely leaves you with a vector space organized around next-token prediction accuracy, not around cosine-similarity-tracks-relevance. Those are different objectives even when they draw on the same underlying knowledge. A pretrained decoder’s last-token vector already encodes rich information about the text before it; nothing in pretraining ever pushed two unrelated documents’ vectors to point in different directions, or two related documents’ vectors to point in similar ones, because next-token prediction never compares two whole documents against each other at all.

Contrastive fine-tuning’s real job, then, is narrower than “teach the model what text means” — it is reorganize an already-rich representation space so that distance in it means semantic distance. That is a geometry-alignment problem, not a knowledge-acquisition problem, and geometry-alignment problems are exactly the kind of thing 0.6% of a model’s parameters (Chapter 3’s LoRA budget) and fewer than 1,000 gradient steps can plausibly solve. Teaching a model to understand language from scratch would need vastly more than that — and did, during pretraining, over trillions of tokens. This distinction is also why SGPT (Chapter 0) could already get partway to a working decoder-based embedder back in 2022 with a comparatively simple weighted-mean-pooling scheme and no contrastive pretraining stage of its own: the semantic substrate was already sitting there, waiting for a readout.

An analogy worth keeping, because it is the right shape even though no analogy is exact: imagine a huge library where every book has already been shelved by an assistant with excellent taste — cookbooks land roughly near other cookbooks, legal texts roughly near other legal texts — but nothing is labeled and the exact ordering within each rough cluster is loose. Pretraining built that library. Full retraining from scratch would mean tearing the whole building down and re-shelving everything from zero — expensive, and unnecessary, because the rough clustering is already good. Contrastive fine-tuning is the much smaller job of walking the existing shelves, tightening the loose ordering, and adding labels precise enough that a stranger (a nearest-neighbor search over cosine similarity) can find the right book fast. The books do not move to a different building. They get reorganized within the one that was already mostly right.

This also resolves a tension worth naming directly. Chapter 0 asked whether a decoder could become an embedder without heavy retraining, and named heavy retraining as one of three predicted failure modes if the idea were wrong. Chapter 3 already showed the fine-tuning is light — 0.6% of parameters, under 1,000 steps. This chapter is the mechanistic explanation for why that lightness was even possible: the job being done is reorganization of an existing representation, not construction of a new one from scratch.

Where the instruction text actually comes from at request time

Chapter 2’s taxonomy is not just a data-generation device — it directly supplies the instruction strings a deployed model uses at inference time. A serving layer built on top of this recipe typically keeps a small lookup table, one entry per task type, and formats the query with whichever instruction matches the caller’s intent:

python
TASK_INSTRUCTIONS = {
    "web_search":      "Given a web search query, retrieve relevant passages that answer the query",
    "qa":              "Given a question, retrieve passages that answer the question",
    "classification":  "Classify the sentiment expressed in the given review text",
    "sts":             "Retrieve semantically similar text to the given sentence",
    "bitext":          "Retrieve parallel sentences that are a translation of the given text",
}

def embed_query(llm, text, task):
    instruction = TASK_INSTRUCTIONS[task]                       # one sentence, swapped per request
    prompt = f"Instruct: {instruction}\nQuery: {text}"
    return llm.embed(prompt)                                    # Ch.1's last-token pooling reads this off

Notice this is precisely Chapter 1’s asymmetric instruction design paying off at serving time: the document side of the index was embedded once, with no instruction, and stays untouched no matter which row of TASK_INSTRUCTIONS a given request selects. Swapping which capability a live system offers is a one-line dictionary edit, not a re-index.

The instruction ablation says the same thing from a different angle

Recall the instruction template from Chapter 1: “Instruct: {task_definition}” before the query. The paper tested removing it, and separately tested replacing it with a terse hand-crafted tag in the style older embedders use, like “query:”.

Notice the two failure modes cost exactly the same — having no instruction and having a meaningless keyword tag are equally bad. That is the tell. A tag like query: is, to a decoder that learned language from reading language, just another rare token; it carries no more usable information than having nothing there at all. A full sentence — “retrieve passages that answer this question” — is something the model can actually read and use to condition how it processes the rest of the input, because reading and using sentences is precisely the skill next-token pretraining spent trillions of tokens building. Bidirectional encoders, trained on masked-span prediction rather than reading full sentences end-to-end, generally cannot exploit rich instructions the same way — which is why older embedders settled for terse tags in the first place. It was the right choice for their architecture. It is the wrong choice for this one.

Put a number on how much a single sentence is worth here:

For comparison, initializing from a weaker base decoder (LLaMA-2-7B instead of Mistral-7B, same everything else) costs 1.6 points — noticeably less than getting the instruction wrong. Which base decoder you start from matters. How you talk to it, at least in this comparison, matters more.

Why “negligible impact” is a better result than “small improvement”

It is worth sitting for a moment on exactly what kind of finding this chapter’s central ablation is. Most ablations in machine learning papers report a tradeoff: technique X helps quality but costs compute, or saves compute but costs quality. This one does not. Contrastive pretraining costs real time and real infrastructure — it is an entire extra training stage, on top of the LoRA fine-tune Chapter 3 already runs, over billions of weakly-supervised pairs. Finding that it has “negligible impact” for a decoder-only model, rather than merely a small positive one, means the stage can be dropped with essentially no quality tax at all. That combination — cheaper and not worse — is the rarer and more valuable kind of result, and it is the specific reason E5-Mistral’s recipe (Chapters 1–3) could skip straight from a frozen pretrained decoder to a light contrastive fine-tune, with no intermediate stage standing between them at all.

One paper table underlies four chapters

It is worth pausing to notice something structural before the quiz. Chapters 1 through 4 have each pulled one row — sometimes two — out of what is, in the paper, a single ablation table (Table 5), every row measured against the exact same 64.5 default configuration. Laid out together, the whole table tells one coherent story about where quality actually comes from:

Rank these deltas by size and a clear hierarchy falls out: the instruction ablation costs the most (−4.2), then the training-data ablation (−1.8) and the base-model ablation (−1.6) land in a comparable middle tier, and the pooling and LoRA-rank ablations are all under a single point. That ordering is the paper's own implicit answer to “what actually matters here”: how you talk to the model (instructions) and what you train it on (data, base model) dominate; the specific pooling strategy and adapter capacity are second-order corrections on top of a foundation that was mostly already right. This is the same hierarchy Chapter 3 stated in words — “data and instruction wording outweigh raw capacity” — now visible directly in one table's own sorted magnitudes, without needing to take that claim on faith.

Chapter 5: English Wins, Then Loses

Chapters 1 through 4 explained the mechanism. This chapter asks how good the result actually is — and where it stops being good.

The English leaderboard climb

On the MTEB benchmark (56 English datasets, the version current at publication), each generation of embedding model gained a few points over the last:

Read the early rows of this table as a short history of the field, not just a leaderboard. GloVe averages static, context-free word vectors — the pre-transformer baseline. SimCSE’s unsupervised variant trains purely by treating two different dropout masks applied to the same sentence as a positive pair, needing no labels at all; its supervised variant adds real NLI entailment pairs and gains 3.2 points from that labeled signal alone. Every row after that adds either more parameters, more (better) training data, or both. E5-Mistral’s jump is notable less for its size — the intermediate rows scaled up plenty too — and more for skipping an entire pipeline stage (contrastive pretraining, Chapter 4) that every one of those intermediate models still relied on.

It is also worth sizing the win against the model it beat. BGE-large-en-v1.5 is roughly 335 million parameters; E5-Mistral is roughly 7 billion — about 21 times larger — for a 2.4-point gain. On raw parameter efficiency, that is not a flattering trade, and it foreshadows the exact reversal Chapter 8 shows in full: a much smaller, differently-trained model beating a 7B decoder outright once the benchmark stops being English. Keep both numbers in mind at once. E5-Mistral’s size buys it real headroom in English; Chapter 8 will show that same size buying it comparatively little once the evaluation language changes.

That final row deserves a caveat the paper itself is careful to include. “Full data” means the synthetic corpus from Chapter 2 plus the thirteen public datasets from Chapter 3. Two leaner settings tell a more honest story about how much of that 66.6 is coming from the synthetic-data idea specifically, versus the labeled data mixed in alongside it:

Synthetic data alone does not beat the prior state of the art — it lands respectably close to it, using no human labels whatsoever. The new SOTA number specifically requires adding the labeled mixture back in. This is a more interesting and more honest result than “synthetic data alone is magic” would have been: the synthetic recipe closes most of the gap to a fully-labeled pipeline, and the labeled data still buys real, measurable extra quality on top.

Beating the commercial APIs, in the open

BEIR is worth naming precisely, since Chapter 0’s opening RAG scenario was a retrieval problem specifically, and BEIR is exactly the retrieval slice of MTEB — 15 of the 56 datasets, spanning search-style queries against corpora from Wikipedia to scientific abstracts to StackExchange threads, the same “find the relevant paragraph” job Chapter 0’s support-ticket assistant needed. Every other MTEB category (classification, clustering, pair-classification, reranking, STS, summarization) measures a genuinely different skill; BEIR is the one that maps most directly onto this session’s motivating use case. The paper’s own comparison table names this row group precisely as “commercial models and the model that tops the MTEB leaderboard” — three closed, pay-per-token APIs, plus one openly-available model that simply happened to be the strongest thing on the public leaderboard as of December 2023. On the BEIR retrieval subset, E5-Mistral edges past all four:

Put a relative number on the gap, computed the honest way rather than picking whichever comparison looks most dramatic. Against the closest of the four competitors by BEIR score, voyage-lite-01-instruct:

And against the weakest of the four, UAE-Large-V1 — remembering that this one is not a commercial API at all, just the strongest open model on the leaderboard at the time:

Neither number is a landslide, and that is worth sitting with rather than smoothing over. The paper’s own text calls this result a win “by a significant margin” — read that adjective with the same skepticism this session has been building toward every other headline claim. A 2–4% relative BEIR gap over the strongest available alternatives is a real win (E5-Mistral does lead all four rows), but it is a narrow one, not a rout. The much larger, more dramatic-sounding number in this story belongs somewhere else entirely: not to this leaderboard comparison, but to the architectural story Chapters 1–4 told about how that narrow English win was produced — no multi-stage weakly-supervised pretraining, no hand-curated dataset, one light LoRA fine-tune under a thousand gradient steps. The margin of the win is modest. The cost of earning it is what actually changed.

There is a practical dimension to this result beyond the leaderboard number, and it applies squarely to the three genuinely commercial rows. Each is a closed API: you send text over the network, pay per token, and have no visibility into how the model was trained, no ability to fine-tune it further for your domain, and a hard dependency on a third party’s uptime. E5-Mistral is fully open — weights, training recipe, and the synthetic-data generation code are all published. Beating those three commercial systems, even narrowly, while being self-hostable is a genuinely different kind of result than beating them while staying closed would have been; it changes what a team building on top of it can actually do next. UAE-Large-V1 does not carry that same practical argument, since it was already open too — its presence in this table is a reminder that “beat the commercial APIs” and “beat the entire field” are two different, easily conflated claims, and this particular table only cleanly supports the first one.

Where the leaner-data table's own gains actually come from, by hand

Chapter 2 already showed the “full data” MTEB average (66.6) against a “synthetic only” baseline (63.1). Revisit that same three-row table from a moment ago and do the subtraction step by step, because each gap answers a different question about where the final number actually comes from.

Notice the second gap is larger than the first, even though MS-MARCO is by far the largest single dataset in the mixture. That is not a contradiction — it means the marginal value of labeled data here comes mostly from task-shape diversity (HotpotQA’s multi-hop questions, FEVER’s claim-evidence pairs, MIRACL’s multilingual retrieval, and the rest of Chapter 3’s thirteen-dataset table), not from raw example count. One very large dataset of one shape is worth less, added last, than eleven smaller datasets covering shapes the synthetic recipe and MS-MARCO alone had not yet exercised.

Now the pivot: MIRACL, 8 languages, 2 stories

Every number above is (near-)entirely English. Before turning to MIRACL specifically, it is worth noting that the paper does not wait until an outside benchmark to spot this pattern — its own bitext-mining results (pairing sentences across two languages, Chapter 2’s “symmetric” taxonomy shape) already show the same shape the MIRACL table below is about to make precise. The paper’s own words, describing that result: E5-Mistral “excels in bitext mining for high-resource languages only.” That sentence is sitting in the paper that also reports the 66.6 MTEB headline number, three sections earlier than the multilingual results Chapter 5 is built around. The gap this chapter is about to quantify was not a surprise discovered by later critics — it is a pattern the authors themselves already named, in their own results section, before this session ever got to MMTEB.

The paper also evaluates on MIRACL, a human-annotated multilingual retrieval benchmark across 18 languages, and reports nDCG@10 for four higher-resource and four lower-resource languages (chosen by number of candidate documents in the corpus):

en/fr/es/ru = higher-resource; te/hi/bn/sw = lower-resource, selected by MIRACL corpus size — nDCG@10 on the dev set.

Averaging both groups by hand

Compute the average score across the four higher-resource languages for the two models that matter most here:

E5-Mistral leads by 58.1 − 56.9 ≈ +1.2 points on higher-resource languages. Now the same computation on the four lower-resource languages:

On lower-resource languages, E5-Mistral trails by 74.4 − 66.2 ≈ −8.2 points. The exact same model swings from +1.2 points ahead to 8.2 points behind — a roughly 9.4-point reversal — depending purely on which four languages you choose to average over.

Push one level deeper, because the same warning this paragraph is about to give applies recursively to the 4-language average itself. Look at E5-Mistral’s own scores within the low-resource group: 73.9 on Telugu, but only 52.1 on Hindi — a spread of 73.9 − 52.1 = 21.8 points between two languages that both got lumped into the single “low-resource” average of 66.2 above. A team serving Telugu-speaking users and a team serving Hindi-speaking users would draw opposite conclusions about this exact same model, and neither team would learn that from the 4-language average alone. Averages hide variance one level at a time; Chapter 6 is about to show this same pattern recur once more, at the scale of an entire benchmark instead of four languages.

What MIRACL actually measures, briefly

MIRACL is a human-annotated retrieval benchmark: for each of its 18 languages, native speakers wrote real search queries against a Wikipedia-derived corpus in that language, and other annotators judged which passages actually answer each query. The metric reported above, nDCG@10, rewards a system for ranking the truly relevant passages near the top of its first ten results, with diminishing credit the further down the ranking a relevant passage falls. The full derivation of nDCG — the discounting formula, why position 1 counts more than position 10, and how it is computed by hand on a toy ranking — belongs to the site’s Embedding Benchmarks gleam; the fact worth carrying forward here is simply that these are real human judgments in each language, not machine-translated proxies, which is exactly what makes the 8.2-point gap in the next paragraph a trustworthy signal rather than a translation artifact.

One purely illustrative pass, small enough to check by hand and not a number either paper reports, is enough to see why position matters and not just presence. Suppose a query has exactly one truly relevant document, and a model ranks it first, second, or third out of three candidates. nDCG@10 rewards relevance with a discount that shrinks as rank grows — specifically 1 ÷ log₂(rank + 1) for a binary-relevant hit, normalized against the best possible ordering:

Since a single relevant document at rank 1 is the best possible ordering here, that 1.00 is also the normalizing denominator (the “ideal DCG” the metric divides by), so ranking the relevant document first scores a perfect nDCG@10 of 1.00, second place scores 0.63, and third place scores 0.50 — despite the model finding the exact same relevant document in all three cases. This is the whole reason a retrieval metric cannot just count “did the model find the right answer somewhere in its top 10” the way a simpler recall metric would; MIRACL’s nDCG@10, and every nDCG@10 score in the tables below, is quietly answering the harder question of how close to the top the model put the right answer, not merely whether it appeared at all.

Hold this number. It is a small, honest, four-language demonstration of exactly the effect Chapter 6 is about to show you at the scale of an entire field’s worth of embedding models, evaluated across 250+ languages instead of four.

Chapter 6: Enter MMTEB

Chapter 5 ended on a four-language, one-dataset demonstration that a single “state of the art” number can hide an 8-point reversal depending on which languages you check. MTEB itself — the benchmark E5-Mistral topped at 66.6 in Chapter 5 — covers 56 datasets, almost entirely in English. A number from it is, strictly, a claim about English.

What MMTEB actually is

The Massive Multilingual Text Embedding Benchmark (Enevoldsen et al., February 2025) is a large-scale, community-built expansion of MTEB — coordinated openly on GitHub, with roughly two hundred named contributors credited through a transparent point system. Its headline scope: 500+ quality-controlled tasks across 250+ languages, the largest multilingual collection of embedding-evaluation tasks assembled to date, including several genuinely new task types — instruction-following retrieval, long-document retrieval, and code retrieval — that did not exist in the original MTEB at all.

Who built this, and how two hundred people avoided chaos

A benchmark this size could not be built by one lab. MMTEB was coordinated as an open, GitHub-managed collaboration: contributors proposed tasks, submitted implementations as pull requests, and were credited through a transparent point system specifically designed so that contribution — not seniority or institutional affiliation — determined authorship order, a structure borrowed from other large-scale collaborative NLP resource papers. The practical effect for this session: many of the low-resource-language tasks inside MMTEB exist specifically because a native speaker of that language chose to contribute one, not because a well-resourced lab decided the language was worth covering. That is a structurally different way of closing Chapter 0’s coverage problem than either the old curated-dataset recipe or E5-Mistral’s own synthetic-data recipe — neither of which requires the language community itself to be represented among the people building the resource.

The quality gate: catching broken tasks before they ship

A benchmark with 500+ community-submitted tasks needs an automatic smell test, because manual review of every submission does not scale. MMTEB’s: run every candidate task through two small baseline multilingual models — multilingual-e5-small and MiniLM-L12 — and flag the task for manual review if either model scores within 2% of random chance, scores near-perfect, or if the two models score nearly identically to each other. Any of those three patterns usually means the task is broken (a bug in the implementation, mislabeled data, or a task so trivial or so impossible that no model’s score is informative) rather than that both models happen to be equally good or bad. A task can still be kept after review — sometimes a near-random score really is just an inherently hard task — but nothing ships without the check running first.

Walk through a concrete failure this check is built to catch. Suppose a contributor submits a new classification task, but a bug in their label-parsing code accidentally shuffles the label column relative to the text column before evaluation runs. Every model, however good, is now being scored against essentially random labels — so both multilingual-e5-small and MiniLM-L12 land within 2% of chance-level accuracy, despite being reasonably strong models on every other task in the pool. That pattern trips the first quality-gate rule immediately, flags the task, and a reviewer opens the implementation and finds the shuffled column before a single external user ever sees a misleading “every model fails this task” result on the public leaderboard. Contrast that with a genuinely hard task — say, disambiguating two closely related dialects with almost no lexical differences — where both baselines might also land near chance, but for the honest reason that the task really is that difficult. The automatic check cannot tell these two cases apart by itself; that is exactly why it routes both to a human reviewer rather than silently keeping or silently discarding either one.

What existed before MMTEB unified it

MMTEB did not invent multilingual embedding evaluation from nothing — it consolidates a landscape that was previously scattered across separate, incompatible efforts. Scandinavian, Chinese, Polish, and French each already had their own standalone embedding benchmark, built by different groups, at different scales, with different conventions for what counted as a task. English had MTEB. Retrieval specifically had BEIR. None of them shared infrastructure, and none of them could be compared against each other on equal footing, because a model that topped the French benchmark and a model that topped BEIR were never run through the same evaluation code with the same task-quality checks. MMTEB folds all of these into one codebase and one submission process, which is a large part of why 500+ tasks were achievable at all — much of the raw task content already existed, scattered; what MMTEB added was the shared plumbing, the quality gate above, and the downsampling machinery Chapter 7 covers, that make hundreds of previously incompatible efforts runnable and comparable side by side.

The headline finding, and where you have already seen it

Across MMTEB’s flagship multilingual benchmarks, the single best-performing publicly available model overall is multilingual-e5-large-instruct — a model with only 560 million parameters — beating both GritLM-7B and e5-mistral-7b-instruct, the two roughly 7-billion-parameter, Mistral-7B-based models that dominate the English-only leaderboard.

This is Chapter 5’s 8.2-point MIRACL gap, replayed at a completely different scale — not four low-resource languages inside one dataset, but 132–343 tasks spanning 250+ languages. Same underlying cause (breadth of multilingual pretraining beats sheer parameter count, once you leave the languages a model was disproportionately trained on), vastly larger body of evidence. Chapter 8 puts you in front of the actual numbers behind this claim.

How MMTEB turns 500+ tasks into a usable benchmark

Five hundred tasks is too many to run on every model release. MMTEB builds several smaller, targeted benchmarks from the full pool through a three-stage funnel: start from an Initial Scope (everything plausibly relevant to the benchmark’s goal), cut to a Refined Scope (remove machine-translated data, tasks with unclear licensing, overly narrow domains), then apply Task Selection to remove the most redundant remaining tasks.

Look at where MTEB(Multilingual)’s reduction actually happens. Refining 500+ down to 343 (removing machine-translated data, unclear licensing, overly narrow domains) removes roughly:

A small honesty note, in the same spirit as this session's other contamination and leakage checks: MMTEB's own headline table (Table 1) reports the Initial Scope as “>500,” which is what the 31% figure above uses. The paper's own body text describing that same step (Section 2.4) instead states the precise count as 550 tasks — which would put the reduction at (550−343)÷550 ≈ 37.6% instead. This is an inconsistency inside the paper itself, not something introduced here; both numbers are directly quoted from the source, and the qualitative point — a substantial minority of tasks removed on quality grounds, a much larger majority removed for redundancy next — holds either way.

But Task Selection — the redundancy-based pruning from the previous section — removes far more:

So most of the funnel’s shrinkage is not about bad data at all — it is the deliberate, statistically justified removal of tasks that were not adding independent signal. That distinction matters if you ever hear “MMTEB only kept a quarter of its tasks” and wonder whether that means a quarter of the tasks were unreliable. It does not; it means three-quarters of them were redundant with the quarter that remained.

Run the same two-stage split on MTEB(Europe)’s row from the table above, and the proportions shift in a way worth noticing:

Both cuts are proportionally larger than MTEB(Multilingual)’s 31% and 62%. That is not noise — European languages are exactly the languages most likely to already have several independently-built, overlapping embedding benchmarks (Chapter 6’s “what existed before MMTEB unified it” point, a few paragraphs up: Scandinavian, Polish, French each had their own). More existing infrastructure for a language region means more machine-translated or duplicate-effort datasets competing for the same slot, which shows up here as a proportionally bigger quality cut, and then more genuinely similar tasks once past that gate, which shows up as a proportionally bigger redundancy cut too. The funnel is not applying one universal ratio to every region; it is responding to how much overlapping prior work already existed for that region specifically.

That last row matters directly to this session. MTEB(eng, v1) is the exact benchmark E5-Mistral topped at 66.6 in Chapter 5. MTEB(eng, v2) is a deliberate, zero-shot rebuild that excludes MS-MARCO and Natural Questions — specifically because those two datasets are commonly used as training data. Recall Chapter 3: E5-Mistral itself trains on MS-MARCO. A model that fine-tuned on MS-MARCO has, in a real sense, already seen something close to the MTEB(eng, v1) test distribution during training. MTEB(eng, v2) exists to close that specific leak.

Task selection, mechanically: remove whatever is most predictable

Rather than cutting tasks arbitrarily, MMTEB frames “which tasks are redundant” as a regression problem. Using backward selection: hold out one task, fit a linear regression estimator on every model’s scores across all the other tasks, and use it to predict the held-out task’s score. Repeat for every task, then permanently remove whichever task was easiest to predict from the rest — i.e., the task that added the least new information, because its score was already implied by the others. Continue removing the most-predictable remaining task until the Spearman correlation between predicted and observed scores for the most-predictable task drops below a threshold (0.8 for MTEB(Multilingual), a stricter 0.9 for the smaller, more mature MTEB(eng, v2)).

Walk through why this is smarter than removing tasks at random with a small toy case. Suppose two clustering tasks, both built from news-article headlines in closely related domains, happen to rank all twelve candidate models in nearly the same order — call them task A and task B. Fit a regression predicting task B’s scores from every other task’s scores (task A included): because A and B are so similar, that regression predicts B almost perfectly, giving a high Spearman correlation between predicted and observed scores. B gets removed — not because it is a bad task, but because keeping A already tells you almost everything B would have told you. Now suppose task C is a low-resource bitext-mining task whose model ranking looks nothing like any retrieval or clustering task in the pool. No combination of the other tasks predicts C’s scores well; its Spearman correlation stays low, and it survives every round of pruning. The algorithm is, in effect, automatically identifying which tasks are measuring something genuinely different from what is already covered — exactly the kind of task a hand-curated 132-task shortlist would want to protect, and exactly the kind random subsampling could easily discard by accident.

Pure statistical predictability, left unchecked, would happily prune every task from an entire low-resource language if that language’s few tasks all happened to correlate strongly with a handful of others — technically redundant by the numbers, but the last remaining measurement of anything at all for that language. MMTEB guards against this with two explicit constraints layered on top of the backward-selection algorithm, not left to fall out of the statistics alone. First, the algorithm is never allowed to remove a task if doing so would eliminate a language from its task category entirely — language coverage inside each category is protected even when a task looks statistically redundant. Second, a task is not removed if the mean-squared error between its predicted and observed scores exceeds 0.5 standard deviations, specifically to avoid a subtler failure mode: overindexing on tasks that happen to be easy to predict rather than genuinely redundant. A task can look predictable simply because most models cluster near the same score on it, which is a sign of low task difficulty, not low information content. Both guardrails exist because a purely statistical redundancy filter, run without constraints, would optimize for the wrong thing — minimizing correlation among the tasks that remain, rather than maximizing the genuine diversity of languages and difficulty levels the benchmark is trying to protect.

One more step closes the loop after the statistical pruning finishes: “the selected tasks were reviewed, if possible, by contributors who spoke the target language,” and the selection criteria could still be revised or individual tasks manually swapped for higher-quality alternatives based on that human pass. Backward selection decides what is statistically redundant; it does not decide what is a good task in the target language. Those are different questions, and MMTEB keeps a human answering the second one even after the algorithm has finished answering the first.

The model roster this benchmark actually runs

MMTEB evaluates a deliberately spread-out roster so the effect of scale and instruction-tuning can be isolated: LaBSE (trained on paraphrase corpora specifically for bitext mining), MPNet and MiniLM in both English-only and multilingual variants (small, efficient encoders that predate the LLM-embedder era), the multilingual-E5 family across four sizes (small, base, large, and large-instruct — the same lineage as Chapter 5’s MIRACL comparison, now instruction-tuned), and two Mistral-7B-based models representing the LLM-embedder approach this whole session has been building: GritLM-7B and e5-mistral-7b-instruct. Spanning encoder scale from roughly 20 million parameters up to 7 billion, and pretraining approach from narrowly bilingual to hundred-language-explicit to predominantly-English, is exactly what makes Chapter 8’s comparison meaningful — the roster was built to expose scale-versus-multilinguality tradeoffs, not just to crown a single winner.

Chapter 7: The Cost of Evaluating Everywhere

Five hundred tasks across 250+ languages is not free to run. The paper itself notes that MTEB(eng, v1) already took up to two days on a single A100 GPU for moderately sized LLMs — and MMTEB is roughly nine times the task count, with languages layered on top. If evaluation cost scaled linearly with that growth, most of the low-resource-language communities MMTEB was specifically built to include — who often have less compute available, not more — would be locked out of using their own benchmark. Three concrete optimizations fix this, each with real, checkable numbers.

Put a rough back-of-envelope shape on the problem before the fixes, purely illustrative and not a number the paper computes this way: two days per model on MTEB(eng, v1) alone, scaled naively by roughly nine times the task count for the full MMTEB pool, lands somewhere around eighteen days of A100 time per model — and that estimate is still only the English-adjacent slice, before any of the 250+ languages are added. Whatever the precise multiplier, the direction is unambiguous: unoptimized, a benchmark this size is not something most academic labs or low-resource-language communities could run even once, let alone every time a new model is released.

This is not an abstract fairness concern; it is the same accessibility argument from Chapter 6 made concrete in hardware terms. An A100 or H100 GPU is expensive cloud infrastructure, typically rented by the hour, and largely out of reach for an individual contributor working on their own language’s bitext-mining task in their spare time. A benchmark whose evaluation cost assumes that hardware quietly re-creates exactly the coverage problem Chapter 0 opened with — except now the gatekeeper is compute budget instead of curated-dataset access. The three optimizations below exist specifically so that the person best positioned to judge whether a low-resource language task is any good — a speaker of that language — can also afford to run it.

The scale of the problem is visible even one benchmark earlier, inside E5-Mistral’s own paper. Section 4.2 reports that evaluating one trained model on the original 56-dataset MTEB(eng, v1) — not training it, just running it through the benchmark — takes about 3 days on 8 V100 GPUs, because so much of the cost is encoding a large number of documents for the retrieval datasets. That is the cost of checking one model against one, mostly-English benchmark. MMTEB multiplies both the task count and the language count on top of that baseline, which is exactly why Chapter 6 and this chapter both treat evaluation cost as a first-class engineering problem rather than an afterthought.

1. Clustering: encode once, bootstrap ten times

Clustering evaluation (k-means, scored with the v-measure metric) is normally repeated across several sampled sets of documents, encoding a fresh sample every time. MMTEB instead encodes a single 4% subsample of the corpus once, then bootstraps 10 different cluster-sets from that same cached subsample, drawing without replacement within it, instead of re-encoding ten fresh full-size samples. Reported result: a 16.11× average speedup across tasks, with relative model rankings preserved (average Spearman correlation 0.96 against the un-optimized version).

Check the ceiling this could theoretically reach. Ten fresh full-size samples cost 10 × 100% = 1000% of a corpus’s worth of encoding. Encoding 4% once and reusing it ten times costs just 4% total:

The measured 16.11× is far below that ceiling — a healthy reminder that a clean algorithmic argument rarely survives contact with a real, heterogeneous benchmark suite untouched. Not every clustering task is dominated by encoding cost the same way, and the reported figure averages across many tasks with different corpus sizes, not the single idealized case used to compute the ceiling.

2. Retrieval: keep only what a strong system would ever surface

Some retrieval corpora contain millions of documents. MMTEB uses TREC pooling: run the corpus through three strong systems — BM25 (lexical coverage), multilingual-e5-large (a strong BERT-scale multilingual baseline), and e5-mistral-7b-instruct (the strongest instruction-based model available) — keep each system’s top 250 ranked documents per query, and merge the three pools into one smaller representative candidate set. Anything none of the three systems ranked near the top is dropped, on the reasoning that a document no strong system surfaces is extremely unlikely to matter for judging how well a new model ranks the genuinely competitive candidates. Queries are additionally capped at 1,000 per dataset for datasets that had more.

The paper reports this reduces “the largest datasets from over 5 million documents to a maximum of 250,000.” That ceiling is not a separately-tuned number — it falls straight out of the two parameters just stated:

And the reduction on the largest datasets:

The paper checks two separate properties before accepting this shortcut, not just one. The first is the same rank-preservation check every other optimization in this chapter uses. The second is subtler: it also checks that absolute scores stay reasonably close to their un-pooled values, not just their relative order. This matters because researchers and practitioners often compare a new model’s score against a number reported in an older paper, months or years apart — a pooling method that preserved ranking perfectly but shifted every model’s absolute score by twenty points would still quietly break that kind of longitudinal comparison, even though it would pass a pure rank-correlation check.

TREC pooling itself is not a new invention — it comes from decades of information-retrieval evaluation campaigns that faced the identical problem long before embedding models existed: no one can judge relevance for every document in a huge corpus by hand, so judge only the documents any strong system actually surfaced, and assume the rest are irrelevant. MMTEB adapts the same trick, using models instead of human judges as the pooling systems. Be honest about what this assumption costs: if a genuinely relevant document happens to rank outside the top 250 for all three pooling systems, it is permanently treated as irrelevant for every model evaluated against this reduced benchmark afterward — including a hypothetical future model with a genuinely different retrieval strategy that might have found it. This is a real, acknowledged limitation of pooling-based evaluation in general, not specific to MMTEB, and it is part of why the paper checks Spearman correlation against the un-pooled version rather than simply asserting the shortcut is safe.

3. Bitext mining: cache the sentence, not the language pair

Bitext-mining datasets like Flores reuse the exact same sentences across many language-pair tasks — the same English sentence appears in the English–Hindi pair, the English–Bosnian pair, and dozens of others. Naively evaluating N language pairs that all share an English side re-embeds those identical English sentences N separate times, so cost scales quadratically with the number of languages covered. MMTEB caches each unique sentence’s embedding and reuses it across every pair it appears in, dropping the scaling to linear. For the English side of Flores specifically: MTEB(eng, v1)’s uncached approach needed 410,000 document embeddings; MMTEB’s cached approach needs just 1,012.

Concept → realization: the cache is one dictionary

The idea “cache the sentence, not the language pair” is simple enough to sound like it should not need code, but the actual implementation detail worth seeing is what the cache is keyed on. It is not keyed on (language, task) — it is keyed on the sentence’s own text, so any two tasks that happen to share a sentence automatically share the embedding, with no coordination between the tasks required.

python
embedding_cache = {}                            # dict[str, np.ndarray], keyed on raw sentence text

def embed_cached(sentence, model):
    if sentence not in embedding_cache:
        embedding_cache[sentence] = model.embed(sentence)   # only computed once, ever
    return embedding_cache[sentence]

# English-Hindi task and English-Bosnian task both call this on the SAME English sentence --
# the second call is a dict lookup, not a forward pass through the model
for lang_pair in ["en-hi", "en-bs", "en-ta", ...]:            # dozens of pairs, one shared English side
    en_vecs = [embed_cached(s, model) for s in flores_english_sentences]
    other_vecs = [model.embed(s) for s in flores_other_side(lang_pair)]  # always fresh -- not shared

Nothing about this requires a distributed cache, a database, or coordination across evaluation runs — a plain in-memory dictionary is enough, because the entire benchmark run happens in one process. The optimization is not clever engineering; it is noticing that “embed this English sentence” is the same function call with the same argument, dozens of times over, and a dictionary is the textbook fix for exactly that pattern.

Why the headline language count has two different numbers

This caching trick also explains something that otherwise looks like a typo the first time you read it across two different parts of the paper: MMTEB is variously described as covering “250+ languages” and, in one place, as extending “the number of languages to over 1,000 (250 excluding bitext-mining tasks).” Both numbers are correct simultaneously, and the gap between them is exactly the quadratic-versus-linear distinction this section just walked through. Every non-bitext task (retrieval, classification, clustering, STS) touches one language per dataset, so those add up to roughly 250 distinct languages. Every bitext-mining task touches a pair of languages, and Flores-style datasets cover a large grid of language pairs — enough pairs, counted individually, to push the total past 1,000 language appearances even though the count of genuinely distinct languages involved stays much closer to 250. “250+ languages” is the honest headline number for language diversity; “1,000+” is a real number too, but it is counting language-pair slots in the bitext tasks, not additional distinct languages the benchmark somehow found. Reading a benchmark’s own “N languages” claim carefully enough to ask which of these two things N is actually counting is the same discipline Chapter 9 applies to MMTEB’s other headline numbers.

A fourth lever, aimed at contributors rather than at the benchmark itself

The three optimizations above all operate on tasks MMTEB already has, after the fact. The paper describes a fourth, complementary strategy that operates earlier in the pipeline: encouraging smaller dataset submissions in the first place. Rather than accepting a contributor’s full-size dataset and downsampling it centrally, MMTEB asks contributors to stratify-split and downsample before submitting, using a stratified split across the task’s target categories so the smaller version still separates strong models from weak ones reliably. The paper validates this the same way it validates every other shortcut in this chapter: compare model scores before and after downsampling, and confirm the ranking survives.

This is a different kind of optimization from the three above, and worth distinguishing precisely. Clustering, retrieval-pooling, and bitext-caching are all algorithmic tricks applied uniformly, after the fact, by MMTEB’s own evaluation code — a contributor never has to think about them. Encouraging smaller submissions instead changes the incentive at the point of contribution, so that many of the 500+ tasks are never oversized to begin with. A benchmark that only optimized the first three would still face the same growing cost every time a new contributor added a full-size dataset; a benchmark that also nudges submission size stays affordable to extend, not just affordable to run once.

Net effect: a zero-shot English benchmark at 2% of the cost

Combining these optimizations, MTEB(eng, v2) “maintains a similar ranking order as the full-scale version but only requires 2% of the original documents,” per the paper’s own abstract. Empirically, it correlates with the original MTEB(eng, v1) at Spearman 0.90 (p < 0.0001) and Pearson 0.96 (p < 0.0001) despite dropping from 56 to 41 tasks and switching to zero-shot-only evaluation. Concretely, running the full v2 suite on an H100 takes 3.11 hours for GritLM-7B and just 0.81 hours for the much smaller all-MiniLM-L12.

Step back and notice what these three optimizations have in common, because it is a genuinely reusable pattern past this one benchmark: in every case, MMTEB found a way to ask fewer questions of the data (fewer documents encoded, fewer queries pooled, fewer duplicate embeddings computed) while explicitly verifying, via Spearman correlation, that the answer to the one question that actually matters — which model is better — did not change. Cheaper evaluation without that verification step is just noise; cheaper evaluation with it is real engineering.

The cost being cut here is not only measured in dollars and hours. MMTEB explicitly instruments every task with emissions tracking (the codecarbon library), measuring kilograms of CO₂-equivalent per task and reporting a carbon-footprint estimate alongside the benchmark’s other results. This is not decoration — a benchmark that a community intends to run repeatedly, across every future model release, compounds its own compute footprint over time in a way a single paper’s one-off experiment does not. Making each optimization’s savings measurable in Spearman correlation and visible in a carbon number is the same accountability instinct applied to a cost that rarely makes it into a results table at all. Chapter 8 now puts the resulting benchmarks to work.

Step back one more level and this whole chapter is really Chapter 0’s coverage problem, showing up a third time in a third disguise. Chapter 0 opened with curated-dataset coverage: a new language costs a team of human annotators weeks of work. Chapter 2 solved that with synthetic data generation. Chapter 6 opened with task-coverage at benchmark-construction time: 500+ community-submitted tasks are too many to review by hand, so MMTEB built an automatic quality gate. This chapter is the same problem one layer further downstream — evaluation-time coverage: even a well-built benchmark is worthless to a low-resource-language community if running it costs eighteen days of A100 time they do not have. Three algorithmic tricks and one submission-guideline change later, the benchmark that was built for low-resource-language communities is also affordable by them. A benchmark that only the well-resourced can afford to run is, in its own quiet way, exactly the coverage problem this whole session keeps circling back to — just moved from “who can afford to build the training data” to “who can afford to check the result.”

Chapter 8: The Leaderboard Flip (showcase)

Time to see Chapter 5’s 8.2-point MIRACL swing and Chapter 6’s “a 560-million-parameter model beats two 7-billion-parameter models” claim at once, using MMTEB’s own results table across all three of its flagship multilingual benchmarks. Every number in the simulation below is the paper’s reported Borda-count rank and average score for three representative models: multilingual-e5-large-instruct (560M parameters, built on XLM-R), and two Mistral-7B-based models — GritLM-7B and e5-mistral-7b-instruct (the instruction-tuned successor to the model this whole session has been building, now evaluated far outside English).

One connection worth making before the numbers: GritLM is not simply “another 7B decoder embedder like E5-Mistral.” It goes a step further than the interface Chapter 0 sketched — rather than a single model that can be called either as an embedder or a generator with the same weights, GritLM is trained so that one instruction prefix switches it into “represent” mode and another switches it into “generate” mode, natively, without needing separate deployment code paths at all. It is, in a real sense, Chapter 0’s opening question taken to its logical conclusion. That it still loses badly on MTEB(Indic) despite that unification is itself informative: unifying the two jobs into one model does nothing to fix an imbalanced pretraining diet. The two problems — architecture unification and multilingual balance — are independent, and solving one does not solve the other.

Before clicking anything: make a prediction. Based on Chapter 5’s MIRACL result and Chapter 6’s headline finding alone, guess the rank order of the three models on each of the three benchmarks below, then check. Getting it right on the first two benchmarks and wrong on the third is, in itself, useful data about which part of the argument you have actually internalized versus which part you are still taking on faith.

Reading the three panels

On MTEB(Multilingual) — all 132 selected tasks, 250+ languages pooled together — multilingual-e5-large-instruct wins outright: Borda count 1375, average score 63.2, versus GritLM-7B at 1258/60.9 and e5-mistral-7b-instruct at 1233/60.3.

On MTEB(Europe) — 74 tasks, concentrated in higher-resource European languages — the order shuffles. GritLM-7B narrowly takes rank 1 (Borda 757, average 63.0); multilingual-e5-large-instruct drops to rank 2 (732, 62.2); e5-mistral-7b-instruct is rank 3 (725, 61.7). All three sit within about 1.5 average-score points of each other. In higher-resource territory, the 7B Mistral-based models are genuinely competitive — sometimes the best available.

On MTEB(Indic) — 23 tasks, lower-resource languages — the flip becomes a collapse. multilingual-e5-large-instruct dominates (Borda 209, average 70.2). GritLM-7B falls to rank 5 (151, 60.2). e5-mistral-7b-instruct falls to rank 6 (144, 60.0). Both Mistral-based 7-billion-parameter models are now beaten not just by the 560M multilingual model, but by several other, smaller models in the full roster too.

The paper’s own explanation matches the mechanism from Chapter 5: this reversal is “especially pronounced for mid-to-low resource languages” and traces back to pre-training composition — Mistral-7B is predominantly English-pretrained, while XLM-R (the backbone under multilingual-e5-large-instruct) explicitly targets 100 languages during its own pretraining. GritLM-7B additionally does best specifically on retrieval and on the highest-resource European languages, partly because it supports a longer maximum sequence length than the others in this comparison.

The full roster, not just the three headline models

The canvas focuses on three models to stay legible, but MMTEB’s MTEB(Multilingual) table reports twelve, and the full spread is worth seeing at least once. It runs from a 41.4 average for an English-only MiniLM model applied outside its home language, up to the 63.2 that wins:

That is genuinely the full twelve-model roster this time, in the paper’s own rank order. Two rows worth a second look: multilingual-mpnet-base (rank 6, 52.0) sits behind the smaller multilingual-e5-small (rank 7, 55.5) despite being a larger, later-generation encoder — a reminder that within a single family lineage, training recipe and fine-tuning data can matter more than raw parameter count, the same lesson Chapter 3’s LoRA-rank ablation taught inside a single model. And multilingual-MiniLM-L12 (rank 9, 48.8), the smallest multilingual model MMTEB tests, still clears every English-only model in the roster (ranks 10–12) by 6–7 points — touching 100+ languages during pretraining is worth more here than being English-only and larger.

Two patterns pop out of the full table that the three-model comparison alone hides. First, within the multilingual-E5 family itself, instruction-tuning is worth roughly 4–5 points at every size — large-instruct (63.2) beats plain large (58.6) by nearly the same margin the base-to-large size jump is worth, confirming Chapter 6’s claim that instruction-tuning helps consistently, independent of parameter count. Second, the English-only models (all-mpnet-base, all-MiniLM) sit more than 20 points below the multilingual winner — not because they are bad models, but because a model with zero non-English pretraining has, by construction, nothing to offer once the benchmark stops being English. That comparison is the clearest possible illustration of Chapter 0’s closed-menu problem, restated one more time at the model-architecture level instead of the dataset-curation level.

Before jumping to Indic, look at the middle benchmark’s full roster too — the one where the order actually favors the Mistral-based models, so it is worth confirming the full picture there is not being cherry-picked down to three rows either:

The top three really are that close — 63.0, 62.2, 61.7, a spread of 1.3 points across the entire top of the leaderboard — which is exactly what “genuinely competitive” looks like as opposed to “narrowly leading” or “collapsing.” Below rank 3, though, the same pattern from MTEB(Multilingual) reappears: every multilingual model beats every English-only model, and English-only models still sit 7–10 points below the multilingual field’s bottom performer even in European languages — a reminder that “higher-resource” does not mean “English,” it means “more labeled data exists,” and a model with zero non-English pretraining still has nothing to offer outside its one language regardless of how well-resourced the target language is.

The MTEB(Indic) full roster makes the collapse even starker, because it shows exactly which models overtake the two Mistral-based ones, not just that something does:

Every single member of the multilingual-E5 family — including its smallest variant, at a fraction of GritLM-7B and e5-mistral-7b-instruct’s parameter count — outranks both 7B Mistral-based models on Indic languages. This is not one lucky model beating two unlucky ones; it is an entire architecture lineage (XLM-R-based, explicitly pretrained across 100 languages) outperforming an entire other lineage (Mistral-based, predominantly English-pretrained) as a group, on this specific language family. That is a much stronger claim than “the winner happened to be smaller,” and it is the claim the paper’s own Section 4 discussion (Chapter 8’s next section) actually makes.

Notice something else in this table that the Borda-rank ordering alone does not make obvious: multilingual-e5-base (rank 3, 64.6) and multilingual-e5-small (rank 4, 64.7) are nearly tied on raw average — the smaller model's average is fractionally higher — yet the base model still outranks it by Borda count. That is not an error in this table; it is exactly the distinction the callout above just made concrete. Borda count is computed from each model's rank on every one of the 23 individual tasks and then summed, so a model that is consistently a little ahead across most tasks can out-Borda a model with a marginally higher raw average built from a few standout tasks and a few weak ones. The two aggregation methods usually agree on the big picture — they agree completely here about which two families dominate — but they are not the same statistic, and a benchmark table that reports both, side by side, is doing you a favor by letting you see where they diverge.

Instruction-tuning's payoff, measured across all three benchmarks at once

Chapter 6 already flagged that instruction-tuning is worth roughly 4–5 points on MTEB(Multilingual), comparing multilingual-e5-large-instruct against its non-instructed sibling multilingual-e5-large. With all three regional tables now on the page, that comparison can be run three times instead of once — a genuine test of whether the effect is a real, general property of instruction-tuning, or a fluke of one particular benchmark.

Three benchmarks, three different language mixes, three different task compositions — and the same instruction-tuning step is worth somewhere between 3.7 and 4.6 points every single time, with no sign of vanishing or reversing on any of them. Compare that consistency to the leaderboard-flip pattern the rest of this chapter is built around, where the same architectural choice (Mistral-7B’s scale and English-heavy pretraining) swings from a narrow win to a rank-6 collapse depending on the benchmark. Instruction-tuning does not behave like that here; it is a broadly transferable win, not a benchmark-specific artifact. That distinction — some design choices generalize across languages, others do not — is exactly the kind of thing a single leaderboard number can never show you, and exactly why this chapter insists on three tables instead of one.

The trend inside the trend: speaker count, not just region

The paper does not stop at three regional benchmarks — it also plots rank against each language’s number of native speakers directly, across MTEB(Europe) and MTEB(Indic) together. The pattern it finds: the two Mistral-based models’ performance “steadily decreases and becomes more volatile” as the number of native speakers of a language drops, with the effect “especially pronounced” below roughly 300 million speakers — a threshold that excludes even some fairly major world languages, not only obscure ones. multilingual-e5-large-instruct’s ranking, by contrast, stays comparatively flat across that same speaker-count range. This is Chapter 2’s foreshadowing landing exactly where it was aimed: 85% of E5-Mistral’s synthetic training examples went to roughly the top 18 higher-resource languages (Chapter 2), and the model’s measured skill degrades in almost the same shape — steadily, and specifically below a speaker-count line — that the training-data imbalance predicts.

What a product team does with this, concretely

Turn the simulation into a decision. A team serving primarily English and Western European users, where GritLM-7B or e5-mistral-7b-instruct win or are statistically tied, gets to make the choice on other grounds — latency, context length (GritLM’s longer maximum sequence length is a real advantage for long-document use cases), or whether the generation-and-embedding unification described above simplifies their serving stack. A team serving primarily Indic-language users has no such luxury: the data says multilingual-e5-large-instruct is not a close second, it is a clear first, and it is also the cheaper model to run at 560M parameters. A team serving a genuinely global, unpredictable mix of languages should weight toward MTEB(Multilingual)’s pooled ranking specifically because it does not assume any one region dominates — which is exactly the benchmark where multilingual-e5-large-instruct also wins outright. In none of these three cases is “pick the model with the most parameters” the right rule.

Chapter 9: What the Benchmark Hides

Chapters 0–4 built the mechanism: causal decoders can become embedders using last-token pooling, a two-step synthetic data recipe, and a light contrastive fine-tune that mostly just exposes semantics the model already learned during pretraining. Chapters 5–8 tested that mechanism at increasing scale — one English leaderboard, then eight MIRACL languages, then MMTEB’s 250+. Each step revealed more about where the model’s real skill boundary actually sits. This final chapter closes with what even MMTEB, the largest and most carefully built multilingual benchmark to date, admits it still does not fully solve.

MMTEB's own stated limitations

The paper’s conclusion lists three limitations directly, in its own words, and they are worth reading with the same care you would give the headline results.

1. English leakage. MMTEB filters out machine-translated datasets, but it permits human-translated ones. This creates real edge cases: the paper names SIB200ClusteringS2S, a task where labels attached to English samples get carried over onto their human translations. A model that is secretly better at matching English-trained representations, rather than genuinely understanding the target language, could still score well on a task like this — risking, in the paper’s words, “inadvertently encourag[ing] model developers to favor English or translated content” in their own training data, precisely to game this style of task.

2. Credit assignment. The community point-system used to fairly attribute authorship across roughly two hundred contributors awards equal points per dataset submission, regardless of how much work it actually took. A dataset that needed heavy HTML parsing, reformulation, and multiple review rounds earns the same credit as one that was simply already available. This is not a flaw in the benchmark’s scores — it is a governance honesty note, and the paper reports it anyway.

3. Language representation. Despite the “250+ languages, 500+ tasks” headline, the actual distribution is heavily skewed toward high-resource languages. English alone accounts for roughly 290 of the task-language appearances in the full collection — enough that the paper removes English from its own Figure 6 just to keep the chart of the next hundred most-covered languages readable at all. Low-resource languages that are represented tend to be concentrated in only a couple of task categories (bitext-mining, classification) rather than spread evenly across retrieval, clustering, and similarity tasks the way high-resource languages are.

That number is not a criticism to dismiss MMTEB with — it is smaller than MTEB(eng, v1)’s 100%, and the whole point of building MTEB(Multilingual), MTEB(Europe), and MTEB(Indic) as separate benchmarks (Chapter 6) is precisely so a deployment team can choose a slice where English’s dominance does not drown out the signal they actually need. It is, instead, a reminder that “multilingual” is a spectrum, not a binary — MMTEB moved the field meaningfully along that spectrum without claiming to have reached a perfectly balanced endpoint, and says so itself.

E5-Mistral's own honesty check, at the single-model level

This same discipline shows up one level down, inside the E5-Mistral paper itself. Appendix B runs a string-match contamination analysis between the paper’s own training set and every MTEB test set, disregarding case and spacing differences, and sorts every overlap it finds into three honesty buckets.

Concept → realization: what a string-match contamination check actually does

“String-match contamination analysis” sounds like it needs sophisticated tooling. Mechanically, it is close to the simplest thing that could possibly work: normalize whitespace and case, then check whether a test-set string appears anywhere inside the training corpus at all.

python
def normalize(text):
    return " ".join(text.lower().split())        # collapse whitespace, ignore case

def contamination_report(test_items, train_corpus):
    train_set = {normalize(t) for t in train_corpus}    # O(1) membership lookup, built once
    hits = []
    for item in test_items:
        if normalize(item.text) in train_set:
            hits.append(item)                       # a literal overlap -- now needs a HUMAN to classify it
    return hits                                     # list of overlaps, NOT yet a "contamination" verdict

Notice what the function does not do: it does not decide whether a hit counts as contamination. That judgment — is this a generic phrase, a small disclosed overlap, or shared source material — is exactly the three-bucket classification the table above lays out, and it happens by a person reading each flagged item, not by more code. The mechanical half of an honesty check (find every literal overlap) is genuinely simple and automatable. The interpretive half (decide what each overlap actually means) is not, and papers that skip straight from “ran a contamination check” to “is clean” without describing that second, human step are skipping the part that actually does the honesty work.

The paper is also candid about the limit of what this kind of audit can even check: it can only string-match against E5-Mistral’s own training data, because Mistral-7B and GPT-4’s own pretraining corpora are not public. There is no way, even for the authors, to fully rule out contamination that entered through those upstream models’ pretraining rather than through the paper’s own fine-tuning mixture. Publishing that limitation alongside the audit — rather than only publishing the audit’s reassuring parts — is itself the behavior worth learning from here. The lesson generalizes past this one paper: even work reporting a new state-of-the-art number ran its own contamination check and published both the findings and the blind spot, rather than letting the reader assume the number was clean by default.

A live example: even this session's own source paper got revised

One more instance of exactly this discipline is worth naming directly, because it happened inside the preparation of this very session and is a genuinely honest way to close a chapter about not trusting a number just because it is in a published paper. E5-Mistral’s own commercial-comparison table (Chapter 5) was not static. The version of the paper submitted in December 2023 and revised in January 2024 compared against “OpenAI Ada-002” (49.3 BEIR, 61.0 MTEB average). By the time the authors revised the paper again in May 2024, OpenAI had deprecated Ada-002 and released a newer embedding family; the paper’s own table was quietly updated to compare against “OpenAI text-embedding-3-large” (55.4 BEIR, 64.6 MTEB average) instead — a stronger, more current commercial baseline, and the specific numbers Chapter 5 above now uses.

Two things are worth taking from this. First, mechanically: a paper’s arXiv listing is not one fixed document — it is a version history, and “the current version” can mean something meaningfully different from what a citation, a blog post, or someone’s memory of the paper described months earlier. Checking which version a number comes from is not paranoia; it is the same discipline this whole session has applied to benchmarks, applied one layer up, to the paper itself. Second, and more directly relevant to everything Chapters 5–9 just walked through: this is a small, concrete demonstration that even a comparison table meant to showcase a strong result gets revisited and corrected as the world around it changes. A commercial API can be deprecated. A benchmark can be superseded by a more careful successor. The specific numbers in any single table, including every table in this session, have a shelf life — and the habit worth keeping past this session is not memorizing today’s numbers, but knowing where to go check them again.

A short checklist for reading the next embedding-model announcement

Every claim in this session traced back to a specific ablation table, a specific benchmark, or a specific appendix — not to a single headline number. Turn that habit into a checklist for the next model release you read about, whether it is a paper, a blog post, or a leaderboard screenshot.

Every row in that table has the same shape: a question that a single leaderboard screenshot cannot answer, and a specific chapter of this session that shows you how to answer it anyway, using numbers that were already published — not new experiments you have to run yourself. That reusability is the actual point of spending nine chapters inside two papers’ ablation tables and appendices instead of just quoting their headline results.

Which benchmark answers which question

One number to hold onto from each chapter

Ten chapters, two papers, dozens of tables. If only one number per chapter survives in memory a month from now, these are the ones worth keeping — each is the single fact that chapter’s entire argument hinges on.

Read down that right-hand column once more and notice the shape of the whole session: Chapters 0–4 are all about a mechanism working better than expected (cheap fine-tuning, negligible pretraining cost, a wrong pooling choice partially self-correcting). Chapters 5–9 are all about the same mechanism’s result mattering less than a single leaderboard number suggested, once the evaluation widens. Both halves are true at once, about the exact same model. That combination — a real engineering win, with a real scope limit — is a more useful thing to walk away with than either half alone.

What to actually do with all of this, Monday morning

Before trusting any single embedding-model leaderboard number for a real deployment: (1) identify which benchmark produced it — MTEB(eng, v1)? MTEB(Multilingual)? something narrower? (2) check whether that benchmark’s language mix resembles your actual users’ languages, the way Chapter 8’s Indic-vs-English comparison should now make instinctive; and (3) check whether the model’s training data plausibly overlaps the benchmark’s test sets, the way MTEB(eng, v2) was purpose-built to rule out. None of this requires re-deriving nDCG or nDCG@10 by hand every time — reuse the machinery this session and its companion lessons already built: Borda-count aggregation for combining many tasks fairly, TREC-pooled retrieval for evaluating at a sane cost, and inter-task correlation for telling a genuinely new measurement from a redundant one.

There is a fourth item worth adding to that list, quieter than the other three but just as load-bearing: build a small in-domain evaluation set of your own, in your own users’ languages, before you trust any external benchmark’s ranking for your specific deployment. Neither MTEB, MIRACL, nor MMTEB was built with your particular corpus or your particular query distribution in mind — they were built to compare models fairly against each other in general, which is a different goal than telling you which model is best for your product specifically. This is precisely the “Building Your Own Evaluation” chapter the Embedding Benchmarks gleam walks through end to end, and it is the natural next step after everything derived in this session.

The one-paragraph version, for a colleague who only has a minute

A decoder-only LLM can become a strong text embedder by taking its last hidden state as a summary vector, fine-tuning it lightly (0.6% of parameters) with a contrastive loss on data the model itself helped generate, and that light fine-tuning works because pretraining already built the semantics — fine-tuning only reorganizes them into a geometry cosine similarity can read. That recipe produced the best English text embedder available in early 2024. It is not, by itself, evidence about any other language, and when tested across 250+ of them by MMTEB, the English winner fell behind a model twelve and a half times smaller almost everywhere outside English and a handful of related languages. Neither half of that sentence is optional if you want to actually understand what “state of the art” meant here.

Where to go from here

• E5 and the Modern Text-Embedder Recipe (Veanors) — the full paper-grade derivation this session drew from: CCPairs and consistency filtering, batch size as the real contrastive-learning workhorse, hard-negative mining and distillation, the query:/passage: prefix era that came before instructions, and a from-scratch InfoNCE derivation with a different worked example.
• Embedding Benchmarks: MTEB, BEIR, and Honest Evaluation (Gleams) — nDCG, recall@k, MRR, MAP and Spearman correlation derived by hand from zero, plus a full chapter on leaderboard contamination and how to build your own small honest evaluation set.
• Vector Embeddings (Gleams) — the foundational lesson on what an embedding is and how cosine similarity works, if any part of Chapters 1–3 above moved too fast.
• Past this session: the field kept moving after E5-Mistral. LLM2Vec (BehnamGhader et al., 2024) goes further than Chapter 1's pooling choice by performing surgery on the causal mask itself — enabling bidirectional attention post-hoc and adding a masked-next-token-prediction adaptation stage, rather than accepting the causal-decoder coverage tradeoff Chapter 1 derived. NV-Embed (Lee et al., 2024) replaces last-token pooling entirely with a trained latent-attention pooling layer and a two-stage instruction-tuning curriculum. Both are covered in depth inside the E5 Veanors lesson linked above.