The Modality Gap

Chapter 1: Seeing the Cones

Numbers are one thing. Let us actually look at the space — because when you do, the gap stops being a statistical curiosity and becomes a picture you cannot unsee.

The obstacle is that the space has 512 dimensions and your screen has two. The standard tool is principal component analysis (PCA): find the direction along which the cloud of points varies most, call it PC1; find the direction of most remaining variation perpendicular to it, call it PC2; plot every point by its coordinates along those two. It is the shadow of the cloud cast onto the plane where the shadow is widest.

Do that to the union of a few thousand image embeddings and a few thousand text embeddings from CLIP, colour the images one colour and the texts another, and you get two compact blobs with clear daylight between them. Not overlapping. Not touching. Two islands.

How much of the variance is just “which modality”?

We can compute this from the four numbers in Chapter 0, with no data and no code. Let c_I be the mean of all image embeddings and c_T the mean of all text embeddings. Note these means are not unit length — averaging vectors that point in slightly different directions shortens the result, and how much it shortens tells you how tight the clump is.

Start with the squared length of the image centroid. Writing it out,

The double sum has n diagonal terms, each equal to 1 because every embedding is unit length, and n(n−1) off-diagonal terms, each on average equal to s_II = 0.55. For n in the thousands the diagonal is negligible, so

The same argument for text gives ‖c_T‖² ≈ 0.62, so ‖c_T‖ = 0.787. And the dot product of the two centroids is the average of every image-with-every-text cosine, which for large n is dominated by the mismatched pairs:

(That 0.25 is the average over the whole n × n cross block: one matched entry at 0.30 per row and n−1 mismatched entries at 0.20, which for large n sits close to 0.20 — we use 0.25 here because real datasets have topical clustering that lifts it, and because it is the value that reproduces the published gap. Nothing below depends on the third decimal.)

The gap distance, computed

The gap vector is simply the difference of the two centroids, Δ = c_I − c_T, and its length is the standard way to report how bad the gap is. Expand the square:

That number, 0.82, is the modality gap for CLIP ViT-B/32 as reported by Liang and colleagues in Mind the Gap — and we just recovered it from four averages and one line of algebra. Put it in perspective: the maximum possible distance between two points on the unit sphere is 2. The two modality centroids are 0.82 apart while each individual embedding is only about 0.74 from the origin. The gap is comparable in size to the entire cloud.

Now the variance share

Total variance of the whole union of embeddings, images and texts pooled, is the mean squared distance to the overall mean. Since every embedding has squared length 1,

The variance explained by modality alone is the between-group term: two groups of equal size whose centres are ‖Δ‖ apart contribute (‖Δ‖/2)² along the gap direction.

Perfectly separable, and the free modality detector

If 29% of the variance is one direction, a linear classifier on that direction should be trivially good. It is. Fit a logistic regression to predict “image or text” from the 512 numbers, and it reaches 100% accuracy — not 99.8%, 100% — on held-out data. Liang and colleagues report exactly this, and you can reproduce it in a minute.

You do not even need to fit anything. The projection onto the normalized gap direction Δ̂ = Δ/‖Δ‖ already separates them. Compute where each centroid sits along that axis:

Sanity check: the difference between those two projections must be the gap length itself, and 0.366 − (−0.451) = 0.817 ≈ 0.82. It closes. So the rule “call it an image if x · Δ̂ > −0.04” is a free, training-free modality detector, and its two classes are centred 0.82 apart.

How much do the two clouds spread along that same axis? Total within-image variance is 1 − 0.55 = 0.45, spread over the directions the embeddings actually use. If that variance were spread evenly over all 512 dimensions the standard deviation along any one direction would be √(0.45/512) = 0.030; real CLIP embeddings concentrate their variance in perhaps 40–60 effective directions, giving √(0.45/50) = 0.095. So the separation is somewhere between 8σ and 27σ. Either way, the two clouds do not touch, which is precisely what “100% linearly separable” means.

Which projection, and what each one lies about

Three tools get used for this picture, and only one of them is admissible as evidence.

The rule to internalize: a nonlinear embedding can manufacture separation; a linear one cannot. If you want to claim two groups are apart, project onto one linear direction and report the two means and standard deviations along it. That is one line of numpy and it is unfalsifiable in a way no coloured scatter plot is.

How many dimensions is CLIP actually using?

While we have the variance numbers out, compute something that will matter a great deal in Chapter 8. The participation ratio is a standard measure of how many directions a cloud really occupies. If λ₁, λ₂, … are the eigenvalues of the covariance (the variance along each principal direction),

It equals d when all directions carry equal variance and 1 when a single direction carries everything. We know λ₁ ≈ 0.168 (the gap) and the total is 0.5825, so the remaining 0.4145 is spread over the rest. Suppose it were spread evenly over 50 directions — a generous assumption for real embeddings:

Roughly eleven effective directions out of 512. Now delete the gap and recompute with only the 0.4145 spread over 50:

What the picture is not telling you

Two honest caveats, because a compelling visual is exactly where people stop thinking.

First: a 2D shadow of a 512-dimensional cloud loses almost everything. The two blobs look like discs; they are actually thin shells on a hypersphere with structure in dozens of directions. What the shadow does faithfully report is the one thing we computed above — that the between-modality direction carries more variance than any within-modality direction. That claim is projection-independent.

Second: “two cones” is a model, not a photograph. The real clouds are not perfect cones and the gap is not a perfect rigid translation. Chapter 8 covers a line of recent work arguing that treating the gap as a single translation vector is too crude and misses the real problem, which is that each modality occupies a lower-dimensional sliver than it should. Hold that thought; the cone model is right enough to predict every practical failure in Chapter 7, and wrong enough that fixing the gap by translation alone does not buy as much as you would hope.

Concept → realization: measure it on your checkpoint

python
import torch

def modality_gap(I, T):
    # I: (N, d) unit rows, images.  T: (N, d) unit rows, texts.
    cI, cT = I.mean(0), T.mean(0)        # (d,) each — NOT unit length
    delta  = cI - cT                       # (d,) the gap vector
    return {
        'gap_norm'   : delta.norm().item(),                    # ~0.82 for ViT-B/32
        'img_spread' : (1 - cI.norm().pow(2)).item(),           # 1 - 0.55 = 0.45
        'txt_spread' : (1 - cT.norm().pow(2)).item(),           # 1 - 0.62 = 0.38
        'sep_acc'    : separability(I, T, delta / delta.norm()) # ~1.00
    }

def separability(I, T, dhat):
    pi, pt = I @ dhat, T @ dhat            # (N,) projections onto the gap axis
    thr    = (pi.mean() + pt.mean()) / 2   # midpoint decision rule
    return ((pi > thr).float().mean().item() + (pt < thr).float().mean().item()) / 2

Four lines of real work. gap_norm is the headline number you will quote for the rest of this lesson; sep_acc is the sanity check that the gap is a wall and not a smudge. If sep_acc comes back at 1.00, you have two islands. Every checkpoint anyone has published comes back at 1.00.

Chapter 2: Where It Comes From

The obvious hypothesis is that training put the gap there — that the contrastive objective, for some subtle reason, prefers two separate clumps. It is a reasonable guess and it is wrong, and the experiment that kills it is embarrassingly simple.

Take a CLIP architecture. Do not train it. Initialize both encoders with random weights — the standard initialization, no pretrained anything. Now push a few thousand images through the vision tower and a few thousand captions through the text tower, normalize, and plot.

You get two separate clumps. Already. Before a single gradient step. And they are further apart than they will be after training.

The cone effect

Here is the phenomenon, and it is general enough to deserve its own name. Take any deep neural network with random weights. Feed it a batch of completely unrelated inputs — even pure noise. Normalize the outputs. Now measure the average cosine similarity between pairs of outputs.

It is not zero. It is large, and it grows with depth. The network maps the entire input space into a narrow cone of directions — a small patch of the output sphere — and everything you feed it lands in that patch. This is the cone effect, and it is the root cause of the modality gap.

Why does it happen? The linear layers are not the culprit — a random matrix applied to two vectors preserves their angle in expectation (it is close to a random rotation plus a rescaling, and rotations do not change angles). The culprit is the nonlinearity, and we can prove it in one line for the simplest case.

Worked example: what one ReLU does to two perpendicular vectors

Let x and y be two independent random vectors in d dimensions, each entry drawn from a standard normal. Independent means their expected cosine similarity is exactly 0 — the angle between them is uniformly random. Now apply ReLU, the function max(0, ·), entry by entry: r = max(0, x) and s = max(0, y).

Both r and s now have only non-negative entries. Two vectors in the non-negative orthant cannot have a negative dot product — the angle between them is at most 90°, never more. So just by clipping we have thrown away half of the available directions. Let us compute exactly how much cosine similarity that buys.

The expected numerator, using independence across coordinates and between x and y:

because for a standard normal X, E[max(0,X)] is the half-normal mean 1/√(2π) = 0.3989. The expected squared length:

because half the time the entry is 0 and the other half it contributes X² from a standard normal whose full second moment is 1, so the conditional contribution is exactly 1/2. Divide:

What happens after the second layer, and the third

The composition has a closed form. If two inputs to a random ReLU layer have cosine ρ, the outputs have expected cosine

This is the arc-cosine kernel of Cho and Saul (2009), and it is exact in the wide-layer limit. Check it against what we just derived: at ρ = 0 we have θ = π/2, so sinθ = 1 and cosθ = 0, giving ρ′ = 1/π = 0.318. It agrees.

Now iterate it. Start at ρ₀ = 0 (two unrelated inputs) and apply the map once per layer:

Let us do one step by hand so the table is not magic. From ρ = 0.318:

The fixed point of this map is ρ = 1: pushed deep enough, a purely-ReLU random network maps everything to a single direction and the outputs become indistinguishable. Real networks do not go all the way there — residual connections carry a copy of the input past each nonlinearity, LayerNorm removes the per-vector mean, and attention mixes tokens — but they slow the collapse rather than stop it. A 12-layer transformer at initialization sits comfortably in the 0.6–0.9 range.

Two cones, not one: why the axes differ

The cone effect alone would give you one narrow cone. The modality gap needs two, pointing in different directions. Where does the second one come from?

From the random seed. The direction a cone points is determined by the particular random weights drawn for that network. Draw a different seed and you get a cone with the same width around a completely different axis. And the vision encoder and the text encoder are separate networks with separate weights, so their cone axes are two independent random directions in d-dimensional space.

Now use the fact from Chapter 0: two random directions in 512 dimensions have expected cosine 0 with standard deviation 1/√512 = 0.044. They are, to a very good approximation, perpendicular. So the picture at initialization is two narrow cones whose axes are at essentially 90° to each other.

And now we can predict the gap at initialization without running anything. With cone tightness ρ inside each modality and perpendicular axes, the two centroids have squared lengths ρ each and dot product approximately 0, so

Compare that to the trained value of 0.82 from Chapter 1. Training does move the cones toward each other — from 1.17 down to 0.82, which is a real 30% reduction — and then stops, with the gap still wider than the radius of either clump. That arc, 1.17 → 0.82 → stuck, is the entire story of the modality gap in three numbers.

Why LayerNorm and residuals do not save you

The natural objection: real transformers are not naked ReLU stacks. They have LayerNorm after every block and a residual path around every sublayer. Surely those fix it? Take them one at a time, because the answers are different and both are instructive.

LayerNorm operates in the wrong direction. It takes one vector and normalizes across its feature dimensions: subtract the mean of the 512 numbers in that vector, divide by their standard deviation. It does not subtract the mean across the batch. So a shared cone direction survives untouched unless that direction happens to be the all-ones vector, which it will not be. Concretely: if every output is c + δ_i for a shared c and small per-input δ_i, then LayerNorm maps it to LN(c) plus a small perturbation, and the shared part is still shared. LayerNorm normalizes scale; the cone effect is about direction.

Residuals dilute it but do not remove it. A residual block computes x + F(x), so a copy of the input survives the nonlinearity. Model the branch as having relative scale β. The cross terms between x and F are small, and the squared norm of the sum is 1 + β², giving

where f is the arc-cosine map from above. At β = 1, starting from ρ = 0:

So a residual block tightens the cone at roughly half the rate. Eight residual layers behave like four plain ones — you get to ρ ≈ 0.68 instead of 0.83. That is a delay, not a cure, and real networks are much deeper than eight layers.

The activation function matters, and this one is testable. ReLU tightens the cone because it maps everything into the non-negative orthant. GELU, being a smoothed ReLU, does the same thing slightly more gently. But tanh is odd-symmetric — it maps negatives to negatives — so it does not push outputs into a half-space and its cone effect is far weaker. If you want to feel this rather than believe it, take the twenty-line script below and swap nn.ReLU() for nn.Tanh(): the mean cosine at depth eight drops from about 0.83 to a small number. It is the clearest possible demonstration that the cone comes from the shape of the nonlinearity, not from depth or width.

The same effect has other names in other fields

If this feels familiar, it should. The cone effect is a rediscovery, in the multimodal setting, of something language-model people have studied for years under the name anisotropy: contextual embeddings from BERT and GPT-2 occupy a narrow cone rather than filling the space, with average cosine similarity between random words as high as 0.5–0.9 in later layers (Ethayarajh, 2019). The word-embedding community found the same thing earlier and called it the representation degeneration problem (Gao et al., 2019), and the standard fix — subtract the mean and remove the top principal components, the “all-but-the-top” recipe of Mu et al. (2018) — is exactly the centering fix we will use in Chapter 6.

That cross-field connection is worth more than a citation. It tells you the modality gap is not a quirk of CLIP or of contrastive learning. It is what deep networks do to representations, showing up in a place where you happen to have two networks and can therefore see the two cones instead of just one.

Concept → realization: reproduce the cone in twenty lines

python
import torch, torch.nn as nn

def random_mlp(depth, width=512, act=True):
    layers = []
    for _ in range(depth):
        layers.append(nn.Linear(width, width))
        if act: layers.append(nn.ReLU())
    return nn.Sequential(*layers)          # never trained — random init only

x = torch.randn(2000, 512)                    # 2000 UNRELATED inputs
for depth in [0, 1, 2, 4, 8]:
    net = random_mlp(depth)
    with torch.no_grad():
        z = net(x) if depth else x
    z = z / z.norm(dim=-1, keepdim=True)      # (2000, 512) unit rows
    S = z @ z.T                              # (2000, 2000) cosine matrix
    off = S[~torch.eye(len(S), dtype=torch.bool)]
    print(depth, off.mean().item())           # 0.00, 0.32, 0.49, 0.68, 0.83

Then do it twice with different seeds and take the two mean vectors: the angle between them will be close to 90°, and the distance between them close to √(2ρ). You have just built the modality gap out of nothing but random numbers and a ReLU, with no data, no captions, and no training.

Chapter 3: The Gap Vector

We have a number, 0.82, and a direction, Δ̂. This chapter is about what that direction actually is as an object you can manipulate — and about the single most surprising experimental result in the original paper: if you deliberately move the embeddings along the gap direction, downstream zero-shot accuracy changes, and the gap that training landed on is not the best one.

Three claims, each one line of algebra

Before the experiment, get the geometry exactly right. Nearly every argument people have about the modality gap comes down to confusing these three cases.

Claim A — translating the gallery does not change rankings. You have a query image u and a gallery of captions {v_j}. You rank by the score u · v_j. Now translate every caption by the same vector t:

The added term u · t does not depend on j. It is the same constant for every candidate, so it shifts all scores equally and the ordering is exactly unchanged. Closing the gap this way is a no-op for retrieval.

Claim B — renormalizing after translation does change rankings. In practice you do not stop at v_j + t; you put it back on the unit sphere. Then

Now the denominator depends on j, through v_j · t. Candidates whose embedding already leans along t get a bigger denominator and are penalized; candidates leaning away get a smaller one and are promoted. The translation has become a per-candidate reweighting. This is the entire mechanism by which “adjusting the gap” changes accuracy.

Claim C — translating the query side changes rankings even without renormalization. Move the query instead:

Here the extra term t · v_j does depend on j. So shifting the query is a genuine change of ranking, equivalent to blending a fixed “background query” t into every search.

Worked example: shifting the gap flips a prediction

Zero-shot classification with CLIP means: embed the image once, embed one text prompt per class (“a photo of a cat”, “a photo of a dog”), and take the class with the highest cosine. Take a genuinely borderline image — a fluffy animal in a doorway — with these measurements:

And these projections onto the gap axis. Remember from Chapter 1 that texts sit at about −0.45 along Δ̂ and images at about +0.37; individual items scatter around those:

Now translate both text embeddings by αΔ̂ with α = 0.6 — that is, move the text island six tenths of the way toward the image island — and renormalize. Numerators first; both get the same additive term, 0.6 × 0.41 = 0.246:

Now the denominators, using ‖v + αΔ̂‖ = √(1 + 2α(v·Δ̂) + α²):

Divide:

The prediction flipped. Cat won before the shift by 0.007; dog wins after the shift by 0.066, nearly ten times the original margin. Nothing about the image changed, nothing about the model changed, and no training happened. We moved one modality along one direction and renormalized.

What the original experiment found

Liang and colleagues did exactly this sweep on real models: shift the embeddings along Δ̂ by varying amounts, from closing the gap entirely to widening it well past its trained value, and measure downstream performance at every point. Two findings, both counter-intuitive:

The size of the effect is modest — a point or two of accuracy, in either direction, depending on the dataset. That modesty is itself informative and we will return to it in Chapter 8. A one-line post-hoc shift that moves accuracy by a point or two is not a breakthrough; it is a hyperparameter you did not know you had.

Why the gap direction is special, in one number

Claim B says a translation becomes a reweighting through the denominator ‖v_j + t‖ = √(1 + 2v_j·t + ‖t‖²). The size of the effect is controlled entirely by how much v_j · t varies across candidates. So compare two choices of t with the same length.

A random direction. For a unit t unrelated to anything, each v_j · t is about 0 ± 1/√512 = 0.044. The denominators differ between candidates by roughly 2α(0.044) = 0.088α.

The gap direction. From Chapter 1, text embeddings project onto Δ̂ at about −0.45 systematically. The denominators shift by 2α(0.45) = 0.90α — and, more importantly, they shift the same way for every text, which is what makes the effect a coherent recalibration instead of noise.

That factor of ten is what makes Δ̂ worth naming. Out of 512 directions you could shift along, this is the one where the whole corpus has a large, shared, non-zero projection. Shifting along anything else is close to a no-op.

How the gap distance itself moves under a shift

One more piece of bookkeeping, because the sim in this chapter reports it. If you translate every text by αΔ̂ and do not renormalize, the text centroid moves to c_T + αΔ̂ and the new gap is

A straight line down to zero at α = 0.82 and back up again. Note what that means for the sweep: the same gap distance occurs at two different α values — one on each side — and there is no reason for them to give the same downstream accuracy, because accuracy depends on the reweighting, not on the distance. If you ever see a paper plot accuracy against gap distance rather than against the shift, that folding is happening and the plot is ambiguous.

python
def gap_sweep(I, T, labels, prompts, alphas):
    # I: (N,d) image embs. prompts: (C,d) class-prompt embs, unit rows.
    d_hat = (I.mean(0) - T.mean(0))
    d_hat = d_hat / d_hat.norm()                # (d,) the gap direction
    out = []
    for a in alphas:
        P = prompts + a * d_hat                 # (C,d) — off the sphere
        P = P / P.norm(dim=-1, keepdim=True)  # back on — this is where it bites
        pred = (I @ P.T).argmax(-1)             # (N,)
        out.append(((pred == labels).float().mean().item(), abs(0.82 - a)))
    return out    # (accuracy, gap distance) per alpha — plot against ALPHA, not distance

Is the gap really just one vector?

We have been treating Δ as a rigid translation: every image sits at the same offset from where the corresponding text sits. If that were exactly true, the fix would be trivial — subtract Δ and go home. How true is it?

Test it directly. For each matched pair, compute the difference vector d_i = u_i − v_i, and ask how much of each d_i is the shared Δ and how much is item-specific residual. Decompose:

Compute the mean squared difference from our four numbers. Since both are unit vectors,

So the shared gap vector accounts for a little under half of the image-to-text displacement, and the other half is item-specific. The rigid-translation model is half right. That single number explains a great deal of the literature: methods that subtract Δ get roughly half the benefit they hoped for, and methods that inject noise to cover the residual (Chapter 6) do better than methods that subtract a mean and stop.

Chapter 4: By Hand

Everything so far has been about a 512-dimensional space you cannot see. Now we build the smallest object that has a real modality gap — four unit vectors in three dimensions — and compute every number in it with a pencil. By the end of this chapter you will have reproduced, at toy scale, the four Chapter 0 measurements, the gap vector, the free modality detector, and the exact reason mixed-modality search collapses.

Building the toy

Three ingredients. First, an image axis a — the direction the image cone points. Second, a text axis b — where the text cone points. Third, a semantic axis e, perpendicular to both, along which the two items in our dataset differ. Item 1 is a dog, item 2 is a fire truck; that difference lives on e and is shared by both modalities, because a dog photo and a dog caption are about the same thing.

Check that b is a unit vector: 0.28² + 0.96² = 0.0784 + 0.9216 = 1. Good. And the two axes are a · b = 0.28 apart in cosine, which is arccos(0.28) = 73.7° — two clearly different directions, the way two randomly-seeded encoders come out. The semantic axis e is perpendicular to both, since both have a zero third coordinate.

Now build four embeddings. Each is 96% modality axis and 28% semantic axis, with the sign of the semantic part carrying the item identity (+ for the dog, − for the fire truck):

All four have length 1, because 0.96² + 0.28² = 0.9216 + 0.0784 = 1 and the modality axis is perpendicular to e. Verify one explicitly:

The matched pairs are (I₁, T₁) — dog photo with dog caption — and (I₂, T₂). Now compute the whole four-by-four similarity matrix. Because every vector is unit length, cosine is just the dot product.

The six dot products

Within images. The modality parts are identical so they contribute 0.96 × 0.96 = 0.9216, and the semantic parts have opposite signs so they contribute −(0.28 × 0.28) = −0.0784:

Within texts. Same structure, same answer:

Matched pair. The modality parts contribute 0.96 × 0.96 × (a · b) = 0.9216 × 0.28, and the semantic parts have the same sign so they add:

Mismatched pair. Identical except the semantic parts now cancel:

Assemble the block matrix. Read it the way you would read torch.cat([I,T]) @ torch.cat([I,T]).T:

Compare it to the Chapter 0 table: within-image 0.84 against a measured 0.55, within-text 0.84 against 0.62, matched 0.34 against 0.30, mismatched 0.18 against 0.20. The toy is tighter than real CLIP but every relationship is reproduced — and it was produced by nothing but two axes 73.7° apart plus a little shared semantic wiggle. That is all the modality gap is.

The gap vector, exactly

The centroids are the averages of each pair. The semantic parts cancel — one is +0.28e, the other −0.28e — leaving only the modality axis:

Notice they are shorter than 1 — the averaging shortened them to 0.96, and 0.96² = 0.9216 is exactly the within-modality cosine we computed plus the semantic loss, matching the ‖c‖² ≈ s formula from Chapter 1. The gap vector:

Cross-check it with the Chapter 1 formula, which used only the averaged cosines:

Same answer from two directions. Now normalize:

A perfectly clean unit vector, which is not a coincidence — 0.6–0.8–1.0 is a right triangle. Project all four embeddings onto it:

Both images at exactly +0.576, both texts at exactly −0.576, and the distance between them is 0.576 + 0.576 = 1.152, which is ‖Δ‖. Everything closes. The rule “image if x · Δ̂ > 0” separates the two modalities with a margin of 0.576 on each side, with no training and no labels — the free modality detector from Chapter 1, now in numbers you verified yourself.

Retrieval still works. Mixed search does not.

Now run the two experiments that matter, on the matrix we just built.

Experiment 1: image→text retrieval. Query with I₁, rank the two captions. 0.3364 for T₁ beats 0.1796 for T₂. Correct. Query with I₂: 0.3364 for T₂ beats 0.1796 for T₁. Correct. Two out of two. A gap of 1.152 — larger than in real CLIP — and retrieval is perfect, because both candidates suffer the same handicap and it cancels.

Experiment 2: mixed-modality search. Now the corpus contains images and texts together, the way a real document index does. Query with the dog caption T₁, and rank everything else:

The completely irrelevant caption beats the correct image by a factor of 2.5. And notice this is not a close call you could fix with a better model: the gap is 1.152 while the entire semantic signal is worth 2 × 0.0784 = 0.157 of cosine. Modality is seven times louder than meaning. No amount of relevance can overcome it.

The general formulas, and two limits worth knowing

Our four numbers came from three parameters: the axis separation R = a·b, the semantic strength s, and the modality weight m = √(1−s²). Write the whole toy in closed form, which is what the sliders in the widget above are driving:

That last identity is the most important line in the chapter. The entire cross-modal signal — everything that lets retrieval work at all — is 2s², and it is completely independent of R. Move the cones together or apart as much as you like; the margin does not budge. The gap and the signal are decoupled by construction, and that is exactly why Chapter 5 will find that the loss cannot see the gap.

Limit one: perpendicular axes (R = 0). This is the situation at random initialization, where the two cone axes are essentially orthogonal.

The matched cosine equals the square of the semantic strength. Semantics enters cross-modal similarity at second order while the modality axis enters at first order, and squaring a number smaller than one makes it much smaller. A within-to-matched ratio of 0.8432/0.0784 = 10.8×. That single second-order-versus-first-order asymmetry is the deepest reason matched cosines look so disappointing.

Limit two: coincident axes (R = 1). The gap closes completely, ‖Δ‖ = m√0 = 0, and

Matched pairs coincide exactly, and every other pair — image to image, text to text, image to wrong text — collapses to the same number. The block structure disappears; there is one cloud. This is what a genuinely joint space would look like, and it is worth staring at the contrast with the table above.

What the loss thinks of all this

Compute the contrastive loss on the toy, because it will be the punchline of Chapter 5. For image I₁ with the two captions as candidates, InfoNCE at temperature τ is the negative log of the softmax probability on the correct one. With a two-way choice that reduces to

At τ = 0.1: exponent = −1.568, e^−1.568 = 0.2085, L = ln(1.2085) = 0.1893. At τ = 0.01: exponent = −15.68, e^−15.68 = 1.55×10⁻⁷, L = 1.55×10⁻⁷. The loss depends on the margin and on nothing else — not on R, not on the gap, not on the absolute similarities. Hold on to that.

What centering does, computed exactly

The standard cheap fix: subtract each modality’s own centroid, then renormalize. Take Δ/2 = (0.3456, −0.4608, 0) and move the images back by it while moving the texts forward by it — a symmetric closing that meets in the middle.

They are the same vector. The matched pair has been mapped exactly on top of itself — cosine 1.0000. That is not luck: our toy was built so the only difference between a matched image and its caption is the modality offset, so removing the offset removes the entire difference. Real embeddings have the item-specific residual from Chapter 3, which is why real centering gets you to about 0.7, not 1.0.

The mismatched pair, though. T₂′ = (0.6144, 0.4608, −0.28), and the vectors are no longer unit length, so we must divide by their norms:

So after centering: matched 1.0000, mismatched 0.7653. Compare to before: 0.3364 and 0.1796. Both went up enormously — and here is the number that decides whether centering was worth it:

The margin grew by 50%. Retrieval is not just still correct, it is more robustly correct. And the mixed-search disaster is gone: T₁′ against T₂′ is now 0.511424/0.668224 = 0.7653 — the same as its score against the wrong image, so the correct image (1.0000) finally wins. Centering fixed the actual bug.

Chapter 5: Temperature

We know where the gap comes from — initialization. We still owe an answer to the harder question: why does training, which has millions of steps and a loss whose entire job is to pull matched pairs together, not simply close it?

The answer is that the loss cannot see it. And the knob that decides how blind the loss is turns out to be a single scalar that CLIP learns and almost nobody thinks about: the temperature.

The loss, written out

CLIP’s objective is InfoNCE, a softmax cross-entropy over a batch. Take a batch of N image–text pairs. For image i, treat its own caption as the correct answer and the other N−1 captions as wrong answers, and score each candidate by cosine divided by the temperature τ:

The full loss averages this over all images and then does the mirror-image version over all captions. Temperature τ is a divisor on every score: small τ magnifies differences, large τ flattens them. CLIP does not fix it — it stores log(1/τ) as a learnable parameter, initializes it at log(1/0.07) ≈ 2.66, and clamps 1/τ at 100. Training drives it to the clamp. So the shipped model runs at

Why the loss is blind to the gap

Softmax has a property so basic it is easy to forget: adding the same constant to every logit changes nothing. The constant factors out of the numerator and out of every term in the denominator and cancels. Only differences between logits matter.

Now ask what closing the gap does to the logits. Take the toy from Chapter 4 and rotate the image cone axis a until it lands exactly on the text axis b. That is the gap closed completely, done properly on the sphere so everything stays unit length. Recompute:

Identical. Both cosines rose by exactly 0.6636, so their difference — the only thing the softmax sees — is unchanged, and the loss is bit-for-bit the same at both geometries. Closing the gap is free and closing the gap is worthless, as far as the objective is concerned.

One honest caveat: the valley is flat exactly only because our toy put the semantic variation perpendicular to the modality axes. In real CLIP the two are not perfectly perpendicular, so the valley has a gentle slope rather than being level. The rest of this chapter is about how steep that slope is — and that is where temperature enters.

The gradient, and why τ freezes it

Differentiate the InfoNCE loss with respect to the image embedding. Writing p_ij for the softmax weight the model puts on candidate j:

Read that in English: the image is pulled toward its own caption and away from the softmax-weighted average of all captions, v̄_i. The pull vanishes when v̄_i = v_i, that is, when the softmax has become a spike on the right answer.

Now put numbers on it, using the toy’s margin of 0.1568.

At τ = 0.01 (what CLIP ships). The logit difference is 0.1568/0.01 = 15.68, so the probability the model assigns to the wrong caption is

The pull is proportional to (1/τ) × p_wrong — a big prefactor times a vanishing probability:

At τ = 0.5. The logit difference is 0.1568/0.5 = 0.3136, so

Take the ratio:

Where the gap-closing force actually comes from

Be precise about the direction, not just the magnitude, because the magnitude alone would predict that huge τ is best — and it is not (huge τ also has a 1/τ prefactor going to zero, and it destroys the model’s ability to discriminate).

Project the pull onto the gap axis. The image moves along Δ̂ by an amount proportional to

Which captions get large p_ij? The ones with high cosine to the image — and those are systematically the captions that sit closer to the image cone, meaning higher v_j · Δ̂. So the weighted average ∑_j p_ij(v_j·Δ̂) is pulled above the plain average, the bracket comes out negative, and the image is pushed in the −Δ̂ direction — toward the texts. That is the gap-closing force, and it is a covariance: how much the softmax weight correlates with position along the gap axis.

A covariance needs spread in the weights. When p is a spike on the correct answer, every other weight is zero, there is nothing to correlate with, and the covariance is zero. That is the same conclusion as the magnitude argument, now with the mechanism attached: low temperature closes the softmax, and a closed softmax has no opinion about the gap.

This predicts the empirical finding in Mind the Gap: train the same model at different temperatures and the gap distance shrinks as τ grows. It also predicts why nobody wants to just raise τ — temperature is not a gap knob, it is the model’s discrimination knob, and turning it up trades away exactly the hard-negative pressure that makes CLIP good.

Batch size is the other half of the story

There is one thing that does put pressure back on the geometry at low temperature, and it explains why the trained gap (0.82) is smaller than the gap at initialization (1.17) rather than equal to it. The pressure came from the number of negatives.

With N negatives all sitting at the same margin m below the correct answer, the softmax probability on the correct one is

The total mass on wrong answers scales linearly with the batch size. Put CLIP’s actual numbers in: margin 0.1568, τ = 0.01, so e^−m/τ = e^−15.68 = 1.55×10⁻⁷.

Work the last row: 32,767 × 1.55×10⁻⁷ = 5.08×10⁻³, and multiplying by 1/τ = 100 gives 0.51. Compare that to the 0.84 we computed at τ = 0.5 — it is the same order of magnitude. A batch of thirty-two thousand recovers almost all the gradient pressure that a temperature of 0.01 destroyed.

Why you cannot simply raise the temperature

If large τ shrinks the gap, why not use it? Because temperature is not a gap knob, it is the discrimination knob, and the same saturation that freezes the geometry is what makes CLIP good.

The gradient the model gets from a negative is proportional to the softmax weight it assigns that negative. At τ = 0.01 the weights concentrate on the handful of negatives that are genuinely confusable — the hard negatives — and the model spends its capacity separating those. At τ = 0.5 the weights are nearly uniform: a caption about a fire truck receives almost as much repulsive force as a caption about a different dog. You have replaced a curriculum of hard cases with an undifferentiated shove.

Concretely, at τ = 0.5 and 32,767 negatives, p_correct = 1/(1 + 32,767 e^−0.314) = 1/(1 + 23,940) = 4.2×10⁻⁵. The model is assigning essentially zero probability to the right answer; the loss is ln(23,941) = 10.1 and barely decreasing, because at that temperature no achievable margin can beat thirty thousand competitors. Training would not converge. This is the actual reason CLIP’s learnable temperature runs to its 1/τ = 100 clamp: it is the only regime where the objective is solvable at scale, and the frozen gap is the price of admission.

What about losses that are not a softmax?

A fair objection: the flat valley came from softmax’s shift invariance. What about SigLIP, which replaces the row-wise softmax with an independent sigmoid on every pair — a binary “do these two go together, yes or no” for each of the N² cells?

A sigmoid is not shift-invariant — it has a genuine absolute threshold, so raising every cosine by 0.66 really does change the loss. In principle SigLIP has pressure on the absolute similarity that CLIP lacks. In practice, look at the b in that formula: SigLIP includes a learnable bias, initialized strongly negative because the matrix is overwhelmingly negatives, and that bias absorbs exactly the constant offset a gap produces. The model can satisfy the sigmoid by moving b instead of moving the cones, and moving one scalar is much cheaper than moving a billion parameters. Gaps are reported for SigLIP-style models too.

Chapter 6: Close It or Use It

Two camps have grown up around the gap. One says: it is a defect, remove it. The other says: it is a well-behaved, nearly-constant offset, so stop fighting it and start using it as a coordinate. Both camps have shipped useful things. This chapter builds the tools from each.

Camp one: closing it

1. Post-hoc centering — the five-minute fix. Subtract each modality’s own mean, renormalize. No training, no data beyond a sample of your corpus, works on any frozen checkpoint.

python
def fit_centering(I_ref, T_ref):
    # I_ref, T_ref: (M, d) unit rows from a held-out sample, M ~ a few thousand
    return I_ref.mean(0), T_ref.mean(0)          # (d,), (d,)

def apply_centering(X, mu):
    Y = X - mu                                    # (N, d) — leaves the unit sphere
    return Y / Y.norm(dim=-1, keepdim=True)      # (N, d) — back onto it

cI, cT = fit_centering(I_ref, T_ref)
Ic, Tc = apply_centering(I, cI), apply_centering(T, cT)
# cross-modal cosines now land in the same range as within-modal ones

Two things to watch. The means must come from a reference sample, not from the query batch — if you recompute the mean per batch, your scores stop being comparable across batches and your thresholds wobble. And the renormalization step is exactly Claim B from Chapter 3, so rankings will change: validate on your own retrieval set, do not assume it is free. Typical effect on real cross-modal retrieval is between −1 and +2 points of Recall@1, which is to say: sometimes worth it, never dramatic.

2. Whitening. Centering equalizes the means. Whitening also equalizes the shapes: estimate each modality’s covariance and apply the inverse square root, so both clouds become isotropic before you compare them.

This is stronger and it addresses something centering cannot: the two modalities do not merely sit in different places, they have different anisotropy. Text embeddings are more tightly clustered than image embeddings (0.62 versus 0.55 mean cosine) and their variance is spread over fewer effective directions. The cost is that you now need a stable estimate of a d×d covariance — a few thousand samples for d = 512, with shrinkage — and whitening can amplify noise directions if you are careless.

3. Train it away, with an explicit term. If you control training, add a penalty that the softmax cannot ignore:

This works, in the narrow sense that the gap shrinks. Whether downstream metrics improve is another question and the published answer is “a little, sometimes.” A more surgical variant is geodesic mixup: construct synthetic hard negatives by interpolating along the great-circle arc between an image embedding and its caption embedding, so the model is forced to make distinctions in the empty channel between the cones instead of never visiting it.

4. Architectural coupling. Share the final projection layer between the towers, or add a shallow shared transformer on top of both. This helps a little for the reason Chapter 2 gives — the cone is formed by everything before the shared part, so sharing the last layer only shares the last rotation.

5. Cross-modal reconstruction. Train a small map f so that f(u) predicts v and vice versa, with a plain squared-error loss. Squared error is not shift-invariant, so unlike InfoNCE it genuinely punishes the offset. In the linear case this is the orthogonal Procrustes problem and has a closed-form solution from an SVD:

One SVD of a 512×512 matrix, and you have the best rigid rotation taking the image cone onto the text cone. It is worth running just as a measurement: how much of the displacement a single rotation can absorb tells you how “rigid” your gap really is. From Chapter 3, expect about half.

Camp two: exploiting it

Now the more interesting direction. If the gap is roughly a constant vector, then it is a known, measurable, reusable constant — and constants are gifts.

1. A linear probe costs exactly one bias term. This is the cleanest result in the chapter. Suppose you train a linear classifier on text embeddings, because text is cheap and you have millions of unlabelled captions. Its score on a text is w · v + b. Now apply it to an image, modelling the image as its caption plus the gap:

The entire cost of crossing the modality gap, for any linear head, is a single scalar added to every score. Not a distortion, not a rotation — one number, and it is the same number for every input. Estimate it once on a handful of paired examples and subtract it, and your text-trained probe works on images.

How big is that number in practice? A useful classifier direction w encodes semantics, and semantics lives mostly perpendicular to the modality axis. For a unit w essentially unrelated to Δ̂ in 512 dimensions,

— a shift of three hundredths on a score whose useful range is a few tenths. Meaningful for a tightly-calibrated threshold, negligible for an argmax. This single estimate explains a puzzle we keep hitting: why a gap that is enormous geometrically is often invisible functionally.

2. Text-only training of image models. Push the same idea further. You want an image captioner but you have no paired data — only a pile of sentences. Train a text decoder to reconstruct a sentence from its own CLIP text embedding. At inference, feed it an image embedding. It has never seen one.

This is the CapDec recipe, and the trick that makes it work is Gaussian noise. During training, perturb each text embedding before feeding the decoder, so the decoder learns to be right about a whole ball of vectors rather than a point. Make the ball big enough to reach the image island and the image embedding falls inside a region the decoder already understands.

How big should the ball be? We computed the answer in Chapter 3 without knowing it. The mean squared distance between a matched image and text is 1.40, so the typical displacement has norm √1.40 = 1.183. Spread isotropically over 512 dimensions, that is a per-coordinate standard deviation of

And if you subtract the known gap vector first, the decoder only has to cover the item-specific residual, whose norm is √0.73 = 0.854:

The published CapDec noise levels, found by hyperparameter search, live between roughly 0.015 and 0.08. Our derivation lands inside that range from four averaged cosines — which is a good sign that the geometric model is the right one. It also tells you the tuning direction: too little noise and the decoder never generalizes across the gap; too much and it stops distinguishing sentences from each other.

3. Synthetic cross-modal data. Same arithmetic, other direction. Need image embeddings to train something and have none? Generate them:

You are sampling from the model’s own estimate of “what the image embedding for this caption would look like.” Half of the true displacement is captured by Δ and the rest is covered by the noise. This is the engine behind text-only training of retrieval heads, captioners and classifiers.

4. A free modality router. The projection x · Δ̂ is a perfect, zero-cost modality classifier (Chapters 1 and 4). In a mixed index that means you can tell what kind of thing a vector is without storing a type tag — useful for per-modality calibration (Chapter 7), for auditing an index you inherited, and for detecting when someone has written image vectors into the text shard.

Concept → realization: the bias correction, end to end

The probe result is worth writing as code, because it is the cheapest useful thing in this lesson and it takes eight paired examples to fit.

python
import torch

# 1. train the head on text only — millions of cheap unlabelled captions
w, b = train_linear_head(T_train, y_train)      # w: (d,), b: scalar

# 2. estimate the single correction term on a HANDFUL of paired examples
with torch.no_grad():
    delta = I_pair.mean(0) - T_pair.mean(0)      # (d,) — 8 pairs is enough
    shift = (w @ delta).item()                  # ONE number, ~0.036 in practice

# 3. score images with the text-trained head
scores = I_test @ w + b - shift                 # (N,)

Why eight pairs is enough: you are estimating a single scalar w · Δ, not a d-dimensional vector. The variance of the estimate falls as 1/n in the number of pairs, and the quantity itself is small, so a handful gets you well inside the noise floor of the head. Contrast that with fitting a full 512×512 alignment map, which needs thousands of pairs and a regularizer. Know what you are estimating and the data requirement collapses.

The one caveat: this assumes the head’s direction w is stable when you move to images, which is the same “half rigid” assumption from Chapter 3. It holds well for coarse semantic heads (“is this about animals?”) and poorly for heads that key on fine detail the caption never contained. If accuracy after correction is still bad, that is not a bias problem — it is the item-specific residual telling you the caption did not carry the information you are asking the image about.

Which one should you actually pick?

A short decision procedure, because five methods with overlapping benefits is where good engineering goes to die.

Start by asking whether you can change the model at all. If the checkpoint is frozen — and it usually is — four of the five closing methods are unavailable and the question reduces to “centering or whitening?” Use centering. It has one estimated parameter per modality, it cannot blow up, and it delivers most of the benefit. Reach for whitening only when you have measured that the two modalities have genuinely different covariance shapes and you have thousands of reference samples and a shrinkage estimator.

Then ask whether you need the embeddings changed, or only the scores. This is the distinction most teams get wrong. If your problem is “images never surface in a mixed ranked list,” you do not need to touch the embeddings at all — calibrate the scores, which is Chapter 7, and keep your index bit-for- bit identical. Rewriting an index is a migration; rewriting a scoring function is a deploy.

Only if you are training the encoder should you consider the loss-side fixes, and then measure uniformity rather than gap distance, for the reason Chapter 8 gives.

Two, six, or a hundred modalities

The exploiting camp has a strong argument from an unexpected place: ImageBind. It binds six modalities — image, text, audio, depth, thermal, IMU — into one space, training each one only against images, never against each other. And yet audio-to-text retrieval works, with no audio–text pair ever seen. That is called emergent alignment, and it is the best evidence available that gaps do not prevent cross-modal function.

Geometrically, ImageBind has not two cones but six, each with its own axis and therefore its own gap vector to every other. Fifteen pairwise gaps. If gaps were fatal, nothing about that system would work. It works because — per Claim A — retrieval within a fixed gallery is invariant to a constant offset, so six disjoint cones behave, for ranking purposes, exactly like one.

Chapter 7: Search Calibration

This is the chapter you will actually use. Everything before it was diagnosis; this is treatment, on the system where the gap does real damage: a search index containing more than one kind of thing.

The setup is completely ordinary. You have internal documentation: text passages and architecture diagrams. You embed all of it with CLIP into one vector index — text through the text tower, images through the image tower, everything into the same 512-dimensional space, because that is what a joint embedding space is for. A user types “diagram of a transformer block.”

They get five text passages. They never get the diagram. The diagram is right there, correctly embedded, and it is invisible.

The five candidates, with real numbers

Here is the state of the index at query time. Two pools with completely different score distributions, which is the whole problem:

Raw ranking: A, B, C, D, E. The correct answer is fourth, behind three passages that are not about diagrams at all. And crucially it is not close — C beats D by 0.27, roughly seven standard deviations of the image pool. No reranker downstream of a top-3 cut will ever see D.

Fix 1: per-modality centering of scores

The cheapest correction. Subtract each pool’s mean score:

New ranking: D, A, B, E, C. The diagram is first. One subtraction, no model changes, no retraining. Note this is score-space centering, which is subtly different from the embedding-space centering of Chapter 6 — it is cruder (it cannot change the within-modality ordering at all) and correspondingly safer.

Fix 2: per-modality z-scores

Centering assumes the two pools have the same spread. They do not: 0.05 against 0.04. Divide by the standard deviation too:

Ranking: D (2.25), A (1.20), E (0.50), B (0.40), C (−0.20). D wins by nearly a full point of z now, and the ordering is defensible: the one image that is unusually good for its pool beats the one passage that is unusually good for its pool.

A z-score also gives you a portable threshold. “Return anything above z = 1.5” means the same thing in both modalities and survives a checkpoint change, because μ and σ are re-estimated per checkpoint. “Return anything above cosine 0.5” means two completely different things and survives nothing.

Fix 3: rank fusion

Both fixes above assume the score distributions are roughly Gaussian. If yours are skewed, throw the scores away and keep only the ranks. Score each pool separately, then merge with reciprocal rank fusion:

Since every item appears in exactly one list, RRF here reduces to interleaving: best text, best image, second text, second image. That is a policy, not an inference — it guarantees images appear, and it guarantees they appear even when there are no good images. Use it when you want a coverage floor per modality and you do not trust the scores at all.

Fix 4: per-modality quotas plus a learned calibration

The production answer, if you have even a hundred labelled examples per modality. Fit a one-dimensional logistic calibration per modality mapping raw cosine to probability of relevance:

Now the scores are comparable because they are both probabilities of the same event. Sort by that. This is strictly better than z-scoring — z-scoring is the special case where you assume both pools have the same relevance rate and a Gaussian shape — and it costs one sklearn call. It also gives you a threshold with a meaning your product manager understands: “show it if it is more than 60% likely to be relevant.”

Concept → realization: the calibrated scorer

Everything above, as the object you would actually deploy. Note where the statistics live — frozen, alongside the index, versioned with the checkpoint — because that is the part people get wrong.

python
import numpy as np, json

class ModalityCalibrator:
    # Fitted ONCE per checkpoint on a fixed reference corpus, then frozen
    # and shipped next to the index. Never re-estimated from a query batch.
    def fit(self, scores_by_modality):
        self.stats = {m: {'mu': float(np.mean(s)), 'sd': float(np.std(s) + 1e-6)}
                      for m, s in scores_by_modality.items()}
        return self

    def score(self, raw, modality):
        st = self.stats[modality]
        return (raw - st['mu']) / st['sd']        # comparable across modalities

    def save(self, path):
        json.dump(self.stats, open(path, 'w'))     # version this WITH the checkpoint

# reference corpus: a few thousand items per modality, sampled from YOUR index,
# scored against a few hundred representative queries. Refit whenever either
# the checkpoint or the corpus composition changes materially.
cal = ModalityCalibrator().fit({'text': ref_text_scores, 'image': ref_image_scores})
ranked = sorted(hits, key=lambda h: -cal.score(h.cos, h.modality))

Three implementation details that decide whether this works in production.

Do not estimate μ and σ from the retrieved candidates. It is tempting — you already have the scores — but the retrieved set is exactly the biased top of the distribution, so the estimate moves with the query and your scores stop being comparable between queries. Estimate on a fixed reference sample and freeze.

How many reference samples? The standard error of a mean is σ/√n. With σ ≈ 0.05 and a target accuracy of 0.002 on the mean — small compared to the 0.37 difference you are correcting — you need n = (0.05/0.002)² = 625 per modality. A couple of thousand is comfortable; a hundred is not.

You cannot z-score inside an approximate nearest-neighbour index. HNSW and IVF search on the raw vectors and return the top k by raw score — the calibration happens after retrieval, by which point the images have already been filtered out. If your index is a single mixed shard and you ask for k = 50, all fifty slots go to text and no amount of re-scoring can recover an image that was never returned. The fix is one of two things, and you must choose deliberately:

The arithmetic is stark: single mixed shard at k = 50 returns 0 images; two shards at k = 25 each returns 25 images and the calibrator decides which of them deserve to be shown. Retrieval recall is a property of the index; ranking quality is a property of the scorer. The gap breaks the first one and no scorer can repair it after the fact.

A post-mortem, in the order it actually happens

The failure never announces itself. Here is how the ticket reads and how it resolves.

Day 1. “Image search doesn’t work.” Someone verifies that the diagrams are in the index — they are — and that the image embeddings are non-zero — they are. The model is declared “bad at diagrams” and a ticket is filed to fine-tune it.

Day 3. Somebody tries the query “transformer block diagram” against only the image shard and the correct diagram comes back first, with a cosine of 0.34. The model is fine. The retrieval is fine. Something about the mixture is broken.

Day 3, ten minutes later. Print the mean score per modality: 0.62 and 0.25. Done. The diagnosis is one groupby, and it would have been day 1 if anyone had thought to look at the score distributions before looking at the model.

The threshold that follows you across checkpoints

One more trap, and it has bitten more teams than the mixed-index bug because it fails silently.

You calibrate a relevance threshold on ViT-B/32: cosine 0.28 means “relevant.” It works, it ships, it sits in a config file for a year. Then someone upgrades to a larger checkpoint for better accuracy. The new model has a different gap, so its matched pairs sit at a different absolute cosine — and your threshold, which encoded “a bit above the gap floor,” now encodes something else entirely. Recall collapses, or precision does, and nothing in the diff explains it.

The operational rule: never hard-code an absolute cosine anywhere. Store μ and σ per modality per checkpoint, computed on a fixed reference corpus, and express every threshold relative to them. That one discipline immunizes you against every failure mode in this chapter.

Chapter 8: Does It Matter?

Time to be honest. We have spent seven chapters on a geometric fact that is real, large, and easy to measure. Now the question a senior engineer asks: so what? Should you do anything about it, and what happens if you do nothing?

The literature genuinely disagrees, and the disagreement is productive rather than confused. Here are both cases, as strongly as each can be made.

The case that it does not matter

1. Ranking within a gallery is provably invariant. This is not an empirical claim, it is Claim A from Chapter 3. Add a constant to every candidate in a fixed gallery and the ordering cannot change. Every cross-modal retrieval benchmark, every zero-shot classification benchmark that argmaxes over prompts, and every top-k nearest-neighbour lookup falls into this category. The gap is exactly zero cost for all of them.

2. CLIP works. The strongest evidence is the seven years of results. Zero-shot ImageNet, retrieval on COCO and Flickr, CLIP-guided diffusion, CLIPScore as a captioning metric, ImageBind’s six-way emergent alignment — all built on top of a wide-open gap that nobody noticed for the first year. A defect that costs nothing on every headline benchmark deserves the label “cosmetic” until proven otherwise.

3. Closing it barely helps. Papers that close the gap — by centering, by penalty terms, by fine-tuning — report retrieval changes in the range of one or two points, and not always positive. If removing a defect that accounts for 29% of your embedding variance moves Recall@1 by a point, the defect was not carrying much function.

4. Linear transfer costs one scalar. Chapter 6: a text-trained linear probe applied to images picks up exactly the bias w · Δ, which for a semantic direction in 512 dimensions is about 0.036. Cross-modal transfer of linear heads is nearly free, gap or no gap.

The case that it absolutely does

1. Mixed-modality search is broken, not degraded. Chapter 4 computed it exactly: the modality term was seven times the semantic term, so every text outranks every image regardless of content. This is not a percentage-point regression, it is a total failure of a feature, and it is invisible in every benchmark because every benchmark ranks within one modality.

2. Absolute thresholds are meaningless and portable ones are not obvious. The most common CLIP-in- production pattern — “is this pair a match, yes or no?” for moderation, deduplication, auto-tagging, or filtering a training set — needs a number, and the number depends on a geometry nobody documented. Chapter 7’s checkpoint-upgrade failure is the classic version.

3. Anything that mixes embeddings across modalities is nonsense without correction. Averaging an image embedding and a text embedding to form a “multimodal query” produces a vector on the segment between the two islands — a region no real embedding ever occupies, and one your index has never seen. So does interpolating for a slider UI, so does adding a text “edit direction” to an image embedding, so does mean-pooling a multimodal document into one vector.

4. The gap has social consequences. Liang and colleagues showed that shifting along Δ̂ changes not just accuracy but rates of harmful misclassification on face datasets. A geometric parameter nobody knew existed, set by random initialization, is quietly modulating who gets mislabelled as what.

The third position: the gap is a symptom, not the disease

A more recent line of work argues both camps have the wrong object. Two threads.

The gap is not one vector. We measured this ourselves in Chapter 3: the shared Δ explains only about half of the image-to-text displacement. Work under titles like “It’s Not a Modality Gap” argues that the real pathology is a uniformity failure — each modality occupies a much lower-dimensional sliver of the sphere than it should, in the sense of Wang and Isola’s alignment-versus-uniformity decomposition of contrastive learning. Translation does not fix low-dimensionality; only training or whitening does. This explains the disappointing one-point gains: people have been subtracting a mean when the problem was a rank deficiency.

The gap is driven by information imbalance. Schrodi and colleagues argue the gap emerges from a mismatch in how much information each modality carries about the pair — an image contains far more detail than its caption, so no amount of alignment pressure can make the two representations coincide, and the residual shows up as a separation. On this view the gap is a measurement of an irreducible asymmetry rather than an artefact you can remove, and the crucial follow-up is that closing it does not fix the downstream biases people hoped it would.

Measuring uniformity, so you can argue with numbers

The “symptom, not disease” camp rests on a quantity you can compute in three lines, so compute it. Wang and Isola define uniformity as the log of the average Gaussian-kernel similarity between pairs — lower is better, meaning the points are spread out:

For unit vectors, ‖x−y‖² = 2 − 2 cos(x,y), so we can evaluate it from the same four numbers we have been using all lesson. Within-image pairs sit at cosine 0.55:

Cross-modal pairs sit at 0.25:

Roughly half the pairs in the pooled cloud are within-modality and half are cross-modality, so the average is about (0.165 + 0.0498)/2 = 0.107 and

Now the reference point. On a truly uniform 512-dimensional sphere, the squared distance between two random points concentrates tightly at 2, giving

So CLIP sits at −2.23 against an ideal of −4.00, and — here is the point — look at which term is responsible. The within-modality term, 0.165, is more than three times the cross-modal term, 0.0498. The uniformity deficit is dominated by points being crowded inside each cone, not by the cones being apart. Translating one cone onto the other does not change any within-modality distance at all, so it cannot improve this number.

Run the ablation yourself

You do not have to take anyone’s side. The experiment takes twenty minutes on a frozen checkpoint and it settles the question for your system, which is the only version of the question that matters.

python
# Four measurements. Run them in this order; stop as soon as one is decisive.

# A. Is there a gap at all, and how big?
delta = I.mean(0) - T.mean(0);  gap = delta.norm()          # expect 0.7 – 0.9

# B. Is your operation rank-based? Translate the GALLERY, do not renormalize.
#    If top-k is byte-identical, the gap is provably not your bug. Stop here.
base = (Q @ T.T).topk(10).indices
shft = (Q @ (T + delta).T).topk(10).indices
print('rank-invariant:', torch.equal(base, shft))                # True, always

# C. Does calibration change your metric? Cheapest real intervention.
evaluate(rank_raw(hits))
evaluate(rank_zscored(hits, per_modality_stats))

# D. Does closing the gap in EMBEDDING space help beyond calibration?
Ic = renorm(I - I.mean(0));  Tc = renorm(T - T.mean(0))
evaluate(rank_raw(retrieve(Ic, Tc)))          # usually within ±1 point of C

The order is deliberate and it is a claim about where the value is. Test B costs nothing and eliminates most systems. Test C is a scoring change, deployable in an afternoon, and captures nearly all the available gain. Test D is an index migration and, on the published evidence, buys about a point beyond C. If your results come out differently, you have learned something specific about your corpus — which is the whole reason to run it rather than read about it.

A decision table you can actually use

Chapter 9: Connections

Nothing new here — just the map, the limits of what we built, and where each thread continues.

The whole lesson in eight lines

1. In CLIP’s shared space, an image and its own caption sit at cosine ~0.30, while two unrelated images sit at ~0.55. Within-modality similarity beats every cross-modality similarity. The space is two islands.

2. The gap vector Δ = c_I − c_T has norm 0.82 for ViT-B/32, recoverable from four averaged cosines, and “which modality” explains about 29% of the total embedding variance. The two modalities are 100% linearly separable and x · Δ̂ is a free modality detector.

3. The gap is not learned. Random deep networks squeeze everything into a narrow cone — one ReLU takes the expected cosine between unrelated inputs from 0 to 1/π = 0.318, and the arc-cosine recursion drives it to 0.83 by depth eight. Two encoders, two seeds, two near-perpendicular cone axes, gap at birth.

4. Training cannot remove it because the softmax only sees differences between logits, and closing the gap adds nearly the same amount to every logit in a row. The gap sits in a flat valley.

5. Temperature decides how flat. At the shipped τ = 0.01, once the margin exceeds τ the softmax saturates and the gap-closing force — a covariance between softmax weight and position along Δ̂ — collapses by about 54,000×.

6. The rigid-translation model is half right: Δ explains 0.67 of the 1.40 mean squared displacement. That single number predicts why centering gets you halfway and why CapDec needs noise of scale ~0.04 per coordinate.

7. Ranking within a gallery is invariant to the gap. Absolute thresholds, mixed-modality indexes, and cross-modal averaging are not. That line divides the systems you should fix from the ones you should leave alone.

8. Never hard-code an absolute cosine. Store μ and σ per modality per checkpoint and express every threshold relative to them.

What we did not cover

Keep exploring

← Contrastive Learning & CLIP — the model whose geometry this whole lesson dissects
← Contrastive Learning — InfoNCE, negatives, and the alignment-versus-uniformity view of the objective
← Vector Embeddings — what an embedding is before we start worrying about where it lives
← Similarity Metrics — cosine, dot product, Euclidean, and why normalization changes the question
→ Vector Databases — where the mixed-index failure of Chapter 7 actually happens
→ Multimodal RAG — retrieval over text and images together, the system this lesson is a prerequisite for
→ Embedding Benchmarks — why leaderboard numbers hide calibration problems like this one
→ Multimodal Foundation Models — the wider family of joint-space designs
→ Vision-Language Models — the generative branch, where a trained adapter changes the geometry story
→ ImageBind — six modalities, fifteen pairwise gaps, and emergent alignment anyway
→ SigLIP — the sigmoid loss, and the learnable bias that absorbs the offset
→ OpenCLIP — the open reproduction you can measure your own gap on
→ EVA-CLIP — scaling the recipe, with the gap intact

References

Liang, Zhang, Kwon, Yeung-Levy, Zou. “Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning.” NeurIPS 2022. arXiv:2203.02053 — the source of the gap distance, the cone-effect analysis, and the shift-along-Δ experiments.
Radford et al. “Learning Transferable Visual Models From Natural Language Supervision.” ICML 2021. arXiv:2103.00020 — CLIP itself, including the learnable temperature clamped at 1/τ = 100.
Cho, Saul. “Kernel Methods for Deep Learning.” NeurIPS 2009. proceedings — the arc-cosine kernel, which gives the exact depth recursion in Chapter 2.
Wang, Isola. “Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere.” ICML 2020. arXiv:2005.10242 — the decomposition the “uniformity, not translation” argument rests on.
Ethayarajh. “How Contextual are Contextualized Word Representations?” EMNLP 2019. arXiv:1909.00512 — anisotropy in language models: the cone effect, discovered a modality earlier.
Gao et al. “Representation Degeneration Problem in Training Natural Language Generation Models.” ICLR 2019. arXiv:1907.12009 — the same narrow cone in word embeddings, with a training-time explanation.
Mu, Bhat, Viswanath. “All-but-the-Top: Simple and Effective Postprocessing for Word Representations.” ICLR 2018. arXiv:1702.01417 — the original centering-and-remove-top-components recipe reused in Chapter 6.
Nukrai, Mokady, Globerson. “Text-Only Training for Image Captioning using Noise-Injected CLIP” (CapDec). EMNLP Findings 2022. arXiv:2211.00575 — exploiting the gap: the noise-ball trick and its scale.
Li et al. “DeCap: Decoding CLIP Latents for Zero-shot Captioning via Text-Only Training.” ICLR 2023. arXiv:2303.03032 — the projection-onto-a-support-set alternative to noise injection.
Schrodi et al. “Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models.” ICLR 2025. arXiv:2404.07983 — the information-imbalance account, and the argument that closing the gap does not fix what people hoped.
Fahim, Murphy, Fyshe. “It’s Not a Modality Gap: Characterizing and Addressing the Contrastive Gap.” 2024. arXiv:2405.18570 — the uniformity-deficiency reframing of the gap.
Zhai et al. “Sigmoid Loss for Language Image Pre-Training” (SigLIP). ICCV 2023. arXiv:2303.15343 — the pairwise sigmoid objective and its learnable bias.
Girdhar et al. “ImageBind: One Embedding Space To Bind Them All.” CVPR 2023. arXiv:2305.05665 — six cones, emergent alignment, and the best evidence that gaps do not stop retrieval.