What Contrastive Loss Actually Optimizes

Chapter 1: Why the Sphere

Every contrastive method in this literature does the same small thing right before computing the loss: it divides the feature vector by its own length.

Here h is the network — ResNet plus a projection MLP — and f is what actually enters the loss. One division. It looks like a numerical-stability nicety, the sort of line you would skim past. It is the reason the entire paper is possible, and this chapter is about why.

What the object is

The set of all unit-length vectors in R^m is the unit hypersphere, written S^m−1:

The superscript is m−1, not m, because the surface has one dimension fewer than the space it sits in. In R³, the surface of a ball is S² — two-dimensional, which is why a map of the Earth is a flat sheet. For a 128-dimensional projection head, features live on S¹²⁷.

Two identities do all the work in this lesson, so derive them once and never again. For unit vectors u and v:

The dot product is the cosine of the angle between them, because both lengths are 1. And the squared distance:

Expand the bracket, use ‖u‖² = ‖v‖² = 1, and you are done. This tiny identity is the hinge on which Chapter 4's theorem turns: squared distance and dot product are the same quantity, affinely rescaled. Anything written with one can be rewritten with the other.

Memorise the last row. The maximum possible squared distance on a unit sphere is 4. Every number in Chapters 3 and 5 lives in [0, 4], and knowing the ceiling makes the exponentials legible at a glance.

Reason 1: without normalisation, the norm is a free temperature dial

This is the argument that most people have not actually worked through, so let us do the arithmetic.

The contrastive loss for one anchor with one positive and M negatives is

where s is a similarity score and τ (tau) is the temperature, a positive scalar that sharpens the softmax as it shrinks. Suppose the network outputs unnormalised vectors and we use the raw dot product s = h(x) · h(z). Take a concrete arrangement: all vectors at radius r, the anchor at angle 0°, the positive at 20°, and three negatives at 80°, 150°, and 200°. Set τ = 0.5.

The raw dot product between two vectors of length r at angle θ is r²cosθ. So every logit scales with r². At r = 1:

arithmetic — unit-length features, tau = 0.5cosines:   pos  cos(20)  = 0.9397
           neg1 cos(80)  = 0.1736
           neg2 cos(150) = -0.8660
           neg3 cos(200) = -0.9397

logits = cos / 0.5 = [ 1.8794 , 0.3473 , -1.7321 , -1.8794 ]
exps   =             [ 6.5495 , 1.4152 ,  0.1769 ,  0.1527 ]
sum    = 8.2943
loss   = -ln(6.5495 / 8.2943) = -ln(0.789634) = 0.2362

Now scale every vector to r = 2 — without moving a single one of them angularly. Every logit is multiplied by r² = 4:

arithmetic — same angles, radius 2logits = 4 * [ 1.8794 , 0.3473 , -1.7321 , -1.8794 ]
       =     [ 7.5175 , 1.3892 , -6.9282 , -7.5175 ]
exps   =     [ 1840.04 , 4.0116 ,  0.00098 , 0.00054 ]
sum    = 1844.050
loss   = -ln(1840.04 / 1844.050) = -ln(0.997824) = 0.0022

Normalisation closes that escape hatch. On the sphere, s = cosθ is bounded in [−1, 1] and depends on nothing but the angular arrangement. The only way to reduce the loss is to move points relative to each other — which is to say, to learn.

Reason 2: a compact space is the only place "uniform" means anything

This reason is quieter and more fundamental. The paper's entire second metric is "the features should be uniformly distributed". Ask yourself what that would mean in R¹²⁸.

It means nothing. There is no uniform probability distribution on an unbounded space. Try to build one: assign equal density everywhere, integrate over infinite volume, and the total is infinite, not 1. You cannot normalise it. Uniform on R^m does not exist.

On a sphere it does exist, and it is unique. The sphere is compact (closed and bounded — finite surface area) and homogeneous (every point looks like every other point; rotations move any point to any other). Those two facts together give you exactly one rotation-invariant probability measure, the normalised surface measure, written σ_m−1. It is the thing you get by picking a point "at random on the sphere."

Reason 3: bounded similarities keep the exponential from exploding

A practical point, but a real one. The loss exponentiates s/τ. With τ = 0.07 — SimCLR's setting for some configurations — and unnormalised features whose dot products can reach 40, you are computing e⁵⁷¹. That overflows a float32 (which tops out near e⁸⁸) and you get inf, then nan, then a dead run.

Normalised, the extreme case is e^1/0.07 = e^14.29 ≈ 1.6 × 10⁶. Comfortable. This is why models like CLAP and CLIP that learn the temperature also clamp it: the bounded similarity gives you a known worst case, and you keep it that way.

Reason 4: the sphere is a good place for a linear probe

The last reason is about the downstream task, and it is the one the paper leans on when it argues that these properties should help, not just that they are what the loss does.

A linear probe fits w and asks whether w · f(x) separates the classes. On the sphere, w · f(x) = ‖w‖ cos(angle between w and the feature). Up to the constant ‖w‖, a linear classifier on the sphere is an angular threshold: it carves the sphere with a hyperplane through the origin, producing two caps. If a class occupies a compact cap, a linear probe finds it.

That is precisely the arrangement alignment produces: each semantic group compressed into a small region. And uniformity ensures those regions are pushed apart rather than piled on top of each other. The two properties are not arbitrary aesthetic preferences — they are exactly the conditions under which the linear probe you are going to run at evaluation time can succeed.

An uncomfortable fact about high-dimensional spheres

One more piece of geometry, because it explains temperature settings you have probably copied without understanding. Take two points drawn independently and uniformly from S^m−1. What is their dot product, typically?

By symmetry the mean is 0. The variance works out to exactly 1/m. So the typical magnitude of the cosine between two random features is

At m = 128, the spread of the "background" similarities is about 0.09. Now recall the loss divides by τ. If τ = 1, the logits from random negatives span roughly ±0.09 — the softmax over them is nearly flat, and the loss barely distinguishes anything. If τ = 0.07, that same spread becomes ±1.3 in logit space, which is a meaningful softmax. The temperature's job is to rescale the natural angular noise floor of the sphere into a usable dynamic range, and the noise floor is set by the dimension.

How much room is there, actually?

Uniformity is a claim that spreading features out preserves information. That claim is only interesting if the sphere has enough room to be worth spreading into, so let us measure the room.

The tool is a concentration inequality. For u and v drawn independently and uniformly from S^d−1, the probability that their cosine exceeds a threshold ε falls off exponentially in the dimension:

Read the right side as "the fraction of the sphere lying inside a cap of cosine radius ε around any fixed point." Evaluate it at d = 128:

The same inequality explains a second thing you have certainly noticed: contrastive models trained in high dimension report very small cosine similarities between unrelated items — 0.05, 0.1 — and beginners read that as "the model thinks nothing is similar to anything." It does not. Those numbers are the background level of a high-dimensional sphere, and a cosine of 0.3 that would look unimpressive in two dimensions is, at d = 128, a one-in-three-hundred event. Always calibrate similarity scores against 1/√d, never against 1.

Normalise what, exactly?

One implementation detail that trips people up. Modern contrastive models have two stages: a backbone h producing, say, 2048-dimensional features, and a small projection head g producing 128 dimensions. The normalisation goes on the output of g, and the loss operates there. But the representation you actually use downstream is the backbone output h, before the head.

So every property this paper analyses — alignment, uniformity, position on the plane — is a property of a space that gets thrown away at deployment. That sounds absurd and is one of the field's genuinely odd empirical facts: features before the projection head consistently give better linear probes than features after it. Chapter 8 revisits this with the alignment-uniformity lens, which gives it a satisfying explanation. For now, just be precise about which vector you are measuring, and report it, because the two spaces have very different metric values.

Summary before we build

Chapter 2: Alignment

Alignment is the easy half, and it is worth doing slowly anyway, because the easy half contains the one parameter people misconfigure.

Where positive pairs come from

Everything starts with a distribution the paper calls p_pos: the distribution over pairs (x, y) that we have declared to be "the same thing." In vision, x and y are two random augmentations of one image — a crop, a colour jitter, a flip. In the sentence-embedding setting they might be two adjacent sentences. In CLIP-style models they are an image and its caption.

The paper requires one technical condition on p_pos, and it is worth naming because it is easy to violate by accident: the two marginals must match. If you look only at x, you should see p_data; if you look only at y, you should also see p_data. Formally p_pos is symmetric and both of its marginals equal p_data.

The metric, derived

We want a number that is small when positive pairs land in the same place. The obvious candidate is the distance between them. Average it over the pair distribution and you are done:

Three observations, each of which you should be able to reconstruct.

It is bounded. On the sphere, distances live in [0, 2], so ℓ_align ∈ [0, 2^α]. For the default α = 2, the range is [0, 4]. A value of 0 means perfect alignment. A value of 4 means every positive pair is antipodal, which would take active malice to achieve.

Its minimum is exactly "perfectly aligned." The metric is zero if and only if f(x) = f(y) almost surely for positive pairs — the paper's definition of a perfectly aligned encoder. Since a distance is non-negative and its average is zero only when it is zero almost everywhere, there is no slack in this statement.

It says nothing about anything else. The constant encoder f(x) = u₀ achieves ℓ_align = 0. So does an encoder that maps every cat and every truck to the same point. Alignment is a purely local property of the pair distribution and is completely blind to whether the representation retains information. Hold that thought; it is why Chapter 3 exists.

What α actually controls

Now the part people get wrong. α is not a "strength" knob — that is what λ is for when you combine the two losses. α changes which positive pairs the gradient cares about.

Take two positive pairs in a batch. Pair A landed 0.2 apart (already well aligned). Pair B landed 1.2 apart (badly aligned — roughly 70°). Compute the loss under three settings of α:

arithmetic — two positive pairs at distances 0.2 and 1.2alpha = 0.5:   (0.2^0.5 + 1.2^0.5)/2 = (0.4472 + 1.0954)/2 = 0.7713
alpha = 1.0:   (0.2     + 1.2    )/2 = (0.2000 + 1.2000)/2 = 0.7000
alpha = 2.0:   (0.2^2   + 1.2^2  )/2 = (0.0400 + 1.4400)/2 = 0.7400

The loss values are almost the same. That is not the interesting part. The interesting part is the derivative, because the derivative is what gradient descent uses. For a single pair at distance d, the contribution is d^α, so

Evaluate that at both distances:

Alignment as invariance — and the trap inside it

There is a second, more conceptual way to read this metric. Because p_pos is generated by augmentation, saying "positive pairs map to the same point" is the same as saying the encoder is invariant to the augmentation distribution. Whatever the augmentation destroys, the encoder is being told to ignore.

This is where the semantics secretly enter. The loss has no idea what a dog is. But the crop-and-jitter pipeline says: colour is not identity, position is not identity, scale is not identity, but texture and shape are. Alignment then forces the encoder to build features that survive exactly those nuisances. The augmentation set is the entire prior, smuggled in through p_pos.

What the number looks like in practice

A metric with no reference values is useless, so fix the scale before you ever log one.

The no-information reference is 2.0. If the encoder's output were statistically independent of its input — a random projection of noise — then f(x) and f(y) are two independent uniform points on the sphere, so E[f(x)·f(y)] = 0 and

That is the same 2.0 as the "untrained" corner of Chapter 7's plane, and it holds at any dimension. A freshly initialised network usually sits a little below it, because random convolutional features of two crops of the same photo are weakly correlated, but 2 is the ceiling you should think of.

The perfect-invariance reference is 0.0, achievable and undesirable on its own.

Trained models land between roughly 0.2 and 0.8, and where exactly depends almost entirely on how aggressive your augmentations are, not on how good your model is. That is the single most important thing to internalise about this metric: ℓ_align is not a quality score. A model trained with weak augmentations will have better alignment and worse features. Only compare alignment values across runs that share an augmentation pipeline.

Alignment is not intra-class compactness

A frequent confusion, and clearing it up sharpens what the objective actually is.

Alignment is defined over positive pairs, which in self-supervised learning means augmentations of the same instance. It says nothing about whether two different photographs of golden retrievers land near each other. As far as ℓ_align is concerned, those two photos are unrelated inputs, and Chapter 3's uniformity term will actively push them apart.

That the resulting features are nonetheless class-compact is an emergent phenomenon, driven by the fact that a finite-capacity network cannot separate two golden retrievers as cheaply as it can separate a retriever from a fire truck. The objective supplies a uniform push; the architecture decides where the push is easy. Chapter 8 returns to this as the framework's central unexplained step, and it is the reason a supervised contrastive loss — which redefines p_pos to include same-label pairs — is a genuinely different objective rather than a tweak.

The identity that makes Chapter 4 work

One last piece of bookkeeping, and it is the reason α = 2 is the natural default rather than an arbitrary one. Use the identity from Chapter 1:

Take the expectation over positive pairs and divide by τ:

Read the left side: it is "minimise the negative cosine of positive pairs", which is exactly the numerator of the contrastive loss in disguise. Read the right side: it is the alignment metric with α = 2, multiplied by a positive constant, minus a constant that does not depend on f.

These are the same objective. Not similar — identical, up to an affine transformation that cannot change which encoder is optimal. When Chapter 4 splits the contrastive loss in two, the first half is not "something like alignment". It is ℓ_align(f; 2), and that is why 2 is the canonical exponent.

What perfect alignment does to the sphere

Picture the endgame. If ℓ_align = 0 exactly, then every positive pair sits on the same point. Positive-pairing induces an equivalence relation on the data — "x ~ y if they are augmentations of the same underlying image" — and a perfectly aligned encoder is constant on each equivalence class. So the feature cloud is not really a cloud at all; it is a finite set of atoms, one per image in the dataset.

That is worth sitting with. A perfectly aligned encoder on a dataset of N images produces at most N distinct points on the sphere, no matter how rich the network. The continuous-looking feature distribution you see in a t-SNE plot is a consequence of imperfect alignment, not of the objective. And the question of where those N atoms should go — that is precisely the question uniformity answers, and precisely the question that classical physicists have been asking about electrons on a sphere for a century.

Chapter 3: Uniformity

Alignment took one paragraph to motivate and one line to write. Uniformity takes a chapter, because "spread the points out" is a phrase with dozens of plausible formalisations and almost all of them are broken. Watching two of them break is the fastest way to understand why the paper's choice is the right one.

Attempt 1: maximise the average pairwise distance

The most natural idea. If you want points spread out, push apart every pair. So define a loss that minimises the negative average squared distance, i.e. maximises

where x and y are drawn independently. Expand using Chapter 1's identity, and remember that for independent draws the expectation of a product is the product of expectations:

derivation — average pairwise squared distance on the sphereE[ ||u - v||^2 ]  =  E[ 2 - 2 u.v ]
                  =  2 - 2 E[u . v]
                  =  2 - 2 E[u] . E[v]        # u, v independent
                  =  2 - 2 ||E[u]||^2         # same distribution, so E[u] = E[v]

Stop and read that last line, because it is devastating. The average pairwise distance depends on the feature distribution only through its mean vector. Every other detail — shape, spread, number of modes, dimension actually used — is invisible to it. Maximising the average distance is exactly and only the instruction "make the mean zero."

Here is the counterexample that kills it. Put half your probability mass on a single point u₀ and the other half on −u₀. Total collapse to two atoms — the representation retains exactly one bit about the input.

Verify the two-atom row by hand rather than trusting the formula. Draw two points independently: with probability ½ they land on the same atom (distance² = 0) and with probability ½ on opposite atoms (distance² = 4). Average: ½(0) + ½(4) = 2. Identical to the uniform distribution's score. The objective cannot tell them apart.

Attempt 2: maximise the entropy of the feature distribution

The theoretically correct thing to want. The uniform distribution on a compact space is precisely the maximum-entropy distribution, so "maximise differential entropy" has exactly the right answer.

It is also unusable, for three reasons that any practitioner will recognise. Estimating differential entropy in 128 dimensions from a batch of 256 samples is statistically hopeless — nearest-neighbour estimators have variance that swamps the signal at these sample sizes. The estimators that do exist are not smoothly differentiable, so backpropagating through them is painful. And entropy on a manifold requires care about the reference measure, which invites subtle bugs.

We want something with entropy's minimiser and a sample mean's tractability. That is what a pairwise potential gives you.

The paper's answer: the pairwise Gaussian potential

Define the Gaussian potential kernel (also called the radial basis function or heat kernel) between two points on the sphere:

Think of it as a proximity score with a tunable reach. Two points on top of each other score G = 1. Points at maximum separation (squared distance 4) score e^−4t, which at t = 2 is 0.000335 — effectively zero. In between it falls off smoothly.

Now average this over all pairs of independently drawn data points and take the logarithm:

This is the total pairwise potential energy of the feature cloud, log-scaled. Every pair of points that sits close together contributes energy; pairs that are far apart contribute nothing. Minimising it means arranging the points so that no two of them are crowded.

Why this kernel: the uniqueness result

Here is the theorem that justifies the whole construction. The paper states it as Proposition 1.

Unique is a strong word and it is meant literally: not "one of the minimisers", not "a minimiser up to rotation" — there is exactly one distribution that achieves the minimum, and it is the one you wanted. Let us see why, in two moves.

Move 1: the energy is strictly convex. A kernel K is called strictly positive definite if, for every signed measure ν that is not zero but has total mass zero, the double integral ∫∫ K dν dν is strictly positive. The Gaussian kernel has this property on the sphere. Now take two different distributions μ₀ and μ₁, and interpolate: μ_λ = (1−λ)μ₀ + λμ₁. Write ν = μ₁ − μ₀, which has total mass 1 − 1 = 0. Expanding the energy E(μ) = ∫∫G_t dμ dμ:

The coefficient of λ² is strictly positive by strict positive definiteness. So E, viewed along any straight line between two distinct distributions, is a strictly convex parabola. A strictly convex function on a convex set has at most one minimiser. That is the uniqueness, and it came entirely from one property of the kernel.

Move 2: symmetry forces the minimiser to be uniform. G_t(u,v) depends only on the distance between u and v, so rotating both points leaves it unchanged. Therefore if you rotate the whole distribution, its energy is unchanged: E(R#μ) = E(μ) for every rotation R. Suppose μ* is the unique minimiser. Then R#μ* is also a minimiser, so by uniqueness R#μ* = μ*, for every rotation. A probability measure on the sphere invariant under all rotations is the surface measure, and nothing else. Hence μ* = σ_m−1. ∎

The same lens explains why Attempt 1 failed

This is the satisfying part. The linear "spread" objective −E[u·v] is a kernel too, and its harmonic spectrum has exactly one non-zero entry: degree 1. It sees the mean and is blind to every other degree, which is why a two-atom distribution slips past it undetected. The Gaussian kernel's spectrum is positive at every degree, so no distribution can hide from it.

Run the numbers side by side, on the circle, with t = 2. The uniform distribution's Gaussian energy on S¹ has a closed form — E_σ[G_t] = e^−2t I₀(2t), where I₀ is the modified Bessel function of the first kind:

arithmetic — uniform vs. two-atom collapse, t = 2# Two antipodal atoms, half mass each.
# Independent draws: P(same atom) = 1/2 -> d^2 = 0 ; P(opposite) = 1/2 -> d^2 = 4
E[G_2] = 0.5 * exp(0) + 0.5 * exp(-8)
       = 0.5 * 1.000000 + 0.5 * 0.000335
       = 0.500168
l_uniform = ln(0.500168) = -0.6928

# Genuinely uniform on the circle.
E[G_2] = exp(-2t) * I_0(2t) = exp(-4) * I_0(4)
       = 0.0183156 * 11.301922
       = 0.207001
l_uniform = ln(0.207001) = -1.5750

# Gap = 0.882 nats. The Gaussian kernel is not fooled.
# Under the LINEAR objective both score exactly 2.000. Tie.

Deriving the uniform reference value by hand

The number −1.5751 in that code block deserves to be earned rather than quoted, because it is the reference against which every other configuration is judged, and the derivation is a nice piece of undergraduate calculus.

On the circle, parametrise a point by its angle. Two independent uniform draws differ by an angle θ that is itself uniform on [0, 2π). Their squared distance is 2 − 2cosθ, so

derivation — the uniform distribution's Gaussian energy on the circleE[G_t]  =  (1/2pi) INTEGRAL_0^{2pi} exp( -t (2 - 2 cos(theta)) ) d(theta)

        =  exp(-2t) * (1/2pi) INTEGRAL_0^{2pi} exp( 2t cos(theta) ) d(theta)
                      \____________  this is the integral definition of I_0(2t)  ____________/

        =  exp(-2t) * I_0(2t)

The modified Bessel function I₀ is defined by that integral, which is the sort of coincidence that stops being a coincidence once you notice that exponentials of cosines are how rotationally symmetric things on circles always end up. Its power series lets us evaluate it by hand:

At t = 2 we need I₀(4), so z²/4 = 4. Each term is 4^k/(k!)²:

arithmetic — I_0(4) from its series, nine termsk=0:  1     / (0!)^2 = 1     / 1        = 1.000000
k=1:  4     / (1!)^2 = 4     / 1        = 4.000000
k=2:  16    / (2!)^2 = 16    / 4        = 4.000000
k=3:  64    / (3!)^2 = 64    / 36       = 1.777778
k=4:  256   / (4!)^2 = 256   / 576      = 0.444444
k=5:  1024  / (5!)^2 = 1024  / 14400    = 0.071111
k=6:  4096  / (6!)^2 = 4096  / 518400   = 0.007901
k=7:  16384 / (7!)^2 = 16384 / 25401600 = 0.000645
k=8:  65536 / (8!)^2 = 65536 / 1.6257e9 = 0.000040
                                    sum = 11.301922

E[G_2] = exp(-4) * 11.301922 = 0.0183156 * 11.301922 = 0.207001
l_uniform = ln(0.207001) = -1.5750

Notice how fast it converges: nine terms give six correct digits. Notice also the shape of the terms — they rise, peak around k = 2, then collapse. That peak sits near k ≈ z/2, which is the same "the exponential is dominated by a narrow band" behaviour that will explain the sampling-noise problem in Chapter 4.

The kernel spectrum, tabulated

One table to make the "strictly positive definite" idea concrete. Every rotation-invariant kernel on the sphere decomposes into spherical-harmonic degrees, and what determines whether it can be fooled is which degrees it gives weight to.

Read the last two rows together and the paper's choice looks obvious rather than arbitrary. The Riesz family works, but its valid range of exponents depends on the dimension you happen to be in, and outside that range the objective is literally infinite for the distribution you want. The Gaussian kernel is bounded above by 1, bounded below by e^−4t, smooth, differentiable, and correct for every t and every m. It is the boring choice, which in numerical work is the highest compliment available.

Press Minimise from the two-atom preset and watch the points fan out into an even ring. That animation is the entire content of Proposition 1, run numerically: descent on the Gaussian energy has one destination.

From distributions to batches: Proposition 2

Proposition 1 is about probability measures, and a training batch is a finite set of M points. The bridge is the paper's Proposition 2: if you take the M-point configuration that minimises the average pairwise Gaussian potential, and let M grow, the empirical distribution of those points converges weak* to σ_m−1.

"Weak* convergence" means: for every continuous test function g, the average of g over your M optimal points converges to the integral of g against the uniform measure. Informally, the point cloud becomes indistinguishable from a uniform sample under any smooth measurement you could make.

That is what licenses the practical estimator. In code you compute the mean potential over the C(M,2) distinct pairs in the batch, which is a consistent estimate of the population double integral (the diagonal, where x = y, has probability zero under a continuous feature distribution). Minimising the batch quantity drives you toward the population minimiser.

What t does, at both extremes

The width t is not a nuisance parameter. It selects which classical problem you are solving, and both limits are worth deriving because Chapter 8 uses them to explain temperature.

As t → 0, uniformity degenerates into Attempt 1. For small t, e^−td2 ≈ 1 − td², so

Minimising that is minimising ‖E[u]‖² — the mean-zero condition, with the two-atom degeneracy fully restored. The theory survives at any t > 0, but the discriminative power of the metric vanishes as t → 0.

As t → ∞, uniformity becomes best packing. With a huge t, the sum ∑ e^−td2 is completely dominated by the single smallest distance, so log of the mean ≈ −t · min_i≠j d_ij² plus a constant. Minimising that means maximising the minimum pairwise distance — the Tammes problem of packing spherical caps, posed by a botanist in 1930 while counting pollen-grain pores.

Every number in that table is computed from the two closed forms above; the Kernel width t slider in the simulation reproduces them live. The moral: t = 2 is not magic, but it does sit in the useful middle, and both extremes fail in ways you can now name.

Chapter 4: The Asymptotic Theorem

We have two metrics that we invented because they seemed like the right things to want. Now comes the claim that makes this a paper about contrastive learning rather than a paper about sphere geometry: the contrastive loss, in the limit of many negatives, is exactly these two metrics added together.

The derivation is four lines of algebra plus one appeal to the law of large numbers. We are going to walk every step, including the parts that are usually waved through, because the error terms are where the practical lessons live.

The loss we are analysing

Write the contrastive loss with an explicit number of negatives M:

The outer expectation is over a positive pair (x, y) drawn from p_pos and M independent negatives x_i⁻ drawn from p_data. This is InfoNCE, NT-Xent, and the MoCo loss, all of which are the same object with different sampling schemes for the negatives.

Step 1: split the log

The only trick in the whole derivation is that −log(a/b) = −log a + log b. Abbreviate the positive similarity as s₊ = f(x)·f(y) and the i-th negative similarity as s_i = f(x_i⁻)·f(x):

Already something is visible. The first term wants the positive pair's similarity to be large — that is alignment, and Chapter 2 proved it is exactly the α = 2 alignment metric up to an affine map. The second term wants the total exponentiated similarity to everything to be small. That is a repulsion. But it is not yet in a form we recognise, because the sum grows with M.

Step 2: normalise by log M

Pull a factor of M out of the sum so that what remains is an average rather than a total:

derivation — the whole theorem, in four linesl = -s+/tau + log( exp(s+/tau) + SUM_i exp(s_i/tau) )

  = -s+/tau + log( M * [ (1/M) SUM_i exp(s_i/tau)  +  exp(s+/tau)/M ] )

  = -s+/tau + log M + log( (1/M) SUM_i exp(s_i/tau)  +  exp(s+/tau)/M )
                             \_________  A_M  _________/    \__ B_M __/

l - log M = -s+/tau + log( A_M + B_M )

The subtraction of log M is not a cosmetic normalisation. It has a physical meaning: log M is, to leading order, the loss a completely uninformative encoder achieves. If every similarity were identical, the softmax would be uniform over the M+1 candidates and the loss would be exactly log(M+1) — which differs from log M by log(1 + 1/M), a quantity that is 0.004 at M = 256 and vanishes in the limit. So ℓ − log M reads as "how much better than guessing", and it is the only version of the loss that can converge to anything finite as M grows. It is also, satisfyingly, the same log M that capped the mutual-information bound in Chapter 0 — the two appearances are the same quantity seen from opposite ends.

Step 3: let M go to infinity

Look at the two pieces inside the log.

A_M is a sample average. The M negatives are i.i.d. draws from p_data, and e^s_i/τ is a bounded function of each one — bounded because s_i ∈ [−1, 1] on the sphere, so the summand lives in [e^−1/τ, e^1/τ]. Bounded random variables satisfy the law of large numbers, so

This is the step where the sphere earns its keep for the third time. Without normalisation, s_i is unbounded, e^s_i/τ may have infinite mean, and the law of large numbers may simply fail to apply.

B_M vanishes. B_M = e^s₊/τ/M ≤ e^1/τ/M → 0. The positive pair, which occupies a full slot in the denominator, becomes negligible once there are enough negatives. Note the rate: this term is O(1/M).

Putting them together and taking the outer expectation gives the paper's Theorem 1.

Step 4: recognising the two terms

Term one is alignment. This is Chapter 2's identity, reused verbatim:

A positive multiple of the alignment metric, minus a constant. Minimising one is minimising the other. There is no approximation here at all.

Term two is uniformity. Substitute u·v = 1 − ½‖u−v‖² into the exponent:

derivation — the Gaussian kernel appears uninvitedexp( u.v / tau )  =  exp( (1 - 0.5*||u-v||^2) / tau )

                  =  exp(1/tau) * exp( -||u-v||^2 / (2 tau) )

                  =  exp(1/tau) * G_t(u, v)        with   t = 1/(2 tau)

Read that. Nobody chose a Gaussian kernel. The contrastive loss exponentiates a dot product; a dot product on the sphere is an affine function of squared distance; therefore the exponential of a dot product is a Gaussian potential. The kernel Chapter 3 justified on abstract grounds is the one the loss was already using.

Take logs and the constant slides out:

The one place to be honest about a gap

Compare the theorem's second term with the metric from Chapter 3, carefully:

These are not the same number. By Jensen's inequality, since log is concave, E[log Z] ≤ log E[Z], so the theorem's term is always less than or equal to ℓ_uniform. The two coincide exactly when the inner quantity E_x⁻[G_t(f(x), f(x⁻))] is the same for every anchor x — that is, when every point on the sphere experiences the same total potential from the rest of the cloud.

Which is exactly what the uniform distribution does. By rotational symmetry, a uniform cloud looks identical from every one of its points, so the Jensen gap closes precisely at the target. That is why both quantities are minimised by the same perfectly uniform encoder, and why the paper is comfortable optimising the outer-log version: it is a cleaner, single-scalar, lower-variance object whose minimiser is the one the theorem points at.

The convergence rate, and where the error actually comes from

Statement (3) of the theorem says the deviation decays as O(M^−1/2). We already have both error sources in hand, so let us see which one is responsible.

Source B (the positive in the denominator): B_M = e^s₊/τ/M, which is O(1/M).

Source A (sampling noise in the partition function): A_M is a mean of M i.i.d. bounded variables, so by the central limit theorem A_M − A_∞ = O_p(M^−1/2). Passing through the log with a first-order expansion, log A_M − log A_∞ ≈ (A_M − A_∞)/A_∞, still O(M^−1/2).

M^−1/2 is slower than M⁻¹, so sampling noise in the partition function is what sets the rate. This has a direct practical reading: the reason large batches help is not primarily that they remove the positive from the denominator. It is that they reduce the variance of the estimate of the repulsive field. Doubling the batch reduces that noise by a factor of √2, which is exactly the disappointing scaling everyone has observed.

Doing it by hand at M = 3

Time to put digits on all of this. Toy setup: features live on a circle. The anchor sits at 0°. Its positive sits at 20°. Negatives are drawn uniformly from the circle; our unlucky draw gives three at 90°, 180°, and 270°. Set τ = 1, so t = 1/(2τ) = 0.5.

arithmetic — the finite-M losss+  = cos(20)  = 0.9397        exp(s+/tau)  = exp(0.9397) = 2.559195
s1  = cos(90)  = 0.0000        exp(s1/tau)  = 1.000000
s2  = cos(180) = -1.0000       exp(s2/tau)  = 0.367879
s3  = cos(270) = 0.0000        exp(s3/tau)  = 1.000000

denominator = 2.559195 + 1.000000 + 0.367879 + 1.000000 = 4.927074
l           = -ln( 2.559195 / 4.927074 ) = -ln(0.519385) = 0.6551

# reference: an uninformative encoder scores ln(M+1) = ln 4 = 1.3863
l - ln M = 0.6551 - 1.0986 = -0.4436

Now the limit. Because the negatives are uniform on the circle, the population partition function has a closed form — the average of e^cosθ/τ over a uniform angle is the modified Bessel function I₀(1/τ):

arithmetic — the M -> infinity limitA_inf = E_theta[ exp(cos(theta)/tau) ] = I_0(1/1) = I_0(1) = 1.266066

limit = -s+/tau + ln(A_inf)
      = -0.9397 + ln(1.266066)
      = -0.9397 + 0.235915
      = -0.7038

So at M = 3 we are at −0.4436 and the limit is −0.7038: a gap of 0.2602 nats. Decompose it into the two sources we identified, using A₃ = (1.000000 + 0.367879 + 1.000000)/3 = 0.789293 and B₃ = 2.559195/3 = 0.853065:

arithmetic — where the 0.2603 came froml - ln M = -s+/tau + ln( A_3 + B_3 )
         = -0.9397 + ln( 0.789293 + 0.853065 )
         = -0.9397 + ln( 1.642358 ) = -0.9397 + 0.496133 = -0.4436   # checks out

# Source B: the positive occupying a slot in the denominator.
ln(A_3 + B_3) - ln(A_3) = 0.496133 - (-0.236617) = +0.7328

# Source A: three unlucky negatives underestimate the partition function.
ln(A_3) - ln(A_inf)     = -0.236617 - 0.235915  = -0.4725

# Total: +0.7328 - 0.4725 = +0.2602.  Matches the gap.

The other extreme: what happens at M = 1

Small-M behaviour is where the asymptotic story is least applicable and most instructive, so run the same toy with a single negative at 90°.

arithmetic — one positive, one negative, tau = 1denominator = exp(0.9397) + exp(0.0) = 2.559195 + 1.000000 = 3.559195
l           = -ln( 2.559195 / 3.559195 ) = -ln(0.719038) = 0.3298
l - ln(1)   = 0.3298 - 0 = 0.3298      # log M = 0, so no normalisation at all

limit       = -0.7038
gap         = 1.0336 nats

The gap is now larger than the limit itself, and it has a structural cause rather than a statistical one. At M = 1 the positive occupies half the denominator, so the loss can never fall below log 2 × something — and since a cross-entropy is non-negative while the asymptotic objective is comfortably negative, the two are not even in the same range. At M = 1 you are not approximating the asymptotic objective badly; you are optimising a different function.

This is the quantitative content of the folklore that contrastive learning "needs many negatives". Tabulate the gap across the whole range on this toy:

The values for M = 16 and M = 256 are Monte-Carlo estimates that the simulation below reproduces exactly; the first two rows are the hand computations above. Note how quickly it converges here and remember Chapter 8's warning: this toy uses τ = 1 on a circle. At τ = 0.07 in 128 dimensions the constant in front of M^−1/2 is enormously larger, and M = 256 is nowhere near enough.

Three assumptions the theorem makes that your training loop breaks

The derivation assumed the M negatives are independent draws from p_data, encoded by the current f. Every popular implementation violates this, and it is worth knowing how.

What the theorem does not say

Three caveats, all of which the paper is upfront about and all of which get misquoted.

Chapter 5: Four Points, by Hand

Two metrics, one theorem, zero intuition for what the numbers feel like. This chapter fixes that with the smallest possible non-trivial example: four features on a circle, worked entirely by hand, with every exponential written out.

The setup

Take a dataset of two images. Each is augmented twice, giving four views and therefore four features. Positive pairs are {1, 2} and {3, 4}. The encoder maps to S¹, the unit circle, so every feature is an angle. Use the paper's defaults: α = 2 and t = 2.

We need two formulas and nothing else. From Chapter 1, for points at angles θ_i and θ_j:

Alignment averages d² over the two positive pairs. Uniformity takes e^−2d2 over all six distinct pairs (that is what torch.pdist gives you), averages, and logs.

Configuration A: total collapse

Everything at 0°. All six pairwise distances are zero.

arithmetic — all four points at 0 degreesalignment:  d(1,2)^2 = 0 , d(3,4)^2 = 0
            l_align = (0 + 0)/2 = 0.0000   # perfect

uniformity: all six pairs have d^2 = 0, so G = exp(0) = 1
            mean = 6/6 = 1.0000
            l_uniform = ln(1.0000) = 0.0000   # the worst possible value

total (lambda = 1) = 0.0000 + 0.0000 = 0.0000

Note the ceiling: because G ≤ 1 for every pair, the mean is at most 1 and ℓ_uniform is at most 0. Zero is the collapse signature. If you ever log a uniformity value at or near 0.00, your encoder has died — no further diagnosis needed.

Configuration B: perfectly aligned, two atoms

Points 1 and 2 both at 0°; points 3 and 4 both at 180°. Alignment is still perfect, but the representation now carries one bit.

arithmetic — pairs collapsed onto antipodal atomsalignment:  d(1,2)^2 = 2 - 2cos(0) = 0 ; d(3,4)^2 = 0
            l_align = 0.0000

uniformity: (1,2): d^2 = 0      -> G = 1.000000
            (3,4): d^2 = 0      -> G = 1.000000
            (1,3): d^2 = 4      -> G = exp(-8) = 0.000335
            (1,4): d^2 = 4      -> G = 0.000335
            (2,3): d^2 = 4      -> G = 0.000335
            (2,4): d^2 = 4      -> G = 0.000335
            mean = (1 + 1 + 4*0.000335)/6 = 2.001342/6 = 0.333557
            l_uniform = ln(0.333557) = -1.0979

total (lambda = 1) = 0.0000 - 1.0979 = -1.0979

Better. The two forces are already visible: alignment is indifferent between A and B, and uniformity is the only thing that prefers B.

Configuration C: the perfect square

Points at −45°, +45°, 135°, 225°. This is the most uniform arrangement four points on a circle can achieve, and it destroys alignment: each positive pair is now 90° apart.

arithmetic — the four-point squarealignment:  d(1,2)^2 = 2 - 2cos(90) = 2 ; d(3,4)^2 = 2
            l_align = (2 + 2)/2 = 2.0000

uniformity: four pairs at 90 deg  -> d^2 = 2 -> G = exp(-4) = 0.018316
            two  pairs at 180 deg -> d^2 = 4 -> G = exp(-8) = 0.000335
            mean = (4*0.018316 + 2*0.000335)/6 = 0.073933/6 = 0.012322
            l_uniform = ln(0.012322) = -4.3963

total (lambda = 1) = 2.0000 - 4.3963 = -2.3963

Better still, on the sum — and this should surprise you. The square has the worst possible alignment for a configuration of this shape, and it still beats the perfectly-aligned Configuration B by 1.30 nats. At λ = 1 and t = 2, uniformity is the dominant force.

Configuration D: the actual optimum

Neither extreme is best. Parametrise the family: put the two positive pairs symmetrically around 0° and 180°, each pair separated by 2δ. Configuration B is δ = 0; Configuration C is δ = 45°. Sweep δ and compute:

Work the winning row so you trust it. At δ = 41° the points sit at −41°, 41°, 139°, 221°:

arithmetic — delta = 41 degrees, the optimum at lambda = 1within-pair gap = 82 deg
  d^2 = 2 - 2cos(82) = 2 - 2(0.139173) = 1.721654
  l_align = 1.7217

six pairs:
  (1,2)  82 deg -> d^2 = 1.721654 -> G = exp(-3.443307) = 0.031959
  (3,4)  82 deg -> d^2 = 1.721654 -> G = 0.031959
  (1,3) 180 deg -> d^2 = 4.000000 -> G = exp(-8.000000) = 0.000335
  (2,4) 180 deg -> d^2 = 4.000000 -> G = 0.000335
  (1,4)  98 deg -> d^2 = 2.278346 -> G = exp(-4.556693) = 0.010497
  (2,3)  98 deg -> d^2 = 2.278346 -> G = 0.010497

  mean = (0.031959 + 0.031959 + 0.000335 + 0.000335 + 0.010497 + 0.010497)/6
       = 0.085582/6 = 0.014264
  l_uniform = ln(0.014264) = -4.2500

total = 1.7217 - 4.2500 = -2.5284   # beats the square by 0.13 nats

Where the settlement sits: a bifurcation you can derive

Slide λ down in the simulation and something abrupt happens: below a certain value the optimum snaps to δ = 0, and the two positive pairs collapse onto single points. That threshold is computable in a few lines, and the answer is clean enough to be worth memorising.

Let φ = 2δ be the within-pair gap, and expand both metrics for small φ. Using cosφ ≈ 1 − φ²/2:

derivation — when does the collapsed solution stop being optimal?alignment:   l_align = 2 - 2cos(phi) ~= phi^2

uniformity:  two within-pair distances  d^2 ~= phi^2
             two antipodal              d^2  = 4
             two cross                  d^2 ~= 4 - phi^2

  mean G = [ 2 exp(-t phi^2) + 2 exp(-4t) + 2 exp(-4t) exp(t phi^2) ] / 6

  for large t the exp(-4t) terms are negligible:
  mean G ~= (1/3) exp(-t phi^2)          ->     l_uniform ~= -ln 3 - t phi^2

total J(phi) = phi^2 + lambda * ( -ln 3 - t phi^2 )
             = const + phi^2 * ( 1 - lambda * t )

The sign of (1 − λt) decides everything. If λt < 1, the coefficient is positive, so φ = 0 is a minimum and the pairs collapse. If λt > 1, the coefficient is negative, φ = 0 becomes a maximum, and the configuration splits open.

Keeping the antipodal e^−4t terms shifts this to λ* = 0.5005, so the approximation is good to one part in a thousand. Try it in the simulation: set t = 2, put λ at 0.45, press Minimise, and the points fuse into two atoms. Nudge λ to 0.55 and they split.

Sweeping λ: the settlement moves, and it moves abruptly

The threshold calculation says something happens at λt = 1. Compute the actual optimum at several λ (holding t = 2) and you can watch it happen.

Look at the jump between λ = 0.50 and λ = 0.55. The optimum does not slide out from 0° gradually; it leaps to 34°. Above the threshold, the configuration snaps open almost all the way, and further increases in λ buy only a few more degrees.

The same calculation also explains the shape of the tradeoff frontier you will meet in Chapter 7. Between λ = 0.55 and λ = 2, alignment worsens from 1.25 to 1.86 while uniformity improves from −3.55 to −4.36. Every extra unit of uniformity is bought with alignment, and the exchange rate gets steadily worse. That is what a frontier looks like when you plot it.

Extending to eight points, and why it barely changes

Four points on a circle is a toy; the reason to trust it is that nothing qualitative changes when you scale it up. Take eight points with four positive pairs on S¹. Two reference configurations:

arithmetic — 8 points, 4 positive pairs, t = 2, 28 distinct pairs# Config P: pairs fully collapsed onto 4 atoms, atoms 90 degrees apart
  4 pairs at d^2 = 0  -> G = 1
 16 pairs at d^2 = 2  -> G = exp(-4) = 0.018316
  8 pairs at d^2 = 4  -> G = exp(-8) = 0.000335
  mean = (4 + 0.293056 + 0.002684)/28 = 4.295740/28 = 0.153419
  l_align = 0.0000    l_uniform = ln(0.153419) = -1.8746

# Config Q: all eight evenly spaced, 45 degrees apart
  8 pairs at  45 deg -> d^2 = 0.585786 -> G = 0.309867
  8 pairs at  90 deg -> d^2 = 2.000000 -> G = 0.018316
  8 pairs at 135 deg -> d^2 = 3.414214 -> G = 0.001081
  4 pairs at 180 deg -> d^2 = 4.000000 -> G = 0.000335
  mean = (2.478936 + 0.146528 + 0.008648 + 0.001342)/28 = 0.094123
  l_align = 2 - 2cos(45) = 0.585786    l_uniform = ln(0.094123) = -2.3629

# At lambda = 1:  Config P total = -1.8746 ; Config Q total = -1.7771
# Now P wins -- because with four classes instead of two, collapsing
# the pairs still leaves the atoms well spread.

The flip is instructive. With eight points and four pair-classes, the collapsed-pairs configuration is already reasonably uniform — four atoms 90° apart is not crowded — so at λ = 1 it beats the fully-spread configuration, where with four points it lost. (Neither is the true optimum: running gradient descent at λ = 1 lands at ℓ_align ≈ 0.25, ℓ_uniform ≈ −2.20, total ≈ −1.95 — an interior compromise that beats both, exactly as Chapter 7's frontier will show.) The general principle: the more distinct positive-pair classes your data has, the less the two forces conflict, because a finite point set with many points approximates the uniform measure better. On a real dataset with a million instances, the tension that dominates this toy is faint, and both metrics can be driven low simultaneously. The toy exaggerates the conflict precisely because it is small.

You can watch both configurations in the eight-point simulation of Chapter 7, which optimises exactly this setup at a range of λ and plots where each one lands.

A sanity check you should run on your own code

These hand numbers are a unit test. Any implementation of the two metrics must reproduce them, and getting a mismatch localises the bug immediately.

Chapter 6: Optimising the Two Metrics Directly

Here is the moment the paper stops being an analysis and becomes an experiment. If alignment and uniformity are genuinely what InfoNCE is optimising, then writing them down and optimising them openly should work at least as well. If it does not, the analysis was a story.

The entire implementation

python — the paper's reference implementation, verbatim in substanceimport torch

def lalign(x, y, alpha=2):
    # x, y: (N, d) L2-normalised features of the two views. Row i of x
    # and row i of y are a positive pair.
    return (x - y).norm(dim=1).pow(alpha).mean()

def lunif(x, t=2):
    # pdist gives the N(N-1)/2 DISTINCT pairwise distances -- no self-pairs.
    sq_pdist = torch.pdist(x, p=2).pow(2)
    return sq_pdist.mul(-t).exp().mean().log()

loss = lalign(x, y) + lam * (lunif(x) + lunif(y)) / 2

Four lines. Walk them once with shapes, because every one of them is a decision.

The numerical trap

There is exactly one way this code bites you, and it is worth pre-empting. The uniformity term computes e^{−t d2} and then takes a log of the mean. If every pair is far apart and t is large, every term underflows to zero, the mean is zero, and the log is −∞. Your loss becomes -inf, then nan.

The fix is the standard one: never exponentiate before you have to. Fold the mean and the log into a single log-sum-exp, which subtracts the maximum internally and is exact in the region where the naive form dies.

python — the numerically safe form (use this under AMP)def lunif_stable(x, t=2):
    sq_pdist = torch.pdist(x, p=2).pow(2)               # (N(N-1)/2,)
    # log( mean_k exp(-t d_k^2) ) = logsumexp_k(-t d_k^2) - log(K)
    return torch.logsumexp(-t * sq_pdist, dim=0) - torch.log(
        torch.tensor(float(sq_pdist.numel()), device=x.device))

Mathematically identical, numerically bulletproof. Cast the pairwise distances to float32 even under mixed precision; the cost is negligible next to the encoder forward pass.

What the paper actually ran

The experiments cover vision and language, and the hyperparameters are worth tabulating because they show that λ and t are genuinely dataset-dependent rather than universal constants.

Two things jump out. The vision settings sit near λ ≈ 1 with t = 2 or 3, which by Chapter 5's threshold λt > 1 puts them comfortably on the non-collapsed side. The sentence-embedding setting is very different — λ = 0.1 with t = 5, so λt = 0.5, i.e. below the toy's threshold. That is not a mistake; text positive pairs (adjacent sentences) are far noisier than augmented crops, so the useful regime is much more alignment-dominated. The same lens, different settlement.

The results

Porting an existing InfoNCE run, step by step

Suppose you already have a tuned SimCLR-style run: τ = 0.1, batch 512, 128-dimensional head. You want the two-metric form without re-tuning from scratch. Chapter 4's identities make this a calculation rather than a search.

Two of those rows catch bugs that are otherwise invisible. Step 3 matters because people memorise −3.94 as "the good value" and then panic when a t = 5 run reports −9.5; the reference moves with t, and comparing metric values across different t is meaningless. Step 4 matters because the failure is silent: under automatic mixed precision the uniformity term simply becomes a constant −∞ that contributes no gradient, and your run collapses while the loss looks superficially fine.

What you get in exchange, practically

Beyond the accuracy, the reformulation buys engineering leverage that a single entangled scalar cannot.

A training checklist

Chapter 7: The Tradeoff Plane

Two numbers per encoder means every encoder is a point. Put ℓ_uniform on the horizontal axis and ℓ_align on the vertical, and you have a map of the entire design space of contrastive representation learning. This chapter is about learning to read it, because once you can, a plot that took a week to produce answers questions that used to take a month of ablations.

Orienting yourself

Both metrics are "lower is better", so down and to the left is good. Fix the ranges from what we already know.

The four corners

The paper's headline plot is exactly this: hundreds of encoders trained with varying losses and hyperparameters, each plotted at its (ℓ_uniform, ℓ_align) coordinates and coloured by downstream accuracy. The colouring is not noise. Accuracy varies smoothly across the plane, and the best models occupy a compact region, which is the empirical content of the claim that these two numbers are what matters.

The frontier is real, and it is not a line you can cross

Chapter 5 proved it in miniature: with finitely many positive-pair classes, perfect alignment and perfect uniformity cannot both hold. Perfect alignment collapses each class to a single point, and a finite set of points is not the uniform measure. So the reachable region of the plane has a boundary — a Pareto frontier — and every training run is a trajectory that eventually parks somewhere along it.

How far the frontier extends depends on things you control:

Read the last two rows together. Everything except capacity is a slide along the frontier; capacity and training time are what move the frontier itself. That is a useful triage rule when a run underperforms: first ask whether you are at a bad point on a good frontier (retune λ, t, batch) or on a bad frontier (bigger model, longer schedule, better augmentations).

Reading real models: the sentence-embedding case

The clearest published use of this plane is in the sentence-embedding literature, where the two pathologies are unusually well separated. Mean-pooled BERT embeddings are famously anisotropic — they occupy a narrow cone, so almost any two sentences have cosine similarity above 0.6 regardless of meaning. In alignment-uniformity language, that is good alignment, terrible uniformity: the top-left of the useful band, near the collapsed corner on the uniformity axis.

A reading protocol

Log both metrics each epoch and the plane becomes a diagnostic instrument.

Chapter 8: What It Predicts, and What It Cannot See

A good explanation earns its keep by predicting things nobody fed into it. This chapter collects the predictions the alignment-uniformity lens makes for free, and then — because a lesson that only lists a theory's wins is advertising — the places where it is silent or wrong.

Prediction 1: everything temperature does

We have the identity t = 1/(2τ) from Chapter 4 and the two limiting behaviours of t from Chapter 3. Combine them and every empirical fact about temperature in contrastive learning falls out.

Prediction 2: the returns to batch size scale as √M

Chapter 4 established that the convergence bottleneck is sampling noise in the partition function, decaying as O(M^−1/2). Two consequences that people usually discover the expensive way:

Doubling the batch buys a factor of √2, not 2. The famous diminishing returns of large-batch contrastive learning are not mysterious; they are the central limit theorem.

The constant in front of M^−1/2 grows as τ shrinks. The variance of e^s/τ across negatives explodes when 1/τ is large, because the expectation is dominated by rare near-collisions. Put numbers on it: in d = 128, similarities between random features are roughly Gaussian with standard deviation 1/√128 = 0.088, so s/τ has standard deviation 0.088/τ. The relative standard deviation of a single term e^s/τ is then √(e^(0.088/τ)2 − 1):

Read the last row: at τ = 0.03, a 256-sample estimate of the repulsive field carries a relative error approaching 500%. Low temperature and small batches are mathematically incompatible, and the alignment-uniformity framing tells you why rather than just that.

Prediction 3: what stronger augmentation costs

Augmentations define p_pos, which appears only in the alignment term. Predicted effect of turning up the augmentation strength: ℓ_align worsens, ℓ_uniform is essentially untouched, and downstream accuracy follows an inverted U — too weak and the encoder has learned no invariance worth having, too strong and it is being asked for invariance to things that carry the signal. The plane makes the diagnosis mechanical: if accuracy is falling and only the vertical coordinate moved, augmentation is your culprit.

Prediction 4: why the projection head gets thrown away

Chapter 1 flagged the oddity: the loss shapes the post-projection space, but everyone uses the pre-projection backbone features downstream, and they work better. The alignment-uniformity lens gives this a clean account.

The loss is applied at the head's output, so it is that space that gets driven toward the negotiated settlement — tight positive pairs, uniform marginal. In particular, the head is rewarded for discarding any information that distinguishes two views of the same image, because keeping it costs alignment. Colour statistics, crop position, blur level: all of it is nuisance under the augmentation pipeline, and all of it should be destroyed by the time you reach the loss.

But some of that information is useful for some downstream task. Colour is nuisance for "is this a dog" and essential for "is this a ripe tomato". The backbone, sitting one nonlinearity earlier, has not yet been forced to throw it away — the head does the discarding, and discarding is what the head is for.

Limit 1: uniformity is a weak detector of dimensional collapse

Here is where the lens has a genuine blind spot, and it is worth quantifying rather than gesturing at.

Dimensional collapse is the failure where features remain spread out but occupy only a low-dimensional subspace of the sphere — the covariance eigenvalue spectrum has a handful of large values and a long tail of near-zeros. Jing et al. (2022) documented it in trained contrastive models. Does uniformity catch it?

Use the formula from Chapter 7. Features uniformly spread over a k-dimensional sub-sphere score ℓ_uniform = −2t + 2t²/k. At t = 2:

Limit 2: uniformity is not necessary

The most direct challenge to the framework arrived within a year of it. BYOL and SimSiam train self-supervised encoders with no negatives at all. There is no repulsive term, nothing that could be called a uniformity objective, and by every naive argument they should collapse instantly. They do not, and they match or beat contrastive methods.

What prevents collapse there is architectural — a predictor head, a stop-gradient, an exponential-moving-average target network — rather than a term in the loss. The honest conclusion: alignment and uniformity describe what the contrastive family of losses optimises, not what self-supervised learning in general requires. Uniformity is one way to avoid collapse. It is not the only way, and the paper does not claim otherwise.

The related methods are worth naming because they show the two forces recurring under different names. Barlow Twins and VICReg replace the repulsive term with explicit variance and decorrelation penalties on the feature dimensions. Read structurally, those are anti-collapse mechanisms operating on the covariance spectrum rather than on pairwise distances — a different answer to the same question that alignment-uniformity poses, and one that, per Limit 1, targets exactly the failure uniformity is bad at seeing.

Limit 3: instance uniformity fights class structure

A subtle one, and arguably the deepest. ℓ_uniform is computed over the marginal distribution of instances. It is minimised when every image is as far as possible from every other image, including two photographs of the same breed of dog.

But the representation we want for a linear probe is the opposite: images of the same class should be clustered. A perfect classifier's feature distribution is not uniform at all; it is a small number of tight blobs. The perfectly uniform encoder is, from the classifier's point of view, a hash function.

Limit 4: the theory is silent on augmentations, which is where the prior lives

Alignment is defined relative to p_pos, and p_pos is whatever you decided it should be. Change the augmentation pipeline and you change what "the same thing" means, which changes the entire semantics of the learned space — while every equation in the paper stays word-for-word identical.

Concretely: strip colour jitter from SimCLR's pipeline and it learns colour-histogram features, because colour becomes a legitimate cue for instance identity. Add heavy colour jitter and colour is destroyed, which is right for ImageNet objects and catastrophic for bird species or histopathology stains. Neither case changes ℓ_align or ℓ_uniform's definition by one symbol. The framework explains the mechanism; the augmentations decide what the mechanism is applied to, and nearly all of the domain knowledge in self-supervised learning lives there.

Limit 5: the analysis is over unconstrained encoders

Every theorem in the paper is a statement about functionals over all measurable f. Real encoders are a ResNet trained by SGD from a specific initialisation, and that constraint set is not a technicality — it is, per Limit 3, doing much of the work. The paper is careful to say "if perfectly uniform encoders exist"; the practical answer is usually that they do not, and what you get instead is the network's biased approximation to them, which is precisely why the representation is useful.

Chapter 9: Legacy & Cheat Sheet

Some papers are remembered for a model. This one is remembered for a plot. Six years on, if you open a paper about contrastive learning, sentence embeddings, or representation collapse, there is a good chance you will find a two-axis scatter with alignment on one axis and uniformity on the other. That is the legacy: not a technique, a coordinate system.

The complete cheat sheet

The five equations worth memorising

Six things people get wrong about this paper

The papers this one made possible

The five-sentence summary

Build it yourself — the afternoon recipe

References

1. Wang, T., Isola, P. "Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere," ICML 2020 — arXiv:2005.10242. The paper this lesson is built on. Code at github.com/SsnL/align_uniform.
2. van den Oord, A., Li, Y., Vinyals, O. "Representation Learning with Contrastive Predictive Coding," 2018 — arXiv:1807.03748. Where InfoNCE and the log M bound come from.
3. Tschannen, M., Djolonga, J., Rubenstein, P., Gelly, S., Lucic, M. "On Mutual Information Maximization in Representation Learning," ICLR 2020 — arXiv:1907.13625. The paper that broke the MI story empirically.
4. Chen, T., Kornblith, S., Norouzi, M., Hinton, G. "A Simple Framework for Contrastive Learning of Visual Representations" (SimCLR), ICML 2020 — arXiv:2002.05709. The normalisation, the temperature, the batch size.
5. He, K., Fan, H., Wu, Y., Xie, S., Girshick, R. "Momentum Contrast for Unsupervised Visual Representation Learning" (MoCo), CVPR 2020 — arXiv:1911.05722. The queue that makes M large.
6. Gao, T., Yao, X., Chen, D. "SimCSE: Simple Contrastive Learning of Sentence Embeddings," EMNLP 2021 — arXiv:2104.08821. The alignment-uniformity plane as a standard NLP diagnostic.
7. Wang, F., Liu, H. "Understanding the Behaviour of Contrastive Loss," CVPR 2021 — arXiv:2012.09740. Hardness-awareness and the uniformity-tolerance dilemma.
8. Jing, L., Vincent, P., LeCun, Y., Tian, Y. "Understanding Dimensional Collapse in Contrastive Self-supervised Learning," ICLR 2022 — arXiv:2110.09348. The failure mode uniformity does not see.
9. Zimmermann, R. S., Sharma, Y., Schneider, S., Bethge, M., Brendel, W. "Contrastive Learning Inverts the Data Generating Process," ICML 2021 — arXiv:2102.08850. Why the uniform-on-the-sphere solution turns out to be semantic.
10. Borodachov, S., Hardin, D., Saff, E. Discrete Energy on Rectifiable Sets, Springer 2019. The potential-theory background for Propositions 1 and 2 — strictly positive definite kernels, minimal energy, and weak* convergence of optimal point configurations.