Hyperbolic Embeddings

Chapter 1: The Poincaré Disk, Built From One Rule

We need a space with more room than the plane. The obstacle is that we also need to store points in it as ordinary tuples of floats, and draw them on ordinary screens. Those two demands look contradictory: how do you fit a space that is bigger than the plane inside a picture that is smaller than the plane?

The answer is the oldest trick in cartography. A world map fits an entire sphere onto a rectangle by distorting scale — on a Mercator map, Greenland is the size of Africa because the map stretches things near the poles. Nobody is confused by this. You read the map knowing that a centimetre near the equator and a centimetre near the pole mean different real distances.

We are going to do the same thing in reverse. We will take an infinite space and squeeze it into the open unit disk, and we will pay for the squeeze by declaring that a centimetre near the rim represents an enormous real distance. The disk is a map. The space it depicts is unbounded.

The one rule

Let B be the open unit disk: all points x with ‖x‖ < 1. (Open: the boundary circle itself is not in the space. We will see why in a moment.) Now the single definition that generates everything:

That factor λ_x is called the conformal factor. Everything about hyperbolic embeddings — the distance formula, the exploding gradients, the numerical failures, the reason generality lives at the centre — is that one fraction and its consequences.

Read it as a shrinking ruler. At the centre λ = 2, so your ruler is normal-ish. As you drift outward the denominator 1 − ‖x‖² collapses toward zero and the ruler shrinks toward nothing, so it takes more and more of them to cross the same Euclidean gap.

Notice the pattern in the last column: for ‖x‖ close to 1, since 1 − r² = (1−r)(1+r) ≈ 2(1−r), we get λ ≈ 1/(1−r). Each extra nine in the radius multiplies the scale by ten. Hold on to that, because in Chapter 7 it becomes the reason float32 dies.

Deriving the distance from the centre — the whole derivation, by hand

Now we can compute a real distance instead of asserting one. Walk in a straight line from the centre out to a point at Euclidean radius r. Chop the walk into tiny steps dt. Each step at radius t counts as λ(t) dt. Add them up:

That integral is elementary. Split the fraction with partial fractions — check the algebra yourself by putting the right-hand side over a common denominator:

Each piece integrates to a logarithm: ∫ dt/(1−t) = −ln(1−t) and ∫ dt/(1+t) = ln(1+t). Evaluate from 0 to r, and both terms vanish at 0 because ln 1 = 0:

That is the entire derivation. No hand-waving, no appeal to authority — one substitution and two logarithms. And it immediately answers the question we opened with.

Four distances you should be able to produce from memory

Plug numbers into ln((1+r)/(1−r)) and something delightful happens.

Look at the gaps: 2.9444 → 5.2933 is a jump of 2.349; 5.2933 → 7.6004 is a jump of 2.307. Both are converging on ln 10 = 2.3026. Each additional nine in the Euclidean radius is worth a fixed 2.30 units of true distance. The Euclidean radius is a logarithmic gauge of the real radius, in exactly the way a decibel is a logarithmic gauge of power.

Invert it and you get the formula you will actually use when placing things by hand:

Want a node at true radius 6.3? Then ‖x‖ = tanh(3.15) = 0.99633. Want one at 9.8? tanh(4.9) = 0.99989. Two nodes 3.5 units apart in the hierarchy differ in stored radius by 0.0036. Chapter 4 makes that uncomfortable fact into a design lesson, and Chapter 7 makes it into a bug report.

The general two-point distance

Distances from the centre are the easy case because the path is a straight radial line. For two arbitrary points the shortest path bends (we will see why in a moment), and the closed form is:

where arcosh(z) = ln(z + √(z² − 1)) is the inverse hyperbolic cosine. Do not take it on faith — check that it reduces correctly. Set u = 0. Then ‖u−v‖² = r² and 1−‖u‖² = 1, so the argument is

and the standard identity cosh s = (1 + tanh²(s/2))/(1 − tanh²(s/2)) says that arcosh of that is exactly 2 artanh(r). The general formula contains the radial one. Good.

Three structural things to notice in that formula, because each one shows up later as an engineering consequence:

Worked example 1: the same shape, twice, at two radii

This is the calculation that makes hyperbolic space click. Take two points at the same radius, ninety degrees apart. Then do it again, further out. Nothing about the shape changes — only where it sits.

Near the centre. u = (0.5, 0), v = (0, 0.5).

For comparison, the distance out to each of them from the centre is ln 3 = 1.0986, so travelling u → centre → v would cost 2.1972. The direct route (1.6807) is a clear shortcut. So far this behaves like an ordinary plane.

Far out. u = (0.99, 0), v = (0, 0.99). Same angle, same shape, bigger radius.

Now compare with the through-the-centre route: 2 × ln 199 = 2 × 5.2933 = 10.5866. The direct path saves only

against 23% at radius 0.5. Far from the centre, going “through the root” is almost free. That is precisely how distance behaves in a tree: to get from one leaf to a distant leaf you climb to the common ancestor and come back down, and there is no shortcut across. Hyperbolic space reproduces that behaviour without anybody programming it in. It falls out of the shrinking ruler.

Why straight lines bow inward

If the ruler is longer near the centre, then a path that detours toward the centre gets to use the cheap ruler for part of its journey. So shortest paths — geodesics — bow inward. It is the same reason a flight from London to Tokyo goes over Siberia rather than along the latitude line: the map is not the territory.

The exact shape is beautiful and easy to compute. Every geodesic of the Poincaré disk is either a diameter (a straight line through the centre — no detour available, it is already using the cheapest route) or an arc of a circle that meets the boundary circle at right angles. A circle with centre c and radius ρ is orthogonal to the unit circle exactly when ‖c‖² = 1 + ρ². Combine that with “the circle passes through u and v” and you get two linear equations for the centre:

Derivation, in one line: ‖c−u‖² = ρ² = ‖c‖² − 1, expand the left side, and the ‖c‖² terms cancel. That is exactly what the simulation below solves, twice per frame, to draw the arcs you are about to drag around.

Concept → realization: the distance function you will actually ship

python
import torch

def poincare_dist(u, v, eps=1e-7):
    """u, v: (..., d) tensors with ||x|| < 1.  Returns (...,) distances."""
    su = u.pow(2).sum(-1)                 # ||u||^2
    sv = v.pow(2).sum(-1)                 # ||v||^2
    duv = (u - v).pow(2).sum(-1)          # ||u - v||^2
    # clamp keeps the denominator away from 0 when a point drifts to the rim
    alpha = (1 - su).clamp_min(eps)
    beta  = (1 - sv).clamp_min(eps)
    z = 1 + 2 * duv / (alpha * beta)
    return torch.acosh(z.clamp_min(1 + eps))   # acosh is undefined below 1

Two clamps, and both are load-bearing. The first stops a point that has crept to ‖x‖ = 1 from producing a division by zero. The second stops floating-point round-off from handing acosh an argument of 0.9999999 — which is mathematically impossible but numerically routine when u and v are nearly equal — and getting nan back. A single nan in an embedding table poisons every subsequent batch through the shared parameters. Chapter 7 returns to this with the arithmetic of why it happens.

Chapter 2: Exponential Room

Chapter 0 said the plane fails because supply grows linearly while demand grows exponentially. Chapter 1 built a space with a shrinking ruler. Now we cash the cheque: we compute the supply in that space and find that it, too, grows exponentially. Then a binary tree fits, and we will know exactly how much edge length to give it.

Deriving the circumference of a hyperbolic circle

Take the circle of true radius s centred at the origin. On the map it appears as an ordinary Euclidean circle of radius r = tanh(s/2) — we derived that inversion in Chapter 1. Its drawn circumference is 2πr. But every bit of that drawn arc sits at radius r, where the conformal factor is λ = 2/(1 − r²). So the true length is the drawn length times the factor:

Now use the identity 1 − tanh²u = 1/cosh²u (which is just cosh² − sinh² = 1 divided through by cosh²). Substituting u = s/2:

where the last step is the double-angle identity sinh 2u = 2 sinh u cosh u. Put it together:

Three lines of algebra, no magic. And by the same argument the area enclosed is A(s) = ∫₀^s 2π sinh t dt = 2π(cosh s − 1).

Why sinh is the whole point

Recall sinh s = (e^s − e^−s)/2. For small s this is approximately s, so hyperbolic space looks flat close up — as any curved space does. For large s the second term dies and

The circumference grows exponentially with the radius. Every additional unit of radius multiplies the available perimeter by e ≈ 2.718. That is the supply curve we were missing.

At radius 20 — walk twenty steps from the root — a hyperbolic plane offers you a billion and a half units of perimeter to arrange things on. The flat plane offers 126. Same two coordinates per point.

The fitting rule: e versus b

Now put demand and supply side by side, per level, for a tree with branching factor b and edge length ℓ.

Room per node is therefore ≈ π e^kℓ / b^k = π (e^ℓ/b)^k. Everything hinges on whether the base e^ℓ/b exceeds 1:

That is the design rule, and it is actionable. A binary tree needs edges longer than ln 2 = 0.693. A ten-ary tree needs ln 10 = 2.303. A tree where one node has a thousand children needs ln 1000 = 6.91 per level. Choose the edge length above that threshold and the tree fits with room to spare, forever, at any depth.

Check it against a concrete case. Binary tree, ℓ = 1, so e/2 = 1.359 — room per node should grow by 36% per level:

Read the two right-hand columns as two different worlds. One is collapsing toward zero; the other is expanding. And 14.57 / 7.87 = 1.85 = 1.359², exactly the compounding the formula predicted over two levels. The arithmetic holds.

Play with the edge-length slider until the rule bites

Set branching to 4 and edge length to 0.7. Since ln 4 = 1.386 and 0.7 < 1.386, the rule says this must fail — and the readout confirms it: the closest pair of leaves shrinks level by level, exactly like the flat case, just more slowly. Now push the slider past 1.4 and the collapse reverses. There is no gentle degradation around the threshold; it is a phase change between “shrinking geometrically” and “growing geometrically.”

This is the first genuinely useful engineering knob in the lesson. If your hierarchy has high-fan-out nodes — a category with 2,000 direct children is common in product catalogues — then a model that learns short edges near the root will crush those children together no matter how long you train. The fix is not more data; it is more radius, which in practice means letting norms grow (Chapter 3) or raising the curvature magnitude.

Why the pictures in the papers all look the same

Every Poincaré embedding figure you have seen has the same look: a few words floating near the middle and a dense confetti of words smeared around the rim. Newcomers read that as “the model collapsed everything to the boundary.” It did not. That appearance is required by the map.

Here is the arithmetic. Suppose two leaves at true radius 9.8 are 4.2 apart — three levels up to their common ancestor and three back down at ℓ = 0.7. Their stored radius is tanh(4.9) = 0.999889, so 1 − r² = 0.000222. Two points at the same radius separated by angle φ satisfy

(that is the general formula with ‖u−v‖² = 4r² sin²(φ/2), the chord of an isoceles triangle). Solve for φ with d = 4.2, so cosh d = 33.35:

Two concepts a comfortable four units apart are drawn one twentieth of a degree apart. On a 1000-pixel figure that is under a pixel. The confetti is the map projection, not the model. The same fact, seen from the other side, is Chapter 7’s numerical catastrophe: if 0.05° of angle carries four units of meaning, then losing a few bits of angular precision loses whole levels of the hierarchy.

Concept → realization: place a tree by hand, no training

You do not always need to learn an embedding. If you already have the tree, you can place it combinatorially in a few lines — this is Sarkar’s construction, and it is a good baseline and a good initialiser.

python
import numpy as np

def place(node, angle, half_width, depth, ell, out):
    """angle: centre of this node's angular slice (radians)
       half_width: half the slice this node owns
       depth: how many edges from the root; true radius = depth * ell"""
    r = np.tanh(depth * ell / 2)              # true radius -> stored radius
    out[node] = (r * np.cos(angle), r * np.sin(angle))
    kids = children[node]
    if not kids: return
    step = 2 * half_width / len(kids)         # split the slice evenly
    for i, k in enumerate(kids):
        a = angle - half_width + (i + 0.5) * step
        place(k, a, step / 2, depth + 1, ell, out)

out = {}
place(root, angle=0.0, half_width=np.pi, depth=0, ell=1.0, out=out)
# out[node] is now a 2-vector inside the unit disk. No gradients involved.

Trace the data flow: the tree structure goes in as children, an adjacency dict; what comes out is a dict of 2-vectors, each with norm strictly below 1 because tanh never reaches it. The only free parameter is ell, and Chapter 2’s rule tells you how to set it: above ln(max branching factor). Every parent–child pair ends up at true distance close to ell, and every leaf-to-leaf distance ends up close to the tree distance times ell. That is distortion near 1, in two dimensions, with no optimiser at all.

Why then does anyone train? Because most real data is not a clean tree given to you in advance. It is a noisy partial order with cycles, missing edges and multiple parents — and that is what Chapter 3 handles.

Chapter 3: Embedding WordNet

Time to train something. The 2017 paper by Maximilian Nickel and Douwe Kiela — Poincaré Embeddings for Learning Hierarchical Representations — is the origin of this whole line of work, and its setup is small enough to hold in your head entirely.

The data, exactly

WordNet is a hand-built lexical database. Its noun hierarchy is a hypernymy graph: an edge u → v means “u is a kind of v”, as in dalmatian → dog. Take the transitive closure — if dalmatian is a dog and a dog is a carnivore, add the edge dalmatian → carnivore directly. That gives

Two remarks on that choice, because it is not innocent. First, taking the closure means the model sees dalmatian → entity as a positive pair, so it is trained to place remote ancestors nearby too — which is what pulls generic words to the centre. Second, WordNet nouns are not a perfect tree; a few synsets have more than one parent. The model does not care. It never assumes a tree; it just fits distances to a set of pairs.

The objective, and why it is a ranking loss

For every observed pair (u, v) we want d(u, v) to be small relative to the distance from u to things that are not its hypernyms. Sample a handful of such non-neighbours — the negatives — and write a softmax over the negated distances:

where N(u) is the true v plus ten sampled negatives. If you have met InfoNCE or word2vec’s negative sampling, this is the same object with one substitution: the score is −d instead of a dot product. That substitution is the entire hyperbolic contribution at the loss level.

It is a relative criterion, which matters. Nothing in the loss says “put dalmatian at distance 0.7 from dog.” It only says “closer than to trombone.” The absolute scale — how far out the leaves end up — is decided by the geometry and the optimiser, not by you. That is a feature: the model discovers how much radius the hierarchy needs.

Worked example: one loss term, by hand

Take a positive pair currently at d = 1.2 and five sampled negatives at distances 2.8, 3.0, 3.5, 4.0 and 5.0. Exponentiate the negated distances:

Now suppose the positive drifts outward so d = 2.5. Then e^−2.5 = 0.08209, the denominator becomes 0.24793, p = 0.3311 and the loss is −ln 0.3311 = 1.1054. A drift of 1.3 units cost us a factor of 2.5 in loss. The gradient will pull it back.

And notice the shape of the push on the negatives. Their gradient weights are their softmax shares:

The nearest negative gets a push nine times stronger than the farthest. The loss spends almost all of its effort on the confusions that are actually happening and essentially ignores the obviously-wrong candidates. This is why ten negatives is enough on a graph with 82,115 nodes: you do not need to push everything away, only the current intruders.

Riemannian gradient: why the plain gradient is the wrong direction

Here is the part that trips people up. Your autograd gives you ∂ℓ/∂θ — the derivative with respect to the stored coordinates. But coordinates are map coordinates, and we established in Chapter 1 that the map is distorted. Following the coordinate gradient is like navigating by a Mercator map and concluding that walking one centimetre near the pole covers the same ground as one centimetre at the equator.

The correction is exact. In the Poincaré ball the metric is λ² times the identity, so converting a coordinate gradient into a true direction of steepest ascent means dividing by λ²:

because λ = 2/(1 − ‖θ‖²) means 1/λ² = (1 − ‖θ‖²)²/4. Write α = 1 − ‖θ‖² from here on. The update rule is then

with ε = 10⁻⁵. That is the whole optimiser. Two lines on top of SGD.

The rescale factor is enormous, and it has to be

The distance function’s own gradient has Euclidean norm 2/α — you can verify this on the radial case, where d = ln((1+r)/(1−r)) and dd/dr = 2/(1−r²). So the coordinate gradient blows up near the rim, and the factor α²/4 shrinks there. Multiply them:

The true step length is η, everywhere, at every radius. That is the whole design goal of a Riemannian optimiser, achieved exactly. In coordinates, with η = 0.1:

What happens if you skip it — two concrete disasters

Disaster one: plain SGD. At ‖θ‖ = 0.999 the raw gradient has norm 2/0.001999 = 1,000.5. A step of η = 0.1 moves the point 100 units in coordinates — a hundred times the radius of the entire disk. The projection catches it and slams it to 1 − 10⁻⁵. Every deep node ends up pinned at the rim in a single epoch, all of them at the same radius, hierarchy annihilated. The loss will not even look bad, because a rim full of points can still satisfy many relative ranking constraints.

Disaster two: plain Adam. Adam normalises away the gradient magnitude, so its coordinate step is about the learning rate regardless of radius — say 0.001. Convert to true distance by multiplying by λ:

The same nominal learning rate moves outer nodes 375× faster in real distance than inner ones. At ℓ = 0.7 per level, one Adam step teleports a leaf 1.4 levels. Your leaves thrash while your root barely moves. This is the single most common reason a first hyperbolic experiment produces garbage, and the fix is one line: use a Riemannian optimiser (geoopt.RiemannianAdam and friends do the rescaling and the parallel transport of momentum for you).

Burn-in: the trick that looks like a hack and is not

Nickel and Kiela initialise every embedding uniformly in [−0.001, 0.001] — essentially all 82,115 nouns stacked on the origin — and then run ten epochs at one tenth of the learning rate before training properly. That is called the burn-in, and the reason it is necessary is a direct consequence of the geometry.

Ask how hard it is to change a node’s angle once it has drifted outward. Moving by Δφ radians at true radius s covers an arc of true length sinh(s) · Δφ. So to swing 0.1 radians:

Angular mobility falls by roughly 750× between radius 1.1 and radius 7.6. Once a node has travelled out, it is angularly frozen — it can still slide radially, but it can no longer change which branch of the tree it belongs to. The angular layout is therefore decided early or not at all. Burn-in is a deliberate period in which everything stays near the origin, cheap to rotate, while the loss sorts out who belongs beside whom. Only then are norms allowed to grow and the hierarchy to stretch out.

Concept → realization: the full training step

python
import torch

EPS = 1e-5

def project(x, eps=EPS):
    """Force every row strictly inside the unit ball. Shape in = shape out."""
    norm = x.norm(dim=-1, keepdim=True).clamp_min(1e-12)
    factor = (1 - eps) / norm
    return torch.where(norm >= 1, x * factor, x)

def rsgd_step(emb, lr):
    """Riemannian SGD on nn.Embedding weights living in the Poincare ball."""
    with torch.no_grad():
        w, g = emb.weight, emb.weight.grad          # both (n, d)
        alpha = 1 - w.pow(2).sum(-1, keepdim=True)  # (n, 1)
        w -= lr * (alpha.pow(2) / 4) * g              # THE metric correction
        emb.weight.data = project(w)
        emb.weight.grad = None

# ---- one training iteration ----------------------------------------
u  = emb(idx_u)                       # (B, d)      the child
v  = emb(idx_v)                       # (B, d)      the true hypernym
vn = emb(idx_neg)                     # (B, K, d)   K sampled non-hypernyms

d_pos = poincare_dist(u, v)                          # (B,)
d_neg = poincare_dist(u.unsqueeze(1), vn)             # (B, K)
logits = torch.cat([-d_pos.unsqueeze(1), -d_neg], 1)  # (B, 1+K)
loss = torch.nn.functional.cross_entropy(
           logits, torch.zeros(len(u), dtype=torch.long))  # target = index 0
loss.backward()
rsgd_step(emb, lr=0.1 if epoch >= burn_in else 0.01)

Trace the shapes once and the whole method is demystified. In: two index tensors and a negative-sample tensor. Out: a scalar. The only non-standard lines are poincare_dist (Chapter 1), the alpha.pow(2)/4 rescale, and project. Everything else is a vanilla contrastive setup.

The result that started the field

On WordNet noun reconstruction — embed the full transitive closure, then for each pair rank the true hypernym against all non-neighbours — the paper reports roughly:

The comparison to stare at is row 2 against row 3. Forty times fewer parameters, roughly five times the MAP. And the Poincaré model has already saturated at dimension 5 — going to 200 buys it almost nothing, because the hierarchy simply does not need more than a couple of dimensions once the curvature is right. The mean rank of about 5 means that on average the true hypernym is the fifth-closest point out of 82,115.

Chapter 4 opens that trained model up and reads it.

Chapter 4: Reading a Poincaré Embedding

Training finished. You have 82,115 five-dimensional vectors, every one with norm below 1. What do they mean? A Euclidean embedding gives you one readout — distance means relatedness — and nothing else. A hyperbolic embedding gives you two, and the second one is free.

Why generality drifts to the centre — the mechanism, not the slogan

It is easy to say “the root ends up in the middle” and hard to say why, so here is the argument.

The training loss requires mammal to be close to all of its thousands of descendants. Those descendants are spread over a wide range of directions, because they are also being pushed apart from each other. So mammal must find a location that is simultaneously close to points spread over a wide angular fan.

In hyperbolic space that location is uniquely determined, and there is a formula for it. For two points at true radii s_u, s_v separated by angle φ, once both radii are more than about 3, the distance is very accurately

Check it against the exact calculation from Chapter 1: two points at r = 0.99 (true radius 5.2933) and φ = 90°. The formula gives 5.2933 + 5.2933 + 2 ln(0.7071) = 10.5866 − 0.6931 = 9.8935. The exact arcosh computation gave 9.89351. Agreement to five significant figures.

Now rewrite it. Define h = −ln sin(φ/2), a positive number that gets larger as the angle gets smaller. Then

That is literally the distance formula in a tree: depth(u) + depth(v) − 2 × depth(lowest common ancestor). Hyperbolic geometry has handed us an LCA, and its depth is −ln sin(φ/2) — determined entirely by the angle between the two points. Points that share a direction share a deep ancestor; points on opposite sides share only the root.

Worked example: placing a WordNet chain by hand

Take the real hypernym chain for dalmatian and give each level a true radius of 0.7 × its depth. Convert with ‖x‖ = tanh(s/2):

Stare at the right-hand column. mammal and dalmatian are five levels apart — five whole steps of the hierarchy — and their stored norms differ by 0.999889 − 0.996334 = 0.0036. Between dog and dalmatian, one level, the difference is 0.000112.

Worked example: recovering the common ancestor from the angle

Put dalmatian and siamese cat in the picture. Both are at level 14, so true radius 9.8. Their lowest common ancestor in WordNet is carnivore at level 11, so a correct embedding should give them

What angle does that require? Invert the tree formula: h = −ln sin(φ/2) = 7.7, so sin(φ/2) = e^−7.7 = 4.5 × 10⁻⁴, giving

which is exactly the number Chapter 2 got by the other route, from the exact distance formula. Two independent calculations, same answer — the asymptotic tree formula and the exact metric agree.

And now the practical reading: an ancestor three levels up costs you a factor of e^3×0.7 = 8 in angular resolution. Every level of depth you want to resolve divides the angular budget by e^ℓ. That is the same exponential from Chapter 2, seen from inside the trained model.

Turning two readouts into a directed prediction

Distance is symmetric, so it alone cannot answer “is u a kind of v, or the other way round?” Nickel and Kiela combine the two readouts into an asymmetric score:

with a large constant α (they used 1000). Read it as: start from the negated distance, then penalise the hypothesis if the claimed parent v is further out than the claimed child u. Work an example with the chain above. dog has norm 0.999777, dalmatian 0.999889, and suppose their distance is 0.7.

Higher is better, so the correct direction wins by 0.156. It works — but look how thin the margin is, because the norm difference is only 0.000112 and needs a multiplier of a thousand to matter at all. This is a patch, not a principle: the geometry gives you an order, and we are reconstructing it from a scalar with a hand-tuned constant. Chapter 5 replaces the patch with a construction that encodes the order directly.

How the model is actually scored

Two protocols, and it is worth knowing exactly what they measure.

Reconstruction embeds the complete relation set and asks whether the geometry can represent it at all — a capacity test. For each observed pair (u, v) you rank v against all of u’s non-neighbours by distance, and record the rank.

Link prediction holds out a slice of edges and asks whether the geometry generalises — whether unseen is-a relations land in the right place. Reconstruction can be gamed by capacity; link prediction cannot.

Both report MAP, mean average precision. Compute an average precision by hand so the number stops being an abstraction. Suppose a node has three true hypernyms and, after sorting all candidates by distance, they appear at positions 1, 3 and 6:

MAP is that, averaged over all nodes. A MAP of 0.82 with mean rank 5 on 82,115 candidates means the true parent is essentially always in the top handful — the top 0.006% of the vocabulary.

What the norm does not tell you

Be honest about the limits of the readout, because “norm equals generality” is repeated far more often than it is qualified.

That last row deserves emphasis. The Poincaré ball has a rich group of distance-preserving maps (Möbius transformations) that move the origin anywhere. The loss cannot see them, so “the origin is the root” is a convention that training only approximately respects — it holds because initialisation starts everyone at the origin, not because the objective demands it. If your downstream code reads norms as depths, pin the root.

Chapter 5: Entailment Cones — Order as Geometry

Chapter 4 ended on an awkward note. Hierarchy is an order: dalmatian is under dog, and dog is emphatically not under dalmatian. But distance is symmetric — d(u,v) = d(v,u) by definition — so the direction had to be smuggled back in through a hand-tuned norm term.

Octavian Ganea, Gary Bécigneul and Thomas Hofmann asked the better question in 2018: what geometric object is asymmetric by construction? Their answer is a cone. Attach to every point x a region C(x) spreading outward from it, and define

Membership in a set is not symmetric. The direction is now built into the geometry rather than bolted onto a score, and “list everything under mammal” becomes an actual geometric query instead of a threshold on a distance.

What the cone field must satisfy

You cannot draw arbitrary cones. If this is to model a hierarchy, the relation must be a partial order, and the crucial requirement is transitivity: if w is under v and v is under u, then w must be under u. In cone language that says

— the cones must nest. A cone can never poke out of a cone it sits inside. That is a strong constraint, and it is what determines the shape. If you also ask for cones that are symmetric about the outward radial direction (so that “downward” means “away from the root”), the nesting requirement pins the half-aperture to a single functional form:

with a single free constant K; the paper uses K = 0.1. Everything about the behaviour of entailment cones is in that fraction (1 − r²)/r, which is large near the origin and small far out.

That table is the hierarchy, written as a picture. Generic concepts own wide cones and therefore many descendants; specific concepts own needles and therefore almost none. And you get the aperture for free from the position — there is no per-node width parameter to learn.

There is a hole at the centre, and it is unavoidable

For arcsin to be defined we need K(1 − r²)/r ≤ 1. Solve Kr² + r − K ≥ 0 for the positive root:

So with K = 0.1 no point may sit within about 0.099 of the origin — there is a small forbidden disk in the middle. The interpretation is honest and worth stating: there is no single top element. The cone construction cannot represent a node that entails literally everything, because that would need an aperture of 180°, which the formula caps at 90° near the hole’s edge. In practice you place the root just outside the hole and accept that it is an extremely general concept rather than a universal one.

The angle you have to measure

To test membership you need the angle at x between two directions: the outward radial direction (the cone’s axis) and the geodesic heading from x toward y. Call it Ξ(x, y). Because geodesics are circular arcs, the closed form is not pretty, but it is elementary:

Do not accept a formula like that without testing it. Take x = (0.3, 0) and y = (0.6, 0) — y is directly outward from x, so the answer must be exactly zero.

The formula survives its own sanity check, numerator and denominator agreeing to four digits by hand.

The energy, and a worked pair

Membership is the test Ξ(x,y) ≤ ψ(x). Turn it into something differentiable by measuring the violation:

Zero when y is inside x’s cone; otherwise the number of radians y would have to swing to get in. Work an example. With x = (0.3, 0) the aperture is ψ = arcsin(0.1 × 0.91 / 0.3) = arcsin(0.3033) = 0.3082 rad = 17.66°.

Note the third row: (0.35, 0.30) is at a larger radius than x and quite close to it in ordinary distance, yet it is emphatically not a descendant. Radius alone would have said “more specific, therefore probably below.” The cone says “more specific, but down a different branch,” which is the distinction Chapter 4’s norm-based score could only approximate.

The loss

Positives should have zero energy; negatives should have energy above a margin γ:

This is a max-margin hinge, and it behaves differently from Chapter 3’s softmax in a way that matters. The softmax always has gradient — every negative contributes something forever. The hinge switches off: once a positive is inside the cone and a negative is γ outside, they produce exactly zero gradient. The model stops pushing things it has already got right, which is why cone training converges to a crisp configuration instead of a perpetual tug-of-war.

The cost of that crispness is a cold start. If the initial layout is random, most positives are wildly outside their cones and most negatives are too, so the gradient signal is dominated by noise and the geometry has no reason to organise. Ganea et al. therefore initialise from a trained Poincaré embedding and let the cone loss refine it. Read that as a two-stage recipe: distance-based training to arrange the space, then cone-based training to sharpen the order.

Cones versus distance, honestly

The reported gains are concentrated exactly where you would predict from that table: in link prediction with sparse supervision. Train on only the direct WordNet edges — no transitive closure — and ask the model to infer the closure. A distance model has to be told that dalmatian is close to entity; a cone model gets it from nesting, because dalmatian ∈ C(dog) ⊆ … ⊆ C(entity) follows from the geometry. As you reveal more of the closure during training, the advantage narrows — the distance model can eventually be shown everything the cone model deduced.

Chapter 6: MERU — Hyperbolic Vision-Language

Everything so far has been about explicit hierarchies: WordNet, a taxonomy, a graph you were handed. The 2023 paper Hyperbolic Image-Text Representations — the model is called MERU — asks a bolder question. Images and their captions form a hierarchy too, one nobody wrote down. Can a vision-language model discover it?

The hierarchy nobody labelled

Look at a photograph of a dalmatian asleep on a red sofa in a sunlit room. Now list some true captions:

That is a chain from general to specific, and it points one way. Text is more generic than the images it describes — a caption always underspecifies, because it omits the exact lighting, the exact pose, the exact fabric of the sofa. An image is a leaf; a caption is an interior node.

Why CLIP cannot represent that

CLIP encodes an image and a caption, L2-normalises both, and trains with a contrastive loss on cosine similarity. That normalisation is not a detail — it is a geometric commitment. It forces every embedding onto the surface of a unit sphere.

You see the symptom in practice. Ask CLIP for the images nearest to “a dog” and you get dogs; ask for the captions nearest to a dog photo and you get dog captions. Fine. But there is no operation that says “give me the more general concept” or “is this caption a valid generalisation of that one?” The space simply does not have that axis.

MERU’s change: one space, one root, two encoders

MERU keeps the two encoders (a vision transformer and a text transformer) and keeps the contrastive objective. It changes the space and adds one loss term.

The Lorentz model, and why they switched off the disk

MERU does not use the Poincaré ball. It uses the Lorentz model (also called the hyperboloid model), and the reason is numerical — a preview of Chapter 7.

In the Lorentz model a point of n-dimensional hyperbolic space is a vector in Rⁿ⁺¹ with one special “time” coordinate x₀, constrained to lie on the sheet

where c > 0 is the curvature magnitude (MERU makes it learnable). Only the n space coordinates are stored; the time coordinate is recomputed as x₀ = √(1/c + ‖x_space‖²). The distance is

The contrast with the disk is the point. In the Poincaré ball, moving far from the root drives ‖x‖ toward 1 and the quantity 1 − ‖x‖² toward zero — a catastrophic cancellation, two nearly equal numbers subtracted, losing precision exactly where your leaves live. In the Lorentz model, moving far from the root makes the coordinates grow: a point at distance s has x₀ = cosh(s)/√c. Large numbers keep their relative precision. Nothing is cancelling.

Worked example: the tangent norm is the specificity

The exponential map at the root, with c = 1, is

Take an encoder output with ‖v‖ = 0.8. Then cosh(0.8) = 1.33743 and sinh(0.8) = 0.88811. Check the constraint first — it should land on the sheet:

(exactly −1 up to rounding, since cosh² − sinh² = 1). Now its distance from the root, which is the point (1, 0, …, 0):

The entailment term

Contrastive loss alone would place matching pairs near each other and say nothing about order, so MERU adds a cone term of exactly Chapter 5’s shape, transplanted to the hyperboloid. The half-aperture at x is

— again shrinking as you move outward, again with a forbidden region near the root (you need ‖x_space‖ ≥ 2K = 0.2 for the arcsin to exist). At ‖v‖ = 0.8 the space norm is sinh(0.8) = 0.888, so

The loss then penalises max(0, exterior angle − aperture) for each (text, image) pair, with the text as the apex. In words: every caption must contain its images inside its cone. Since a narrower cone means a smaller radius is needed to contain a given spread, the pressure pushes text inward and images outward, and the general-to-specific axis organises itself.

What it bought, stated without hype

MERU was trained on RedCaps, a public corpus of roughly 12 million image–text pairs, against a CLIP baseline trained on the identical data with the identical encoders — the only honest way to attribute a difference to the geometry. Two findings:

That is the mature reading. The geometry did not make the model smarter; it made the model’s representation structured, and gave it an operation — “generalise” — that a spherical space cannot express at all. Whether that structure is worth the engineering is exactly the question of Chapter 7.

Concept → realization: the encoder head that changes

python
import torch

# ---- CLIP head: normalise onto the sphere, score with cosine -------
z_img = torch.nn.functional.normalize(img_enc(pixels), dim=-1)   # (B, n), ||z|| = 1
z_txt = torch.nn.functional.normalize(txt_enc(tokens), dim=-1)   # (B, n), ||z|| = 1
logits = z_img @ z_txt.T / tau                                 # (B, B)

# ---- MERU head: do NOT normalise; lift onto the hyperboloid -------
v_img = img_enc(pixels)            # (B, n) — the NORM now carries specificity
v_txt = txt_enc(tokens)            # (B, n)

def exp_map0(v, c):
    """Tangent vector at the root -> space coords on the hyperboloid. (B, n)"""
    n = v.norm(dim=-1, keepdim=True).clamp_min(1e-8)
    return torch.sinh(c.sqrt() * n) * v / (c.sqrt() * n)

def lorentz_dist(x, y, c):
    x0 = (1/c + x.pow(2).sum(-1)).sqrt()     # time coord, recomputed not stored
    y0 = (1/c + y.pow(2).sum(-1)).sqrt()
    ip = -x0 * y0 + (x * y).sum(-1)         # Lorentzian inner product
    return torch.acosh((-c * ip).clamp_min(1 + 1e-7)) / c.sqrt()

x_img, x_txt = exp_map0(v_img, c), exp_map0(v_txt, c)
logits = -lorentz_dist(x_img.unsqueeze(1), x_txt.unsqueeze(0), c) / tau   # (B, B)
loss = contrastive(logits) + w * entailment(x_txt, x_img, c)   # text is the apex

The diff is four lines and one deletion. The deletion is the important one: removing normalize. Everything else follows from letting the encoder’s output length mean something.

Chapter 7: When It Wins, and When It Is a Gimmick

Six chapters of enthusiasm. Now the audit. Hyperbolic embeddings are one of the few ideas in representation learning with a genuinely principled motivation, and they are also one of the most over-applied. This chapter gives you a measurement to run before you commit, an honest accounting of what it costs, and the arithmetic of the failure mode that catches everyone.

The measurement: is your data actually tree-like?

“Tree-like” is not a vibe; it is a measurable quantity called Gromov δ-hyperbolicity. Take any four points w, x, y, z and form the three ways of pairing them up:

The four-point condition says: the two largest of those three sums differ by at most 2δ. The smallest δ that works for every quadruple is the space’s δ-hyperbolicity. Two hand calculations settle what it means.

Case 1 — four leaves of a binary tree. Leaves a, b are siblings; c, d are siblings in the other half. So d(a,b) = 2, d(c,d) = 2, and every cross pair is 4.

Case 2 — a four-cycle. Four nodes in a square with unit edges: adjacent pairs are 1 apart, opposite pairs are 2 apart.

Normalise by the diameter to compare across datasets: δ_rel = 2δ/diameter. The tree gives 0. The square gives 2 × 1 / 2 = 1, the maximum. A square is the purest possible non-tree: there are two equally good routes between opposite corners, and a tree never offers a choice of routes.

Where the win actually is: dimension, not accuracy

The most common overclaim is “hyperbolic embeddings are more accurate.” The precise claim is narrower and more useful: hyperbolic embeddings reach a given quality at far lower dimension. Plot WordNet reconstruction MAP against dimension and the shape is unmistakable.

Read the practical consequence off the chart. If you are storing 82,115 vectors, the difference between 5 and 200 dimensions is 410,575 floats versus 16,423,000 — 1.6 MB versus 66 MB, and a fortieth of the memory bandwidth on every lookup. For a taxonomy served at query time inside a recommender or a retrieval reranker, that is the entire business case. For a research benchmark where you were going to use 768 dimensions anyway, there is no business case at all.

The engineering tax, itemised

Every one of these is real, and none of them appears in the abstract of a paper.

The precision ceiling, computed

At true radius s, the quantity everything depends on is α = 1 − ‖x‖² = sech²(s/2) ≈ 4e^−s. It decays exponentially. Meanwhile your float type can only resolve numbers near 1 down to its machine epsilon. Set them equal:

Past that radius, 1 − ‖x‖² rounds to exactly zero and your distances become inf and then nan. And it is worse than a hard wall, because the relative error in α is ε_machine/α = ε e^s/4, which degrades long before the wall:

Now convert to hierarchy levels, which is what you actually care about. Chapter 2 said a tree with branching factor b needs edge length ℓ > ln b. So the number of levels you can afford is s_max / ℓ:

And be clear that the Lorentz model is a different trade, not a free lunch. There the coordinates grow like cosh(s) so far-apart points are fine — but the distance is arcosh(−c⟨x,y⟩_L), and for two nearby points that inner product is a difference of two huge and nearly equal numbers. The cancellation moved from “far from the origin” to “close together.” No model of hyperbolic space is uniformly well-conditioned; pick the one whose bad case you do not have.

The decision procedure

That last row is the one people skip. The literature is full of hyperbolic variants reported against Euclidean baselines at a different dimension, a different optimiser, or a different training budget. The geometry is a hypothesis about the data, and hypotheses need controls. Run the matched baseline before you believe your own result — the way MERU ran a CLIP baseline on identical data and reported honest parity.

Chapter 8: Geometry Is a Design Axis

The narrow lesson of this course is “use hyperbolic space for hierarchies.” The broad lesson is much more useful: the space is a hyperparameter, you have always been setting it, and until now you were setting it by default.

One formula for all three

Chapter 2 measured a space by the circumference of a circle of radius r. That single measurement organises every constant-curvature geometry into one family:

Three formulas, one knob. Turn K positive and the circumference eventually turns around and shrinks back to zero — you are on a sphere and you have come back to the far side. Set it to zero and you get the straight line 2πr. Turn it negative and you get exponential growth.

Two more geometries that are not curvature at all

Curvature is one axis. There is a second family that encodes order without any curvature, and it is worth knowing because it often beats hyperbolic space on the specific job of predicting order.

Order embeddings (Vendrov et al., 2016) put every concept in the non-negative orthant and define

with the violation penalty E(u,v) = ‖max(0, v − u)‖². Work it: let dalmatian = (3.0, 5.0, 2.0) and dog = (1.0, 4.0, 2.0).

Asymmetric, transitive by construction (coordinatewise ≥ is a partial order), and the origin is a universal top element with no forbidden hole. Notice what this is, geometrically: an entailment cone with a fixed aperture, axis-aligned, the same everywhere. Chapter 5’s hyperbolic cone is the same idea with the aperture allowed to depend on position — which is what buys the exponential room.

Box embeddings (Vilnis et al., 2018) go one step further and represent each concept as an axis-aligned box, with containment as the order and volume as probability:

Worked: let box(dog) = [0,4] × [0,3], area 12, and box(dalmatian) = [1,2] × [0,3], area 3, entirely inside it. The intersection has area 3, so

A calibrated conditional probability, in both directions, from a volume ratio. Neither the Poincaré distance nor the cone energy can give you that — they give an order, not a measure. The price is a notorious optimisation problem: when two boxes are disjoint the intersection volume is exactly zero, so the gradient is exactly zero and they never find each other. The fix is smoothed or Gumbel boxes, which soften the corners so a gradient survives.

Product spaces (Gu et al., 2019) refuse to choose. Represent a point as a tuple across several factors — say two hyperbolic planes, a sphere and eight Euclidean dimensions — and define the squared distance as the sum of the squared distances in each factor. Real data is rarely uniformly one shape: a social graph is tree-like around hubs and grid-like inside dense communities. The signature (how many factors of each curvature) becomes something you search over, guided by the local δ_rel from Chapter 7.

The choosing table

Chapter 9: Connections

Nothing new here — just the map of where this sits, what it does not solve, and where to go next.

The lesson in eight sentences

Where to go next on this site

Honest limitations

Four things this lesson taught you to do, and the reasons each might not be the right move.

References

1. Nickel, M. & Kiela, D. “Poincaré Embeddings for Learning Hierarchical Representations.” NeurIPS, 2017. arXiv:1705.08039 — Chapters 3–4.
2. Nickel, M. & Kiela, D. “Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry.” ICML, 2018. arXiv:1806.03417 — the numerical argument of Chapters 6–7.
3. Ganea, O., Bécigneul, G. & Hofmann, T. “Hyperbolic Entailment Cones for Learning Hierarchical Embeddings.” ICML, 2018. arXiv:1804.01882 — Chapter 5, including the aperture and angle formulas.
4. Desai, K., Nickel, M., Rajpal, T., Johnson, J. & Vedantam, R. “Hyperbolic Image-Text Representations” (MERU). ICML, 2023. arXiv:2304.09172 — Chapter 6.
5. Sala, F., De Sa, C., Gu, A. & Ré, C. “Representation Tradeoffs for Hyperbolic Embeddings.” ICML, 2018. arXiv:1804.03329 — the precision analysis behind Chapter 7’s ceiling.
6. Chami, I., Ying, R., Ré, C. & Leskovec, J. “Hyperbolic Graph Convolutional Neural Networks.” NeurIPS, 2019. arXiv:1910.12933 — δ-hyperbolicity used as a model-selection signal.
7. Khrulkov, V. et al. “Hyperbolic Image Embeddings.” CVPR, 2020. arXiv:1904.02239 — measuring δ on vision datasets.
8. Vendrov, I., Kiros, R., Fidler, S. & Urtasun, R. “Order-Embeddings of Images and Language.” ICLR, 2016. arXiv:1511.06361 — Chapter 8.
9. Vilnis, L., Li, X., Murty, S. & McCallum, A. “Probabilistic Embedding of Knowledge Graphs with Box Lattice Measures.” ACL, 2018. arXiv:1805.06627 — box embeddings and calibrated conditionals.
10. Gu, A., Sala, F., Gunel, B. & Ré, C. “Learning Mixed-Curvature Representations in Product Spaces.” ICLR, 2019 — product manifolds and signature search.
11. Sarkar, R. “Low Distortion Delaunay Embedding of Trees in Hyperbolic Plane.” Graph Drawing, 2011 — the 1+ε construction of Chapter 2.
12. Bourgain, J. “The Metrical Interpretation of Superreflexivity in Banach Spaces.” Israel J. Math., 1986 — the √(log h) lower bound on Euclidean tree distortion quoted in Chapter 0.
13. geoopt — Riemannian optimisation for PyTorch (Riemannian Adam, the Poincaré ball and Lorentz manifolds). github.com/geoopt/geoopt