Attention Sinks & Streaming

Chapter 1: What the Cache Holds

Chapter 0 asked you to trust a rough number: about half a megabyte of cache per token. Trust nothing in this lesson without deriving it. Open the box.

Why K and V, and not Q

Self-attention at one layer takes the hidden state for every token and produces three projections: a query vector Q (what am I looking for), a key vector K (what do I contain, for others to match against), and a value vector V (what I actually contribute if someone attends to me). Attention for the current token is computed by comparing its query against every previous token's key, turning those comparisons into weights with softmax, and mixing the previous tokens' values by those weights.

Now think about what changes from one decoding step to the next. When the model is about to generate token number 501, it needs the keys and values of tokens 1 through 500 — and those never change once computed, because a token's key and value depend only on that token's own hidden state at that layer, not on what gets generated afterward. But it only ever needs the query of the newest token, computed fresh at each step. There is nothing to gain by caching queries; there is everything to gain by caching keys and values, because recomputing them for all 500 prior tokens on every single new step would make generation cost grow quadratically in the sequence length. Cache K and V once, reuse them forever — that is the entire reason the KV cache exists.

The byte arithmetic, for a real model

Use Llama-2-7B's published architecture — it is one of the four model families this paper actually tests, so the numbers are not arbitrary. Hidden size d = 4,096, split across 32 attention heads (so each head is 128-dimensional), and 32 transformer layers. Store everything in half precision, 2 bytes per number, which is standard for serving.

Step 1 — one token, one layer. You need to store a key vector of width 4,096 and a value vector of width 4,096 — the per-head split doesn't change the total width, it's still 4,096 numbers each:

Step 2 — across all 32 layers. Every layer keeps its own independent K and V for that token, because attention happens separately at every layer:

The same derivation, cross-checked in a different unit

A good habit for any back-of-envelope calculation: redo it in a completely different unit and confirm the two answers agree. Step 1 and Step 2 worked in bytes. Redo them in bits, where each fp16 number is 16 bits instead of 2 bytes:

Same answer, reached by a different path. This isn't padding — cross-checking a derivation in a second unit system is a genuinely useful habit for catching silent factor-of-8 or factor-of-1024 errors (bits versus bytes, KiB versus KB) before they propagate into a much larger downstream number, like the 1.9 TiB figure this chapter builds toward next.

That is the exact number Chapter 0 asked you to trust, and now you have derived it, not guessed it. Every single token — whether typed by the user or generated by the model — costs the GPU exactly 512 KiB of cache, forever, under dense attention, for as long as it stays in the cache.

Where the 4,096 and 32 actually come from

Don't let those two numbers pass as trivia. Hidden size (4,096) is the width of the vector a transformer uses to represent a single token's meaning at one layer — every token, at every layer, is one 4,096-number list, and that list is what gets projected into Q, K, and V. Layer count (32) is how many times that representation gets refined, attention block after attention block, before the model produces a prediction. Both numbers are architectural choices baked in at pretraining time; you cannot change them without retraining the model, and every downstream KV-cache calculation in this lesson inherits them.

There's a subtlety worth making explicit, because it resurfaces the moment you read a modern inference paper: splitting the hidden size across attention heads — 32 heads of 128 dimensions each, in this model — does not change the byte cost of the cache at all. Whether the model used 32 heads of 128 dimensions, 64 heads of 64 dimensions, or a single head of 4,096 dimensions, the total width being cached per token per layer is still 4,096 numbers for K and 4,096 for V. Head count changes how attention computes its weighted sums — it is irrelevant to how many bytes the cache stores. That's why Step 1's arithmetic above never needed to mention head count at all.

Two worked consequences

The day-long chat, precisely. Chapter 0 estimated 48,000 tokens in a working day. At 512 KiB each:

Add the model's own weights — 7 × 10⁹ parameters × 2 bytes ≈ 13.0 GiB — and a single conversation on a single model instance is already pressing against a 24 GiB consumer card, before you account for activation memory or a second concurrent user.

The paper's own headline number. StreamingLLM's central experiment runs a model stably out to 4 million tokens. Ask what dense attention would cost at that length:

Nearly two terabytes of cache, for one conversation, on one model. No GPU cluster you are likely to be handed makes that a sane engineering choice. Whatever the fix is, it cannot be “keep everything.”

A third worked consequence, at human scale. Zoom back in from millions of tokens to a single ordinary exchange — a user typing one message and reading the model's reply, roughly 50 tokens combined, a size you could estimate from any real chat transcript. At 512 KiB per token, that one exchange alone costs:

25 MiB sounds negligible — and it is, for a single exchange. But it's exactly the granularity at which Chapter 0's 48,000-token, 23.4 GiB estimate was built: roughly 960 such exchanges, each depositing another 25 MiB that never comes back out under dense attention. No individual step in that accumulation looks alarming. The sum, after a working day, is what alarmed Chapter 0. That gap — between a harmless-looking marginal cost and an unsustainable running total — is the entire reason unbounded growth is dangerous in a way a single large allocation wouldn't be: nothing ever throws an error until, eventually, everything does at once.

In code, so the shapes are undeniable

python
import torch

d_model, n_layers, dtype_bytes = 4096, 32, 2   # Llama-2-7B, fp16

def kv_cache_bytes(n_tokens):
    per_token_per_layer = 2 * d_model * dtype_bytes     # K and V
    return per_token_per_layer * n_layers * n_tokens

print(kv_cache_bytes(1) / 1024)              # 512.0   KiB, one token
print(kv_cache_bytes(48_000) / 1024**3)     # 23.4    GiB, one workday
print(kv_cache_bytes(4_000_000) / 1024**4)  # ~1.9    TiB, the paper's headline length

# real KV caches in torch look like this per layer:
#   k_cache: (batch, n_heads, seq_len, head_dim)
#   v_cache: (batch, n_heads, seq_len, head_dim)
# seq_len is the axis that grows by 1 every decoding step -- and the
# axis this whole lesson is about learning to bound.

The axis this lesson is not about — batch size

One more axis is worth naming so it doesn't get confused with seq_len later: a production server rarely runs one conversation at a time. It batches several requests together on the GPU to use the hardware efficiently, and the KV cache formula picks up one more multiplicative factor for that:

Everything in this chapter has implicitly held batch_size = 1. Double the number of simultaneous conversations a GPU serves, and the cache bill doubles too, independent of — and stacked on top of — whatever seq_len is doing. This lesson is entirely about taming seq_len, because that's the axis with no ceiling at all; batch_size is a knob an operator chooses and can always turn down. Keep the two straight: one is a design decision, the other is a runaway variable.

At a batch of 32 — a modest number for a busy endpoint — the cache alone, at just 1,024 tokens per conversation, is already competing with the weights for space on the card. Batch size makes the ceiling this lesson builds toward matter even more, not less: whatever per-conversation bound Chapter 5 lands on gets multiplied by however many conversations are being served at once.

The general formula, in code

python
def kv_cache_bytes_general(n_tokens, n_kv_heads, head_dim, n_layers, dtype_bytes=2):
    return 2 * n_kv_heads * head_dim * dtype_bytes * n_layers * n_tokens

# Llama-2-7B, as used throughout this lesson: 32 KV heads, 128-dim each
print(kv_cache_bytes_general(1, n_kv_heads=32, head_dim=128, n_layers=32) / 1024)
# 512.0 KiB -- matches Chapter 1's original derivation exactly

# a hypothetical variant with only 8 KV heads (same head_dim, same query heads)
print(kv_cache_bytes_general(1, n_kv_heads=8, head_dim=128, n_layers=32) / 1024)
# 128.0 KiB -- 4x smaller, purely from decoupling KV head count from query head count

One function, two calls, and the entire relationship between architecture and cache cost from earlier in this chapter becomes something you can actually run and check, rather than take on faith.

A fourth worked consequence: the cost of doing nothing

One more scenario worth pricing out, because it surprises people the first time they hit it in production: a conversation the cache does not know is idle. A user opens a chat, exchanges a few messages, then leaves the tab open without typing for the rest of the day. Does the cache shrink while nothing happens?

No. The KV cache's size is a function of how many tokens have been processed, not how recently, or how frequently. An idle conversation's cache costs exactly the same as an active one with the same token count — the 512 KiB/token bill was already paid the moment each token was generated, and nothing about later inactivity refunds it. A service holding open 1,000 idle conversations, each having accumulated 2,000 tokens before going quiet, is still paying:

This is why production serving systems need an eviction or timeout policy for the conversation level, not just the token level this lesson focuses on — a separate, complementary concern from everything Chapters 2 through 9 build, but one that becomes obvious the moment you price out what an idle session actually costs.

Chapter 2: Window Attention Fails

Chapter 1 gave you the bill: 512 KiB per token, unbounded. The engineering instinct that follows is universal — cap it. Keep a fixed-size sliding window of the most recent L tokens' KV pairs; when a new token arrives, evict the single oldest one. Memory is now flat forever, at L × 512 KiB. This is window attention, and it is the first thing anyone tries.

The experiment that breaks the intuition

The paper runs this exact test on Llama-2-13B against the PG-19 benchmark — a test set of full-length books, used here as one long 65,000-token stream, exactly the “long, continuous text” a streaming deployment would actually see. Cache size is fixed at 1,024 recent tokens. Language-modeling perplexity is measured throughout — recall from Chapter 0 that this is a “how surprised was the model” score, and that ordinary dense attention on ordinary text usually sits in the single digits.

Read that gap again. Nothing about the model changed between those two rows — same weights, same 1,024-token budget, same text. The only difference is whether four specific tokens, from the very beginning of the stream, sixty-thousand-plus tokens in the past by the end, are kept or evicted. Keeping them is the difference between a broken model and a working one.

A second experiment, same conclusion

Table 1's 65,000-token single-book test isn't the only place this collapse shows up. The paper runs a second, complementary check on a shorter 20,000-token stream, comparing three approaches side by side: dense attention (cache everything), window attention (cap and evict), and the fix this lesson builds toward starting in Chapter 5. The pattern holds across both experiments, which matters — a single unlucky test could be a fluke; two independent tests agreeing is a diagnosis.

Dense attention is fine for a while, then fails once the input length crosses the model's pretraining window — a genuinely different failure mode from window attention's, worth distinguishing now so it doesn't get confused later. Dense attention breaks because the model has never seen positions that far out during training; window attention breaks because of what gets evicted, independent of how far out the absolute position goes. They fail for different reasons, at different moments, and (as you'll see in Chapter 5) they call for different kinds of fixes. The approach previewed here tracks a full-recomputation oracle almost exactly across the entire 20,000-token run — the same three-curve comparison Chapter 8's showcase simulation revisits with real wall-clock timing numbers.

Just how broken is 5,158?

Perplexity and loss are related by an exponential: PPL = e^loss, where loss is the average cross-entropy in nats (natural-log units) per token. That means the gap between these two rows is not additive, it's exponential, and it is worth converting back to see how large it really is.

Nearly ten extra bits of “surprise” on every single token, once the cache starts discarding the initial ones. For comparison, going from a strong model to random guessing over a 50,000-word vocabulary is about 15.6 bits (log₂ 50,000). Window attention has traveled most of the way to random, just by throwing away four tokens.

What a perplexity of 5,158 actually means for one prediction

Bits per token is one way to feel the size of this gap. Here's a second, more concrete one: perplexity has a direct interpretation as roughly the inverse of the probability the model assigned to the token that actually came next, averaged (geometrically) across the whole stream. That inversion is close enough to exact to be worth working by hand.

Before the collapse (PPL ≈ 5.40):

After the collapse (PPL ≈ 5,158.07):

Read those side by side. Before eviction, the model on average puts something like one-fifth of its total probability mass on the exact token about to appear — a healthy, confident language model on ordinary text. After eviction, that number falls under 1-in-5,000 — the model is, on average, barely better than guessing uniformly across a large vocabulary. This is the same collapse the bits-per-token version described a moment ago, restated in the currency the model actually outputs at every step: a probability distribution over the next word.

Exactly when does it happen?

This is the detail that turns out to matter enormously: the collapse is not gradual and it is not caused by “too little context.” It happens at one precise moment — the instant the cache first evicts token number 1 to make room for a new one. Before that moment, with the cache still holding the true beginning of the stream, perplexity sits at its normal low value. The step after, it does not degrade slightly; it jumps by roughly a thousandfold, in that single step, and stays broken for the rest of the stream.

The one line of code that produces this

python
def window_step(kv_cache, new_kv, L):
    kv_cache.append(new_kv)
    if len(kv_cache) > L:
        kv_cache.pop(0)          # <-- this one line is the entire bug
    return kv_cache

That is the whole mechanism — a fixed-size deque. It looks completely reasonable. It is the single most natural thing to write. And pop(0), the very first time it fires, is the moment the model starts producing garbage. Something about token 0 — not its content, we haven't even discussed what it says — is load-bearing in a way nothing else in the cache is.

Perplexity, from the definition, in code

Every number in this chapter's table is the same one-line formula, applied to a different model, cache configuration, or stretch of text. Seeing it as code, not just as a callout from Chapter 0, makes it concrete:

python
import math

def perplexity(token_losses):
    # token_losses: per-token cross-entropy in nats, e.g. from
    # F.cross_entropy(logits, targets, reduction='none')
    mean_loss = sum(token_losses) / len(token_losses)
    return math.exp(mean_loss)          # PPL = e^(mean cross-entropy)

print(perplexity([1.69] * 100))   # ~5.42, matches the 4+1020 row above
print(perplexity([8.55] * 100))   # ~5,166, matches the 0+1024 row above

Two things worth noticing in that snippet. First, perplexity is computed after averaging the losses, not by averaging perplexities directly — because of the exponential, those give different, non-equal answers, and the field's convention is always average-then-exponentiate. Second, this function is oblivious to why a given token's loss is high or low; it just reports the aggregate. That's exactly why Chapters 3 and 4 have to go further than perplexity numbers alone — the metric tells you something broke, not what broke.

Would a smarter eviction policy fix this without knowing why?

Before Chapter 3 introduces the actual explanation, it's worth taking one more naive idea seriously, because it's the second thing anyone tries after “keep only the recent tokens” fails: what if, instead of always evicting the single oldest token, the cache evicted whichever token currently has the lowest attention weight — a content-aware policy, rather than a purely positional one?

This sounds more sophisticated, and it would be, if the problem were about which content is worth keeping. But nothing in this chapter's collapse has been about content yet — the collapse happens the instant position 0 leaves the cache, regardless of what word sits there. A content-aware policy built around “keep whatever currently has high attention” could easily be the policy most likely to evict the sink token early: a sink token, by construction, often receives near-zero attention from nearby tokens doing local, syntax-level work in the shallow layers — and a naive importance score computed the wrong way could rank it as disposable precisely because of that. Chapter 3 shows why that read is backwards: the sink's apparent unimportance in a local, shallow-layer sense is exactly what makes it structurally load-bearing everywhere else, in the deeper layers this chapter's metric can't see into.

What if eviction targeted the middle of the window instead of either end — keeping both very-old and very-recent tokens, hedging between two things that might matter? It would, by accident, delay this chapter's specific collapse, simply because it also delays evicting position 0. But delay is all it buys. Once the window has processed enough tokens that even the earliest position eventually reaches the “middle” and gets evicted, the exact same renormalization catastrophe from Chapter 4 fires, for the exact same reason. No eviction order heuristic fixes this on its own, because the problem was never about which position gets evicted last — it's about position 0 ever being evicted at all. The only policy that actually works is the one that never evicts it, which is what Chapter 5 builds.

What perplexity alone can't tell you

Notice what this chapter's numbers do and don't explain. They tell you precisely when the model breaks (the step token 1 is evicted) and precisely how badly (roughly a thousandfold). They tell you nothing about why evicting one specific token does this much damage, or whether it's really about that token specifically versus its position, versus something about the mechanism that would happen no matter which token sat there. Perplexity is a single aggregate number computed after the fact — it cannot, by construction, point a finger at which internal computation caused the damage.

Answering that question requires looking somewhere perplexity never looks: inside the attention mechanism itself, at the actual weights it assigns to each token, before they get mixed together into a prediction. That's the shift Chapter 3 makes — from measuring the symptom to inspecting the mechanism directly.

How much "safe" runway does a given window size actually buy?

Quantify the delay this chapter's misconception callout already named qualitatively. A window of size L holds the first L tokens without ever needing to evict anything — eviction only starts on token L+1. Converting that token count into real time, using Chapter 0's 100-tokens-per-minute estimate for active back-and-forth:

Even the largest window this chapter's simulation offers buys barely twenty minutes of safe operation before the exact same catastrophic collapse this chapter measured. Doubling the window again, to 4,096, buys roughly forty minutes. The runway scales linearly with window size; the requirement (Chapter 0's day-long chat, or StreamingLLM's eventual 4-million-token target) does not. This is the same shape of argument Chapter 0 already made against “just buy a bigger GPU” — a bigger window is a bigger constant against a problem that needs to survive an amount of time with no upper bound.

Why continuous books, not shuffled excerpts

One detail of the experimental setup worth noticing: the test text is real, continuous books — PG-19, 100 full public-domain novels — run through as one long stream, not a shuffled collection of short, disconnected excerpts. That choice matters for what the perplexity number actually measures. A model reading a real, continuous narrative depends on long-range coherence in a way a shuffled test set wouldn't force it to; if window attention only failed on artificially disjointed text, the result would say much less about a real streaming deployment, where a genuine, continuous conversation is exactly the kind of long, coherent stream being modeled. Testing on real books is what makes this chapter's numbers a credible stand-in for Chapter 0's day-long chat, rather than a result that happens to hold only in a contrived benchmark setting.

Chapter 3: The Sink Observation

Chapter 2 left a dangling question: what is so special about the first few tokens? Not "special" in the sense of interesting content — these are usually a beginning-of-sequence marker and a couple of ordinary words. Special in some structural sense the model itself has learned.

The authors answer this by looking directly at where attention weight goes, averaged over 256 sentences fed through Llama-2-7B, each 16 tokens long — a full map of, for every layer and every attention head, how much attention mass lands on each position in the sequence.

What the attention maps show

Two clean regimes, split by depth. In the bottom two layers, attention is local — each token mostly attends to its near neighbors, which is exactly what you'd expect from a language model tracking local syntax and nearby context. But from the third layer onward, across every single layer and every single head, a large fraction of attention mass consistently lands on the very first token of the sequence — regardless of what that token is, regardless of what the query token is, almost regardless of anything else about the sentence.

The paper names this phenomenon the attention sink: a token that absorbs attention weight the way a drain absorbs water, not because anything flows to it for a reason, but because the mechanism forces something to end up there.

The bottom two layers aren't a contradiction — they're doing a different job

It's worth pausing on the shallow layers specifically, because “local” attention isn't the odd case here — deep-layer sink attention is. In layers 0 and 1, each token's attention weight falls off roughly exponentially with distance from the query: very high on immediate neighbors, small a few positions away, negligible further out. That's the behavior you'd expect from a layer whose job is tracking local syntax — which word modifies which, what part of speech is likely next, short-range agreement between nearby words. There's no need for a sink there because there's no leftover attention to park: the query has plenty of genuinely relevant nearby candidates to spread its budget across, so softmax's forced normalization (Chapter 4's subject) never runs short of something real to attend to.

The sink phenomenon only shows up once a layer's job has shifted from “what's nearby” to something requiring a broader, more abstract view of the sequence — where a specific query frequently has no single strong local match, and the unclaimed probability mass has to go somewhere. This split between early-layer local processing and later-layer abstraction is not unique to attention sinks; it's the same progression most deep networks show between low-level feature extraction and higher-level composition. Attention sinks are what that transition looks like once you're staring at a softmax's raw numerical output instead of a hand-wavy description of “abstraction.”

Putting a number on it: how big is “a large fraction,” exactly?

“A large fraction of attention mass” is a qualitative claim so far. The paper backs it with a direct measurement at a longer, more demanding length than the 16-token windows used for the clean visualization in this chapter's widget: for sequences of 4,096 tokens, they measure the attention score — already normalized, post-softmax — that the very last token in each sequence assigns back to the very first token, at every layer, averaged over 256 sequences, with error bars showing the spread across attention heads within each layer.

The result: except for the same bottom two layers this chapter's widget marks as “local,” the first token's attention score is often more than half of the querying token's entire attention budget — on its own, competing against 4,095 other candidate positions. That holds no matter how far downstream the querying token sits; a token 4,095 positions later in the sequence still routes the majority of its one softmax output — which, remember, must sum to exactly 1 across everything — onto a single, fixed, far-away position. The same qualitative shape survives at the shorter 128-token length too, so this isn't an artifact of the short windows used for the illustration above; it's the same pattern at every scale the paper checked.

Is this a Llama-2 quirk, or a Transformer-wide one?

One more control is worth knowing about, because it changes how far you should generalize this chapter's finding. Everything so far used Llama-2-7B, a decoder-only, autoregressive model. The authors also examined an encoder Transformer — BERT, which sees the entire input sequence at once rather than one token at a time — and found the same qualitative behavior: attention concentrates disproportionately on a single, consistently-present token, in BERT's case its special [SEP] separator token, across most layers.

The control experiment: is it position, or is it meaning?

There are two live hypotheses for why the first tokens get this treatment. Either (1) their semantic content genuinely matters — maybe a beginning-of-sequence token really does carry information the model needs throughout. Or (2) the model has learned a bias toward the absolute position zero, independent of what token sits there.

The paper distinguishes these with a clean substitution test: take the same 4+1020 window-attention setup from Chapter 2, but replace the four kept initial tokens with four copies of the linebreak character "\n" — a token that carries essentially no semantic content about the passage that follows.

5.60 versus 5.40 — a small, unremarkable difference. Four meaningless linebreak tokens, occupying the first four positions, restore almost all of the lost performance, nearly matching the real initial tokens. That answers the question: it is position, not semantics. The model doesn't care what's written at the start of the sequence. It cares that there is something sitting at positions 0, 1, 2, and 3, because it learned during training to route unwanted attention there — to those addresses, not to that content.

Quantifying just how small the semantic gap really is

Put a number on “5.60 versus 5.40 is unremarkable” instead of just asserting it. Compare each condition's perplexity to the 4+1020 real-token baseline as a relative change — how much worse, in percentage terms, each substitution makes things:

Swapping the content at positions 0–3 for something semantically empty costs a rounding-error-sized 3.7% penalty. Removing those positions from the cache entirely costs a penalty roughly 25,800 times larger. Whatever information is lost when the real tokens are swapped for linebreaks is negligible next to what's lost when the positions themselves vanish. That ratio is the “position, not semantics” argument, expressed as a single number instead of an intuition about two similar-looking table rows.

What the measurement loop actually looks like

python
# conceptual sketch of the attention-map measurement behind Figure 2 / Appendix F
sink_share = []
for sentence in sentences[:256]:                    # 256 sampled sentences
    attn = model.get_attention_weights(sentence)       # post-softmax, per layer/head
    for layer in range(n_layers):
        for head in range(n_heads):
            last_token_row = attn[layer][head][-1]          # query = final token
            sink_share.append(last_token_row[0])       # its weight on position 0

print(sum(sink_share) / len(sink_share))     # averages past 50% for layer >= 2

Nothing exotic is happening here: pull the post-softmax attention weights the model already computes on every forward pass, index the column for position 0, and average. The entire “attention sink” finding is an empirical measurement of numbers the model was already producing all along — it took someone looking at the right slice of them.

Concept → realization: what this means for engineering, before we've even explained it

You don't need the mechanism yet to act on this finding — it already tells you exactly what to build. If the model's need is for positions 0–3 to be occupied by anything, then the fix isn't “never evict old tokens” (impossible, per Chapter 0) and it isn't “pick a smarter eviction policy that guesses which tokens are semantically important” (unnecessary work, since importance isn't the variable). The fix is narrower and cheaper: pin exactly those first few slots, evict everything else normally. That is StreamingLLM in one sentence, and Chapter 5 builds it. But first, Chapter 4 earns the why — because an engineer who understands why the sink exists can predict how many tokens to pin, whether it generalizes across model families, and what happens if you try to train it away. All three of those questions get answered by one equation: softmax.

Why 256 sentences, why 16 tokens

The sample size and sequence length behind this chapter's measurements aren't arbitrary. Any single sentence's attention map is noisy — a particular query might have an unusually strong local match that temporarily masks the sink behavior, or an unusual token might behave atypically. Averaging over 256 independently sampled sentences is what turns a noisy, sentence-specific pattern into a stable, reportable one; run the same measurement on 5 sentences instead and the sink's dominance would still be visible, just noisier — harder to state as cleanly as “a large fraction, consistently, across every layer and head.”

The 16-token length is a visualization choice, not a claim that sinks only exist at that scale — the longer-sequence measurement earlier in this chapter confirms the same pattern holds at sequence lengths hundreds of times longer. Short sequences make for a legible heatmap; long sequences make for a rigorous quantitative claim. The paper uses both, for what each is good at.

Not every head agrees equally

One nuance worth holding onto so this chapter's claim doesn't get overstated: the longer-sequence measurement reports error bars — the standard deviation of the first token's attention score across different heads within the same layer. That spread is real. Not every one of a layer's attention heads dumps exactly the same share of its budget onto the sink; some heads lean into the sink pattern more heavily than others, even within one layer. The claim being made is that the average is consistently high and the pattern is consistently present across layers and heads in aggregate — not that every individual head behaves identically. That's a normal, expected kind of variation for a learned system: the underlying softmax constraint applies to every head equally, but how strongly any one head ends up specializing into a sink-heavy role is itself something learned, and learned things vary from head to head.

A preview of the precision Chapter 4 brings

Everything in this chapter has described the sink phenomenon qualitatively — “a large fraction,” “often more than half.” That's the right level of precision for an observation, but it raises an obvious next question: can the exact share be predicted ahead of time, from first principles, rather than just measured after the fact? Chapter 4 answers yes, and the answer is short enough to preview here: given a set of raw attention scores (logits) before softmax, the sink's share is fully determined by nothing more than those logits and the softmax formula itself — no separate “sink mechanism” needs to exist anywhere in the model's weights beyond ordinary attention. A worked example there computes a sink absorbing 99.48% of the attention weight from five logits handed to you explicitly, by hand, in under a page. If this chapter's “large fraction” language felt unsatisfying, that's the fix.

What the linebreak control doesn't test

Be precise about the boundaries of what this control experiment actually establishes. It tests one substitute token, a linebreak, at one set of positions, on one text distribution. It does not test whether literally any token at all would work as well — an extremely rare subword, or a token the model essentially never saw during pretraining, might behave differently, since the model's learned key vector for a token's embedding is itself a function of how much (and what kind of) training exposure that token received. The position-not-semantics conclusion this chapter draws is well-supported by the evidence given, but it's a claim about ordinary tokens occupying the sink positions, not a universal claim about every possible token id a vocabulary might contain. That's a reasonable, honestly-scoped claim, not a weakness in the argument — good empirical claims usually come with exactly this kind of boundary attached.

Chapter 4: Why Sinks Emerge

Chapter 3 established the fact: early positions absorb attention regardless of content. Now derive why, from the one equation every attention layer runs.

Softmax has nowhere else to put it

For a query attending over N keys with raw similarity scores (logits) x₁, x₂, …, x_N, softmax turns them into attention weights:

Read the constraint this equation imposes, because it's the whole argument: the outputs are forced to sum to exactly 1, always, no matter what the inputs are. There is no option for the model to say “none of these tokens are relevant right now, distribute zero attention.” If the query genuinely has no strong match among the keys — the paper's own phrasing is x₁ ≫ x_j for the rest in the failure case, but the softer, more common case is simply no candidate stands out — the probability mass still has to land somewhere, in full, every single time.

That's a strange asymmetry for a learned system to sit inside of. It means every head, at every layer, at every step, is forced to manufacture a full probability distribution even on the steps where it has nothing useful to say. A well-trained model has to decide where to park that unneeded mass so that it does the least damage to the tokens actually being predicted — and it turns out to reliably choose a small set of fixed, always-available parking spots, rather than smearing it randomly across whatever happens to be nearby.

Why the parking spot is specifically the beginning

This is the autoregressive-visibility argument, and it's simple once stated: token 1 is visible to every query that ever gets computed in that sequence — token 2's query can see it, token 500's query can see it, token 50,000's query can see it. Token 500, by contrast, is only ever visible to queries from position 500 onward. Position 1 gets trained as a candidate attention target far more often, across far more contexts, than any later position does — simply by virtue of being present for every prediction that follows it. A location that is always in scope is the cheapest, most reliable place for the optimizer to learn to route the excess mass, because it's the one spot guaranteed not to disappear from view.

How much training exposure does each position actually get?

Put a number on “far more often, across far more contexts.” Consider one training sequence of length N. Because attention in an autoregressive model is strictly causal — token t can only attend to positions 1 through t — a position p is a legal attention target for every query at position t ≥ p. Position p is therefore a candidate for exactly N − p + 1 different queries over the course of that one sequence.

Take a concrete N = 100,000-token training sequence and compare three positions directly:

Position 1 is exposed as a candidate attention target roughly 50,000 times more often than the second-to-last position, purely as a mechanical consequence of where it sits in a causal sequence — before any content, gradient signal, or learned preference enters the picture at all. An optimizer doesn't need to “decide” position 1 is special; position 1 is mechanically exposed to vastly more training signal than any position near the end, in every sequence the model ever trains on. That imbalance, repeated across billions of training sequences, is what turns “always in scope” into “reliably reused as the parking spot for excess attention.”

Now derive the off-by-catastrophe, by hand

Here is the piece that finally explains Chapter 2's thousandfold collapse. Take a small, concrete example: one query attending over 5 keys, where the first key really is dominant (the paper's stated failure condition, x₁ ≫ x_j) and the other four have unremarkable, middling logits — a query with no strong match among the recent tokens, which is common in ordinary text.

Step 1 — softmax with the sink present. Compute e^x for each: e⁸ ≈ 2,980.96, e² ≈ 7.39, e¹ ≈ 2.72, e⁰ = 1, e^1.5 ≈ 4.48. Sum:

Read it: the sink token (position 1) absorbs 99.48% of the attention weight, exactly as observed in Chapter 3. The four ordinary tokens split the remaining 0.52% between them, in tiny, uneven fractions — 0.25%, 0.09%, 0.03%, and 0.15%, each comfortably under half a percent, tracking the size of their own logits relative to one another. This is normal, working attention. The model is fine.

Step 2 — now evict the sink (this is pop(0) from Chapter 2's code, applied at the attention-math level: token 1's key and value are simply gone, so it can no longer be a candidate at all). Recompute softmax over the same four remaining logits — nothing about their content changed:

Look at position 2 specifically. Its raw logit never moved — still 2, exactly as relevant or irrelevant to the query as it always was. But its attention weight went from 0.25% to 47.40% — a 192× jump, purely from renormalizing a denominator that used to be dominated by a term that no longer exists.

The catastrophe generalizes — a second worked example, larger cache

The example above used 5 keys to keep the arithmetic legible. Confirm the shape holds at a size closer to a real cache by working a second, larger example by hand: 9 keys, one dominant sink and eight ordinary tokens with small, varied logits.

With the sink present. e⁹ ≈ 8,103.08. The eight ordinary terms: e^1.2≈3.32, e^0.8≈2.23, e^1.5≈4.48, e^0.3≈1.35, e^1.1≈3.00, e^0.6≈1.82, e^0.9≈2.46, e^1.4≈4.06, summing to 22.72. Total Z ≈ 8,103.08 + 22.72 = 8,125.80.

Evict the sink, renormalize over the remaining eight. New Z = 22.72.

0.041% → 14.6% is a 356× jump — even larger than the 5-key example's 192×, because there were more competing tokens sharing the tiny leftover slice before eviction, so each one's individual jump upon renormalization is larger still. Adding more ordinary tokens to the cache doesn't dampen the catastrophe; if anything, each individual survivor's relative jump grows, because each one started from an even smaller baseline share.

The two conditions, side by side, in code

python
import math

def softmax(logits):
    exps = [math.exp(x) for x in logits]
    z = sum(exps)
    return [e / z for e in exps]

logits_with_sink    = [8, 2, 1, 0, 1.5]
logits_sink_evicted = [2, 1, 0, 1.5]     # same 4 tail logits, sink's entry just gone

print([round(w, 4) for w in softmax(logits_with_sink)])
# [0.9948, 0.0025, 0.0009, 0.0003, 0.0015]
print([round(w, 4) for w in softmax(logits_sink_evicted)])
# [0.4740, 0.1743, 0.0641, 0.2875]  -- exactly the renormalized values above

Ten lines of code reproduce the entire renormalization catastrophe. There is no hidden mechanism, no special case, no extra logic anywhere in a real model that “notices” the sink is gone and compensates. The jump in every surviving token's weight is purely what dividing by a smaller denominator does — the same softmax function, called on one fewer input.

When does the sink NOT dominate? The other side of the mechanism

Every worked example in this chapter assumed the failure condition the paper states explicitly: x₁ ≫ x_j, the sink's logit overwhelming everything else. It's worth being precise about when that condition actually holds, because it isn't every single query at every single step — Chapter 3 already showed the bottom two layers are the counterexample, where local, content-driven attention dominates and the sink barely registers.

The pattern is conditional on whether the query has a genuinely strong match among the other candidates. When it does — a pronoun resolving to its antecedent, a closing bracket matching an opener, a word continuing a well-established local pattern — softmax happily assigns most of its mass to that real match, and the sink's share stays small, the same way it would for any token with a low logit. The sink phenomenon isn't “the model always mostly attends to token 0”; it's “whenever the model doesn't have anything better to do with a share of its forced attention budget, token 0 is where that leftover reliably lands.”

This distinction matters for reading Chapter 3's attention-map finding correctly. The claim that deep layers heavily attend to the initial token across all layers and heads is a statement about the average, aggregated over hundreds of sentences and many query positions — not a claim that every single query, on every single step, sends the majority of its attention to position 0 regardless of content. On the specific steps where a real match exists, that real match still wins. The sink is the model's answer to the steps where nothing wins outright — and this chapter's math explains why softmax guarantees there will always be some such steps, no matter how well-trained the model is.

Every survivor jumps by the exact same factor — a closed form

Both worked examples in this chapter computed a single number — 192×, then 356× — as “the” jump factor, without stating explicitly that it's the same factor for every surviving token, regardless of that token's own logit. That's not a coincidence; it falls straight out of the algebra. Write survivor i's weight before eviction as w_i = e^x_i ÷ Z, where Z is the full sum including the sink. After evicting the sink, the new weight is e^x_i ÷ Z′, where Z′ = Z − e^x_sink. The ratio of new to old:

The e^x_i term — the one piece of the expression specific to survivor i — cancels out completely. The jump factor is Z ÷ Z′, a single number that depends only on the sink's own share of the original total, and applies identically to every surviving token, no matter how large or small that token's individual logit was. Check it against the 5-key example: Z=2,996.55, Z′=15.59, so Z÷Z′ ≈ 192.2 — matching the 192× jump computed earlier for one specific position, and matching every other surviving position's jump too, if you check them against the table. The 9-key example's 356× is the same formula, just starting from a different, larger sink share (Z÷Z′ = 8,125.80÷22.72 ≈ 357.6).

Chapter 5: The StreamingLLM Recipe

Everything so far has been diagnosis. Now build the fix. It is disarmingly small, which is exactly the point — a good diagnosis usually makes the fix look almost too simple.

The recipe, in one sentence

The paper's default, empirically justified in Chapter 6, is 4 sink tokens. That's not a round number chosen for convenience — it's the smallest count that reliably restores normal perplexity, as you already saw in Chapter 2's 4+1020 row: 5.40, versus dense-equivalent quality, versus 5,158 for zero sink tokens.

Cache arithmetic, tied back to Chapter 1

Use the exact 4+1020 configuration from Chapter 2 — total cache size 1,024 tokens — and Chapter 1's exact byte formula, 512 KiB per token for a Llama-2-7B–class architecture:

That's the entire memory footprint, and it is a constant — it does not matter whether the conversation is 2,000 tokens long or 4,000,000. Compare against Chapter 1's dense-attention numbers: 23.4 GiB for one workday, 1.9 TiB at 4 million tokens. StreamingLLM's cache is smaller than either of those by several orders of magnitude, and stays that size permanently, which is exactly the property Chapter 0 said was missing.

Put a precise ratio on “several orders of magnitude,” using Chapter 1's own headline number — dense attention's cost at the paper's tested 4-million-token length, about 1.9 TiB — against this chapter's constant 512 MiB:

Nearly four thousand times less memory, for a model that runs stably for the same 4 million tokens and beyond — and that ratio only grows the longer the conversation runs, because StreamingLLM's cache size never moves while dense attention's keeps climbing without limit. This is the same asymptotic story Chapter 8 revisits for speed instead of memory: a quantity held flat, set against one left unbounded, with the gap between them widening the longer the stream continues.

The subtle part: positions inside the cache are not positions in the text

Here's a detail easy to get wrong and expensive to get wrong. Say the rolling window has evicted tokens 4 and 5, and the cache currently holds original-text positions [0, 1, 2, 3, 6, 7, 8], about to process token 9. What position number should the model assign to each cached entry when computing attention?

The tempting answer is “use the real text position” — [0, 1, 2, 3, 6, 7, 8, 9]. This is wrong, and it's wrong for a precise reason: most modern LLMs use relative positional encodings (RoPE, ALiBi), where what the model actually learned to interpret is the distance between a query and a key, not their absolute indices. A gap of [3 → 6] tells the model “these are 3 positions apart,” when in the model's actual experience during pretraining, tokens 3 and 6 in a contiguous stream really would be 3 apart — but here, two tokens were silently deleted from between them. The model has never seen this exact distance pattern paired with this content pattern during training.

StreamingLLM's fix: assign positions by slot in the cache, not by the token's original index. The seven cached tokens above get relabeled [0, 1, 2, 3, 4, 5, 6], and the new ninth token becomes position 7 — contiguous, gap-free, exactly the kind of sequence the model was trained on. The original text positions are simply forgotten; only cache-relative order survives.

Two flavors of relative position encoding, one shared fix

Chapter 0 flagged that this session deliberately tests two different positional-encoding schemes across its four model families: Llama-2, Falcon, and Pythia use RoPE; MPT uses ALiBi. The RoPE gap-repair story above is one instance of a more general requirement — the fix is versatile enough to slot into any autoregressive model built on a relative positional scheme, and both of these qualify, even though they encode relative position through mechanically different routes.

RoPE encodes position by rotating each query and key vector by an angle proportional to its position, before the dot product happens — two vectors end up with a relative rotation proportional to the distance between them, which is what the model actually learned to read. ALiBi instead leaves the vectors alone and subtracts a penalty directly from the raw attention logit, proportional to the distance between query and key, before softmax runs. Different mechanism, same underlying dependency: both methods bake the notion of distance into the attention computation, and both would be handed a distance they never trained on if the true text position — with its silently growing gap — were used instead of the cache-relative one. Relabeling to a contiguous cache-relative index repairs both, because both only ever consume a distance between two tokens, never an absolute coordinate.

The sink tokens' own position never moves

One detail easy to miss: the sink tokens are not exempt from this relabeling scheme — they're just permanently pinned to the front of it. The 4 sink tokens always occupy cache-relative positions 0, 1, 2, and 3, for the entire lifetime of the stream, no matter how many millions of real tokens have streamed past them by the original text's own count. Their rotation angle (under RoPE) or distance-penalty term (under ALiBi) relative to whatever the current query is, is therefore always computed as though they were still the literal first four tokens of the sequence — which, in cache-relative terms, they permanently are, even while in real text-position terms they may be tens of millions of positions in the past.

This is exactly what lets the sink tokens keep doing their job indefinitely: from the model's own point of view, at every single decoding step, there is always something occupying the extremely-early, extremely-heavily-trained positions its attention learned to rely on — because the cache manufactures that appearance by construction, permanently, regardless of how long the actual conversation has run.

In code

python
class StreamingCache:
    def __init__(self, n_sink=4, window=1020):
        self.n_sink, self.window = n_sink, window
        self.sink_kv  = []        # filled once, first n_sink tokens, then frozen
        self.recent_kv = []       # deque, capped at `window`

    def append(self, k, v):
        if len(self.sink_kv) < self.n_sink:
            self.sink_kv.append((k, v))          # never touched again
        else:
            self.recent_kv.append((k, v))
            if len(self.recent_kv) > self.window:
                self.recent_kv.pop(0)      # same eviction as Ch2 -- but now it never touches the sink

    def assembled(self):
        # cache-relative positions: 0..n_sink-1, then contiguous from there
        return self.sink_kv + self.recent_kv     # total size: always n_sink + window, forever

Compare this to Chapter 2's window_step. The eviction line is unchanged — pop(0) is still there. The only new idea is that pop(0) now operates on recent_kv, a list that has already been quietly protected from ever containing the sink. That one-line guard is the entire difference between 5,158 and 5.40.

Position reassignment, in code

python
def assign_cache_positions(sink_kv, recent_kv):
    # cache-relative positions: sink block first, always 0..n_sink-1,
    # then the recent window, contiguous, no gaps -- regardless of
    # how many real tokens were evicted to get here.
    positions = list(range(len(sink_kv))) \
              + list(range(len(sink_kv), len(sink_kv) + len(recent_kv)))
    return positions          # fed to RoPE/ALiBi instead of the true text index

# RoPE specifically: keys are cached UNROTATED, and rotated fresh
# at attention time using the cache-relative angle -- never the
# angle from when the token was first computed.
def apply_rope(key, cache_relative_pos, rope_freqs):
    angle = cache_relative_pos * rope_freqs
    return rotate(key, angle)   # recomputed every step a token's slot shifts

The second function is the detail the earlier callout warned about in prose: rotate at attention time, using the cache-relative position, never the position the token was originally computed at. Cache the un-rotated key; rotate it fresh, every time, against whatever slot it currently occupies. ALiBi's version is even simpler — there's no key to keep un-rotated, just a bias term recomputed from cache_relative_pos at attention time instead of being baked into anything stored.

What happens before the cache is even full

One edge case worth tracing through explicitly: what does StreamingCache do for the very first 1,024 tokens of a brand-new conversation, before the rolling window has ever needed to evict anything?

Nothing special, and that's the point. The append method's if branch fills sink_kv first, for the first n_sink tokens — identical to what dense attention would do for those same tokens, since nothing has been evicted yet. Every token after that goes into recent_kv, which only starts popping once it exceeds window. For an entire short conversation that never reaches 1,024 tokens, StreamingCache and plain dense attention are behaviorally identical — same tokens kept, same positions, same everything. The two only diverge the moment a conversation actually runs long enough to need eviction at all.

This matters for how to think about the “cost” of switching to StreamingLLM: there isn't one, for short interactions. A five-message chat pays exactly the same memory and compute StreamingLLM and dense attention both would, because the fixed-size cache never fills up enough to diverge from unbounded caching in the first place. The entire benefit is asymptotic — it shows up precisely when, and only when, a conversation grows past the cache size, which is exactly the regime dense attention can't survive at all.

Tracing the code by hand, token by token

Concept-check the StreamingCache class above by hand-tracing what it does for a specific handful of tokens, rather than trusting the class definition on faith. Use n_sink=4, window=6 (small numbers, to keep the trace short), and feed it tokens 0 through 11 in order:

After 12 tokens, assembled() returns sink_kv + recent_kv = [0,1,2,3,6,7,8,9,10,11] — ten entries total, exactly n_sink + window = 4 + 6, holding steady no matter how many more tokens arrive after this. Tokens 4 and 5 are the ones silently gone, evicted the moment tokens 10 and 11 respectively pushed the window past its limit — exactly the [0,1,2,3, gone, gone, 6,7,8,…] pattern the widget above animates continuously as the slider moves.

Chapter 6: How Many Sinks?

Chapter 5 asserted 4 sink tokens as the default. Don't take that on faith — the paper ablates it directly, sweeping the sink count from 0 up to 8 across three model families, and the results are worth reading closely because they don't all say the same thing.

Look at that: for these three families, a single sink token already recovers almost all the quality — going from 1 to 8 barely moves the number. That would suggest “why does the paper recommend 4, just use 1?” The next table is why.

Different story. Llama-2 keeps improving noticeably from 1 sink (11.88) through 2 (10.51) to 4 (9.59), and only flattens out after that — adding a fifth through eighth token buys just 0.05 more. One sink token is not enough for this model family; four is close to the point of diminishing returns; eight is wasted budget.

Diminishing returns, quantified

“Close to the point of diminishing returns” is worth pinning down numerically instead of taking on faith. Compute the relative improvement Llama-2-7B gets from each additional sink token, as a percentage drop in perplexity relative to the previous step:

The pattern is unambiguous: each additional sink token buys noticeably less than the one before it, and by the jump from 4 to 8 the return is under one percent — arguably not worth chasing at all. The curve doesn't fall off a cliff after 4 tokens; it just keeps decaying smoothly, and 4 is simply where the paper judged the remaining gains too small to justify pursuing further.

Weigh that against the cost side of the ledger. Each additional sink token costs, from Chapter 1's own byte formula, another 512 KiB of permanently pinned cache — going from 1 sink to 4 costs an extra 1.5 MiB total, against a multi-hundred-megabyte cache budget. The decision to stop at 4 isn't really a memory decision at all; the marginal KiB cost of extra sinks stays negligible well past 8. It's a decision about when an ablation curve has flattened enough that further tuning stops being worth the effort of re-verifying it per model family.

Why the models disagree

The paper has a specific, satisfying explanation for exactly this discrepancy, and it comes back to Chapter 4's argument about training exposure. Llama-2 does prepend a beginning-of-sequence token, <s>, to each paragraph during pretraining — but that prepending happens before the text gets chunked into fixed-length training windows. The practical effect is that <s> does not reliably land at position 0 of the windows the model actually trains on; a mostly random token ends up occupying that slot instead, window after window.

Without one consistent, always-present anchor token, the model can't learn to dump all its excess attention on a single designated address — it has to spread the job across several of the early positions instead, because none of them individually shows up in every training example. That's a model that never had the chance to specialize one slot, so it specializes several, loosely. Chapter 7 pre-empts this problem entirely by giving the model exactly the one consistent anchor it's missing, from the start of pretraining.

What “applied before chunking” looks like concretely

The chunking detail is easy to skim past, so make it concrete. Pretraining doesn't feed the model one paragraph at a time; it concatenates a huge stream of text (with a <s> token stitched onto the front of every paragraph) and then slices that entire stream into fixed-length training windows, back to back, with no regard for where a paragraph happened to start.

Picture three short paragraphs, each prefixed with <s>, concatenated into one long stream, then cut into fixed 12-token training windows:

Window 1's position 0 genuinely is a <s> token — lucky, this time. But the next window's position 0 is P3b, an ordinary content token, because the paragraph boundary fell in the middle of that window instead of at its start. Across millions of training windows, position 0 lands on <s> only for whichever windows happen to align with a paragraph boundary by chance — the rest of the time it's whatever ordinary token happened to fall there instead. There is no single token the model can reliably learn to treat as “the sink” at position 0, because position 0 isn't reliably the same kind of token from one training window to the next. Chapter 7's fix removes exactly this source of inconsistency.

A related, independent line of work

This diagnosis didn't arrive in isolation. Concurrent research on the same length-generalization failure (Han et al., 2023) approached it from a more theoretical angle and landed on a structurally similar picture: giving models a “Λ”-shaped attention pattern — heavy weight at the very start of the sequence and near the current position, sparse in between — combined with reconfiguring how position distances are computed, improves length generalization. It's a different derivation, arrived at independently, converging on the same practical shape: a small, fixed number of early positions matter disproportionately, and protecting them specifically — rather than trying to guess which content anywhere in a long sequence is semantically important — is what stabilizes the model.

That kind of convergence — two separate research efforts, different starting assumptions, overlapping conclusions — is a stronger signal than either finding on its own. It suggests the phenomenon this lesson has been building toward isn't an artifact of one team's specific experimental setup; it's a real, reproducible property of how these models were trained.

The failure mode you'd hit by under-provisioning

python
# tempting "optimization": save a few KB, use 1 sink token everywhere
cache = StreamingCache(n_sink=1, window=1023)
# fine on Falcon-7B / Pythia-12B (Table above: 1 sink already near-converged)
# SILENTLY WORSE on Llama-2-7B: 11.88 vs. the achievable 9.59 -- a ~24% higher
# loss that a quick smoke test on a short prompt will not surface, because
# the degradation compounds over a LONG stream, not a short one.

The ablation sweep, in code

python
# conceptual sketch of the sweep behind this chapter's tables
for n_sink in [0, 1, 2, 4, 8]:
    cache = StreamingCache(n_sink=n_sink, window=total_cache - n_sink)
    ppl = evaluate_perplexity(model, long_text, cache)
    print(f"n_sink={n_sink}: PPL={ppl:.2f}")

# same StreamingCache class Chapter 5 already built -- only n_sink
# changes. Eviction, position relabeling, RoPE re-rotation are all
# untouched by this sweep.

The entire ablation this chapter reports is one constructor argument, swept across five values, on the exact StreamingCache class Chapter 5 already built. There's no separate mechanism for “how many sinks” — it's the same fixed-plus-rolling design, just with the fixed block resized, run once per model family and once per candidate sink count.

Why 0, 1, 2, 4, 8 specifically?

The swept values double each time (after the first step, 0→1) — a standard ablation-design choice when you don't know in advance where a curve will flatten. Linear steps (1, 2, 3, 4, 5…) would waste measurements densely sampling a region that turns out uninteresting, or under-sample the region where the interesting transition actually happens. Doubling covers a wide range of the design space — a thousand-fold range takes just ten doublings — with far fewer experiments than a linear sweep would need to cover the same range, which matters when each measurement means a full forward pass over hundreds of thousands of tokens, per model family, per candidate count.

Reading Falcon and Pythia's flatness correctly

Falcon-7B's numbers — 12.12, 12.12, 12.12, 12.12 across 1, 2, 4, and 8 sinks — aren't just “close.” They're identical to two decimal places, for four different cache configurations. That level of exactness is itself informative: it's very unlikely a real, continuous quantity would land on the exact same rounded value four separate times by coincidence. What's actually happening is that Falcon has converged to a floor determined by everything except the sink-token count — the recent window's own information content, the model's baseline capability on this text — and additional sink tokens beyond the first are doing something too small to move the needle at this level of precision.

Pythia-12B's numbers move slightly more (11.95 → 12.09 → 12.09 → 12.02) — small enough to plausibly be measurement noise rather than a real trend, since it isn't even monotonic (12.09 repeated, then a slight improvement to 12.02 at 8 sinks, moving the “wrong” direction if you expected strict monotonic decay). Llama-2-7B's numbers, by contrast, move in a clean, monotonically-decreasing, clearly-not-noise pattern: 11.88, 10.51, 9.59, 9.54. Learning to tell a real signal from noise inside a results table is as much a part of this skill as reading the big collapses Chapter 2 covered.

The diminishing returns, in bits instead of percentages

Chapter 2 converted a perplexity gap into bits using loss = ln(PPL). Apply the same conversion here, on Llama-2-7B's own numbers, to see the diminishing-returns story in the same units Chapter 2 used for the window-attention collapse:

Compare that to Chapter 2's collapse: 9.9 bits per token, lost the instant the sink is evicted entirely. Going from 1 sink token to 4 recovers about 0.31 bits — roughly 3% of the size of the original catastrophe. The sink-count ablation this chapter runs is fine-tuning around the edges of an already-mostly- solved problem; Chapter 2's collapse was the problem. Keep that scale difference in mind: 4 versus 1 sink tokens is a real, measurable, worth-having improvement, and it is nowhere near as consequential as having any sink tokens at all.

Same marginal cost, very different marginal value

One more comparison worth making explicit: adding one more sink token and adding one more recent-window token cost exactly the same — 512 KiB each, per Chapter 1's formula, since both are just one more slot in the same cache. But their marginal value is wildly different. Going from 1 to 4 sink tokens on Llama-2-7B recovered roughly 2.3 points of perplexity (11.88 to 9.59) across 3 extra slots — about 0.77 points per slot. Chapter 8's cache-size ablation, by contrast, shows going from 508 to 1,020 recent-window tokens on the same model recovering only about 0.41 points of perplexity across 512 extra slots — roughly 0.0008 points per slot. Sink-token slots, in this regime, are close to a thousand times more valuable per byte than ordinary window slots.

That asymmetry is the entire justification for treating the sink block as a special, separately-tuned parameter rather than folding it into “just make the cache bigger.” A byte spent on the first few positions is doing structural, stabilizing work; a byte spent on the 500th recent token is doing ordinary, diminishing-returns context work. Two very different jobs, sharing the same cost unit, is exactly why this lesson spends an entire chapter isolating the sink-count question from the window-size question Chapter 8 asks separately.

What if you can't run the ablation yourself?

Not every engineer deploying StreamingLLM on a new model has the compute budget to run this chapter's full sweep. There's a reasonable fallback: check whether the model's tokenizer prepends a beginning-of-sequence token, and if the pretraining pipeline is documented, check whether that token was applied before or after chunking into fixed-length training windows — the exact distinction diagnosed above for Llama-2. A model with a consistently-placed BOS token going into every training window is a reasonable prior for needing fewer sink tokens; a model where BOS gets lost in chunking (or has no BOS token at all) is a reasonable prior for needing the full 4. Absent that information entirely, defaulting to 4 — this chapter's safe choice — costs about 1.5 MiB of extra pinned cache relative to using 1, which is cheap insurance against silently under-provisioning a model built more like Llama-2 than like Falcon.

Chapter 7: The Learnable Sink

Chapter 6 ended on a diagnosis: models without one consistent, always-present token learn a blurrier, less efficient version of sink behavior, spread across several positions instead of concentrated in one. If that's the cause, there's an obvious experiment — give the model that consistent token, on purpose, from the first step of pretraining, and see if a single sink token becomes enough.

The intervention

The authors pretrain three 160-million-parameter language models from scratch, identical in every setting except one thing:

• Vanilla — standard softmax attention, nothing added.
• Zero Sink — replace ordinary softmax with a variant, SoftMax₁, that has a built-in escape valve for unneeded mass:

Notice the “1 +” sitting in the denominator, next to the real terms. That extra 1 is exactly equivalent to prepending one phantom token whose key and value are all zeros and whose logit is e⁰ = 1 — a fixed, permanent place for softmax to park unneeded mass without it ever landing on a real token. It costs nothing (no extra parameters, no extra token in the sequence), and it directly targets Chapter 4's diagnosis: give the equation an escape hatch that doesn't require sacrificing a real position.

• Sink Token — simplest of all: literally prepend one extra, learnable placeholder token to every training sample, at position 0, consistently, every single time. Let the model's own parameters figure out what that token's key and value should be.

How the intervention was actually run

This isn't a thought experiment — the authors pretrained real models to test it. All three 160-million-parameter models were trained from scratch on an 8×A6000 NVIDIA GPU server, using the deduplicated Pile dataset and the Pythia-160M codebase's own training recipe as the base configuration. The only setting they changed was the batch size, reduced to 256; everything else — learning-rate schedule, model initialization, the order data was shown to the model — was held identical to Pythia's own defaults, and every model trained for the same 143,000 steps.

That level of control matters for trusting the comparison. If the Sink Token model had also gotten a different learning-rate schedule, or seen data in a different order, any perplexity difference at the end could be explained by a dozen confounding factors instead of the one variable actually being tested — whether a dedicated placeholder token exists at position 0. Holding literally everything else fixed is what lets the paper attribute the entire streaming-perplexity gap to that one architectural choice, and nothing else.

The result

Read the middle column of row two: with the dedicated Learnable Sink token kept, perplexity is already 18.01 — matching or beating the best the Vanilla model ever achieves, even with 4 tokens (18.05). The model that was given one consistent anchor from the start needs only that one token to fully stabilize. The diagnosis from Chapter 6 held.

Two more things this table is honest about. First, Zero Sink genuinely helps — 29,214 is smaller than you'd guess for a model with no mitigation at all, but it's still catastrophic, and it still needs multiple real initial tokens (19.90 at 1, dropping further at 2 and 4) to fully recover. A fixed, content-free escape valve of exactly “1 unit” isn't flexible enough to absorb however much excess mass the model actually wants to dump; a token with trainable content is. Second — and this is the detail worth sitting with — look at the Learnable Sink model's own 0 + 1024 row: 1,235. Evict the model's own dedicated sink token, the one thing it was specifically trained to rely on, and it fails badly too, just like the vanilla models fail without their improvised ones. The sink token is doing real, load-bearing work. There is no free lunch here — only a cheaper, more reliable place to pay the same bill.

The full seven-benchmark comparison

The callout above gave two spot numbers. Here is the complete picture — all seven zero-shot benchmarks the paper reports, comparing the Vanilla 160M model against the one pretrained with a dedicated Sink Token:

Every single one of the seven benchmarks ticks up, not down, with the sink token added — a small margin each time (0.3 to 1.0 points), well within the range you'd expect from ordinary training-run variance rather than a genuine capability gain. But the direction is what matters: there is no benchmark on which adding the sink token cost anything measurable. Combined with the matching convergence curves, this is about as clean a “free” architectural change as pretraining research produces.

Does a second sink token help further?

If one dedicated sink token is such a clean win, the obvious next question is whether two would be even better. The authors ran that experiment too, pretraining a third 160M model with two learnable placeholder tokens instead of one.

Read the middle row carefully: the 2-sink model, given back only one of its two dedicated sink tokens (the 1+1023 configuration), scores 25.73 — clearly worse than the 1-sink model's 18.01 at the same configuration, and worse than its own 18.05 once both of its sinks are restored (2+1022). The model with two sink tokens apparently learned to split its excess attention across both of them, the same way an un-modified model splits it across several improvised early positions — so keeping only one of its two sinks leaves the mechanism half-starved. Two sinks isn't strictly better than one; it's just a new commitment that has to be honored in full or not at all.

On the 7 zero-shot benchmarks (not shown in full here), the 2-sink model doesn't clearly beat the 1-sink model either — some scores tick up fractionally, others tick down, a wash rather than a trend. Combined with the streaming-perplexity result above, the paper's conclusion is that a single sink token is both necessary and sufficient: necessary, because zero sinks catastrophically fails even for a model trained with the mechanism available; sufficient, because a second one buys nothing on either metric while adding a second commitment you must always honor.

Why 160 million parameters, specifically

The choice of scale here is practical, not incidental. Pretraining language models from scratch is expensive — every model in this chapter's tables had to be trained separately, to full convergence (143,000 steps each), just to run one clean ablation. 160 million parameters is small enough to iterate on an 8-GPU server in a reasonable amount of wall-clock time, while still being a real Transformer language model exhibiting the same qualitative attention behavior Chapters 3 and 4 documented at 7-billion-parameter scale.

This is a common and reasonable pattern in this kind of research: validate a mechanism at small, cheap scale first, where a clean controlled experiment is affordable, then trust that the same mechanism (softmax's normalization constraint, unchanged by model size) applies at the larger scales where full ablations would be prohibitively expensive. Chapters 1 through 6 already demonstrated the underlying attention-sink phenomenon directly at 7B-and-larger scale; this chapter's 160M experiment is testing the fix, not re-testing whether the problem exists.

The Zero Sink math, worked with Chapter 4's own numbers

Make Equation 2 concrete by reusing Chapter 4's exact worked-example logits, [8, 2, 1, 0, 1.5], and asking what Zero Sink's SoftMax₁ would compute instead of ordinary softmax:

Reuse Chapter 4's exponentials — e⁸≈2,980.96, e²≈7.39, e¹≈2.72, e⁰=1, e^1.5≈4.48 — but now add a phantom “1” to the denominator instead of a fifth real logit:

Compare to Chapter 4's ordinary-softmax weights with the sink present: 99.48% versus 99.45% here — nearly identical, because when a real dominant logit (8) already exists, the phantom “+1” term is tiny by comparison and barely changes anything. The real difference shows up in the opposite case: imagine no logit is dominant at all — say all five real logits are small and similar, with no natural sink candidate. Ordinary softmax still has to sum its five outputs to exactly 1, spreading mass across whatever tokens exist even though none of them earned it. SoftMax₁, in that same scenario, sends a real, meaningful chunk of the total budget into the phantom term instead, leaving the five genuine tokens with a smaller, more honest combined share. That's the entire mechanism Zero Sink offers: an escape hatch that exists even when nothing in the real context deserves the leftover mass — which is exactly why it helps to some extent without fully solving the problem the way a trainable, contentful Sink Token does.

What actually changes in the training pipeline

Concretely, in code, the Sink Token intervention is a small, mechanical change to how each training example is assembled — not a change to the attention math itself:

python
def prepare_example(token_ids, sink_token_id):
    # vanilla: token_ids unchanged
    # with a dedicated sink: prepend the SAME learnable token id,
    # every single sample, before any chunking happens
    return [sink_token_id] + token_ids

# sink_token_id's embedding is a normal, trainable row in the
# embedding table -- gradients flow into it like any other token,
# it just always occupies position 0, on every sample, without
# exception, unlike Llama-2's <s> which gets lost in chunking
# (Chapter 6).

That's the entire intervention: one extra token id, prepended before chunking (not after, unlike the <s> problem Chapter 6 diagnosed), consistently, on every sample, for the whole run. Nothing about the attention layers, the loss function, or the optimizer changes at all. The fix lives entirely in data preparation — which is part of why it's cheap enough to recommend as a default going forward.

What this recommends going forward

Put together, this chapter argues for a design change upstream of deployment entirely: models intended for future streaming use should be pretrained with a dedicated sink token from the start, rather than relying on a lucky, inconsistent early token and needing 4 (or more, or an unknown number) reintroduced tokens at inference time to patch around it. It's a one-line change to the training data pipeline that converts an emergent, unreliable property into a designed, guaranteed one.

Neither fix in this chapter is retrofittable

Worth stating plainly, because it's easy to conflate this chapter with Chapters 5–6: neither Zero Sink nor Sink Token can be bolted onto an already-pretrained checkpoint the way StreamingLLM's cache management can. Both require the change to be present from the very first step of pretraining — Zero Sink changes the attention formula itself, and Sink Token changes what every single training example looks like. Neither is something you can apply to Llama-2-7B's public weights tomorrow morning.

That's precisely why this lesson kept the two tracks separate. Chapters 5 and 6 solve the problem you have today, for a model someone else already trained: patch the cache, no retraining, working within the hour. This chapter solves the problem for whoever trains the next model: build the fix in from day one, so future deployments don't need Chapters 5–6's patch at all, or need a much smaller one (just 1 sink token instead of 4). Both are real, useful interventions; they just operate at different points in a model's lifecycle, and confusing them would mean expecting a cache-management trick to somehow retroactively change what a model learned during training, which it cannot do.

Chapter 8: Evaluation, Honestly (showcase)

A fix that only wins on perplexity numbers, on curated benchmarks, chosen by the people who built it, would be a weak claim. This chapter checks three separate things the paper reports: does it actually go fast, does it actually work on a downstream task an instruction-tuned model would face, and — the honest part — where does it still fall short.

1. The speed claim, and why the shape is quadratic vs. linear

StreamingLLM's only real competitor for correctness is sliding window with recomputation: at every single generation step, throw away the cache entirely and recompute keys and values for the whole recent window from scratch. It gets the right answer — it never has a stale-position problem, because nothing is ever cached and reused — but watch what that costs.

For a window of L tokens, recomputing from scratch means running a fresh forward pass over L tokens on every decoding step, and self-attention inside that forward pass costs O(L²) — the familiar quadratic cost of attention, paid again and again, once per new token generated. StreamingLLM, by contrast, does what ordinary incremental decoding always does: compute the new token's own K, V once, and attend it against the already-cached L entries — cost O(L) per step, no recomputation, because the cache from the previous step is still valid and correctly reused.

Measured on a single NVIDIA A6000 GPU with Llama-2-7B and Llama-2-13B, the paper reports StreamingLLM reaching up to 22.2× the decoding speed of the recomputation baseline as cache size grows, while using a comparable memory footprint to it (the recompute baseline pays a different bill — activation memory for its from-scratch forward pass — that happens to land in a similar range).

Reading the “22.2×” number correctly

It's tempting to read “up to 22.2× faster” as a single fixed multiplier you'd see on every deployment. It isn't — it's the peak of a ratio between two curves with different growth rates, measured at the largest cache size the paper tested. Because recomputation's cost grows as O(L²) and StreamingLLM's grows as O(L), their ratio (L²÷L = L) grows linearly with cache size on its own. At a small cache — say L=128 — both curves start close together and the speedup is modest. At the paper's largest tested cache size, the gap has widened enough to hit 22.2×. Push the cache size larger still, and the ratio keeps climbing, at least in principle, until some other bottleneck (memory bandwidth, kernel launch overhead) takes over.

This is why Chapter 5's own arithmetic matters here too: StreamingLLM's cache size is a design choice, not something forced by the conversation length. A larger recent window buys more speedup relative to the recomputation baseline (because the baseline's cost grows quadratically with that same window), at the cost of more memory per Chapter 1's byte formula. There's a real tradeoff hiding inside “just make the window bigger for more speedup” — and this chapter's own honest-limitation section below already shows a bigger window doesn't reliably buy better quality either. Cache size is a knob with at least three competing consequences — memory, speed relative to the recompute baseline, and quality — and none of them move in a simple, universally-good direction together.

The measurement itself was run on a single NVIDIA A6000 GPU, using the standard Huggingface Transformers library implementation for both methods — not a custom kernel built to favor one side. That matters for trusting the comparison: both baselines got the same off-the-shelf inference stack, so the speedup reflects the algorithmic difference (recompute everything vs. reuse a valid cache) rather than one method benefiting from more careful low-level engineering.

2. Does it survive contact with a real downstream task?

Perplexity is an intrinsic measure; it says nothing directly about whether an instruction-tuned model still gets the right answer. The paper tests this by concatenating question-answer pairs from ARC-Easy and ARC-Challenge into one continuous stream fed to Llama-2-Chat models, scoring exact-match accuracy at each answer as it streams by.

Dense attention runs out of memory outright on this long a stream — it isn't even a contender, exactly Chapter 0's prediction. Window attention doesn't just underperform, it collapses to essentially below random guessing on a 4-or-5-option multiple-choice task — a sign the model isn't just worse, it's producing degenerate, malformed completions once its cache boundary crosses the first tokens. StreamingLLM matches or fractionally exceeds the one-shot ceiling across all three model sizes.

ARC-Easy isn't the only split tested. The harder ARC-Challenge partition, run through the identical streaming setup, tells the same story with just as clean a margin between window attention's failure and StreamingLLM's recovery:

Notice StreamingLLM doesn't just match the one-shot baseline here — on every model size, on both splits, it slightly exceeds it (55.03% vs. 53.16%, 65.61% vs. 63.31%, 80.20% vs. 78.50%, and similarly for ARC-E above). That's not StreamingLLM making the model smarter; it's the streaming format incidentally giving the model access to earlier question-answer pairs from the same conversation as informal few-shot context — a small, real side-benefit of processing questions as one continuous stream rather than scoring each one in isolation.

StreamEval: a harder, more realistic downstream test

The ARC-concatenation test is illuminating but a little artificial — questions and answers packed back to back aren't quite how a real conversation unfolds. The paper also introduces a purpose-built benchmark, StreamEval, closer to how a streaming assistant actually gets used: rather than one giant query at the very end of a long document (the design of the older LongEval benchmark it's adapted from), StreamEval queries the model every 10 lines of new information, and each query's answer sits exactly 20 lines earlier — deliberately modeling the realistic pattern where a user asks about something recent, not about the very start of a session hours ago.

Run on Llama-2-7B-32K-Instruct (a context-extended variant, showing StreamingLLM composes with that technique too), accuracy holds up well as long as the query-answer distance stays inside the cache's recent window, and then falls off sharply once it doesn't. Each line of StreamEval is 23 tokens, so a line distance converts directly to a token distance:

Each column is close to a step function, not a gentle curve: accuracy holds up reasonably well right up until the query-answer distance exceeds that cache configuration's total recent-window size, at which point it drops to exactly zero, not to some degraded-but-nonzero number. A 4+2044 cache (2,048 recent tokens) answers correctly at a 2,300-token distance essentially never — the true answer is no longer physically present anywhere in the cache, sink tokens included, so there is nothing left for the model to retrieve even approximately. This table is the clearest single piece of evidence in the whole paper that StreamingLLM's “memory” is a sliding window with a hard edge, not a gracefully-degrading approximation of one — exactly the property Chapter 9 formalizes as this method's central limitation.

3. The honest limitation: bigger cache does not mean better model

Here is where the paper resists overselling its own result. You'd expect a larger cache — more recent context available — to monotonically improve perplexity. It usually doesn't.

MPT-7B gets steadily worse as cache size grows from 252 to 2,044 recent tokens — 14.12 up to 14.99, monotonically the wrong direction. Falcon-7B improves up to 1,020 recent tokens, then gets slightly worse again at 2,044. Only Pythia is close to flat-to-improving throughout. The paper states the implication plainly: these models “might not maximize the utility of the entire context they receive.” A bigger StreamingLLM cache buys you a bigger window to draw from, but it does not automatically buy you a better-informed prediction — that depends on the base model's own ability to exploit long context, which is a separate, still-open problem this paper does not solve.

A fourth data point: Llama-2-7B's own cache-size ablation

The three-family table above already tells an already-nuanced story. Llama-2-7B, run through the identical ablation at its own set of cache sizes, adds a further wrinkle rather than resolving it:

Here the cache-size increase does help, monotonically, all the way from 508 to 2,044 recent tokens — unlike Falcon or MPT above. But keep doubling past that point, to 4,092 recent tokens, and perplexity gets worse again, back above where it stood at 1,020 tokens. Even the model that behaves the way intuition predicts — bigger cache, better performance — only behaves that way up to a point, and that point isn't something you can guess in advance. It has to be measured, the same as everything else this chapter checks.

Does the honesty hold on real long-document benchmarks too?

One more check, on a genuinely different kind of task. LongBench is a suite covering single-document QA, multi-document QA, and summarization; running it through Llama-2-7B-Chat (a 4K-context model) means comparing StreamingLLM against the standard fallback for inputs longer than the context window — plain truncation, keeping the first 1,750 and last 1,750 tokens and discarding everything in the middle.

StreamingLLM's default cache shape — a small 4-token sink plus a large recent window — loses to plain truncation here, on nearly every task. The reason ties directly back to what these tasks actually need: single- and multi-document QA and summarization often depend on information sitting anywhere across the whole document, including its beginning, not just its most recent tail. A 4-token sink block was sized for stabilizing streaming attention, not for holding a document's opening context. Rerun StreamingLLM with 1,750 sink tokens instead of 4 — matching how much of the beginning the truncation baseline preserves — and the gap closes almost entirely, landing within a point of the truncation baseline across every task.

This is a genuinely important qualifier on top of everything Chapters 5–7 built: the 4-sink-token default is tuned for the streaming-perplexity problem this lesson has been solving, not for every task shape you might throw at a cache-bounded model. Sizing the sink block is itself a task-dependent decision, not a universal constant — 4 tokens is enough to stabilize an endless conversation, but a task that genuinely needs a document's opening content may need far more.

Reading all four families together

Step back and classify each of the four model families by how it responds to a bigger cache, now that this chapter has covered all four:

No two families share the same shape. That diversity is itself the finding: there is no single rule — not “bigger always helps,” not “bigger never helps past some fixed size,” not even “bigger helps up to a size proportional to X” — that predicts all four curves from one shared formula. Whatever determines how well a model exploits additional recent context is a property of that specific model's own training, not something derivable from its architecture family or parameter count alone. The only reliable process is the one below: measure your specific model, on your specific traffic, at a few candidate sizes, before committing to one in production.

Chapter 9: Limits & Connections

Before closing, be precise about what StreamingLLM does not claim to do — the paper's own limitations section is unusually direct about this, and it's worth reading in its own words before paraphrasing.

What streaming does not solve

Unpack why that limitation is fundamental, not just an unfinished corner. The model can only ever attend over what's currently in the cache: 4 sink tokens plus the recent window. Whatever was said 100,000 tokens ago and has since scrolled out of the rolling window is gone — not summarized, not compressed, not stored anywhere the model can retrieve it. StreamingLLM keeps the model coherent and running indefinitely; it does not give the model a memory of everything that happened. Those are different engineering problems, solved by different mechanisms (retrieval, external memory stores, periodic summarization into the prompt) that are orthogonal to, and can be layered on top of, everything in this lesson.

There's a second, more specific consequence worth naming: ask the model a question whose answer requires information from far outside the current window — “what was the very first thing I asked you today?” on hour six of a running session — and StreamingLLM cannot produce that answer correctly, by construction, no matter how well it's implemented. It isn't a bug to fix; it's the tradeoff being made, stated plainly.

Attention sinks beyond this paper: registers in Vision Transformers

Chapters 3 and 7 both promised a closer look at a closely related, independently discovered finding. Around the same time as this paper, separate research on Vision Transformers (Darcet et al., 2023) found that ViTs routinely dump anomalously high attention onto a handful of essentially meaningless patches — typically uninformative background regions of an image — rather than genuinely relevant image content. They named these registers: extra learnable tokens added specifically to give the model somewhere legitimate to route that unwanted attention, instead of hijacking real background patches for the job.

The parallel to attention sinks is direct. In both cases, a softmax-based attention mechanism is structurally forced to allocate its full probability budget on every single step, and in both cases the model responds by designating some token — content-independent, always available — as the place to dump whatever it doesn't need elsewhere. Registers and sinks were discovered independently, in different architectures (bidirectional ViT encoders versus autoregressive decoder-only LLMs), addressing different downstream problems (cleaner attention maps and feature quality for registers; unbounded streaming for sinks) — and yet the fix both groups converged on is structurally the same: add a dedicated, content-free token and let the model learn to use it.

One genuine difference is worth carrying over from Chapter 7: ViT registers keep helping as you add more of them, while a second learnable sink token in a decoder-only model does not. The likely explanation is architectural — registers in a bidirectional encoder can plausibly absorb broad, distributed global-image information useful throughout the whole network, a job with room for more capacity. A decoder-only sink's job is narrower — purely a parking spot for softmax's forced leftover mass — and one reliable, always-present slot is apparently already enough capacity for that.

How this compares to earlier efficient-attention research

StreamingLLM did not arrive in a vacuum. A whole prior line of work — Sparse Transformer, Longformer, the Extended Transformer Construction (ETC), BigBird — already tried to make attention cheaper on long sequences by restricting which tokens are allowed to attend to which, using fixed local windows, strided patterns, or a small set of globally-visible tokens. Each represents a real, meaningful reduction in compute — Sparse Transformer, for instance, cuts attention's cost from O(n²) down to O(n√n).

Three practical problems kept those approaches from solving this paper's problem specifically. First, several of them (Sparse Transformer, ETC) need custom GPU kernels for specialized block-sparse matrix multiplication — real engineering overhead beyond what a standard attention implementation provides. Second, Longformer, ETC, and BigBird all lean on some notion of globally-visible tokens attending bidirectionally, a pattern that doesn't map cleanly onto autoregressive, causal, decoder-only generation — the setting this entire lesson has been in. Third, and most limiting for anyone with an existing deployed model: all of them require training the sparse attention pattern in from scratch. None are compatible with a pretrained dense-attention checkpoint the way this lesson's Chapters 5–6 are.

That third point is the practical center of why this lesson spent ten chapters on a four-token fix instead of a new attention architecture: StreamingLLM works on Llama-2, MPT, Falcon, and Pythia checkpoints that already exist, trained the ordinary way, with dense attention, by people who never heard of attention sinks. Nothing about the weights changes. That's a qualitatively different value proposition than “here's a new architecture, retrain from scratch” — it's “here's a cache-management wrapper for the model you already have.”

Where StreamingLLM sits among the alternatives

That last row matters for how to actually use this lesson: StreamingLLM and context-extension methods are not competitors, they're orthogonal knobs. StreamingLLM decouples how long a conversation can run from the model's pretraining window; context extension changes how much recent context fits inside the rolling window itself. A production system reasonably uses both — a context-extended model as the base, with StreamingLLM wrapped around it for truly unbounded operation.

Where this actually gets used

Bring it back to Chapter 0's opening scenario. The paper frames its intended use around exactly that day-long-assistant case: continuous multi-round dialogue where an LLM needs to keep functioning over long stretches without either resetting its context (losing the recent thread) or paying to recompute recent history from scratch (Chapter 8's slow, correct baseline). A persistent daily assistant, a long customer-support session, or any interface meant to stay open and responsive for hours is exactly the shape of problem this fix targets — not because any of them need infinite memory, but because they need to never crash or degrade, for as long as the session runs. The upside is genuinely practical: lower memory footprint means the same hardware serves more simultaneous users, which is both a cost story and, in the paper's own framing, an accessibility one — cheaper inference lowers the bar for who can afford to run these systems at all.

The numbers that carried this lesson, together in one place

Eleven numbers, one throughline. Every one of them is a consequence of the same softmax constraint Chapter 4 derived from a single equation. The KV cache's unbounded growth, window attention's catastrophic collapse, the existence and number of attention sinks, the fix's four-token minimalism, and its clean 22× speedup are not eleven separate facts to memorize — they're one idea, followed all the way through from a memory problem to a working, deployed system.

If you're implementing this today: a decision guide

Collapse everything from Chapters 1 through 8 into the sequence of decisions an engineer actually has to make, in order:

Every one of those five decisions traces back to a specific chapter's derivation, not a rule of thumb handed down without justification. That's the standard this lesson tried to hold itself to throughout: no number should be trusted until it's been derived, measured, or both.

One last worked check: does the whole pipeline actually add up?

Close the loop with one final sanity check, chaining several of this lesson's numbers together end to end. Take a StreamingLLM deployment on Llama-2-7B, 4 sink tokens, a 1,020-token rolling window (Chapter 5's running example), serving 16 concurrent conversations (Chapter 1's batching axis), each one long enough to have filled its window:

Add the model's own weights (≈13.0 GiB, Chapter 1) and this deployment comfortably fits on a single 24 GiB card with room to spare — 16 simultaneous day-long-capable conversations, each one able to run for 4 million tokens without ever growing past that same 512 MiB, on hardware that Chapter 0 showed couldn't even hold one such conversation's dense-attention cache past about four hours. That comparison, computed from numbers this lesson actually derived rather than asserted, is the entire argument for everything Chapters 5 through 7 built, restated as one final piece of arithmetic.

Connections on this site

This session assumed you already understand how attention and the KV cache work mechanically. If any of that felt shaky, these are the lessons underneath this one:

• Attention & Transformers — the QK^TV mechanism this entire lesson assumes.
• The Transformer — the full architecture the KV cache lives inside.
• CS229S: FlashAttention — a different, complementary answer to attention's quadratic cost, at the kernel level rather than the cache-eviction level.
• CS229S: Transformer Efficiency — the broader landscape of serving-cost tricks this session's KV-cache arithmetic belongs to.
• LLM Inference and the Inference Handbook — where KV caching, batching, and streaming meet real serving systems.
• Attention Variants — other departures from vanilla softmax attention, for context on how unusual (or not) SoftMax₁ really is.
• Long-Context LLMs — the context-extension methods (RoPE interpolation, ALiBi, YaRN) referenced in this chapter's comparison table.

What this paper doesn't test, and honestly can't tell you

Close with the same honesty Chapter 8 modeled throughout. This lesson, following the paper closely, tested text-only, standard-precision (or near-standard) language models on natural-language streams. It says nothing directly about how attention sinks behave in heavily quantized models (where the logits feeding softmax are themselves approximated), in multimodal models mixing text with image or audio tokens, or in attention variants substantially different from the four families tested here. The underlying argument — softmax must sum to 1, so excess mass must land somewhere — is a property of any softmax-normalized attention, and there's good reason (the BERT and ViT evidence above) to expect it generalizes broadly. But “good reason to expect” is not the same claim as “empirically verified,” and this lesson has tried throughout to keep that distinction visible rather than blur it.

What you should be able to do now

If this lesson worked, you should be able to, without looking anything up: derive the per-token byte cost of a KV cache from a model's hidden size, layer count, and precision; explain in one sentence why softmax forces attention sinks to exist; predict, from a single number (the sink's share of total attention), roughly how bad evicting it will be; and describe StreamingLLM's fix precisely enough that someone else could implement it from your description alone. That last one is the real test — not whether you recognize the right multiple-choice answer, but whether you could rebuild the mechanism from scratch, the way Chapter 5's StreamingCache class was built from nothing more than Chapters 1 through 4's diagnosis.

“What information consumes is rather obvious: it consumes the attention of its recipients.” — Herbert Simon, 1971