Scaling Laws: Chinchilla & Precision

Chapter 1: Fixed-Model Curves

Chapter 0 posed the question: given a compute budget C, what (N, D) pair minimizes loss? The paper this session is built on — Training Compute-Optimal Large Language Models, by Hoffmann, Borgeaud, Mensch, and 19 co-authors at DeepMind, published in March 2022 — answers this three separate, independent ways, and only trusts the answer because all three agree. This chapter covers the first.

The method: train the same models many times, differently

Take a range of model sizes, from 70 million up to over 10 billion parameters. For each size, train it four separate times, using four different cosine learning-rate schedules — a training recipe where the learning rate ramps up, then decays smoothly to a small fraction of its peak over some chosen horizon, measured in tokens. The four horizons differ by a factor of 16× from shortest to longest.

For each of these runs, plot the training loss continuously as a function of the number of tokens seen so far. Then trace the lower envelope across every single run and every model size: at each point along the FLOPs axis, which model, trained for how long, achieved the lowest loss for that many FLOPs spent? That envelope is a direct, empirical map from “FLOPs spent” to “best loss achievable,” and critically, it also records which (N, D) pair achieved it at every point.

Why an envelope, and not just “pick the model that ends lowest”

It's worth being precise about why the envelope — the lower boundary across every run's curve, not just each run's final loss — is the right object to trace. A single training run of a 10-billion-parameter model produces one loss curve, but that curve passes through many different (FLOPs spent so far, loss so far) points on its way to its final value. Each of those intermediate points is itself a valid, honest data point: “this many FLOPs were spent, and this was the best loss achieved with them, given this particular model size.” A smaller model's curve might sit lower than a bigger model's curve early on (small models cross the useful-capacity threshold faster, per FLOP, when data is scarce), then get overtaken once the bigger model has seen enough tokens to make its extra capacity pay off. The envelope across every curve, at every FLOPs value, is therefore not one model's curve — it's a composite, stitched from whichever model size is winning at that particular point in compute.

A concrete, toy version of the tracing procedure

Strip this down to two models and see the crossover directly. Say a 100M-parameter model and a 1B-parameter model are both being trained, and (purely illustratively, to make the mechanism visible) their smoothed loss curves as a function of FLOPs spent so far happen to look like:

At small C, the bigger model's larger leading coefficient (5.1 vs. 3.4) makes its loss higher — it hasn't seen enough tokens yet to make its extra capacity worth anything, so the smaller model wins on the envelope. Both curves decay at the same rate here (same exponent, for simplicity), so the bigger model's loss term always stays proportionally larger — in this toy setup it never catches up, which is itself the lesson: model size only pays for itself once it's paired with enough data to use it, and until that crossover point, the honest envelope keeps recording the smaller model as compute-optimal. The real Chapter 2 (IsoFLOP) picks up the more realistic version, where model size interacts with the loss non-trivially rather than through a shared decay rate, and a true crossover does appear.

python
# The envelope-tracing procedure, sketched (not the paper's exact interpolation --
# DeepMind smooths each curve with a spline before comparing; this is the idea).
def trace_envelope(runs, flop_grid):
    """runs: list of (model_size, loss_fn) where loss_fn(flops) -> smoothed loss.
    Returns, for each FLOP value, the (model_size, loss) that wins."""
    envelope = []
    for C in flop_grid:
        best = min(runs, key=lambda r: r[1](C))   # lowest loss at this exact FLOPs value
        envelope.append((C, best[0], best[1](C)))
    return envelope

# 1,500 log-spaced FLOP values, per the paper -- dense enough that the resulting
# (C, N_opt) scatter can be power-law-fit with confidence
flop_grid = [10**(18 + i*0.005) for i in range(1500)]

A subtlety that determines everything: matching the schedule to the run length

Here is the detail that, on its own, explains most of the gap between this paper's conclusion and Kaplan's 2020 one. Kaplan's study used a single, fixed cosine schedule — tuned to decay over 130 billion tokens — for every model, regardless of how long that particular model was actually going to train. If you only train for 10 billion tokens but your learning-rate schedule is tuned to decay smoothly all the way out to 130 billion, your learning rate is still relatively high and undecayed at the point you stop. That is a systematically worse final loss than a model whose schedule was tuned to finish decaying exactly when training stops.

The consequence compounds in a specific direction. Every short run in Kaplan's data was measured with an ill-fitting, too-long schedule, making short-data training look artificially worse than it truly was. That systematically underestimated how good a model trained on less data could be — which nudges the fitted conclusion toward “you need more parameters, not more data,” exactly the direction Kaplan's paper leaned. This paper's own runs instead match the schedule length to each run's actual horizon: train for D tokens, and set the cosine decay to finish right around D tokens too. A supplementary check found that with a matched schedule, the best FLOP-per-loss points across the whole training run consistently fall within the final 15% of that run — almost fully decayed, not mid-schedule.

The result, and the comparison that matters

Fitting power laws Nopt ∝ Ca and Dopt ∝ Cb to the 1,500 sampled envelope points gives:

This single table is the entire disagreement, in two numbers. Kaplan's exponents say: as compute grows 10×, grow the model by 100.73 ≈ 5.4× and the data by only 100.27 ≈ 1.9× — almost all the gain into parameters. This paper's exponents say: grow both by 100.50 = √10 ≈ 3.16× — split evenly, every time, at every scale. Since a ≈ b, the ratio Dopt / Nopt stays roughly constant as compute grows — which is exactly the “fixed tokens per parameter” rule Chapter 4 will pin down numerically.

Worked check: what does 10× more compute actually buy, under each rule?

Take a concrete starting point — a 1-billion-parameter model, and ask what a lab following each rule would build with 10× the compute.

Kaplan's rule builds a model nearly 70% bigger than this paper's rule (5.4B vs 3.16B) for the identical 10× compute budget — and correspondingly starves it of data growth (1.9× vs 3.16×). Every model in Chapter 0's table was built following something close to the Kaplan-shaped recipe. The rest of this session is the receipt for what that cost them.

Where the confidence intervals come from

Table 2's parenthesized ranges — 0.50 (0.488, 0.502) — aren't decoration; they're the honest answer to “how much would this exponent have wobbled if the underlying set of training runs had been slightly different?” The method is bootstrapping: repeatedly resample 80% of the collected training runs (with replacement), refit the power law on each resample, and look at the spread of fitted exponents across 100 such resamples. The reported interval is the 10th-to-90th percentile of that spread.

python
import numpy as np

def bootstrap_exponent(C_vals, N_vals, n_boot=100, frac=0.8):
    """Refit N_opt ~ C^a on 100 resamples of 80% of the data; return the spread."""
    n = len(C_vals)
    exponents = []
    for _ in range(n_boot):
        idx = np.random.choice(n, size=int(frac*n), replace=True)
        # fit log(N) = a*log(C) + const via least squares on the resampled subset
        a, _ = np.polyfit(np.log(C_vals[idx]), np.log(N_vals[idx]), 1)
        exponents.append(a)
    return np.percentile(exponents, [10, 50, 90])

# p10, p50, p90 = bootstrap_exponent(flop_budgets, optimal_N_per_budget)
# -> roughly [0.488, 0.50, 0.502] for Approach 1, per the published table

Notice how tight Approach 1's interval is (0.488–0.502, a spread of only 0.014) compared to Approach 2's (0.462–0.534, a spread over five times wider). Approach 1 draws on 1,500 envelope points from many training curves; Approach 2 draws on only nine IsoFLOP budgets. More independent evidence, tighter bootstrap interval — exactly the behavior you'd want a trustworthy uncertainty estimate to have.

Why models bigger than 10B needed a second, independent check

Approach 1's model grid tops out around 10 billion parameters. That matters, because the paper's own appendix finds a small amount of curvature in the FLOP-to-loss frontier once model sizes get large enough — the relationship isn't a perfectly straight power law forever. Relying on one method, extrapolated far past where it was measured, is exactly the kind of mistake this whole session is about avoiding. That is why Chapter 2 introduces a second, structurally different method — one that goes up to 16 billion parameters and asks the question in a different order — before trusting any conclusion.

Chapter 2: IsoFLOP Profiles

Approach 1 asked: across an entire training run, what's the best loss achieved at each point along the way? Approach 2 asks a cleaner, more direct version of the same question, by inverting the order of operations: fix the compute budget first, then ask which model size is best for that exact budget, using only the fully trained final loss — no envelope-tracing through partial training curves required.

The method: hold FLOPs constant, vary the model

Pick a fixed training-compute budget — the paper uses nine of them, ranging from 6×1018 up to 3×1021 FLOPs. For each budget, train a range of model sizes, choosing the number of training tokens for each one so that 6ND lands on that exact budget — a bigger model trains on proportionally fewer tokens, a smaller one on proportionally more, so every point on an IsoFLOP curve (“iso” meaning equal) costs the same total compute. Each model's cosine schedule is set to match its own token count exactly — the same fix from Chapter 1, applied consistently here too.

Then, for a fixed FLOP budget, plot the final loss against model size. The result is not a straight line — it's a curve with a clear valley. Too few parameters, and the model is too small to represent the data well even with plenty of tokens to see. Too many parameters, and the model barely gets to see any tokens at all before the compute runs out, and undertraining dominates. Somewhere in between is a minimum.

Finding the valley's floor: fit a parabola

On log-log axes (log loss against log model size), each IsoFLOP curve is close enough to parabolic near its minimum that fitting a parabola to the sampled points and solving for its vertex gives a clean, precise estimate of the optimal model size at that budget — without needing to have sampled a model size exactly at the true minimum. This is standard practice for locating the minimum of any smooth, roughly-quadratic curve from a handful of samples, and it's what turns nine discrete FLOP budgets into nine precise (C, Nopt) points, ready for the same power-law fit Chapter 1 used.

Why nine budgets, and why that particular range

The range 6×1018 to 3×1021 FLOPs isn't arbitrary — it spans roughly two and a half orders of magnitude, wide enough to fit a power law with real leverage (a line fit through points clustered in a narrow range is far more sensitive to noise than one fit through points spread widely), but still cheap enough, at the low end, to run many model-size sweeps per budget without burning the paper's entire compute allocation on Approach 2 alone. At the low end, 6×1018 FLOPs is enough to fully train dozens of small models (tens of millions of parameters) many times over, cheap enough to explore a wide range of sizes per IsoFLOP curve. At the high end, 3×1021 FLOPs starts to approach the cost of a real production-scale run, expensive enough that only a handful of model sizes can be afforded per budget — which is exactly why the parabola-fitting trick from the next section matters: at the expensive end of the sweep, you cannot afford to brute-force sample enough points to see the valley floor directly, you have to interpolate it from a few.

Worked example: finding a vertex from exactly three points, by hand

Here is the parabola-fit mechanism, run to completion, with real numbers. Fix a compute budget of C = 1020 FLOPs and sample three model sizes, equally spaced in log10(N) by a step of h = 0.3 — 108.7, 109.0, 109.3 params — using the parametric loss formula Chapter 3 derives (so these losses are exact, not illustrative):

For three points equally spaced by h in x, a parabola's vertex offset from the middle point has a clean closed form — the numerator is a finite-difference estimate of the slope at the middle point, the denominator is a finite-difference estimate of the curvature:

Plug in the three losses:

So the vertex sits at log10(N) ≈ 9.0 − 0.1906 = 8.8095, i.e. Nopt ≈ 6.45×108 ≈ 645 million parameters.

python
def isoflop_vertex(log_N_mid, loss_fn, C, h=0.3):
    """Three-point parabola vertex, equally spaced by h in log10(N)."""
    x1, x2, x3 = log_N_mid - h, log_N_mid, log_N_mid + h
    y1, y2, y3 = [loss_fn(10**x, C) for x in (x1, x2, x3)]
    dx = -h * (y3 - y1) / (2 * (y3 - 2*y2 + y1))
    return 10**(x2 + dx)

def loss_at(N, C, E=1.69, A=406.4, B=410.7, alpha=0.34, beta=0.28):
    D = C / (6*N)
    return E + A*N**(-alpha) + B*D**(-beta)

isoflop_vertex(9.0, loss_at, 1e20)   # -> ~6.45e8, matches the by-hand result above

What if the three sampled points don't straddle the true minimum?

The worked example above deliberately centered its three points on the true optimum. Real experiments don't get that luxury — you choose model sizes to train before knowing where the minimum is. Rerun the same by-hand recipe with all three points sitting to one side of the true minimum instead — log10(N) = 9.3, 9.6, 9.9, all noticeably larger than the true optimum near 8.809:

Running the same finite-difference formula gives a vertex estimate at log10(N) ≈ 8.846 — N ≈ 701 million, versus the true 644 million. About a 9% error, noticeably worse than the earlier example's sub-0.1% but still a reasonable extrapolation from three points that never got close to the actual floor. The lesson generalizes: parabola-vertex fitting degrades gracefully, not catastrophically, when your samples miss the minimum — but it degrades. This is precisely why the paper's own protocol insists on checking, for every one of the nine real IsoFLOP budgets, that “a diverse enough set of model sizes” was trained to see a clear valley, not just a downward or upward slope. Confidence in a fitted minimum is only as good as the evidence that a minimum was actually bracketed.

The result, and why it corroborates Chapter 1

Fitting Nopt ∝ Ca and Dopt ∝ Cb to these nine valley-floor points gives:

Compare this to Approach 1's a = 0.50, b = 0.50. These two numbers were derived from a structurally different experiment — different training runs, different model sizes (up to 16B here versus 10B in Approach 1), a different way of extracting the optimum from the data (parabola-vertex fitting versus envelope-tracing across full training curves). They agree to within their own uncertainty. That agreement is the entire reason this paper's conclusion is trustworthy rather than a fitting artifact of one particular method.

Concept → realization: what “agreement between methods” is actually buying you

It would be easy to read Chapter 1 alone and worry that a ≈ b ≈ 0.50 was somehow baked in by a choice made during Approach 1's envelope-fitting procedure — a hidden assumption, a particular smoothing kernel, an artifact of which 1,500 FLOP values got sampled. Approach 2 doesn't share any of that machinery. It doesn't trace training curves at all; it only ever looks at fully-finished, converged final losses. The fact that two experiments built almost entirely differently land on the same exponents is what makes this a scientific finding rather than a curve-fitting coincidence — and it's exactly the kind of cross-check every one of the papers in this course has, in its own way, insisted on before trusting a number.

Chapter 3: The Parametric Loss

Approaches 1 and 2 both fit power laws after the fact — find the optimal N at each of several compute budgets, then fit a line through those optima in log-log space. Approach 3 does something more ambitious: build a single closed-form function L(N, D) that predicts the loss for any (N, D) pair directly, fit its handful of constants once against every training run from both earlier approaches combined, and then minimize that one function analytically to get Nopt(C) and Dopt(C) in closed form — no re-fitting needed for a new compute budget.

Deriving the functional form from first principles

Start from what the true, ideal loss on this data distribution decomposes into. Write f⋆ for the theoretically best possible next-token predictor — the Bayes-optimal one, limited only by the actual entropy of natural language itself, not by any model's capacity. Write fN for the best a transformer with exactly N parameters could ever achieve, given infinite data and infinite training steps. Write f̄N,D for what a real, finite training run with N parameters and D tokens actually achieves. The realized loss decomposes as three additive, non-negative pieces:

Read the three terms left to right. The first is the irreducible entropy of natural text — no model, however large, however well trained, gets below it, because that's the genuine unpredictability of language itself. The second is the approximation error: how much worse a finite-capacity model of size N is than the theoretical best, even given unlimited data and training. The third is the estimation error: how much worse a real, finitely-trained run on D tokens is than the best that same N-parameter model could achieve with infinite data.

Existing theory gives an expected shape for each of the last two terms. For a broad class of function approximators, the approximation-error term is expected to shrink as roughly 1/√N as capacity grows. The estimation-error term, from classical stochastic-optimization theory, is lower-bounded by roughly 1/√D — the standard convergence rate for first-order methods trained on a finite sample. Both are power laws in their respective variable, just not necessarily with the exponent exactly one-half once fit to real transformer training runs rather than idealized theory. That motivates the proposed functional form:

Three additive terms, matching the three-part decomposition exactly: E is the irreducible entropy floor, A/Nα is the model-capacity penalty that shrinks as parameters grow, and B/Dβ is the finite-data penalty that shrinks as tokens grow. Five free constants — E, A, B, α, β — get fit once, jointly, against every loss measurement from both Approach 1 and Approach 2's runs combined.

How the fit is actually done

Fitting five constants against hundreds of noisy loss measurements is done by minimizing a robust loss (the Huber loss, which behaves like squared error for small residuals but like absolute error for large ones, making it resistant to being thrown off by a handful of outlier runs) between the predicted and observed log loss, using the L-BFGS optimization algorithm, started from a grid of different initial guesses to guard against getting stuck in a bad local minimum. The published, fitted values are:

Both exponents sit below 0.5 — lower than the idealized 1/√N and 1/√D theory predicted, meaning real transformers extract value from extra parameters and extra data a little more slowly than the simplest theoretical bound suggests. The paper explicitly flags this as room for future architectures and training methods to improve on.

Why Huber loss, and why fit in log-space at all

Two design choices in the fitting procedure are easy to skim past and both matter. First: the fit minimizes error in log(L), not raw L. Loss values across the full range of models trained (70M-parameter toy runs up to 16B-parameter ones) span nearly an order of magnitude, from well above 3.0 nats down toward 1.8. Fitting on raw loss would let the huge-model runs, with their small absolute losses, be dominated numerically by whatever noise the small-model runs contribute; fitting on log-loss puts every run on comparable footing, proportionally, regardless of scale.

Second: the Huber loss, not ordinary squared error. Squared error punishes large residuals quadratically — one badly-behaved outlier run (an unstable training curve, a schedule misconfiguration, an unlucky seed) can drag an entire least-squares fit toward accommodating it. Huber loss behaves like squared error close to zero (smooth, differentiable, good gradient signal for L-BFGS) but transitions to behaving like absolute error past a threshold δ, capping how much any single outlier point can influence the fit:

python
def huber(residual, delta=1e-3):
    """delta = 1e-3, per the paper -- residuals are computed in log-loss space,
    so a delta this small is already a large relative miss."""
    r = abs(residual)
    if r <= delta:
        return 0.5 * residual**2
    return delta * (r - 0.5 * delta)

def fit_objective(params, runs):
    E, A, B, alpha, beta = params
    total = 0.0
    for (N, D, L_observed) in runs:
        L_pred = E + A * N**(-alpha) + B * D**(-beta)
        import math
        residual = math.log(L_pred) - math.log(L_observed)
        total += huber(residual)
    return total

# minimize fit_objective over (E,A,B,alpha,beta) via L-BFGS, from a grid of
# starting points -- the paper sweeps alpha,beta in {0, 0.5, ..., 2.0},
# e in {-1, -0.5, ..., 1.0}, a,b in {0, 5, ..., 25} as initial guesses,
# and reports the optimum never landed on the boundary of that sweep

The paper reports that δ = 10−3 was chosen deliberately — a larger δ lets the fit overweight the small-compute regime (where many more runs happen to be cheap to collect) and predict held-out large-scale data poorly; a smaller δ than that made no further difference to the resulting predictions. That's a real, reported robustness check, not a default left untuned.

A worked toy fit: recovering A and B by hand

The real fitting procedure uses hundreds of data points and five simultaneous unknowns solved by numerical optimization — not something to reproduce by hand. But the structure of the problem, once α, β, E are already known (say, from a coarse first pass over the data), collapses into something genuinely tractable: two unknowns, A and B, appearing linearly. Given just two (N, D, L) observations, you can solve for both exactly. Here is that calculation, done in full, using two real points drawn from this session's own worked tables so the result can be checked against the published constants at the end.

The two anchor points. Take a 400-million-parameter model trained on 8.0 billion tokens (predicted loss, using the real fitted formula, 2.8662), and a 67-billion-parameter model trained on 1.5 trillion tokens (predicted loss 1.9348) — both drawn from Table 3 in Chapter 4. Rearranged, each observation gives one linear equation in the two unknowns A and B:

Step 1 — compute the coefficients, by hand, using logs. For point 1 (N=4×108, α=0.34), the coefficient on A is N−0.34 = 1/N0.34. Compute the exponent first, in natural log, because fractional powers are far easier to handle additively:

Repeat the same three-step recipe (log, multiply by the exponent, exponentiate, invert) for the other three coefficients — the arithmetic is identical each time, only the numbers change:

Step 2 — write the two linear equations. Subtracting E = 1.69 from each observed loss:

Step 3 — eliminate A. Scale Point 2's equation so its A coefficient matches Point 1's exactly — multiply through by 0.0011893 ÷ 0.0002085 ≈ 5.7038:

Subtract the (unscaled) Point 1 equation from this scaled version. The A terms cancel exactly, leaving one equation in B alone:

Step 4 — back-substitute for A using Point 1's original equation:

Why this exercise matters beyond checking arithmetic

Two data points and two unknowns is not how you'd actually trust a fit — with only two points, the “fit” is really just exact interpolation, no error bars, no way to detect whether the functional form is even right. That's precisely why the real study used over 400 models and a robust loss with outlier-resistant weighting. But the linear-in-(A,B) structure this hand-derivation exposes is real and load-bearing: it's what makes the full five-parameter fit tractable at all. Fix α, β, E at a reasonable guess, solve the resulting linear system for A, B in closed form, measure the residual, adjust α, β, E, repeat. That's a fast inner loop, and it's exactly the kind of structural insight — not the specific numbers — that a derivation like this is meant to leave you with.

Chapter 4: The Efficient Frontier

With L(N,D) = E + A/Nα + B/Dβ fitted, the compute-optimal allocation Chapter 0 asked for stops being an empirical curve-fit and becomes a calculus problem: minimize L subject to 6ND = C, for any C at all, including budgets far beyond anything actually trained.

The closed-form solution

Substitute D = C/(6N) into the loss, which turns a two-variable constrained problem into a plain one-variable minimization over N alone:

Differentiate with respect to N and set the result to zero:

Move the first term to the right side and divide through. The N−1 factors cancel from both sides, leaving a single power of N isolated:

Raise both sides to the power 1/(α+β) to solve for N itself. This is exactly the closed form:

(Dopt follows immediately by plugging Nopt back into D=C/6N and simplifying the exponents the same way.) Notice immediately that a + b = 1 always, by construction — the two exponents are two pieces of a single whole. Plugging in the fitted constants (α=0.34, β=0.28) gives a = 0.28/0.62 ≈ 0.452 and b = 0.34/0.62 ≈ 0.548 — close to, though not identical to, Approach 1 and 2's directly-fit a ≈ b ≈ 0.50. All three approaches land in the same neighborhood; Approach 3 predicts a slightly more data-heavy split at very large scale, a genuine, documented difference the paper attributes to the small curvature Chapter 2 flagged, which pulls this method's prediction toward smaller optimal model sizes as compute grows very large.

It's worth checking the special case that makes all three approaches collapse into one, since it explains exactly why Approach 1 and 2's exponents came out so close to 0.50 in the first place. If α = β exactly, then a = β/(α+β) = α/(2α) = 0.5 and, by the identical substitution, b = 0.5 too — an exact 50/50 split, with no dependence on the specific value of α at all, only on the two exponents being equal to each other. Approach 1 and 2 don't fit α and β as separate quantities at all — they fit a and b directly from data — and the fact that both landed almost exactly at 0.50 is itself evidence that, whatever the true underlying α and β are, they must be close to equal. Approach 3's own fit (α=0.34, β=0.28) shows they aren't exactly equal — a real, if modest, asymmetry between how quickly extra parameters help versus how quickly extra data helps — and that small asymmetry is the entire source of the small gap between Approach 3's a≈0.452 and Approach 1/2's clean a=0.50.

Worked by hand: where that low-30s-billions number actually comes from

The caution above asserts a gap between two numbers without showing where either one comes from. Here is the first one, G, computed the same way Chapter 3's toy fit was — by hand, with logs, so every digit is traceable rather than taken on faith.

Start from the definition G = (αA / βB)1/(α+β), using Chapter 3's published constants α=0.34, A=406.4, β=0.28, B=410.7. Compute the numerator and denominator first:

Raise this ratio to the power 1/(α+β) = 1/0.62 ≈ 1.61290. As with every fractional-power calculation this session has done by hand, take the log first, multiply, then exponentiate:

Now evaluate Nopt(C) = G · (C/6)a at Gopher's precisely-accounted compute budget from Chapter 5, C = 5.76×1023 FLOPs (not the looser 6ND shortcut), using a = β/(α+β) = 0.28/0.62 ≈ 0.45161:

That is exactly the “low-30-billions” figure the caution above referred to — now with every digit of the arithmetic exposed, computed from nothing but Chapter 3's four published constants and this chapter's own closed-form solution.

Why the exponent, not the prefactor, is where rounding actually bites

It's natural to assume the 32B-versus-40B gap comes from rounding A and B — they are, after all, the constants with the most digits. Test that assumption directly: nudge A from 406.4 to 406.45, a change smaller than the paper's own stated precision, and recompute G and Nopt from scratch.

Barely moves — a 0.01% change in A produced roughly a 0.02% change in the final answer, exactly the proportional sensitivity you'd expect from a number that only ever enters through the prefactor G. The real amplifier lives somewhere else. Compare the exponent this chapter's own α=0.34, β=0.28 fit implies, a = 0.45161, against the exponent the published paper's own Table 2 reports directly for Approach 3: a = 0.46 — a difference of only 0.00839, under one percentage point. Hold G fixed and swap in that one-percentage-point-smaller exponent instead:

A 0.00839 shift in the exponent alone — nothing else touched — multiplies the predicted Nopt by 1.56×, carrying it from 32.19B up past 50B, overshooting the paper's reported ~40B in the other direction. (That overshoot is itself informative: in the real fit, G and the exponent aren't free to move independently the way this test moved them — refitting one shifts the other to partially compensate, which is exactly why the actual gap lands at 40B rather than 32B or 50B.) The lesson generalizes well past this one paper: whenever a fitted quantity gets raised to a fractional power and then multiplied against something astronomically large — here, C/6 itself is 9.6×1022 — any rounding error in the exponent gets multiplied through ln(base), a factor of 52.9 in this case, before it ever reaches the final answer. A rounding error in the prefactor only ever multiplies through by 1. Reading any power-law headline number, the exponent's precision is worth scrutinizing far more carefully than the prefactor's — a reflex worth carrying well past this one session.

python
E, A, B, alpha, beta = 1.69, 406.4, 410.7, 0.34, 0.28

a = beta / (alpha + beta)
b = alpha / (alpha + beta)
G = (alpha * A / (beta * B)) ** (1 / (alpha + beta))

def N_opt(C): return G * (C / 6) ** a
def D_opt(C): return (C / 6) ** b / G

# a much smaller, "academic-scale" budget -- what a university lab, not a
# frontier lab, might actually have access to
C_academic = 2e21
print(N_opt(C_academic) / 1e9, "B params")   # -> ~2.49B
print(D_opt(C_academic) / 1e9, "B tokens")   # -> ~133.6B
print(D_opt(C_academic) / N_opt(C_academic))         # -> ~53.6 tokens/param

At a two-orders-of-magnitude-smaller compute budget than Gopher's — 2×1021 FLOPs, well within reach of a well-funded university cluster — the closed form recommends roughly a 2.5-billion-parameter model on about 134 billion tokens. Notice the ratio, near 54 tokens per parameter, sits well above Table 3's ~20–22 band. That's not a bug — it's a direct, provable consequence of a ≠ b under Approach 3 (a ≈ 0.452 < b ≈ 0.548, versus Approach 1's exact a=b=0.50). Since Dopt/Nopt ∝ Cb−a and b−a > 0 here, Approach 3's own closed form predicts the tokens-per-parameter ratio grows with compute, unlike Approach 1's nearly-flat one — a small but genuine disagreement between the three methods about the frontier's exact shape, on top of the absolute-value sensitivity the caution above already flagged. It's exactly the kind of second-order discrepancy that motivated Section 3.4's cross-checking of all three approaches against each other, rather than reporting any single one alone.

Table 3: the frontier, spelled out at nine scales

Rather than trust the closed form in the abstract, the paper tabulates what Approach 1's fitted power law (a = b = 0.50 exactly) predicts at nine model sizes, from toy-scale to far beyond anything trained in 2022:

Read the rightmost column top to bottom: 20.0, 20.2, 20.5, 22.4, 21.1, 21.1, 21.2, 21.2, 21.6. Across nine orders of magnitude of compute — from a toy model up to a hypothetical 10-trillion-parameter one — the tokens-per-parameter ratio drifts only from about 20 up to about 22. That is the origin of the “20 tokens per parameter” rule of thumb: because a ≈ b, the ratio Dopt/Nopt ∝ Cb−a has an exponent close to zero, so it is nearly — not perfectly, but nearly — scale-invariant.

Spot-check one interior row against the raw C=6ND relationship, the same way Chapter 5 will cross-check Chinchilla itself — take the 10-billion-parameter row:

Table 3 lists 1.23×1022 for that row — matching to three significant figures. Every single row in Table 3 satisfies this same identity exactly, because Nopt and Dopt were generated from the closed-form solution under the constraint 6ND=C, not fit independently and then happened to agree. It's a useful reflex for reading any scaling-law table: the columns should always satisfy the constraint equation the table claims to be optimizing under, and it costs one line of arithmetic to check.

Cross-checking against the other two approaches, at the one budget that mattered

Table 3 reports Approach 1's own numbers at nine model sizes, including the exact one this whole session keeps returning to: 67 billion parameters, 1.5 trillion tokens, for Gopher's compute budget. That's one method's answer. The paper ran the identical exercise with Approaches 2 and 3 — their own, structurally different fitting procedures — and reported their answers for that same budget too, a natural cross-check this session hasn't assembled into one place yet.

Read the first three rows and Chapter 2's curvature callout resurfaces in actual numbers. Approaches 1 and 2, which never assume a functional form for the loss at all, land within 6% of each other (67B vs. 63B). Approach 3, which fits one global closed-form function across every run from both, lands noticeably lower — 40B, over a third smaller. Approach 3's own token count for that budget isn't directly reported by the paper; deriving it by applying the constraint D=C÷6N — the same reflex Chapter 4 keeps returning to — gives 2.4 trillion tokens, a tokens-per-parameter ratio of 60, nearly three times Approach 1 and 2's ~22.

The paper explains this gap directly, and it's the same curvature Chapter 2's closing callout flagged, now made quantitative. Fitting Approach 3's single closed-form function with the Huber loss automatically down-weights training runs at low compute budgets, because those runs happen to carry the largest residuals against the fitted curve; the fit places more weight on the high-compute runs instead. Combined with a small, genuinely observed negative curvature in how Nopt bends against compute at the high end, Approach 3 systematically predicts a smaller optimal model size than Approaches 1 and 2 — not because it's wrong, but because it answers the question with a different, more compressed functional assumption baked in from the start.

This is also the arithmetic behind the “somewhere between 40 and 70 billion” band the next section cites: 40B is Approach 3's answer, 63–67B is Approaches 1 and 2's. Rather than trust any single method, DeepMind built at the upper end of that disagreement — 70B, essentially Approach 1's own anchor point, rounded up slightly — and let the benchmark table in Chapter 5 be the actual tiebreaker between the three predictions, rather than trusting whichever method's number happened to look cleanest on paper.

The rule is a band, not a razor edge

It would be a mistake to treat “20 tokens per parameter” as a single sacred constant. Table 3's own numbers drift from 20.0 to 22.4 to 21.6 depending on scale — a real, if small, effect. And when the paper went to actually verify this prediction by training a real model (Chapter 5), they didn't hit the Approach-1 point exactly: they built a 70B-parameter model on 1.4T tokens (ratio: exactly 20.0), a deliberate, practical choice slightly off Approach 1's precise 67B/1.5T point (ratio 22.4), justified by “both dataset and computational efficiency considerations,” and comfortably inside the 40–70 billion parameter range the paper's own Section 3.4 gives as the full compute-optimal range once all three approaches are considered together. The rule is directionally exact and numerically approximate — which is exactly the right level of precision to carry forward.

Chapter 5: Chinchilla vs. Gopher (showcase)

A derivation is a hypothesis until someone spends the compute to test it. DeepMind already had a 280-billion parameter model, Gopher, trained on 300 billion tokens at a cost of roughly 5.76×1023 FLOPs. Chapters 1–4 predicted that, for that exact compute budget, the optimal model was somewhere between 40 and 70 billion parameters, trained on well over a trillion tokens. So they built one, and named it Chinchilla.

Same compute, radically different shape

Same depth (80 layers each), same key/value dimension — but Chinchilla's hidden width is exactly half Gopher's (8,192 vs. 16,384), with exactly half the attention heads. It is, architecturally, a scaled-down Gopher, trained instead on 1.4 trillion tokens rather than 300 billion — 4.67× the data, on a model with a quarter the parameters.

A detail worth checking by hand: the per-head width didn't change

Look again at the “heads” and “key/value size” columns together. Gopher: 128 heads, each 128-dimensional. Chinchilla: 64 heads, each still 128-dimensional. Divide hidden width by head count for each model:

Identical. Halving dmodel and halving the head count together leaves each individual attention head exactly the same shape it was in Gopher — DeepMind removed entire heads rather than narrowing every existing one. This is a real, deliberate architectural choice, not a side effect of the parameter-scaling recipe: Chapters 1–4 only ever prescribe a total parameter count N, never how to distribute it across heads, layers, or width. Turning “70 billion parameters” into an actual architecture is a separate design decision, made using the field's existing heuristics for width-to-depth ratios (referenced in the paper's related work as prior work by Levine et al.), layered on top of — not derived from — this session's scaling law.

What else changed besides the parameter/token split

Chapter 5's headline comparison isolates one variable — how compute was split between N and D — but an honest read of the paper's own methods section shows three smaller changes rode along with it, each modest on its own, each worth naming so the benchmark table isn't mistaken for a single-cause experiment. Chinchilla switched from Adam to AdamW, an optimizer variant that decouples weight decay from the gradient update; the paper reports this alone improves both language-modeling loss and downstream fine-tuned performance. The tokenizer changed slightly too — a modified SentencePiece tokenizer that skips Unicode NFKC normalization, chosen because it noticeably helps represent mathematical and chemical notation more faithfully; 94.15% of the resulting vocabulary is shared with Gopher's, so this is a small perturbation, not a different tokenizer family. And the training precision setup (bfloat16 compute, float32 optimizer state) matches Gopher's own approach, so precision itself is not a variable this particular comparison isolates — consistent with Chapter 5 living entirely inside Chapters 0–4's two-variable (N,D) world, with Chapter 7's third axis, P, held fixed throughout.

Verify the compute match by hand

Before trusting any benchmark comparison, check the premise: did these two models actually cost the same to train? Using C ≈ 6ND:

The simple 6ND approximation gives Gopher 5.04×1023, a few percent below the paper's own more carefully accounted figure of 5.76×1023 (the precise FLOP count, detailed in the paper's appendix, includes embedding and output-layer costs that the simple 6ND shortcut doesn't fully capture). Chinchilla's 5.88×1023 lands within about 2% of that same precise Gopher figure — close enough to call it a genuinely compute-matched comparison, not an accounting trick.

python
def flops_6nd(N, D): return 6 * N * D

gopher = flops_6nd(280e9, 300e9)       # 5.04e23
chinchilla = flops_6nd(70e9, 1.4e12)    # 5.88e23
print(chinchilla / gopher)   # 1.167 -- within the range this shortcut's own ~2-5% slack explains

# and the tokens-per-parameter each model actually landed on:
print(300e9/280e9)    # Gopher:     1.07 tokens/param
print(1.4e12/70e9)   # Chinchilla: 20.0 tokens/param -- exactly the Chapter 4 rule of thumb

The results, task by task

Every comparison below trains for the identical compute budget, evaluated the identical way. This isn't “a bigger model beat a smaller one” — it's the reverse, at matched cost.

On MMLU specifically, Chinchilla's 67.6% didn't just beat Gopher — it beat the average forecast that 73 competitive human forecasters had made in 2021 for what state-of-the-art accuracy would be by June 2023 (63.4%), arriving a full year early. Out of MMLU's 57 individual subject tasks, Chinchilla outperformed Gopher on 51, tied on 2, and lost on only 4 — college mathematics, econometrics, moral scenarios, and formal logic.

Common sense, checked five different ways

Benchmark wins are more convincing when they replicate across independently designed tasks, rather than resting on one favorable evaluation. The paper's common-sense suite spans five separate benchmarks, each testing a different flavor of everyday reasoning — physical intuition, pronoun resolution, social inference, yes/no factual judgment:

Chinchilla wins outright on four of five, ties on the fifth (PIQA), and beats MT-NLG 530B — 7.6× more parameters — on every task but PIQA too. The one near-tie, PIQA, is itself informative: even where Chinchilla doesn't pull ahead, a 70B model trained compute-optimally matches a 530B model that wasn't, which is a strictly better outcome once training and inference cost are both counted.

What the 4×-smaller model buys beyond the benchmark table

The paper is explicit that pretraining compute is only part of a model's lifetime cost — fine-tuning and, above all, the millions or billions of inference calls a deployed model serves afterward, cost compute too. Because Chinchilla has one quarter of Gopher's parameters, its memory footprint and per-token inference cost are correspondingly smaller — the exact same 4× parameter-count ratio that made it compute-optimal to train also makes it cheaper to serve, for every single call made to it after training ends. That downstream saving compounds for the entire operational lifetime of the model, on top of the accuracy gains above — a genuine, undiluted win on every axis measured, for identical training cost.

Worked example: pricing out one billion served tokens

Put a number on “cheaper to serve.” Inference compute for a single generated token is, to the same 6N-per-token approximation used throughout this session, roughly proportional to parameter count (the factor of 2 rather than 6 sometimes quoted for inference-only, forward-pass-only compute doesn't change the ratio between two models being compared, since it cancels):

The unknown per-token constant factor cancels in the ratio, leaving exactly the parameter-count ratio: 280/70 = 4×. Every billion tokens a deployed Chinchilla serves costs one quarter the compute (and, roughly, one quarter the GPU-time and dollar cost) that the same billion tokens would have cost served from Gopher — on top of Chinchilla answering more of them correctly, per the benchmark tables above. A service serving, say, 50 billion tokens a month recovers the equivalent of 37.5 billion Gopher-scale “token-equivalents” of compute every single month it stays in production, for the rest of the model's deployed life — a savings that starts the day serving begins and never stops compounding.

Chapter 6: Caveats & Overtraining

“20 tokens per parameter” became, almost overnight, the most quoted number to come out of a pretraining paper in years. It is also, taken as a universal rule for how to build a production model, subtly wrong — not because the derivation was flawed, but because the derivation answered a narrower question than the one the field actually needed answered.

Caveat 1: compute-optimal is a pretraining-only optimum

Look again at exactly what Chapter 0 set up: minimize L(N,D) subject to a fixed training compute budget. That is the right question if training is the only cost that matters. It almost never is. A model that gets deployed gets called — by real users, millions or billions of times, for months or years — and every single one of those calls costs inference compute, roughly proportional to the model's parameter count. Chinchilla's own paper acknowledges this directly: part of its stated advantage over Gopher is exactly that its 4×-smaller size makes it cheaper to serve, on top of being cheaper to have trained.

But the 20-tokens-per-parameter rule doesn't account for inference cost at all — it only ever minimizes training FLOPs. If a model is going to be served at enormous scale, it can be worth deliberately overshooting the training-compute-optimal token count — accepting a training run that costs more FLOPs than strictly necessary to reach a given loss, in exchange for a smaller model that is dramatically cheaper to run for the rest of its deployed life. This tradeoff, not covered by Chinchilla-optimality alone, is precisely what Chapter 8's precision-scaling analysis returns to formally.

Worked example: when does inference cost actually catch up to training cost?

Caveat 1 says inference cost matters, but not yet how much serving volume it takes before it matters as much as training did. That threshold has a clean closed form, using nothing but Chapter 0's own forward-pass approximation (≈2N FLOPs per generated token) and the C≈6ND training-cost formula this entire session has built on.

Set the cumulative inference FLOPs for T served tokens equal to the FLOPs already spent training, and solve for T:

The model size N cancels completely. The breakeven serving volume — the point where cumulative inference compute catches up to total training compute — is always exactly three times the number of training tokens, regardless of how many parameters the model has. Plug in the two models from Chapter 5's own comparison:

This cuts both ways, and both directions matter for reading Caveat 1 correctly. Below roughly a trillion served tokens, Gopher's cumulative inference cost hasn't even caught up to what its own training already cost — at that volume, whether it runs at 280B or 70B parameters barely moves the total lifetime compute bill, because training dominates either way. But past that point, every additional served token costs strictly more on Gopher than on Chinchilla, in direct proportion to their 4× parameter gap, and that gap compounds without limit as serving volume keeps growing over a model's deployed years. A model called billions or trillions of times — squarely the regime a frontier-lab deployment operates in — blows past both breakeven points within its first weeks or months in production, which is exactly the regime where Caveat 1's inference-aware correction, and Caveat 2's deliberate overtraining, actually start to pay for themselves.

Caveat 2: the field has already moved past the 20:1 point — on purpose

Later work that studies scaling under this exact inference-aware lens (referenced directly inside the precision paper this session's second half is built on) documents just how far past Chinchilla-optimal real deployed models now sit. Llama-3-8B was trained to a tokens-per-parameter ratio of roughly 2,000 — a hundred times past the ~20:1 Chinchilla-optimal point. The Gemma-2 model family was trained past a ratio of 1,000. These are not mistakes; the field has a name for this deliberate strategy now: overtraining — intentionally training a smaller model on far more data than pure training-compute-optimality would recommend, because the smaller model's lifetime inference savings outweigh the extra up-front training cost.

Worked example: what overtraining Llama-3-8B actually cost, in training FLOPs

Put real numbers on Caveat 2, using nothing but arithmetic already introduced this session. A Chinchilla-optimal 8-billion-parameter model, at 20 tokens per parameter, would train on:

The real Llama-3-8B, at a reported ratio of D/N ≈ 2,000, trained on:

That's 100× the training compute a Chinchilla-optimal 8B model would have used. Ask the question Caveat 1 raised directly: what size model would have been training-compute-optimal for that same, actually-spent 7.68×1023 FLOPs? Using D=20N and C=6ND=120N2, solve for N:

Caveat 3: the data wall

Table 3's own numbers, read forward rather than backward, carry a second warning. A compute-optimal 10-trillion-parameter model would need 216.2 trillion tokens — not a number any lab in 2022 (or today) has anywhere near enough high-quality natural text to reach without repeating data many times over. The paper's own conclusion says this plainly: “the amount of training data that is projected to be needed is far beyond what is currently used to train large models, and underscores the importance of dataset collection in addition to engineering improvements that allow for model scale.” Scaling the parameter count further, compute-optimally, runs headfirst into a wall that has nothing to do with GPUs and everything to do with how much genuinely useful text exists to train on.

Caveat 4: overtraining has its own hidden cost, discovered two years later

There's a sharper twist still, and it's the exact seam where this session's second paper picks up. Overtraining — pushing D/N far past 20:1 to shrink inference cost — looks purely beneficial from everything covered so far: smaller model, cheaper to serve, and (per Chapter 1's loss formula) lower pretraining loss too, since more data always helps loss, holding N fixed. But a model destined for deployment almost never gets served at the same precision it was trained in — it typically gets quantized afterward, compressed to fewer bits per weight, specifically to make that inference even cheaper. The precision-scaling paper this session turns to next found something the Chinchilla paper had no reason to anticipate: the amount a model's quality degrades when quantized after training grows with how much data it saw during pretraining — so aggressively overtrained models, exactly the ones this chapter just showed are individually rational to build, can be the most fragile once compressed for serving. A pretraining choice that looks strictly good under Chapters 0–6's lens can carry a real, hidden cost that only shows up once precision enters the picture — which is exactly where Chapter 7 begins.

A preview number: how much MORE fragile does 100× overtraining make a model?

Chapter 7 will introduce the full formula behind Caveat 4's finding, but one piece of it can be previewed here with nothing more than a ratio — the same way Chapter 0 previewed “20 tokens per parameter” long before Chapter 4 derived it. The precision-scaling paper fits post-training quantization degradation as proportional to DγD, with a fitted exponent γD ≈ 0.51 (Chapter 7 introduces the rest of the formula in full). Holding model size N and the final serving precision fixed, every other factor in the degradation formula cancels between two models of the same size, leaving a clean ratio:

Apply this to the two data budgets already on the table for an 8-billion-parameter model: a Chinchilla-optimal D1 = 160 billion tokens (20 tokens per parameter), versus Llama-3-8B's actual D2 = 16 trillion tokens (2,000 tokens per parameter) — a 100× larger data budget for the identical model size.

Training the same 8-billion-parameter model on 100× more data — exactly the overtraining move Caveat 2 showed was individually the right call, for inference-cost reasons alone — makes that model roughly ten times more fragile to post-training quantization than its Chinchilla-optimal counterpart would have been, at the identical post-quantization serving precision. The multiplier scales with the overtraining ratio itself, not with any particular model:

This is the sharpest version of Caveat 4 this session can state before Chapter 7 builds the machinery properly: the same decision that Caveat 2 showed was a clean win for training-plus-serving cost, in isolation, is also, simultaneously, an order-of-magnitude increase in how much quality that model stands to lose the moment it gets compressed for deployment.

What the Chinchilla paper itself admits it doesn't yet know

It's worth reading the paper's own discussion section directly, because it is more cautious about its own result than the “20 tokens per parameter” sound bite that escaped into the field ever became. Three admissions stand out, each a genuine limit on how far to trust the specific numbers this session has spent five chapters deriving.

Only two data points exist at real scale. Every fitted curve in Chapters 1–4 rests on hundreds of small-to-medium runs (70 million to 16 billion parameters) — but at the scale that actually matters for a frontier lab's decision, there are exactly two directly comparable training runs: Gopher and Chinchilla itself. The paper's own words: “we only have two comparable training runs at large scale (Chinchilla and Gopher), and we do not have additional tests at intermediate scales.” Every one of this session's closed-form predictions above 16B parameters is an extrapolation, not an interpolation — trustworthy because the underlying functional form has behaved well everywhere it has been tested, but never directly verified at, say, 200B or 500B parameters.

The paper suspects it may still be overestimating. This is the sharpest admission, and it connects directly back to the curvature Chapter 2 flagged and Approach 3's smaller prediction in Chapter 4's cross-check table: “we observe some concavity in log Nopt at high compute budgets… This suggests that we may still be overestimating the optimal size of large models.” Read that carefully: even Chinchilla's own 70B parameters, the model that beat Gopher on 51 of 57 MMLU tasks, might still have been bigger than truly optimal for its compute budget. The gap between Approach 3's 40B and Approaches 1/2's 63–67B isn't just measurement noise between three equally-trustworthy methods — by the paper's own account, it may be a genuine signal pointing toward smaller-than-Chinchilla being the real answer at very high compute, one the paper's own three approaches disagree about by a wide enough margin to flag explicitly.

Every run trained on less than one epoch. No model in this entire 400-run study repeated any training data — every token was seen exactly once. The paper flags the multi-epoch regime (reusing data more than once, which later became unavoidable once Caveat 3's data wall started to bind in practice) as explicitly outside what this derivation covers: “future work may consider the multiple epoch regime.” Every number in Chapters 1–5 assumes fresh, never-repeated tokens; nothing here predicts what happens once a lab starts training on the same trillion tokens two or three times over — because it's the simplest case to assume, this session did too, throughout.

None of these three admissions overturn the paper's central finding — parameters and data should scale together, not parameters alone. But they are the honest boundary of what “400 training runs and three independent methods” actually proves, stated in the paper's own words rather than left for a reader to discover the hard way.

Between these two pressures — inference cost pushing toward smaller, more-overtrained models, and a shrinking supply of fresh high-quality text pushing the field toward reusing (or synthesizing) data rather than endlessly growing D past what's compute-optimal — the clean 20:1 rule from Chapter 4 turned out to be less a permanent law and more a snapshot: exactly correct for the narrow question it was built to answer, and a genuine starting point for every harder, more realistic question the field asked next.

Chapter 7: Precision as a Third Axis

Everything through Chapter 6 treated a parameter as a parameter — one unit of model capacity, however many bits it happened to be stored in. In practice it was always stored in some precision, and that choice was never free. Training in 16-bit precision instead of 32-bit roughly halves memory and, on hardware that supports it, roughly halves compute per step too. The frontier line, in other words, has always had a hidden third axis running through it: P, the number of bits used to represent weights, activations, and the attention cache during training and at inference.

This is the subject of Scaling Laws for Precision, a 2024 paper from a nine-author team spanning Harvard, Stanford, MIT, Databricks, and Carnegie Mellon (Kumar, Ankner, Spector, and six further collaborators). It asks the natural follow-up question Chapter 6 leaves open: given that precision costs compute just like parameters and data do, how should a fixed compute budget be split across all three axes — N, D, and P — not just two?

Where precision actually lives inside a transformer

Three separate things get quantized, and they behave differently enough to be worth naming individually. The weights, the model's actual learned parameters. The activations, the intermediate values flowing through matrix multiplications during the forward and backward pass. And the KV cache — you've already met this exact object, at length, in Session 8's derivation of its per-token byte cost; quantizing it is a direct, precision-axis answer to the same memory pressure that session spent an entire lesson diagnosing.

There are two structurally different reasons to quantize. Quantization-aware training keeps weights in low precision throughout training but leaves activations and matrix-multiply hardware in high precision — it saves no training compute at all (the actual multiplications still run at full precision), but it lets weights adapt to living at low precision, so the model can be served cheaply at inference time afterward. Low-precision training goes further, quantizing weights, activations, and the KV cache together, which does save real training compute — because modern accelerators require every input to a matrix multiplication to share the same precision, so only when all three match can the hardware actually run the multiply faster.

First, the simpler question: what happens if you quantize after training?

Before building a scaling law for training in low precision, the paper studies the more common industry practice first: train an ordinary BF16 model, then post-train quantize (PTQ) it down to a lower precision purely for serving, using an off-the-shelf method (GPTQ). Chapter 6's Caveat 4 already previewed the punchline; here is the actual finding, with its shape made explicit. Across models of different sizes trained on different data budgets, then quantized to various post-training precisions, the loss degradation introduced by quantization — the gap between quantized loss and the loss right before quantization — fits:

Read the three pieces. The exponential term says degradation grows sharply as the post-quantization precision Ppost drops — unsurprising, coarser rounding hurts more. The fraction DγD/NγN is the genuinely surprising part: degradation increases with how much data the model was trained on, and decreases with how many parameters it has — in other words, it grows with the tokens-per-parameter ratio D/N this entire session has been optimizing. The fitted constants (γD ≈ 0.51, γN ≈ 0.34, similar in magnitude, which is why the paper describes this as an approximate power law in the ratio D/N itself) came from fitting this exact functional form against real degradation measurements, achieving R2=0.97 — a very tight fit for a real-world measurement of model behavior.

The intuition the paper offers: as a model trains on more and more data, it compresses more information into the same fixed number of weights, packing them more tightly and precisely. A model whose weights are already using their full numerical range efficiently has less slack to absorb the rounding error quantization introduces — the same reason a fully-packed suitcase is harder to close than a half-empty one. This critical-dataset-size point is exactly why Chapter 6 flagged overtraining as carrying a hidden cost: the very strategy that makes a model cheaper to serve (shrink N, grow D) is, by this finding, also the strategy that makes it more fragile once that serving step includes quantization — two forces pulling in opposite directions on the same knob.

Effective parameter count: a saturating exponential

The paper's central move is to model the effect of low-precision weights not as a separate loss term, but as a shrinkage of the effective parameter count itself — a model trained with N real parameters at weight-precision Pw behaves, loss-wise, like a smaller model trained at full precision. Call this shrunk count Neff:

Read this shape carefully, because it recurs everywhere in this chapter. At Pw → ∞ (arbitrarily high precision), the exponential term vanishes and Neff → N — full effective capacity, exactly as expected. At Pw → 0, the exponential term approaches 1 and Neff → 0 — a model quantized to nothing has no effective capacity left, no matter how many nominal parameters it has. In between, the function saturates — each additional bit adds less than the one before it, because you're always closing a shrinking fraction of the remaining gap to full precision. The constant γw controls how fast that saturation happens — a smaller γ means the part of the model is more sensitive to losing bits, saturating to full effective capacity at a lower bit count.

Substituting N &mapsto Neff(N, Pw) into the ordinary Chinchilla loss form gives the paper's core scaling law for quantization-aware training:

This is, structurally, exactly Chapter 3's parametric loss, with one substitution. Nothing about the N-term or D-term changed shape — only what gets plugged in for N did. That reuse of the exact same functional skeleton, verified independently on a completely different model family and a completely different dataset, is itself a striking finding: the Chinchilla form isn't a fact about one paper's specific runs, it's a fact about how transformer loss behaves in general.

Three parts, one multiplicative law

Weights, activations, and the KV cache each get their own saturating-exponential term, with their own fitted sensitivity constant, and the paper finds the three combine multiplicatively, as though their effects on effective capacity were statistically independent:

Read the sign of γ in the exponent carefully before trusting an intuition about what a “small” or “large” value means: since Neff/N = 1−e−P/γ, a smaller γ makes the exponent P/γ larger for the same P, which pushes e−P/γ closer to zero and Neff/N closer to 1 — a smaller γ means the component reaches full effective capacity using fewer bits, not more:

Plugging in numbers makes this concrete and checks the intuition above. At P=4 bits, weights sit at only 77.6% effective capacity — still a long way from full — while the KV cache is already at 98.5%. At P=3, weights are down to 67.4%, while the KV cache is still at 95.6%. Counter to what the raw memory-liability story from Session 8 might suggest, the KV cache is, by this metric, the most forgiving part of the model to compress — it reaches nearly its full effective capacity at bit-widths where the weights are still leaving significant capacity on the table. The two sessions are asking genuinely different questions about the same tensor, and the answers don't transfer: Session 8 asked “what happens if you remove a cached entry entirely,” and found the model breaks catastrophically the moment even one specific entry (the attention sink) is evicted — a question about which positions are kept. This chapter asks “what happens if every kept entry is stored less precisely,” a question about numerical fidelity, and finds the cache tolerates that kind of degradation unusually well. Being irreplaceable if deleted and being robust to rounding are not the same property, and a component can genuinely have both at once.

python
import math

gamma = {'w': 2.6745, 'a': 2.2102, 'kv': 0.9578}

def n_eff(N, P_w, P_a, P_kv):
    factor = 1.0
    for part, P in [('w', P_w), ('a', P_a), ('kv', P_kv)]:
        factor *= (1 - math.exp(-P / gamma[part]))
    return N * factor

N = 70e9
print(n_eff(N, 16, 16, 16) / N)   # ~0.997 -- BF16 everywhere, almost no shrinkage
print(n_eff(N, 8, 8, 8) / N)      # ~0.924 -- FP8 everywhere; of the three, weights (94.98%
                                       # alone at P=8) are contributing the most shrinkage here
print(n_eff(N, 4, 16, 16) / N)    # ~0.775 -- ONLY the weights dropped to 4-bit, everything
                                       # else full precision -- capacity collapses almost as much
                                       # as it would with every part quantized together
print(n_eff(N, 16, 16, 4) / N)   # ~0.981 -- ONLY the kv-cache dropped to 4-bit -- barely moves

Compare the last two lines directly: dropping only the weights to 4-bit costs almost as much effective capacity as dropping every component together, while dropping only the KV cache to 4-bit costs almost nothing. The product is dominated by whichever factor is smallest, and because the weights curve is the slowest to saturate (largest γ), it's the weights — not the cache — that are the bottleneck once precision gets aggressive. If a fixed bit-budget has to be spent unevenly across components, this is exactly the kind of calculation that tells you where spending it buys the most effective capacity back: protect the weights first.

Worked numbers: what does BF16 actually buy you, versus 8-bit or 4-bit?

Take the weights curve alone (γw = 2.6745) and evaluate Neff/N = 1 − e−P/γw at a few precisions used in real deployments:

This table is the numeric backbone of an important, non-obvious claim from Section 4.1 of the paper: the gains from adding bits are large at low precision and saturate around 6–7 bits. Look at the jump from 3-bit to 4-bit — effective capacity rises about 10 percentage points. The jump from 8-bit to 16-bit, doubling the storage cost, buys less than 5 points. Training in BF16 (16-bit) is, by this measure, spending real compute and memory on bits that the model can barely make use of — a claim Chapter 8 turns into a precise, quantitative compute-optimal recommendation.

Chapter 8: Compute-Optimal Precision (showcase)

Chapter 4 solved minN,D L(N,D) subject to 6ND=C. This chapter asks the full three-variable version of the same question: given that compute cost scales linearly in precision too (halving precision roughly halves the FLOPs a training step costs, on hardware that supports it), what is minN,D,P L(N,D,P) subject to a compute constraint that now includes precision explicitly?

Regime 1: precision fixed, only N and D optimized — trade parameters for bits

Hold P fixed at whatever precision you've committed to, and ask how the compute-optimal N and D shift relative to ordinary (full-precision) Chinchilla-optimal values, NCh and DCh, at the same compute budget:

where γ̄ is the average of the three sensitivity constants from Chapter 7. Read the sign of the first exponent: as P drops, the bracket [1−e−P/γ̄] shrinks, and raising a shrinking number to a negative power makes N⋆ grow. Lower precision, at fixed compute, means the compute-optimal choice is more parameters and less data — the opposite direction from what intuition about “wasted compute at low precision” might first suggest, and exactly what the paper states as its headline pretraining recommendation: “if you must train in low precision, increase parameters before data.”

Worked example: exactly how much more, going from 16-bit to 4-bit

Using the fitted constants from Chapter 7 (γw=2.6745, γa=2.2102, γkv=0.9578, so γ̄ = (2.6745+2.2102+0.9578)/3 ≈ 1.9475) and this paper's fitted α = β = 0.4965 (the authors deliberately tie these two exponents together for stability, finding it barely changes the fit), the bracket exponent simplifies cleanly since 3α/(α+β) = 3×0.5 = 1.5 when α=β:

At the same total compute, dropping from 16-bit to 4-bit training means the compute-optimal choice is to train a model with roughly 2.46× more parameters on roughly 41% as much data. This is Chapter 6's overtraining logic running in reverse: there, cheaper inference justified overshooting past the training-optimal token count; here, cheaper-per-FLOP low-precision compute justifies overshooting past the full-precision-optimal parameter count.

python
import math

gamma_bar = (2.6745 + 2.2102 + 0.9578) / 3   # 1.9475
alpha = beta = 0.4965                                # tied, per the paper's fit

def u(P): return 1 - math.exp(-P / gamma_bar)

def N_ratio(P1, P2):
    """N*(P1) / N*(P2) at fixed compute."""
    return (u(P2)/u(P1))**1.5 * (P2/P1)**0.5

print(N_ratio(4, 16))    # 2.456 -- matches the by-hand result
print(N_ratio(3, 16))    # 3.315 -- pushing to 3-bit costs even more parameters
print(N_ratio(3, 7))     # 2.103 -- relative to the fitted 7-bit optimum, not 16-bit

Relative to 16-bit, 3-bit training costs a 3.3× parameter premium. Relative to the compute-optimal 7-bit point Regime 2 derives next, that same 3-bit precision costs a smaller but still substantial 2.1× premium. Both numbers are correct, honest answers to slightly different questions — “compared to the common default” versus “compared to the actual optimum.” Reading a headline ratio without checking which baseline it's measured against is the single easiest way to misquote a scaling-law comparison.

Regime 2: jointly optimize N, D, and P together — the surprising independence result

Now let P float too, jointly minimized alongside N and D at fixed total compute. Setting all three partial derivatives to zero and combining them algebraically eliminates C entirely from the equation that determines the optimal precision — meaning the compute-optimal precision P⋆, under this joint optimization, does not depend on how much compute you have. Whether the budget is small or enormous, the same precision comes out compute-optimal; only N and D should grow as compute grows, exactly as in ordinary two-variable Chinchilla scaling, with P held fixed at that one value throughout.

Fitting this condition against the paper's own 465 training runs gives a concrete answer:

It's worth building intuition for why an interior optimum has to exist here at all, without working through the full implicit equation by hand. Two forces pull in opposite directions as P changes, at fixed compute. Push P down: the [1−e−P/γ̄] bracket shrinks, so Neff shrinks for any fixed N — a real loss, unless N itself grows to compensate (which Regime 1 already showed is compute-optimal to do, up to a point). Push P up instead, toward 16 or 32 bits: each additional bit buys steadily less Neff, per Chapter 7's saturating curve, while every one of those bits still costs the same linear slice of the compute budget — compute that could otherwise have gone toward more tokens or more parameters directly. Too low a P wastes compute on a model whose effective capacity is artificially capped; too high a P wastes compute on marginal bits the saturating curve says barely matter. Somewhere between those two failure modes is a floor, and because both the saturation curve's shape (γ̄) and the loss exponents (α, β) are properties of the architecture and data, not of any particular compute budget, that floor doesn't move as C grows — which is the entire content of Regime 2's independence result.

This single number carries two separate, practically important consequences. First: the field's default of training in 16-bit (BF16) is, by this analysis, spending compute on roughly 2.3× more bits than compute-optimal — consistent with Chapter 7's saturation table, where the jump from 8 bits to 16 bits bought less than 5 percentage points of effective capacity. Second, and more of a warning to the industry's race toward ever-lower precision: pushing training precision below 4 bits requires disproportionately larger models to hold the loss-scaling trend, per Regime 1's N⋆ formula — the “more parameters, less data” trade stops being a mild adjustment and starts requiring model sizes that grow faster than the compute savings from fewer bits can pay for.

Regime 3: the one case where the answer flips — model size fixed in advance

Regimes 1 and 2 both assumed N is free to be chosen. That's not always true. A lab releasing a model family at fixed sizes — Llama-3's 8B/70B/405B, or Gemma-2's fixed size tiers — commits to N in advance, for reasons having nothing to do with this derivation (product tiers, hardware targets, a promised roadmap). In that case, only D and P are free to jointly optimize at fixed compute, and the paper's derivation gives a qualitatively different answer:

When N is pinned, compute-optimal precision grows, slowly (logarithmically), as more compute (and therefore more data, at fixed N) is spent — the opposite conclusion from Regime 2. The intuition: as a fixed-size model gets pushed further and further into the overtraining regime (Chapter 6's territory — huge D/N), precision becomes the only remaining lever available to bring that already-committed model size closer to its own optimal effective-capacity point, since raising N itself is off the table by assumption. Higher precision, not lower, is what a heavily overtrained fixed-size model family needs to stay closer to compute-optimal as its training-data budget keeps growing.

It's worth pausing on the phrase “forward and backward pass computed in bfloat16, with a float32 copy of the weights kept in the optimizer state” — part of Chinchilla's own training setup, described in Chapter 5 — because it's a live instance of the mixed-precision pattern this chapter has been building a scaling law for, two years before this second paper existed to formalize it. Chinchilla itself already wasn't trained in one uniform precision throughout: the matrix multiplications that dominate compute ran in bfloat16 (16-bit), while a master copy of the weights, used only for the optimizer's update step, stayed in float32 (32-bit) to avoid compounding rounding error across millions of small gradient updates. In this session's vocabulary, that's exactly Ptrain=16 for the compute-heavy forward/backward arithmetic, paired with a higher-precision shadow copy for accumulation — a detail every frontier lab already knew mattered in practice, well before anyone had fit a scaling law explaining how much it mattered or why 16 bits, rather than 8 or 32, was the right choice to make it in.

The logarithm is doing real, load-bearing work in that formula, and it's worth feeling what it implies numerically. A logarithmic dependence means precision has to climb only slowly as overtraining gets much more extreme — going from Chinchilla-optimal (D/N ≈ 20) to Llama-3-8B's actual ratio (D/N ≈ 2,000, a 100× increase in the overtraining ratio, per Chapter 6) only shifts log(D/N) by log(100) ≈ 4.6 nats — a modest additive bump in recommended precision, not a 100× multiplicative one. This is exactly the shape you'd want from a “how much should I compensate” correction: a model overtrained two orders of magnitude past Chinchilla-optimal doesn't need a proportionally enormous precision increase to stay near its own optimum, just a modest, bounded one — which is a genuinely reassuring result for anyone maintaining a heavily overtrained production model family.

Chapter 9: Course Synthesis

Twenty-one sessions asked, in one form or another, how to build a better piece of an LLM system — a better tokenizer, a better attention mechanism, a better alignment objective, a better agent harness. This session asked a different kind of question: given a fixed amount of compute, what should you even be trying to build more of? Every other session's answer lives somewhere inside the (N, D, P) space this one just mapped.

Where this session sits relative to the rest of the course

Session 1 opened the course with pretraining data and tokenizers — the raw material that fills the D axis this session spent four chapters optimizing against N. Session 4's Mixture-of-Experts architecture is, read through this session's lens, a way of decoupling N (total parameters) from the compute cost of using them — a third knob Chinchilla's dense-model derivation didn't have available. Sessions 9 and 4 together (Latent Attention, Mixture-of-Experts) are both, structurally, bets that the compute-optimal frontier looks different once architecture stops being held fixed — the same frontier this session derived for one specific dense-transformer family. The Chinchilla paper's own closing section is explicit that this raw material isn't interchangeable by token count alone: “we expect that scaling to larger and larger datasets is only beneficial when the data is high-quality,” and it flags that “larger datasets will require extra care to ensure train-test set overlap is properly accounted for” — a caution this session's own Dopt numbers make sharply concrete. Table 3's 216.2-trillion-token figure at 10-trillion-parameter scale assumes that much high-quality, correctly-deduplicated text actually exists to train on; sourcing and verifying it is a Session 1 problem this session's scaling law simply assumes is already solved.

Session 8's attention sinks and the KV-cache byte arithmetic it built from scratch is the exact object Chapter 7 returned to under a second, completely different lens — and found the opposite of what a naive reading of Session 8 might predict. Session 8 showed the cache is catastrophically fragile to eviction: remove the wrong token and perplexity jumps a thousandfold. Chapter 7 showed the same cache is unusually robust to quantization: its fitted γkv=0.96, smallest of the three model components, means it reaches near-full effective capacity at bit-widths where the weights (γw=2.67) still have a long way to climb. Two sessions, five weeks apart in the syllabus, studying the same tensor under two different kinds of degradation, and finding it behaves in opposite ways under each — a reminder that “how load-bearing is this component” is not one property but many, and the answer depends entirely on how you propose to degrade it.

Sessions 10–15 (GRPO/DAPO, compound systems, code world models, self-play, test-time scaling) all spend compute at inference time rather than training time — extra forward passes, extra reasoning tokens, extra rollouts. None of that compute is priced into the C ≈ 6ND constraint this session built its entire derivation on. Chapter 6's inference-aware overtraining logic is the seam where pretraining-scaling (this session) and inference-time-scaling (Session 15) meet: both are, at bottom, questions about where a finite compute budget buys the most loss reduction, just measured at different points in a model's lifecycle.

Session 16's mode collapse and Session 19's safety work both study what happens to a trained model's behavior — but Chapter 6 of this session already showed that the training recipe itself (how far past Chinchilla-optimal a model is overtrained, what precision it's stored in) changes what that trained model even is, before any alignment or safety intervention touches it. A model's point in (N,D,P)-space is upstream of nearly everything downstream sessions study.

Sessions 5 and 6 (Bayesian Reasoning, DPO & Curiosity) sit slightly to the side of this session's compute accounting — alignment and reasoning objectives are typically applied after pretraining, on a model whose (N, D, P) point is already fixed. But the order of operations matters: Chapter 6 showed that how overtrained a base model is (and, per this chapter's PTQ finding, how it will later be quantized) changes how much room a subsequent DPO or RL pass even has to work with. A base model's scaling-law coordinates aren't erased by whatever training happens next; they're the starting conditions everything downstream inherits.

The four sessions immediately before this one round out the picture further. Session 17's linear transformers and Session 18's diffusion language models are both, in this session's vocabulary, proposals to change the architecture term hidden inside C≈6ND — a linear-attention model's forward pass doesn't cost the same multiple of N per token that a dense-attention transformer's does, so every one of this session's closed-form optima would need to be refit, not merely reused, for either family. Session 20's mechanistic interpretability work (sparse autoencoders, crosscoders) studies what a trained model's weights actually encode — a question this session's scaling laws are silent on by design: L(N,D) predicts how low the loss gets, never what the model represents to get there. And Session 21's calibration work — getting a model's stated confidence to match its actual accuracy — is worth naming explicitly as a question neither this session's papers, nor any pretraining-loss scaling law, can answer: two models with identical L(N,D) can be calibrated very differently, because calibration is a property of the output distribution's shape, not of the scalar loss value averaged over it.

Three years, three answers to “what should a fixed compute budget buy?”

Laid end to end, this session's two papers are chapters in a single, still-unfinished argument about how to spend a training budget. It's worth seeing that argument as a timeline, not just as two separate results.

Notice the pattern in the rightmost column: every correction came from questioning an assumption the previous answer had baked in silently — Kaplan's fixed schedule, Chinchilla's implicit assumption that a parameter is a parameter regardless of the bits it lives in. There is no reason to expect 2024's answer is the last word either; the precision paper's own honest-limits admissions (see “What this session cannot tell you,” below) already flag directions — architectures beyond dense transformers, downstream task performance rather than pretraining loss alone — that the next correction, whenever it arrives, will likely come from questioning.

One last worked number, tying both papers into a single sentence

As a closing exercise, chain together a fact from each half of this session into one figure. Chapter 5 showed Chinchilla training on 1.4T tokens instead of Gopher's 300B bought a compute-matched model that wins on essentially every benchmark. Chapter 7 showed that same extra data, if this model were later heavily post-train quantized, would make it more susceptible to PTQ degradation than a less-thoroughly-trained model would be (Finding 1's DγD term grows with tokens seen). Put together: the exact same training decision — more data, fewer parameters — that made Chinchilla the better unquantized model is, by this session's second paper, also the decision that would make it the more fragile model to compress afterward. Neither paper is wrong; they're answering different questions about the same trained artifact, at different points in its lifecycle, and only by holding both scaling laws in mind at once do you see the full tradeoff a real deployment decision has to weigh.

One synthesis the course hasn't stated yet: the overtraining multiplier is scale-free

Chapter 6 computed one specific number: training an 8-billion-parameter model to Llama-3-8B's actual ratio (2,000 tokens per parameter) costs the same training compute that a training-compute-optimal 80-billion-parameter model would have used — a 10× jump. It's worth checking whether that 10× is a coincidence of Llama-3-8B's specific size, or something that shows up at any scale. Take Chapter 4's own “academic-scale” worked example — a 2.49-billion-parameter model, compute-optimal at 133.6 billion tokens — and push it to the identical 2,000-tokens-per-parameter ratio instead:

Now ask Caveat 1's question at this smaller scale: what size model would that same 7.44×1022 FLOPs have bought, training-compute-optimally, using the 20-tokens-per-parameter rule (D=20N, so C=6N(20N)=120N2)?

10.0× larger than the actual 2.49B model — the identical multiplier Chapter 6 found for Llama-3-8B, at a starting size almost 3,000× smaller. That isn't a coincidence; it falls straight out of the algebra. Whenever a model of size N is trained to a token/parameter ratio r=D/N instead of the Chinchilla-optimal 20, the training-compute-optimal-equivalent size scales as:

The N cancels completely. The training-compute-optimal-equivalent multiplier depends only on the ratio of how far past Chinchilla-optimal a model was pushed — never on the model's actual size.

This is the shape this session keeps promising: not a number to memorize, but a relationship that transfers. A lab building an 8B model and a lab building an 800B model, both choosing to overtrain by the same factor past Chinchilla-optimal, pay the identical proportional training-compute premium for it — the entire calculation this session built, chapter by chapter, reduces to one square root, applicable at any scale that hasn't been trained yet.

The single idea underneath both papers this session covered

Chinchilla and the precision-scaling paper look, on the surface, like they're about different things — one about parameter/data allocation, one about bit-width. They share a method, almost exactly: propose a small, interpretable functional form for how loss depends on some resource, fit its handful of constants against hundreds of real training runs, and then minimize that closed-form function analytically to answer questions no single training run could answer directly — what happens at a compute budget ten thousand times larger than anything actually trained. Both papers are explicit that their fitted numbers are not the point; a different dataset or architecture will fit different constants (the precision paper's own A, B, E, α, β differ substantially from Chinchilla's, because it trained OLMo-style models on Dolma rather than DeepMind's MassiveText). What transfers is the shape — a saturating term for the resource that's scarce, an additive irreducible floor, exponents fit once and reused everywhere.

The same finding, replicated under two more quantization methods

One more piece of evidence for trusting Finding 1: the precision paper doesn't stop at GPTQ, the one post-training quantization method used everywhere else in this session. In an appendix, the authors rerun the exact same experiment — train models across a range of token/parameter ratios, then post-train quantize and measure degradation — using two structurally different techniques instead: AWQ, a modern, activation-aware method, and round-to-nearest, the simplest possible baseline with no sophistication at all. Both replicate the same DγD/NγN growth-with-overtraining pattern, with different fitted constants but the identical qualitative shape.

This is Chapter 2's cross-method-agreement argument, run a third time, on a completely different question. A finding that survives being tested with a sophisticated method, a naive method, and (in the main results) a mid-complexity method, all landing on the same functional shape, is much harder to dismiss as an artifact of one particular implementation choice — exactly the same epistemic move that made Chapter 2's agreement with Chapter 1 convincing rather than coincidental. The paper states this directly: this pattern “should be the default expectation for any newly proposed PTQ technique,” not a quirk of the one method this session happened to lead with.

What this session cannot tell you

In the same spirit as the honest-limits chapter every session in this course has carried: neither paper studied downstream task performance directly through the precision lens (only pretraining loss), neither paper covers architectures radically different from dense or lightly-modified transformers, and both explicitly warn that their numerical constants are specific to their own training setups — a claim this lesson's own worked derivations (Chapter 3's toy fit, Chapter 8's 2.46× ratio) depended on taking at face value. The precision paper's own conclusion adds two more admissions worth carrying forward: it fixes architecture throughout to isolate precision's effect cleanly, even though real low-precision training in practice often pairs with architectural tweaks that close much of the gap it measures — meaning this session's 2.46× and 7-bit numbers describe a controlled comparison, not necessarily the best a lab willing to change more than precision could do. And Chapter 8's clean C∝N·D·P constraint is an idealization the paper itself flags: halving precision rarely buys a full 2× speedup in practice, because real hardware and software carry systems overhead the linear compute model doesn't account for — every ratio this session computed from that constraint is directionally right and numerically optimistic. The functional forms are the durable contribution. Treat any specific number quoted from either paper — 20 tokens per parameter, 7 bits, 2.46× — as a snapshot computed from one particular fit, not a constant of nature.

“All models are wrong, but some are useful.” — George Box

Where to go next

• CS229S: FlashAttention and CS229S: Sparsity & Quantization — the systems-level mechanics that make training at any given precision actually run fast on real hardware.
• CS336: Scaling Laws Basics and CS336: Scaling Laws Details — a from-scratch, implementation-first companion to this session's derivations.
• TinyML: Quantization I and TinyML: Quantization II — the full quantization toolkit (per-channel scales, calibration, integer vs. floating-point schemes) that Chapter 7 only sketched.
• Session 08: Attention Sinks & Streaming — the KV-cache byte arithmetic and eviction-fragility that Chapter 7's quantization-robustness finding (γ_kv=0.96) stands in contrast to.
• Session 04: Mixture-of-Experts — a fourth axis (routed vs. dense capacity) this session's derivation held fixed throughout.
• Session 15: Test-Time Scaling — the inference-time compute axis Chapter 9 flagged as outside this session's C≈6ND constraint entirely.

Or step back further: the Efficient Deep Learning path carries the pruning, quantization, distillation, and deployment threads this session's precision half only opened, and the Quantization deep-dive article (in the main LLM Deep-Dive Series) covers integer and floating-point quantization schemes at implementation depth — the mechanics behind the bits this session spent two chapters putting a scaling law on top of.