The Economics of Inference

Chapter 1: Token Economics

Chapter 0 handed you a price — $0.50 and $1.50 per million tokens — and asked you to trust it. That number came from somewhere. If you're renting GPUs and running the model yourself instead of paying a hosted API, nobody hands you a price per million tokens; you have to derive your own, from two numbers you actually control: what the GPU costs per hour, and how many tokens per second it can produce. This chapter builds that derivation from the ground up.

Why an API price and a derived self-host price are directly comparable at all

It's worth pausing on why it's even fair to put chapter 0's API price and this chapter's derived self-host price side by side, given they come from completely different sources — one a published rate card, the other an arithmetic derivation. Both describe the same underlying thing: the cost of turning one million tokens of model input or output into a billed dollar amount. The API's version bundles in the provider's own hardware cost, margin, and operational overhead behind an opaque number; this chapter's version exposes every one of those layers except margin, which is exactly what makes the two comparable — and what makes the gap between them, once computed, meaningful rather than an apples-to-oranges mismatch.

The formula, before the arithmetic

A GPU rented by the hour is a rate, dollars per hour. A serving system produces tokens at some rate too, tokens per second. Divide one rate by the other and the hours cancel, leaving dollars per token — the exact currency chapter 0's price table was already speaking in.

Every quantity on the right side of that line is something you can look up on a cloud pricing page or measure with a stopwatch on your own serving stack. Nothing on the right side is a guess.

Why per-hour pricing, and not a purchase price

GPUs can be bought outright instead of rented — and large, sustained deployments sometimes do exactly that, since a purchased card's per-hour cost falls once the purchase price is amortized past its rental- equivalent break-even. This chapter rents by the hour deliberately: renting is the more conservative, more broadly applicable assumption (available to a team of any size, with no capital outlay or hardware-lifecycle risk), and every dollar figure downstream is easy to rescale if your own team's hardware situation is different — substitute your own effective $/GPU-hour, amortized however your finance team amortizes it, and the rest of the derivation is unchanged.

Pricing the hardware

Take a concrete, realistic setup: a 70-billion-parameter model — a common production size class — served on two H100 GPUs. On-demand cloud pricing for an H100 typically lands somewhere in the $2–4 per GPU-hour range depending on provider and region; use the low end of that range for a clean number.

A note on what “$4/hour” is actually paying for

It's worth being explicit that this $4/hour is the whole instance — compute, memory, and the networking around it — not a per-GPU add-on price for something else. Cloud GPU pricing sometimes bundles in storage or egress costs separately; this chapter's number assumes those are small relative to the compute cost for a serving workload (mostly reads from local weights, not heavy network transfer), which is realistic for the vast majority of LLM serving setups, though worth checking against your own provider's actual line-item breakdown before trusting the number for a real budget.

Pricing the throughput — the part that actually takes work

The GPU-hour price is the easy half. Tokens per second depends on the model's size, the GPU's memory bandwidth, and how many requests are batched together — and it has to be derived, not guessed, because guessing it wrong by 2× means guessing the entire cost model wrong by 2×.

Each H100 moves data between its memory and its compute units at roughly 3.35 terabytes per second — its HBM bandwidth, the same ceiling that governs the memory-bound decode step from the serving-engines lesson. Two H100s working together give:

The 70B model, stored in half precision (2 bytes per parameter, standard for serving), weighs:

Decoding one token means reading every one of those weight bytes once — that's the memory-bound mechanism the serving-engines lesson derives in full. The time that read takes is the floor on how fast a single decoding step can possibly run, no matter how many requests ride along in that step (up to the point where compute becomes the bottleneck instead — chapter 2 draws that line precisely):

From step time to tokens per second, at a realistic batch size

A step that takes 20.9 ms and serves a batch of B simultaneous requests produces B tokens — one per request — in that same 20.9 ms, as long as the batch is still small enough that the GPU remains memory-bound rather than compute-bound (chapter 3 finds exactly where that line sits; for now, trust that a batch of 64 concurrent requests — a modest, realistic number for a busy endpoint — is comfortably inside it).

Read that number twice: 3,000 tokens per second is not any single user's speed — a single user still gets roughly 1/0.0209≈48 tokens/second, the batch-of-1 ceiling. It's the GPU's total output, summed across all 64 users riding the same memory-bound step for free. Batching is what turns a slow per-user ceiling into a fast aggregate throughput number, and it's the reason the dollar math in this chapter looks nothing like the dollar math you'd get by pricing a single lonely conversation.

The number nobody puts on the pricing page: uptime utilization

There's one more factor before this becomes a real cost number. A rented GPU instance is billed for every hour it exists, whether or not a request happens to be running that second. Real traffic is spiky — gaps between requests, quiet overnight hours, autoscaling lag — so the GPU sits idle some fraction of the time it's being paid for. Call that fraction the uptime utilization, and use a realistic value of 70% for a well-run but not perfectly-tuned service (chapter 8's dashboard lets you dial this directly).

This is a genuinely different knob from the memory-bound-vs-compute-bound ceiling derived above — that one is architecture and hardware, fixed once you pick a model and a GPU. Uptime utilization is operational: it's how well the autoscaling and traffic-shaping around the GPU are tuned, and it's one of the biggest levers in this entire lesson precisely because it costs nothing to improve except engineering attention (a theme chapter 5 returns to explicitly).

A brief pause on units, before the final division

Every quantity assembled so far is in a different native unit — hours, seconds, tokens, bytes. The final formula's job is entirely bookkeeping: convert everything to a common footing (seconds and tokens) and divide. Getting this step wrong — mixing hours and seconds without converting — is the single most common way a back-of-envelope hardware calculation goes wrong by a fixed multiple, which is exactly why the earlier cross-check in a second unit system exists later in this chapter.

Putting it together — the derived $/M-token

That $0.53 is the self-hosted cost of a decode (output) token, on this specific 2×H100, 70B, batch-64, 70%-utilized setup. Compare it to chapter 0's API output price of $1.50/M — nearly 3× higher. That gap is exactly what chapter 4 turns into a real API-vs-self-host decision, once the fixed costs of actually running your own fleet are added in honestly.

Cross-checking the derivation in a different unit

A good habit for any back-of-envelope number: rederive it a second way and confirm the two answers agree. Instead of going through tokens-per-second, go through seconds-per-token — the direct reciprocal — and see if the same $0.53 falls out.

Same answer, reached by dividing in the opposite order. This kind of cross-check is cheap insurance against a silent factor-of-1000 error (per-token versus per-thousand-token versus per-million-token units are the easiest place to lose a decimal point in cost modeling, and the mistake doesn't announce itself — it just quietly makes every downstream number wrong by a fixed multiple).

A quick gut check before moving on

Fifty-three cents per million tokens should feel small in isolation and large in aggregate — both are true at once, and this lesson needs both intuitions simultaneously. It's small enough that a single request (roughly 800 tokens) costs a fraction of a cent, which is why nobody notices any individual request's price. It's large enough that chapter 0's 24-billion-token-a-month app turns that fraction of a cent into a five-figure monthly line item. Neither framing is wrong; they're the same number viewed at two different zoom levels, and losing track of either one is how teams end up surprised by their own bill.

The single-user cost, for comparison

Chapter 1's $0.53/M number assumed batch 64. It's worth pricing the other extreme — a single user, no batching at all — to feel how much of that $0.53 is actually “paid for by sharing.” At batch 1, the same 20.9 ms step now produces exactly one token instead of 64:

Thirty-three dollars per million tokens, unbatched, versus fifty-three cents at batch 64 — a 63× difference, and the GPU, the model, and the GPU-hour price never changed. The entire gap is batching: 64 users sharing one 20.9 ms weight read instead of one user paying for it alone. This is the same mechanism the serving-engines lesson calls continuous batching, priced out here in dollars instead of tokens-per-second.

Choosing hardware: the same formula, three GPU classes

Nothing in the derivation is specific to H100s — swap in any GPU's bandwidth and hourly price and the same six lines produce a comparable number. A quick survey, holding batch (64) and uptime (70%) fixed:

The three land within a few cents of each other — each generation's higher price roughly tracks its higher bandwidth, so the derived $/M-token doesn't move much across a hardware refresh at fixed utilization. That's a genuinely useful, slightly counter-intuitive finding: chasing the newest GPU for its raw bandwidth isn't, by itself, a big cost lever here — the batching and uptime knobs from earlier in this chapter move the number far more than a hardware generation does.

python
def dollars_per_million_tokens(gpu_hourly_cost, tokens_per_sec, uptime_util=1.0):
    effective_tps = tokens_per_sec * uptime_util
    tokens_per_hour = effective_tps * 3600
    return (gpu_hourly_cost / tokens_per_hour) * 1_000_000

# 2x H100, 70B fp16, batch 64, 70% uptime utilization
print(dollars_per_million_tokens(4.0, 3061, 0.70))
# 0.529... -- matches the hand derivation above

# the single-user (batch 1) case, for comparison
print(dollars_per_million_tokens(4.0, 47.8, 0.70))
# 33.17... -- 63x more expensive per token than batch 64, same hardware

def decode_step_ms(weight_gb, agg_bandwidth_tbps):
    return (weight_gb / 1024) / agg_bandwidth_tbps * 1000

for name, bw_tbps, price in [('A100', 4.0, 1.50), ('H100', 6.7, 2.00), ('H200', 9.6, 2.80)]:
    step_ms = decode_step_ms(140, bw_tbps)
    tps = 64 / (step_ms / 1000)
    print(name, round(step_ms,1), round(dollars_per_million_tokens(price*2, tps, 0.70),2))

Reading the formula as a story, one more time

Say the whole chapter back in one sentence, because it's the sentence every later derivation in this lesson is a variation of: a GPU costs a rate per hour, it produces tokens at some other rate, and dividing the first rate by the second (converted to a per-hour footing) gives the price of a token — and any lever that moves either rate moves the price predictably, in a direction you already know before you touch it.

The two knobs that move this number, isolated

None of those rows required a new derivation — each is the same formula with one input changed, which is precisely the point of having derived the formula instead of memorizing a single number.

Batch size, taken further — a preview of chapter 3

The table above showed batch 64 versus batch 128 as a single doubling. Widen that out to a handful of realistic batch sizes and the cost curve becomes visible as a curve, not just two points:

Each row divides the previous row's cost roughly in half — because step time is flat at 20.9 ms across every one of these batch sizes (all comfortably below the compute-bound crossover chapter 3 derives), doubling the batch doubles tokens/step and halves $/M-token, cleanly, every time. That flatness doesn't continue forever; chapter 3 is entirely about finding exactly where it stops and what happens on the other side of that line.

What chapter 1 hands to every later chapter

Three numbers from this chapter recur, unchanged, for the rest of this lesson: the $4/hour instance price, the 20.9 ms memory-bound decode step, and the derived $0.53/M decode rate at batch 64 and 70% uptime. Chapter 2 uses the same step-time logic to derive prefill's rate. Chapter 3 uses the same 20.9 ms floor to find the compute-bound crossover. Chapter 4 uses the $0.53/M rate, blended with chapter 2's prefill rate, as the backbone of the entire API-versus-self-host comparison. None of that downstream work introduces a new formula — it's this chapter's formula, applied to new questions. Keep that habit in mind through the rest of this lesson: whenever a later chapter introduces a new-looking number, check first whether it's actually this same formula wearing a new set of inputs.

Chapter 2: Prefill vs Decode

Chapter 0's API prices weren't symmetric — $0.50/M for input tokens, $1.50/M for output tokens, a 3× gap. That gap isn't a marketing decision. It's downstream of a hardware fact: the GPU processes input tokens (prefill) and output tokens (decode) through completely different bottlenecks, and one of those bottlenecks is intrinsically cheaper per token than the other.

Why this distinction survives every serving optimization in this lesson

It would be tempting to think a sufficiently clever serving system could erase the prefill/decode split entirely — process everything the same way, at the same rate. It can't, and the reason is architectural, not an engineering gap: decode is sequential by necessity, because token N's hidden state genuinely depends on token N−1 having already been produced — there's no way to compute token 501 before token 500 exists. Prefill has no such dependency between input tokens, so it's free to run in parallel. Every technique this lesson touches later — speculative decoding, batching, routing — works within this constraint, not around it.

Two different jobs, wearing the same word “token”

Prefill happens once per request: the model reads the entire prompt — all 500 input tokens from chapter 0's baseline — in a single forward pass, computing every token's key and value at once, in parallel. Decode happens once per output token: the model produces one new token, appends it to the KV cache, and repeats, one token at a time, sequentially, because each new token depends on the one before it.

That difference in shape — “500 tokens at once” versus “1 token, 300 times in a row” — changes which part of the GPU is the bottleneck. Prefill has enough parallel work to keep the GPU's arithmetic units busy the whole time: it's compute-bound. Decode, one token at a time, spends almost all its time waiting on memory bandwidth to re-read the model's weights for each single token: it's memory-bound — exactly the mechanism chapter 1 built the $0.53/M decode price from.

Naming the shared assumption before deriving the new number

Everything in this chapter reuses chapter 1's hardware exactly — same $4/hour instance, same 2×H100, same 70B model — and changes only which phase of inference is being measured. Keeping the hardware fixed while the workload's shape changes is precisely what makes the resulting comparison clean: any difference in the derived price has to come from the compute-bound-versus-memory-bound distinction itself, not from a hidden change in what's being priced.

Deriving the prefill price, the same way chapter 1 derived decode's

Prefill's cost floor comes from FLOPs, not bytes. A standard estimate for a transformer's forward pass — the same approximation used throughout the scaling-laws literature — is about 2 floating-point operations per parameter, per token. For the 70B model:

An H100 has a published peak of roughly 1,000 TFLOPS (rounded) of dense half-precision compute; two of them give 2,000 TFLOPS of peak. No real workload hits peak — call the fraction actually achieved the model FLOPs utilization, or MFU, and use a realistic prefill-time value of 40%, achievable because prefill has plenty of parallel work to fill the pipeline with:

Apply the same 70% uptime utilization chapter 1 used for decode — the GPU sits idle the same fraction of the time regardless of which phase it's running:

The two rates, side by side

That 1.9× is the raw-compute explanation for why input tokens are cheaper than output tokens. Chapter 0's API prices showed a bigger gap — $1.50 versus $0.50, a 3× multiple — and that's an honest, worth-naming discrepancy: real API pricing isn't purely a markup on raw GPU-seconds. Some of the extra output-token premium covers the harder engineering of serving decode at low, predictable per-token latency under real, ragged traffic (chapter 3's whole subject) rather than the batch-64-forever assumption this chapter used for a clean derivation, plus ordinary margin. The direction of the asymmetry — output costs more than input — is real physics; the exact multiple a given API charges also has a business decision baked into it.

Naming the assumption this whole chapter rests on

The 2 FLOPs-per-parameter-per-token estimate deserves one more sentence, since it's doing real work in this chapter's numbers. It's a widely used approximation for a transformer's forward-pass compute, accurate to within a small constant factor across model sizes and architectures — not an exact count of every multiply-add the hardware performs, but close enough that the resulting $/M-token figures are trustworthy to the precision this lesson needs (a few percent, not many multiples). Chapter 5's distillation lever leans on this same approximation scaling linearly with parameter count, so it's worth internalizing here rather than re-deriving later.

Restating the whole chapter as one sentence

Parallel work is cheap because it fills the GPU's arithmetic units efficiently; sequential work is expensive because it pays a fixed memory tax per step regardless of how much arithmetic is available to fill the wait — and that one distinction, not any difference in the tokens themselves, is the entire reason input and output are priced differently everywhere in this lesson and in every real API.

A worked comparison against a real published number

It's worth checking this chapter's derived 1.9× ratio against how real API providers actually price the gap, not just asserting the direction is right. Public pricing for capable hosted models commonly shows output priced somewhere between 2× and 5× input — wider, on average, than this chapter's raw-compute-only 1.9×. Recall chapter 2's own honest caveat: providers also charge for the engineering cost of holding decode's per-token latency low and predictable under real, ragged traffic (the serving-engines lesson's entire subject), which this chapter's clean batch-64-forever assumption doesn't price in at all. The takeaway isn't that either number is wrong — it's that a from-scratch derivation gives you a defensible floor to compare a real price against, and the gap between the floor and the real price is itself informative: a provider charging close to 1.9× is pricing near raw compute cost; one charging 5× has a lot of margin, engineering overhead, or both baked in.

Why compute-bound is intrinsically the cheaper regime

The deeper reason prefill is cheap isn't about prefill specifically — it's that parallel work lets you spend every dollar of GPU-hour on arithmetic the hardware was built to do at full rate, while sequential work forces the GPU to pay the same fixed memory-bandwidth tax over and over, once per step, whether or not there's enough arithmetic to fill the time between those memory reads. Any inference workload that can be restructured from sequential-many-steps into parallel-one-pass inherits this same discount — which is exactly the intuition behind speculative decoding (verifying several draft tokens in one parallel pass instead of decoding them one at a time) and the reason it exists at all.

Sensitivity check: what if MFU is different?

Chapter 2's prefill derivation leaned on one assumption — 40% model FLOPs utilization — pulled from a realistic range for production serving. MFU varies with implementation quality, sequence length, and how well the kernels are tuned; recompute the prefill price at two other plausible values to see how much that one assumption is actually doing:

More than double the MFU from 25% to 55% and prefill's raw cost more than halves — a genuinely large lever, and one that's entirely about serving-software quality (kernel fusion, sequence packing, avoiding padding waste), not hardware. This is exactly the compute-bound analog of chapter 1's uptime-utilization lever: a free-feeling win available to whoever tunes the software carefully, independent of what GPU or model is underneath it.

One more way to say it: the GPU is never confused about which regime it's in

It's worth being clear that prefill and decode aren't a choice the serving system makes — they're determined entirely by the shape of the work arriving. A batch of prompt tokens that can all be processed together is prefill, full stop; a single new token that depends on everything before it is decode, full stop. Nothing about pricing, model size, or GPU choice changes which category a given piece of work falls into — only how expensive that category turns out to be, which is exactly what this chapter derived.

Splitting one request's cost across both phases

Put chapters 1 and 2 together and price a single request end to end, not just an aggregate rate. Take the baseline app's 500-input, 300-output request, priced at self-hosted rates ($0.28/M prefill, $0.53/M decode, both effective at 70% uptime):

Decode is 53% of this single request's dollar cost despite being only 37.5% of its tokens — the same lopsidedness chapter 0 found on the API side, now shown to come from the same underlying hardware asymmetry rather than from pricing policy alone. A request with a much longer prompt shifts that split further toward prefill in token count, but not necessarily in dollars, because prefill tokens are individually cheaper — worth checking explicitly:

Once the prompt is long enough, prefill dominates the request's cost even though its per-token rate is cheaper — sheer token count wins out. This is the arithmetic behind a genuinely important operational fact: for prompt-heavy workloads (RAG with large retrieved context, long documents, big system prompts), optimizing prefill — via caching, chunking, or a shorter context — matters more than optimizing decode, exactly the opposite emphasis from a short-prompt chat app.

Restated as an arithmetic-intensity ratio

There's a cleaner, hardware-native way to see why one phase is compute-bound and the other is memory-bound: compare the FLOPs done per byte read, called arithmetic intensity. Prefill reads the weights once and reuses them across every token in the (parallel) batch of prompt tokens, so its FLOPs-per-byte is high. Decode reads the same weights but produces only one token's worth of arithmetic per read, so its FLOPs-per-byte is low — and the GPU has its own fixed ratio of peak FLOPs to peak bandwidth (roughly 2,000 TFLOPS ÷ 6.7 TB/s ≈ 300 FLOPs/byte for this 2×H100 pair) that acts as the dividing line: workloads above that ratio are compute-bound, workloads below it are memory-bound.

This is the same conclusion chapter 2 already reached from step-time arithmetic, restated in the vocabulary the hardware itself is built around — and it's the same “roofline” framing that explains why batching decode (chapter 3) pushes it toward the compute-bound line from below, while nothing about batching changes prefill's position, since prefill is already there.

python
def request_cost(tok_in, tok_out, prefill_per_m=0.28, decode_per_m=0.53):
    return (tok_in / 1e6) * prefill_per_m + (tok_out / 1e6) * decode_per_m

for tok_in in [500, 4000]:
    cost = request_cost(tok_in, 300)
    decode_frac = (300/1e6*0.53) / cost
    print(tok_in, round(cost, 6), 'decode share:', round(decode_frac*100,1), '%')
# 500  0.000299  decode share: 53.2 %
# 4000 0.001279  decode share: 12.4 %

The chapter's finding, restated as a design principle

Every subsequent optimization this lesson considers — batching, quantization, caching, distillation, routing — either widens or narrows this chapter's 1.9× gap in some way, which is a genuinely useful lens for predicting what a new technique will do to cost before working the arithmetic in full: techniques that add parallelism to something previously sequential (speculative decoding is the clearest example, chapter 5) push decode's economics toward prefill's; techniques that shrink the model uniformly (quantization, distillation) preserve the ratio, since both phases shrink together.

Does the asymmetry hold at other model sizes?

Everything in this chapter used the 70B model. Recompute both phases' effective throughput for the 13B model chapter 5 later uses for distillation, to confirm the 1.9× ratio isn't an artifact of this one parameter count:

The ratio holds exactly, because both the FLOPs-per-token (prefill) and the bytes-per-token (decode) scale linearly with parameter count in this simplified model, so the parameter count cancels out of their ratio. The 1.9× decode premium is a property of this hardware pairing — the specific ratio of peak FLOPs to peak bandwidth on 2×H100 — not of any one model size. Change the GPU and the ratio would move; change only the model size and it won't.

What chapter 2 hands to every later chapter

The $0.28/M prefill rate and the 1.9× decode-vs-prefill ratio derived here reappear directly in chapter 4's blended self-host rate (a mix-weighted combination of exactly these two numbers), in chapter 5's prefix-caching lever (which specifically targets prefill tokens, since a cache hit skips prefill compute entirely), and in chapter 7's modality comparisons (embeddings and image generation are both, structurally, closer to prefill's compute-bound shape than to decode's memory-bound one). The compute-bound-versus-memory-bound distinction this chapter built is the single most-reused idea in the rest of the lesson.

Chapter 3: Batching vs Latency

Chapter 1 picked batch 64 for the decode calculation and moved on. It's time to earn that number. Batching more requests together drives $/M-tokens down — but not forever, and not for free once you push it far enough. This chapter finds exactly where “free” ends and a real cost-versus-latency trade begins.

Restating chapter 1 and 2's numbers as the two endpoints of this chapter's curve

This chapter's curve has two named endpoints already sitting in earlier chapters, worth calling out explicitly before deriving what's between them. Chapter 1's batch 64 point ($0.36/M raw, before uptime discount) and chapter 2's prefill ceiling ($0.20/M raw) are, respectively, a point comfortably inside this chapter's free-batching zone and the asymptote decode approaches once it's pushed far enough past the crossover to become just as compute-bound as prefill always was. Everything in this chapter is the curve connecting those two already-familiar numbers.

Why batching looks free at first

Chapter 1's decode step time — 20.9 ms — came from reading 140 GB of weights once. That read happens regardless of batch size, because every request in the batch shares the same weights; the GPU reads them once and reuses that single read for every sequence riding along. So as long as the extra arithmetic from a bigger batch doesn't itself become the bottleneck, the step time stays pinned at 20.9 ms while the number of tokens produced per step — one per request — keeps climbing. Throughput scales linearly with batch size in this region, and cost per token falls proportionally. This is the entire mechanism behind continuous batching.

The stakes: why this chapter's number is worth deriving carefully

Batch size is one of the few knobs in this entire lesson that a serving team controls directly, with no model change, no eval risk, and no vendor negotiation — it's a config value. That makes getting the crossover right unusually high-leverage: undershoot it and you're leaving free throughput on the table; overshoot it by a lot and you're paying real latency for savings that have already mostly been collected.

Where the free lunch ends: the compute-bound crossover

It can't stay free forever, because a bigger batch also means more arithmetic per step — and eventually that arithmetic itself takes longer than the 20.9 ms memory read it's riding alongside. Find the batch size where that happens by setting the two times equal, reusing chapter 2's per-token compute-time figure of 0.175 ms:

Below batch 119, the GPU is memory-bound: step time stays flat at 20.9 ms no matter the batch size, so per-token latency (ITL, inter-token latency) doesn't get worse as you add more concurrent requests — you're using compute capacity that was otherwise sitting idle. Above batch 119, the GPU is compute-bound: step time starts growing linearly with batch size, and every request in that batch waits longer between tokens.

Two concrete operating points

Batch 32 — comfortably below the crossover. Step time is still the flat 20.9 ms:

Batch 400 — well past the crossover, deep into diminishing returns. Step time is now set by compute, not memory:

Naming the two curves before reading the numbers

Two quantities move in opposite directions as batch size grows, and keeping straight which is which makes everything below easier to read: $/M-tokens only ever falls or flattens as batch grows (more sharing, never less), while ITL only ever holds flat or rises (never falls) — there is no batch size at which making the batch bigger makes an individual request's per-token wait shorter. The trade is real precisely because one curve is monotonically non-increasing and the other is monotonically non-decreasing; if either one reversed direction anywhere, batch size would stop being a genuine trade-off at all.

The trade, stated as a ratio

Going from batch 32 to batch 400 buys roughly 3.7× cheaper tokens at the cost of roughly 3.3× worse per-token latency — a real, symmetric-looking trade, and it lands squarely in the 2–5× range that shows up again and again comparing throughput-optimal and latency-optimal serving configurations in practice. The trade only exists past the crossover batch. Below it, there's no latency cost at all — which is exactly why every serious serving system pushes batch size up to the crossover before it starts trading anything away, and why chapter 1's choice of batch 64 (well under 119) was a genuinely free win, not a compromise.

A note on why 20.9 ms specifically doesn't change across this whole chapter

It's worth re-emphasizing why the memory floor stays pinned at exactly chapter 1's 20.9 ms across every batch size below the crossover: every request in the batch shares the identical set of weights, read once, from the identical HBM at the identical 6.7 TB/s. Nothing about a bigger batch asks the GPU to read more bytes of weight — it only asks it to do more arithmetic with the bytes already in flight, which is precisely the definition of “free” in the memory-bound regime.

The full curve, at finer granularity

Two operating points made the trade concrete; a wider table shows the curve's actual shape — steep gains at first, flattening hard once the crossover is crossed:

Look at the marginal-savings column, not just the cost column: almost all of the available cost reduction is captured by the time batch reaches the crossover at 119 — every doubling past that point buys essentially nothing further in dollars while ITL keeps climbing in direct proportion to batch size. This is the mathematical shape of “diminishing returns,” made concrete: the curve doesn't fail gracefully past the crossover, it goes almost perfectly flat, which means pushing batch size past 119 for this workload is close to pure latency cost with no further cost benefit to show for it.

The other kind of latency this chapter has been ignoring: queueing

Everything above priced step time — how long one decode step takes once a batch is running. Real systems also have to form that batch, which means a scheduler waits some window of time to accumulate enough requests before firing the step. A larger target batch size generally means a longer average wait to fill it, especially at lower traffic rates — a genuinely separate latency cost from ITL, worth naming so the two don't get confused.

A simple worked example: at 1,000,000 requests/day, the average arrival rate is roughly:

Filling a target batch of 32 at that average rate takes roughly 32/11.6 ≈ 2.8 seconds of pure waiting if the scheduler insisted on a full batch before firing — wildly worse than the 20.9 ms step time itself, and the reason real serving systems use a maximum wait timer instead (fire the batch after, say, 50 ms even if it isn't full) rather than waiting for a target size to be reached. This is exactly the scheduling knob the serving-engines lesson covers in depth; it's flagged here only so “latency” in this chapter's cost/latency trade is understood to mean per-token ITL specifically, not the batch-formation wait, which is a separate dial entirely.

python
def step_ms(batch, crossover=119.4, mem_ms=20.9, compute_ms_per_tok=0.175):
    return mem_ms if batch <= crossover else compute_ms_per_tok * batch

def cost_per_m(batch, gpu_hourly=4.0):
    tps = batch / (step_ms(batch) / 1000)
    return (gpu_hourly / (tps * 3600)) * 1_000_000

for b in [8, 32, 64, 119, 200, 400]:
    print(b, round(step_ms(b),1), 'ms  $', round(cost_per_m(b),3), '/M')

Two named strategies, restated in this chapter's terms

“Latency-optimal” serving keeps batch size at or below the crossover, prioritizing a snappy, predictable per-token pace for interactive users — the right choice for a chat product where a person is watching tokens stream in. “Throughput-optimal” serving pushes batch size well past the crossover, accepting slower per-token pacing in exchange for the lowest possible dollar cost — the right choice for offline, non-interactive workloads like nightly report generation or bulk document summarization, where nobody is staring at a cursor waiting for the next word.

Closing the loop with chapter 1's original batch-64 choice

Chapter 1 asked you to trust batch 64 as a representative, realistic starting point. With the full curve in hand, that trust is now earned rather than assumed: batch 64 sits well inside the free-batching zone (46% of the way to the crossover at 119), already capturing the steep, early part of the cost-reduction curve, while leaving the remaining, much smaller marginal gains for a team to decide whether they're worth pursuing given their own latency tolerance — exactly the kind of informed, deliberate choice this chapter's derivation was built to enable.

A concrete rule of thumb for choosing where to sit

Given the shape of the curve derived above, a practical policy falls out naturally: target a batch size at or slightly below the crossover for anything a human is waiting on, and only push meaningfully past it when the workload is genuinely latency-insensitive and the extra dollar savings past the crossover are actually worth collecting for that workload's volume. Because the marginal savings past the crossover are so small (the earlier table's last two rows differ by a fraction of a cent per million tokens), in practice most production systems that serve interactive traffic never bother pushing past it at all — the crossover isn't just a mathematical curiosity, it's close to the actual operating point real serving stacks converge on.

Why the crossover is a genuinely useful number to know, not just a curiosity

It would be easy to treat 119 as a piece of trivia specific to this one hardware-and-model pairing. It isn't — it's the single most actionable number this chapter produces, because it's the one config value a serving team can act on directly, this week, without a model change or a vendor conversation: set the max batch size (or the scheduler's target) at or near it, and the free cost reduction chapters 1–2's batch-64 baseline only partially captured gets fully captured, with the latency cost still at zero.

What determines the crossover for a different model or GPU

119 is not a universal constant — it falls straight out of the ratio between this chapter's two fixed times: the 20.9 ms memory floor and the 0.175 ms compute cost per token. Change either input and the crossover moves predictably:

The one row that actually moves the crossover meaningfully is MFU — the same software-quality lever chapter 2 flagged. Better kernels don't just lower prefill's raw cost; they also push the free-batching zone for decode further out, which is a second, independent reason serving-engineering investment pays for itself twice over.

A brief aside on why this crossover doesn't depend on request length

Nothing in this chapter's derivation referenced how long any individual request's prompt or reply is — the crossover is purely a function of the model and hardware's memory-versus-compute ratio, not of the traffic mix. A product with much longer average replies than this lesson's baseline app faces the exact same 119-batch crossover for its decode phase; what changes for that product is how many total decode steps each request needs (300 output tokens means 300 decode steps at whatever batch size is chosen), not where the free-batching zone ends.

Reframing the trade as a single number: cost elasticity of latency

It's useful to compress the whole curve into one summary statistic: how many percentage points of latency do you spend to buy one percentage point of cost reduction, in the region past the crossover? From batch 119 to batch 400:

Two hundred thirty-five percent worse latency for half a percent cheaper tokens — a genuinely terrible trade, and it makes the earlier batch 32-to-400 comparison ($0.726→$0.194, a real 3.7× win) look far better than pushing all the way from the crossover to batch 400. The lesson generalizes: the useful part of “push batch size for cost” lives in the approach to the crossover, not past it. Past the crossover, keep batching only if the workload genuinely doesn't care about per-token latency at all — the offline case chapter 3 already named — because the dollar reward for doing so has already been collected.

What chapter 3 hands to every later chapter

Chapter 1 picked batch 64 without justifying it; this chapter is that justification — it's comfortably inside the free-batching zone, most of the way to the crossover's savings, with room to spare before latency becomes a real cost. Chapter 6's routing chapter reuses the same shape of reasoning (diminishing returns past a threshold) for how aggressively to route traffic to a cheap model. Chapter 8's dashboard exposes batch size as one of its underlying (if not directly user-facing) assumptions, inherited silently from this chapter's derivation.

Chapter 4: API vs Self-Host

Everything so far has priced tokens two ways: chapter 0's flat API rate, and chapters 1–3's derived self-hosted compute cost. Now put them head to head, honestly — not “compute cost versus sticker price,” but total cost of ownership versus total API bill, at the baseline app's actual volume.

Framing the question this chapter actually answers

“Should we self-host?” is really two questions bundled together, and keeping them separate avoids a common confusion. The first is a rate question — is the self-hosted per-token cost lower than the API's per-token price? Chapters 1 and 2 already answered that: yes, $0.338 blended versus $0.875, comfortably. The second is a volume question — does the app send enough traffic to earn back self-hosting's fixed overhead before the rate advantage has done enough work? That second question is what this chapter actually derives an answer to, and it's the one a rate-only comparison skips past entirely.

The blended rate each option actually charges

Chapter 0's app sends 500 input and 300 output tokens per request — 62.5% input, 37.5% output, by token count. API pricing is a genuine per-unit price, so blending it is a straightforward weighted average:

Self-hosted compute cost is not safe to blend the same way, and the distinction matters: $/M-tokens for prefill and decode are both derived from a shared, single resource — the same GPU-hour — so blending them correctly means blending the throughputs first, weighted by the token mix, then converting the combined throughput to a price. Averaging the two prices directly overstates the true blended cost, because cost and throughput are inversely related, not linearly.

Reading the raw comparison honestly, before fixing it

Take the $0.338-versus-$0.875 comparison at face value for one paragraph, because the instinct it produces is worth naming and then correcting rather than skipping past. A team seeing only those two numbers might reasonably conclude self-hosting is a 2.6× win at any volume, and start migrating immediately. The rest of this chapter exists because that conclusion, while directionally right at high enough volume, is quantitatively wrong at low volume in a way that has burned real teams — the fixed costs below are not a footnote, they're the entire reason a break-even volume exists at all rather than self-hosting simply always winning.

The part self-hosting can't skip: fixed costs

$0.338 versus $0.875 makes self-hosting look like a landslide — 2.6× cheaper. That comparison is incomplete, and incomplete in the direction that gets teams into trouble: it prices only the GPU-seconds spent actually generating tokens, and ignores everything that has to exist around those GPU-seconds for the service to run reliably at all.

Headroom capacity. A production service can't run at exactly the instance count its average traffic needs — a burst, a retry storm, or a single instance failure needs somewhere to land. Budget one extra 2×H100 instance, running 24/7, purely as headroom:

Engineering time. Self-hosting means someone owns capacity planning, autoscaling policy, upgrades, and being paged when it breaks — work an API provider absorbs into their price. Budget a fifth of one engineer's fully-loaded time (a $200k/year role, so 20% of that):

Why blending self-host throughput correctly (not naively) matters here specifically

It's worth pausing on the blending method one more time, because chapter 5 reuses it repeatedly and an error here would propagate through every lever's savings estimate. The wrong method — averaging the two $/M rates directly, weighted by token share — would have given:

The wrong method overstates self-host cost by about 11% ($0.374 vs $0.338) — small-looking here, but it would have shifted the break-even volume, every lever's savings percentage in chapter 5, and the routing math in chapter 6, all in the same direction, compounding a single unit-blending mistake across the rest of the lesson. The reason the naive average is wrong: $/M-token and tok/s are reciprocally related (one goes up exactly as the other goes down for a fixed GPU-hour cost), so averaging costs directly, weighted by a count that's proportional to the wrong side of that reciprocal relationship, silently double-counts the effect. Blending the throughputs — the actual shared, physical resource — and converting to a rate only once, at the end, avoids the error entirely.

The full cost model, as a function of volume

Self-hosting is fixed cost plus a per-token variable rate; the API is pure variable rate with no fixed floor. Write both as functions of monthly token volume V, in millions:

Two straight lines: one starts above zero and climbs shallowly, the other starts at zero and climbs steeply. They must cross exactly once, at the volume where the API's lower fixed cost stops outweighing its higher per-token rate.

Why the two curves cannot be parallel

Before solving, notice something structural: the self-host line's slope ($0.338/M) is necessarily flatter than the API line's slope ($0.875/M), because the API's price already has to cover its own version of chapter 4's fixed costs (the provider's GPUs, engineers, and margin) baked into every token, while this chapter's self-host rate is pure marginal compute cost with the provider-equivalent fixed costs pulled out into the separate F term instead. Two lines with different slopes are mathematically guaranteed to cross exactly once — the interesting question was never whether they cross, only where.

Solving for the crossover, by hand

Below 11.57B tokens/month, the API wins — you haven't sent enough traffic to earn back self-hosting's fixed overhead. Above it, self-hosting wins, and the gap only widens as volume grows, because self-hosting's cost curve is shallower.

Where the baseline app actually sits

Chapter 0's app does 24 billion tokens/month (15B input + 9B output, from 1,000,000 requests/day × 800 tokens/request × 30 days) — more than double the 11.57B crossover. Plug it into both formulas:

That $21,000 is the exact number chapter 0 derived independently, two chapters ago, from a completely different starting point — a good cross-check that nothing drifted along the way. Self-hosting saves:

Reading the equation as an inequality instead of an equality

The break-even equation solved for the exact volume where costs tie. It's often more useful to flip it into an inequality and ask directly: for what volumes does self-hosting win? Rearranging F + cV < pV gives V > F/(p−c) — exactly the same 11.57B number, now read as a threshold rather than a crossing point. Every volume strictly above it favors self-hosting; every volume strictly below it favors the API; the threshold itself is a dead heat.

The gap at three different volumes

The break-even point is a single number, but the size of the win or loss on either side of it grows the further you move from it. Compute both TCOs at three volumes — below, near, and well above the 11.57B crossover — to see the gap widen:

At 5B tokens/month, self-hosting loses money by 44.6% — a team that self-hosted anyway, reasoning only from the raw per-token rate comparison in the previous section, would be paying nearly double what the API would have cost, entirely because they hadn't yet earned back the fixed overhead. At 50B, the self-host advantage nearly doubles in percentage terms compared to the 24B baseline, because the fixed cost $6,213 is a shrinking share of an ever-larger bill while the per-token rate gap ($0.875 vs $0.338) keeps compounding. This is the shape every fixed-cost-plus-variable-cost comparison has: a crossover, then a widening gap in whichever direction you're moving away from it.

Why the gap widens rather than staying constant

It's worth deriving, not just observing from the table, why the percentage gap grows with volume rather than staying fixed. The dollar gap between the two lines is (p−c)V − F — linear in V with a positive slope, since p > c. As V grows, that gap grows without bound, while the percentage gap (dollar gap divided by the larger of the two totals) approaches a ceiling set by (p−c)/p as F becomes negligible relative to the variable terms — roughly (0.875−0.338)/0.875 ≈ 61% in the limit of very high volume, a number neither of this chapter's two finite examples (31.8% at 24B, 47.2% at 50B) has fully reached yet, but is climbing toward.

Sensitivity check: what if the fixed-cost estimate is wrong?

$6,213/month rested on two specific numbers — one headroom instance, one-fifth of one engineer. Real teams disagree about both. Recompute the break-even volume across a plausible range of fixed costs, holding everything else fixed:

A fully-loaded dedicated engineer roughly doubles the break-even volume compared to a part-time allocation — at $12,213 fixed cost, the baseline app's own 24B tokens/month barely clears the new, higher bar. This is the honest reason teams sometimes get burned by self-hosting: they price the GPU-hours accurately and forget that a service someone has to operate, on call, indefinitely, is a real and often underestimated number that moves the break-even substantially.

python
def tco_self_host(vol_m_tokens, fixed=6213, rate=0.338):
    return fixed + rate * vol_m_tokens

def tco_api(vol_m_tokens, rate=0.875):
    return rate * vol_m_tokens

def breakeven(fixed=6213, c=0.338, p=0.875):
    return fixed / (p - c)

for f in [2880, 6213, 12213]:
    print(f, '-> breakeven at', round(breakeven(fixed=f)), 'M tokens/month')

What the break-even formula does and doesn't tell you

The formula V = F ÷ (p − c) answers exactly one question: at what volume does the raw dollar total cross over? It deliberately leaves out at least three things worth naming, because they change the decision without changing the arithmetic:

None of these change the math in this chapter — they change how much weight to put on the math versus other considerations when the decision is close to the break-even line. Far from the line (5B or 50B tokens/ month in the earlier table), the dollar answer is decisive enough that these factors rarely flip the decision. Near the line, they often should be the tiebreaker.

The two failure modes this chapter's derivation prevents

Naming both failure modes explicitly, since a real team can fall into either one depending on which direction they're biased: self-hosting too early, at a volume below the break-even, pays fixed overhead with too little traffic to spread it over — effectively burning money relative to just staying on the API. Staying on the API too long past the break-even overpays a per-token rate that a modest, well-justified fixed investment would have beaten months earlier. The break-even formula is the single number that resolves both failure modes at once, in either direction.

A worked example of growth crossing the line

Suppose the baseline app started smaller — 8B tokens/month six months ago — and has been growing 15% month over month. Project forward and find when it crosses the 11.57B break-even:

Under this growth rate, the app crosses the break-even line in roughly two and a half months — which is exactly the kind of forward-looking number the raw current-month comparison misses entirely, and exactly the kind of number a growth-stage team should be tracking to know when to start the self-hosting migration project before the API bill has already grown past the point where self-hosting would have saved money for months.

What chapter 4 hands to every later chapter

The $14,325/month self-hosted baseline derived here is the number every lever in chapter 5 discounts from, the number chapter 6's routing cascade compares its variable-cost savings against, and the number chapter 8's dashboard reconstructs live from its own sliders. Get this chapter's fixed-cost estimate wrong and every downstream percentage in the rest of the lesson is still directionally correct, but the dollar figures attached to it would shift — which is exactly why the sensitivity table above exists: to make explicit how much of the final number rides on an estimate you should replace with your own team's real numbers, not this lesson's illustrative ones.

Chapter 5: Cost Levers, Ranked

Five techniques, one shared starting point, each priced by hand against it.

Chapter 4 landed on a real number: $14,325/month, self-hosted, for the baseline app. That number is not a floor — it's a starting point, and this chapter's job is to show exactly how much lower it can go, and at what cost in engineering effort and risk, one lever at a time. This chapter prices out five concrete levers, each applied independently to that same $14,325 baseline, so the savings are directly comparable instead of each resting on its own unrelated toy example.

Why every lever below is priced against the same $14,325, not against each other

It's worth being explicit about the comparison method before working through five different techniques, because it's easy to accidentally compare a lever against the wrong baseline. Every row in this chapter answers “what would our bill be if we shipped only this one change, starting from today's unoptimized $14,325/month self-hosted setup?” — not “what does this save on top of whatever else we've already shipped.” That choice makes the five levers directly comparable to each other (same starting point, same units), at the cost of not directly telling you what stacking several of them buys — which is exactly why the worked stacking example above exists as a separate, explicitly flagged calculation.

Lever 1 — Quantization: fp16 → int8

Reuse chapters 1 and 2's fixed hardware one more time — same $4/hour instance, same batch, same uptime — and change only the model's stored precision.

Halving the bytes per weight halves the memory-bound decode step time (chapter 1's 20.9 ms floor is directly proportional to weight bytes) and, on modern tensor cores, roughly halves compute time too. Apply a clean 2× to the blended compute rate:

Near-free: modern serving stacks quantize routinely, output quality loss is usually small to negligible at int8, and this lever needs no code changes to the app calling the model — only to how it's served. This is the single lever on this table most teams should ship first, precisely because it's this cheap to try.

Why quantization is listed first, not by accident

The ordering of these five sections isn't the ranked order (that table comes at the end) — it's roughly the order most teams should evaluate them in, starting with the technique that has the fewest moving parts and the least to verify before shipping. Quantization changes exactly one thing (numeric precision), is supported natively by most serving frameworks, and its quality impact is small and well-studied enough that it's usually the very first lever any serving team reaches for.

Lever 2 — Prefix caching on a shared system prompt

Suppose 200 of the 500 average input tokens are a shared system prompt, reused across most requests. With a 90% cache-hit rate on that prefix, the effective billed input drops:

Monthly volume falls from 24B to (320+300) tokens × 1,000,000 × 30 ÷ 10⁹ = 18.6B tokens. The mix also shifts — input is now 320/620 = 51.6% of tokens, output 48.4% — so recompute blended throughput at the new mix:

Smaller than quantization here because the shared prefix is a modest 40% of this app's average input — an app with a longer, more heavily shared system prompt or few-shot examples would see this lever pay off far more. It's also exact, not approximate: a cache hit skips real, identical compute.

Why prefix caching's win is capped by how the app is actually built

Unlike quantization, prefix caching's savings are a direct function of a product decision made outside the serving stack entirely: how much of the prompt is genuinely identical across requests. A team that restructures its prompts to front-load the shared, static portion (system instructions, tool definitions, few-shot examples) ahead of the per-user variable content directly increases this lever's ceiling — it's one of the few cost levers in this chapter that a product engineer, not a serving engineer, can meaningfully move.

Lever 3 — Distillation to a smaller model

If a task-specific 13B model matches the 70B model's quality closely enough on this product's traffic (a real claim that needs its own evaluation, not assumed), cost scales down roughly linearly with parameter count in this simplified model — both the memory-bound decode floor and the compute-bound prefill floor are proportional to parameter count:

The biggest single lever in this table — and the one with the most real risk attached. Unlike quantization, distillation changes the model's actual weights and behavior; it demands a genuine quality evaluation on this product's own traffic before shipping, not just a benchmark score from someone else's task.

What “task-specific” is doing in this lever's framing

The distillation number above assumed the 13B model “matches quality closely enough”, and that phrase is carrying real weight. A general-purpose 13B model rarely matches a general-purpose 70B model across every possible task — but a 13B model fine-tuned or prompted specifically for one product's narrower task distribution often can, because it no longer needs to be good at everything the larger model was trained for, only at the specific thing this product actually asks of it.

Lever 4 — Speculative decoding

A small draft model proposes several tokens ahead; the big model verifies them all in one parallel (compute-bound, prefill-shaped) pass instead of decoding them one at a time. At an acceptance rate α = 0.7 and k = 4 draft tokens per round, the expected tokens produced per verification round is:

Because verification is still one memory-bound weight-read per round — the same 20.9 ms floor — but now yielding 2.77 tokens instead of 1, decode throughput scales up by roughly that same factor:

Smaller than distillation, but with a guarantee neither quantization nor distillation offers outright: rejection sampling makes speculative decoding's output distribution provably identical to running the target model alone, token for token, in expectation — a free lunch with a mathematical receipt.

What speculative decoding costs that the dollar savings don't show

The $2,424/month saving above is net of one cost this chapter's formula folded in implicitly rather than itemizing: the draft model's own GPU-seconds. A small draft model is cheap to run (the same parameter-linear approximation from lever 3 suggests a modest draft model adds only a few percent to total compute), and that cost was already absorbed into the 2.77× throughput multiplier rather than subtracted separately — worth naming so the $2,424 figure isn't mistaken for a number with no hidden line items at all.

Lever 5 — Batch APIs / off-peak utilization

Suppose 30% of this app's volume (7.2B of 24B tokens/month) is offline-tolerable — nightly reports, bulk classification, anything without a human waiting — and can be scheduled into idle capacity, lifting uptime utilization on that slice from 70% to 95%:

The smallest lever in this table for a self-hosted fleet, because the win here is purely about squeezing better utilization out of hardware you already own — it matters far more on a pay-per-token API, where providers pass along genuine off-peak GPU discounts of 40–50% for batch/asynchronous jobs.

Restating why this is genuinely the smallest lever on a self-hosted fleet

It's worth one more sentence on why off-peak scheduling underperforms its counterparts here specifically: a self-hosted fleet's GPUs exist whether or not off-peak work fills them, so the “savings” from better utilization are really just recovering value from hardware that was already paid for and already idle — a genuine win, but bounded by how much idle capacity actually exists, unlike quantization or distillation, whose savings scale with total volume regardless of when that volume arrives.

Stacking two levers by hand, honestly

The five rows above are each independent, from-baseline comparisons — but a real team will want to know what happens when several run together. Work quantization and speculative decoding stacked, since both are close to free lunches and a natural pair to combine first.

Quantization multiplies the compute rate by 0.5 (as derived above). Speculative decoding multiplies decode throughput by 2.77×, which this chapter approximated as multiplying the decode portion of the blended rate. Stacking them means applying quantization's uniform 0.5× on top of the already-adjusted speculative-decoding rate:

Compare that stacked 36.7% to each lever's solo effect (28.3% and 16.9%): stacking beats either alone, as expected, but 36.7% is meaningfully less than the naive sum of the two percentages (45.2%) or their naive product interpreted as sequential percentage cuts on the ORIGINAL rate. The correct method — applying each multiplicative factor to the rate in turn, not to the baseline dollar figure independently — is the only one that stays dimensionally honest, and it's exactly why chapter 8's dashboard computes combinations this way rather than simply adding percentages off a printed table.

Why levers interact instead of composing cleanly, in the real world

The multiplicative-factor approximation above is still an approximation, and it's worth being honest about where it breaks. Quantized weights change the acceptance rate α a speculative decoding draft model achieves, because the target model's outputs shift slightly under lower precision — usually only a little, but not exactly zero. A distilled model has its own separate, independently-measured latency and throughput profile rather than simply inheriting the 70B model's numbers scaled down. Prefix caching's hit rate can interact with routing, since a router that sends easy, repetitive requests to a small model changes which requests are left hitting the large model's cache. None of these interactions are large enough to invalidate the ranking in this chapter's table, but a team stacking three or more levers in production should measure the combined result directly rather than trust a multiplied-factor estimate past two levers deep.

The full ranking, savings and effort together

Each row is independent — a from-baseline comparison, not a running stack — because real levers interact (quantized weights change the draft/target acceptance math for speculative decoding; a distilled model changes what quantization saves) in ways that would need their own careful re-derivation, not a naive multiplication of five discount factors. Chapter 8's dashboard lets you toggle combinations and see an honest, clearly-labeled approximation of what stacking looks like.

Ranking by effort, not just by savings

Dollar savings alone is an incomplete ranking — a lever that saves less but takes an afternoon to ship is often a better use of a team's time this sprint than one that saves more but needs a multi-week evaluation project. Add an honest effort column to the same five rows:

Reordering by “savings per unit of effort and risk” rather than raw dollars flips the practical sequencing: most teams should ship quantization first (biggest savings-per-hour ratio on the table), then speculative decoding, then prefix caching and batch scheduling as they fit naturally into a roadmap, and treat distillation as a separate, deliberately-scoped project rather than a quick win — even though it's the single largest number in the dollar column.

What happens if a lever's assumption is optimistic

Every lever's savings number rested on an assumption that might not hold in a real deployment. Worth pricing the downside case for the two levers with the widest plausible assumption range: prefix-caching hit rate and speculative-decoding acceptance rate.

Both levers roughly halve their savings under a realistic pessimistic case — still positive, still worth shipping, but a team that budgeted for the optimistic number and got the pessimistic one would be off by several thousand dollars a month against their own forecast. The lesson generalizes beyond these two rows: any lever whose savings depend on a workload-specific parameter (hit rate, acceptance rate) deserves a measured value from your own traffic before it goes into a budget, not a borrowed number from someone else's blog post about their own, different workload.

Chapter 6: Multi-Model Routing

Not every request needs the expensive model. Deciding which ones do is a cost lever in its own right.

Chapter 5's five levers were all in-place optimizations — the same model, made cheaper to run. Chapter 5's levers all shrink the cost of running one model. This chapter asks a different question: does every request even need the big model? A cascade — try a cheap model first, escalate to the expensive one only when needed — can beat every lever in chapter 5 combined, if the traffic mix cooperates.

Why routing is a fundamentally different lever than chapter 5's five

Every lever in chapter 5 answered a version of “how do we make the 70B model's own tokens cheaper.” This chapter asks a prior question chapter 5 never raised: does every request need the 70B model's tokens at all? A support bot handling “what are your hours” and a support bot handling “my payment failed and I was charged twice” are not, cost-wise, the same kind of request — treating them identically means either overpaying for the easy one or under-serving the hard one. Routing is the mechanism for treating them differently, on purpose, with the cost difference measured explicitly rather than left as an unexamined assumption.

The cascade, precisely

This mechanism goes by several names across the industry — model cascading, a router, a gateway pattern — but the underlying arithmetic is identical regardless of what a given vendor calls their version of it.

Route every request through a small, cheap classifier/answer model first. If it's confident, its answer ships as-is. If it isn't, the same full request reruns on the 70B model — paying for both passes on anything that escalates. This is the honest accounting: the small model's pass on an escalated request isn't free, it's a sunk cost.

Restating chapter 0's traffic numbers one more time, in this chapter's terms

The cascade reuses chapter 0's exact traffic shape — 1,000,000 requests/day, 500 in / 300 out tokens each — unchanged. What's new in this chapter is not the traffic, it's the decision, made once per request, about which model gets to see it. Nothing about volume or token length needs to be re-derived; only the routing split is a new input.

Pricing the small model

The same cost derivation from chapters 1 and 2 applies unmodified to any model size — only the parameter count changes, so this is a direct reapplication rather than a new formula.

Use a 3B-parameter model for the first pass — small enough to be dramatically cheaper, large enough to handle straightforward requests competently. Under the same linear-in-parameters approximation chapter 5 used for distillation:

Choosing the small model's size: a real trade, not a free variable

Why 3B specifically, rather than an even smaller, even cheaper 1B model? Compare all three under the same linear-in-parameters approximation:

A 1B classifier's routing overhead is nearly a third of the 3B model's — a real, cheap-sounding improvement. But the smaller the routing model, the less capable it typically is at correctly judging its own confidence, which is exactly the signal the whole cascade depends on to decide who escalates. Model size here is a genuine two-sided trade: smaller saves more on the routing pass itself, but a routing model that's too small to judge its own limits reliably either escalates too much (erasing the savings by sending too much traffic to 70B anyway) or too little (shipping wrong answers with false confidence). 3B is a reasonable middle point for this lesson's illustrative purposes, not a universal answer — the right choice depends on measuring each candidate size's actual escalation accuracy on real traffic.

Pricing the cascade against the baseline

Suppose 60% of the app's 1,000,000 daily requests are straightforward enough for the 3B model to resolve outright, and 40% escalate. Every request pays the small model's full 800-token cost first (that's the routing overhead, paid even by requests that end up escalating):

The 40% that escalate pay again, in full, on the 70B model:

Every number in that chain is a direct application of a formula from an earlier chapter — chapter 0's volume arithmetic, chapter 1's dollars-per-token derivation, applied twice, once per model — combined by a new rule specific to this chapter: some traffic pays once, some pays twice, and the split between them is the one genuinely new input this chapter introduces.

Both stages reuse the exact same $/M-token derivation this lesson has built up across four earlier chapters — nothing about the cascade required a new hardware assumption or a new pricing method, only a new split of traffic across two already-priced models.

Compare to always running the full 24B tokens/month on the 70B model alone — chapter 4's variable compute cost:

Bigger than any single lever in chapter 5 — because routing doesn't make the 70B model cheaper per token, it removes 60% of the traffic from ever touching it at all. Of the $348 spent on the small model, only the 60%-share — roughly $208.80 — is where the real saving lives; the other $139.20 is the honest cost of the routing decision itself, spent even on requests that escalated anyway.

Verify that $208.80/$139.20 split adds back to $348 as a quick sanity check: 0.60 × $348 = $208.80 and 0.40 × $348 = $139.20, summing to exactly $348.00 — the small model's spend simply partitions along the same escalation split as everything else in this chapter, which is a useful property to notice because it means the “wasted” fraction of the routing cost is always exactly the escalation rate, regardless of what that rate happens to be.

The latency cost of escalation, priced the same way as dollars

Escalated requests pay a real latency tax this chapter hasn't priced yet: they run the small model first, then the large model, sequentially, rather than going straight to the large model. Using chapter 1's single-user (batch 1) decode figures as a rough per-request latency proxy for the small model's classification pass:

18 milliseconds is small relative to a full request's total latency (typically hundreds of milliseconds to seconds for a 300-token reply), but it's not zero, and it's paid by every escalated request on top of whatever the 70B model itself takes. A latency-sensitive product with a strict end-to-end SLA needs to budget for this tax explicitly, the same way chapter 3 budgeted for batching's latency cost — routing's dollar savings and its latency cost are two sides of the same coin, exactly like batching's were.

The quality/cost frontier the routing threshold controls

The 60/40 split isn't fixed — it's a threshold the small model's confidence score is checked against, and moving that threshold trades cost against risk directly. A stricter threshold (route more to the big model, escalate more readily) costs more but risks fewer wrong answers from an under-qualified small model. A looser threshold saves more but leans harder on the small model's judgment. There's no single correct threshold — it's a genuine frontier, and the right point on it depends on how costly a wrong answer is for this specific product, not on the cost math alone. A customer-support cascade might tolerate a fairly loose threshold; a medical or legal triage cascade should not, regardless of what the dollar savings look like at that looser setting.

That last row is worth sitting with: routing 100% of traffic to the big model while still paying for the small model's classification pass on every request is worse than not routing at all, because the small-model overhead is pure loss with no offsetting savings. It's a useful sanity check on the whole mechanism — a cascade only pays for itself once a meaningful share of traffic actually resolves at the cheap tier, which is exactly why choosing (and measuring) the escalation threshold correctly matters as much as the cascade's existence in the first place.

Lower the escalation rate and savings climb — but so does the risk of the cheap model quietly answering something it shouldn't have. The gateway pattern that makes this practical routes based on a real confidence signal (the small model's own uncertainty, or a lightweight separate classifier), not a fixed percentage target chosen in advance.

How this connects to chapter 5's levers, one more time

Nothing prevents combining routing with chapter 5's per-model levers — in fact production systems usually do all of it at once, applying the same cost discipline to both the cheap and expensive tiers of a cascade. A 70B model that's also quantized costs $0.169/M instead of $0.338/M for the escalated 40%, cutting the always-70B comparison point itself and, proportionally, every routing percentage computed against it. The two chapters compose the same way chapter 5's own levers composed with each other: correctly, by recomputing the underlying rate before reapplying the routing split, not by adding percentages from two separately-computed tables.

Routing and quantization stacked reach 75.7% off the original baseline — more than either alone (55.7% and 28.3% respectively) and more than the two-lever stack chapter 5 worked out for quantization plus speculative decoding (36.7%), because routing's mechanism (remove traffic entirely) and quantization's mechanism (make remaining traffic cheaper) don't compete for the same savings the way two decode-side optimizations partially do. This is the largest single combined-savings figure anywhere in this lesson.

A three-tier cascade, worked the same way, step by step

Nothing about the cascade mechanism limits it to two models, and it's worth checking that the pattern generalizes cleanly before treating two tiers as the only shape it can take.

Add a middle tier — a 13B model, priced at chapter 5's distillation rate of $0.063/M — between the 3B classifier and the 70B model, and route in three stages: 3B handles the easiest 50%, 13B handles the next 30%, and only the hardest 20% reaches 70B.

Compare to the two-tier cascade's $3,592.80/month and the always-70B baseline's $8,112/month:

A third tier buys another 10.7 percentage points of savings, at the cost of a more complex routing decision (now two thresholds to tune instead of one) and two extra places a misrouted request can end up with a worse-than-necessary answer. Whether that complexity is worth it depends entirely on how cleanly the traffic actually separates into three difficulty bands — a judgment call this arithmetic can inform but not make.

What makes a request “easy” in the first place

This chapter has treated “60% resolved by the small model” as a given input, but real systems have to decide it, request by request. Three common signals, each with a real tradeoff: the small model's own softmax confidence on its answer (cheap to compute, since it's already run; can be miscalibrated — a wrong answer delivered with high confidence is the worst failure mode a router can have); a separate, lightweight classifier trained specifically to predict “will the 70B model's answer differ meaningfully from the 3B model's” (more accurate but adds its own, if small, cost and latency); or simple rule-based heuristics on the request itself (prompt length, presence of certain keywords, previous escalation history for this user) which cost nothing to evaluate but generalize the worst to new traffic patterns.

python
def cascade_cost(vol_m, small_rate=0.0145, big_rate=0.338, escalate_frac=0.40):
    small_pass = vol_m * small_rate            # every request pays this
    big_pass = vol_m * escalate_frac * big_rate  # only escalations pay this too
    return small_pass + big_pass

for esc in [0.20, 0.40, 0.60, 0.80]:
    cost = cascade_cost(24000, escalate_frac=esc)
    always70 = 24000 * 0.338
    print(esc, '-> $', round(cost,2), '  save:', round((1-cost/always70)*100,1), '%')

The failure mode routing introduces that per-model levers never do

Quantization and speculative decoding both come with mathematical guarantees about output fidelity (near-identical or exactly identical, respectively) — routing has no such guarantee built in. A misrouted request doesn't get a slightly worse answer from a compressed version of the right model; it gets an answer from a genuinely different, less capable model that may be confidently wrong in ways the user has no way to detect. This is the single reason routing sits alongside distillation, not alongside quantization, on the risk spectrum this lesson has been tracking chapter by chapter — big savings, real quality surface area, worth a genuine evaluation before the threshold gets tuned aggressively.

Restating the chapter's core claim once more, precisely

Not every dollar of inference cost is created equal, and not every request needs the same amount of model. Routing is the mechanism for acting on that fact deliberately, with the savings measured rather than assumed.

What this chapter hands to chapter 8's dashboard

The cascade's core insight — that not every token needs to touch the expensive model — is the one piece of this lesson's toolkit that chapter 8's dashboard doesn't directly expose as a slider, precisely because a routing threshold needs a real, measured escalation rate from actual traffic, not a number a lesson can hand you generically. What the dashboard does inherit from this chapter is the underlying lesson: before reaching for a per-model lever, ask whether the traffic itself can be triaged first.

Chapter 7: Modality Economics

Four modalities, one method, applied fresh to each one's own physical bottleneck — text tokens were never the whole story for a real production AI product, and the same derivation discipline chapters 1 and 2 built for text carries over completely unchanged.

Every derivation so far has been text tokens in, text tokens out. Production AI products increasingly aren't just that — embeddings, image generation, and speech all get billed too, and each has its own cost floor worth deriving the same way, once, so you stop treating “the AI bill” as one undifferentiated number.

Why every modality gets the same treatment: identify the bottleneck, price the GPU-time

Every derivation in this chapter follows the exact same three-step method chapters 1 and 2 already established for text: figure out what physically bottlenecks the workload (bytes moved, FLOPs performed, or wall-clock audio duration), price the GPU-seconds that bottleneck actually consumes, and compare that floor to what the market charges. Nothing about embeddings, images, or speech requires new cost theory — only a new physical bottleneck to identify.

Embeddings: pennies per million, and mostly margin

An embedding model is tiny compared to a 70B chat model — a common size class is around 100–400 million parameters, three orders of magnitude smaller than this lesson's 70B baseline. Use 110M parameters, half precision:

On a modest GPU with roughly 600 GB/s of memory bandwidth (far cheaper than an H100, at maybe $0.50/hour), and a large, easily-affordable batch of 512 (a tiny model leaves huge headroom before hitting a compute-bound crossover):

That's not a typo — the raw compute cost of embedding a million tokens is close to a hundredth of a cent. Real hosted embedding APIs typically charge somewhere in the $0.01–$0.10/M range — a hundred to a thousand times the compute floor this derivation found. That gap isn't dishonest pricing; it's the cost of reliability, uptime, rate limiting, and support infrastructure sitting on top of a genuinely tiny amount of raw silicon time. Of every cent an embeddings API charges, essentially none of it is GPU-seconds.

What the embeddings number implies for RAG pipelines

This near-zero embedding cost has a direct, practical consequence for any product built around retrieval (RAG): embedding the corpus once and re-embedding it on updates is essentially free at the compute level even for a large corpus. Reuse chapter 0's exact traffic-times-price method one more time, applied to a corpus rather than to daily requests. A million-document corpus, each document averaging 500 tokens, costs:

Five cents. To embed an entire million-document corpus. That is a genuinely striking number, and it's the reason RAG system design almost never treats embedding cost as a real constraint — the actual costs in a RAG pipeline live elsewhere: storage and retrieval infrastructure (a vector database), and the LLM generation step that consumes the retrieved context, which is priced at this lesson's full text-token rates, not embeddings' near-zero ones.

Recompute the same corpus cost at ten times the scale — ten million documents, still 500 tokens each — just to confirm the linearity holds at genuinely large scale, not only at this chapter's illustrative one-million-document example:

This is worth restating as a general planning principle for any multi-modal product: identify the modality that dominates a workflow's cost before optimizing, because intuition about which step “feels expensive” is frequently wrong. Embeddings, which sound like they should be a serious infrastructure cost given how central they are to modern retrieval systems, are in fact the cheapest thing this chapter measured by a wide margin — the earlier voice-assistant breakdown found ASR, not the more conceptually-central generation step, dominating that particular pipeline's bill.

Image generation: steps × GPU-seconds × price

A diffusion model builds an image through a fixed number of denoising steps — a common configuration is 25 steps, each costing roughly 100 ms of GPU compute on hardware in the H100/A100 class:

Hosted image-generation APIs typically charge somewhere in the $0.02–$0.04/image range — roughly 15–30× the raw compute floor. Smaller multiple than embeddings, because image generation genuinely uses more of the GPU per unit billed (a full 2.5 second occupancy per image, versus a fraction of a millisecond per embedded token), so there's proportionally less headroom before compute cost is a real share of the price.

The step-count knob, and how it trades quality against cost

Unlike text generation's token count (usually set by the content, not a quality dial), a diffusion model's step count is a direct, continuous quality/cost knob a product team chooses. Fewer steps means faster, cheaper generation at some quality cost; more steps means slower, pricier generation with diminishing visual returns past a point. Recompute the floor at three step counts:

Even the 50-step high-fidelity mode's raw floor is still under a third of a cent — the entire step-count range stays deep inside the same order of magnitude, which is a useful thing to know before assuming a “high quality” product tier needs meaningfully different unit economics from a “fast draft” tier. The typical $0.02–$0.04 hosted price barely moves across step counts in practice, which is itself evidence that step count isn't the dominant cost driver behind that price — overhead and margin are.

Techniques that reduce the effective step count without a proportional quality loss — distillation applied to the diffusion process itself, a close cousin of chapter 5's model-distillation lever — therefore have a real but bounded ceiling on how much they can save here: even driving step count to zero would only ever recover the raw-floor-to-hosted-price gap this chapter already measured, not eliminate the image-generation bill outright.

Speech: real-time factor sets the floor

Text-to-speech models are usually benchmarked by their real-time factor (RTF) — how long it takes to synthesize versus how long the resulting audio lasts. An RTF of 0.05 means the model generates 20× faster than real time: one hour of audio in three minutes of GPU time.

Hosted TTS/ASR APIs typically charge somewhere in the $5–$15/audio-hour range — 50–150× the raw floor. Notably tighter margins than text or images at the low end of that range, because audio pipelines usually carry heavier streaming, buffering, and real-time-delivery infrastructure than a single-shot text or image request does.

Why speech's RTF framing generalizes better than a per-token count would

Speech doesn't have a natural “token” unit the way text does, which is precisely why RTF is the right lens: it measures cost against the one unit that actually matters to the product — wall-clock audio duration — rather than an internal implementation detail like how many acoustic frames or sub-word units the model happens to process per second. A team comparing two different TTS model architectures should compare their RTFs directly; internal unit counts between architectures often aren't even comparable. This is the same lesson as chapter 3's ITL metric for text — pick the unit that matches what the product actually delivers, not the unit that's easiest for the model itself to report.

The pattern across all four modalities

The multiple tracks how much of a request's actual GPU-time the hosted price is covering versus how much margin, reliability, and operational overhead is layered on top — and it's smallest for the modality that occupies the GPU the longest per billed unit (image generation, at a full 2.5 real seconds/image) and largest for the modality that occupies it the least (embeddings, at a fraction of a millisecond/token). This is a genuinely useful lens for evaluating whether self-hosting a given modality is worth the engineering effort: a wide multiple is a hint there's real margin to capture; a narrow one is a hint the hosted price is already close to the honest floor.

Automatic speech recognition: the mirror image of TTS

Text-to-speech converts text into audio; automatic speech recognition (ASR) converts audio into text — the reverse direction, priced by the same real-time-factor logic. A production ASR model commonly runs at an RTF around 0.1 (10× faster than real time, typically a bit slower than TTS because transcription models tend to be somewhat larger for a given quality bar):

Twice TTS's floor for a comparable GPU class, and landing in the same $5–$15/audio-hour typical hosted range as TTS — the two, being structurally similar sequential-audio-processing tasks, end up with similar economics on both the floor and the market-price side.

The direction of the RTF difference is worth a sentence: ASR generally runs slower (higher RTF, more GPU time per audio-hour) than TTS because transcription has to handle far more acoustic variability — accents, background noise, overlapping speech — that a well-behaved TTS system, generating clean audio from scratch, never has to contend with. More variability to model generally means more parameters and more compute to reach the same quality bar.

A worked pass through a real multi-modal product

Put all four modalities together in one worked scenario: a voice assistant that transcribes a 2-minute user query (ASR), embeds the transcript for retrieval (embeddings), generates a 200-token text reply (this lesson's baseline text economics), and speaks that reply back (TTS). Price one full round-trip, self-hosted, using this chapter's floors:

This single round-trip stitches together every derivation this chapter has built, in the order a real request would actually flow through them — a genuine end-to-end trace, not four separate numbers computed in isolation. Roughly seven-tenths of a cent per full voice interaction, at raw compute cost, is the number this trace produces. ASR dominates this particular breakdown — nearly 93% of the total — simply because it's the step processing the most real-world seconds of audio; the text generation and embedding steps, despite being the “main” work conceptually, are almost free by comparison. This is exactly the kind of cross-modality cost breakdown a team building a voice product needs before deciding where to invest optimization effort — and it would be invisible from a single aggregated monthly bill.

Scale this same round-trip to 100,000 daily voice interactions and the ASR-dominated shape persists exactly:

A number in the same order of magnitude as this lesson's entire text-only baseline app from chapter 0 — a useful reminder that a genuinely multi-modal product's bill can rival a large text-only app's bill even at a fraction of the daily request volume, once audio processing enters the picture.

python
def tts_asr_floor(audio_hours, rtf, gpu_hourly=2.0):
    return audio_hours * rtf * gpu_hourly

def image_floor(steps, ms_per_step, gpu_hourly=2.0):
    seconds = steps * ms_per_step / 1000
    return seconds * (gpu_hourly / 3600)

print('TTS floor/audio-hr:', tts_asr_floor(1, 0.05))   # 0.10
print('ASR floor/audio-hr:', tts_asr_floor(1, 0.10))   # 0.20
print('Image floor/image: ', image_floor(25, 100))     # 0.00139

Why the multiple matters more than the absolute number, when deciding to self-host

A team eyeing self-hosting any of these modalities should look at the multiple, not just the raw floor, to gauge how much margin is actually available to capture. A wide multiple (embeddings, 100–1,000×) means there's a great deal of headroom, but it also usually means the raw dollar amounts at stake are tiny in absolute terms — self-hosting embeddings only becomes worth the engineering effort at genuinely large volume, because 1,000× a near-zero number is still often a small number. A narrower multiple (image generation, 15–30×) means less relative headroom, but the absolute dollar amounts per unit are larger, so the same percentage improvement translates to more real money sooner. The multiple tells you where the margin is; the absolute floor tells you whether it's worth chasing at your actual volume. Both numbers, together, are what a real self-host-versus-API decision for a new modality actually needs — neither one alone is sufficient.

What this chapter changes about how you read any AI product's pricing page

The practical skill this chapter is meant to leave behind: given any AI API's pricing page, for any modality, you now have a method for building a rough floor estimate in under five minutes — find or estimate the model's size and the hardware's rate, apply the compute-bound-or-memory-bound logic from chapters 1 and 2, and compare. A price close to the floor tells you the vendor is optimizing for volume over margin. A price far from the floor tells you either there's real, defensible overhead behind it, or there's room a competitor could undercut on. Either reading is useful; neither was available before this chapter's method existed.

Chapter 8 closes the lesson by putting text's economics back at the center, since that's what most of this lesson's sliders manipulate directly — but every number this chapter derived stays valid and reusable the moment a product's roadmap adds a second modality.

That closes the loop on all four modalities this chapter set out to price, each with its own floor, its own typical market range, and its own reason for the gap between them.

Chapter 8: The Full Bill

Every formula this lesson derived, assembled into one live, traceable number.

Eight chapters have priced out one number at a time — a rate derived from hardware, a ratio derived from a compute-versus-memory bottleneck, a break-even point derived from two intersecting lines, a lever's savings derived against a fixed baseline, a modality's floor derived from its own physical bottleneck.

A real monthly bill is all of those numbers, applied together, to whatever this month's actual traffic, model, and hosting choice happened to be. This chapter's simulation is that: a configurable dashboard over the exact same baseline app, with every lever from this lesson as a live toggle, so the last thing this lesson leaves you with is something you can actually manipulate, not just read.

Why the very first chapter's method is still exactly right, eight chapters later

It's worth pausing on how little has actually changed since chapter 0's opening six lines. Every refinement since then — hardware-derived rates, prefill/decode splits, batching curves, break-even volumes, lever discounts, routing splits, modality floors — has been a more precise way of computing one or both of the two quantities chapter 0 started with: a volume, and a rate. Nothing about the underlying shape of the calculation ever changed; only the sophistication of what feeds into each side of the multiplication grew, chapter by chapter.

What the dashboard computes, in one pass

Given a request volume, an average input/output token split, a hosting mode (API or self-host), and a set of active levers, the dashboard walks the same formulas this lesson already derived, in order:

Every one of these steps is independently checkable against a specific earlier chapter's worked numbers. That's a deliberate design choice, not an accident: a dashboard whose output you can't trace back to a derivation is exactly the kind of opaque tool chapter 0 opened by criticizing (“a billing dashboard... explains nothing”), and this lesson would be undermining its own thesis if its final chapter built one.

Contrast that traceability with the billing dashboard chapter 0 opened by criticizing: that dashboard shows a number and stops. This one shows the number and, at every step along the way, the specific earlier derivation responsible for it — the difference between a receipt and an explanation, restated one final time as this lesson's closing design principle.

Four steps, matching the four sections of this lesson's table of contents almost exactly — chapters 0–2 build the rate, chapter 3 informs the batch-size choice baked into it, chapters 4–6 apply hosting mode and levers, and the result is the number chapter 0 asked for at the very start.

Reading the dashboard's four inputs as four earlier chapters, restated

Each control on this chapter's widget maps directly to a chapter already covered, which is worth making explicit before working through the numbers:

Nothing on this dashboard is a black box — every toggle's effect on the final number is traceable to a specific, hand-derived formula from a specific earlier chapter, which is exactly the property that makes it trustworthy to hand to someone else (a teammate, a finance partner) and let them change the inputs themselves without you standing over their shoulder explaining what any given number means.

A worked pass through the dashboard, by hand

Set the dashboard to the baseline app, self-hosted, with quantization and prefix caching both on (leaving distillation, speculative decoding, and routing off, since those three change model behavior and deserve their own evaluation before flipping on together). This is a deliberate, conservative default: the two levers active are the two this lesson has repeatedly flagged as close to free lunches, and the three left off are the three carrying real quality risk — distillation's model-behavior change, speculative decoding's extra draft-model dependency, and routing's need for a measured escalation threshold. Combine the two active levers' effects multiplicatively on the compute rate — an approximation, since real interactions aren't perfectly independent, but a reasonable first estimate:

Down from the $14,325 no-levers baseline — a 33.1% cut from two of the safest, lowest-risk levers on the table, stacked.

The widget below computes exactly this pass, live, as you toggle its checkboxes — drag them to reproduce this hand-worked $9,580 and confirm the dashboard's arithmetic matches what you just derived on paper. Flip on distillation as well in the dashboard and watch it fall further, with the caveat callout from chapter 5 reappearing every time that lever is active, exactly because it's the one lever this lesson refuses to let you forget carries a real quality question.

A second worked pass: API mode, at a different volume

Flip the dashboard's hosting mode to API and set volume to a smaller app — 0.3M requests/day, well below chapter 4's 11.57B-token break-even — to see the same formulas produce a genuinely different shape of answer:

At this volume the API wins by $2,347/month (27.1%) — the mirror image of chapter 4's baseline result, and exactly what the break-even formula predicted for a volume below the 11.57B crossover. The dashboard doesn't have a special case for “small app” versus “large app” — it's the same two formulas, evaluated at whatever volume is currently set, which is the entire point of having derived them as formulas instead of two disconnected facts.

A third worked pass: all three safe levers, at the break-even volume itself

One more instructive setting: dial volume to exactly 11.57B tokens/month, the chapter 4 break-even, with quantization and prefix caching both on. Since both levers reduce the self-host rate and/or volume without touching the API side, self-hosting should win decisively even at a volume where the unoptimized self-host TCO was merely tied with the API:

Interesting: even after applying prefix caching's volume reduction to both options fairly (caching reduces billed tokens regardless of who's hosting), the two remain nearly tied at this specific volume — because prefix caching's effect on volume is common to both while quantization's effect on rate applies only to the self-hosted side. This is exactly the kind of interaction the dashboard's live recomputation surfaces immediately and a static table would hide: the break-even point itself shifts once a lever changes one side's rate but not the other's, and re-deriving it from the dashboard's current settings is faster and more trustworthy than trying to remember which of chapter 4's original numbers still applies.

Every worked pass, side by side

Five rows, five different combinations of this lesson's inputs, and every single dollar figure in that table traces back to a formula derived somewhere in chapters 0 through 7. That's the payoff of building a lesson around derivation instead of memorized numbers: five scenarios that would otherwise need five separate explanations collapse into one set of reusable formulas, evaluated five times at five different settings.

Notice, too, that the five rows span a 3.33× range ($6,300 to $21,000) purely from changing volume, hosting mode, and lever selection — the same underlying app, the same underlying hardware, the same underlying arithmetic, producing a genuinely wide range of honest answers depending on which real-world decisions get made. That range is precisely why a single memorized number was never going to serve finance's original question well; only the formula, evaluated at the actual chosen settings, could.

What this dashboard does not decide for you

Every number in this lesson answers “what does this cost,” never “is this the right tradeoff.” Chapter 3's latency-versus-throughput choice depends on what your users will tolerate. Chapter 5's distillation lever depends on an eval this lesson can't run for you. Chapter 6's routing threshold depends on how costly a wrong answer actually is in your product. The arithmetic tells you the size of each option. It was never going to tell you which one to want.

That division of labor — arithmetic for the size, judgment for the choice — is the honest place to end this lesson.

Nine chapters bought you the first half, precisely.

The second half was always yours.

A checklist for taking this lesson's method into a real budgeting conversation

Nine chapters distilled into six steps, in the order a real team should actually walk them, each pointing back to the chapter that derives it. Print this table, tape it above a desk, and replace every illustrative number in this lesson with your own product's real one before trusting a single dollar figure from it.

Every row replaces one of this lesson's illustrative numbers with a real, measured one from your own product. The formulas don't change; only the inputs do — which is the entire reason this lesson insisted on deriving every number from scratch instead of asking you to memorize $14,325 as if it were a universal constant.

The single sentence to answer finance with, nine chapters later

Chapter 0 opened with finance's one-line question: what does the AI feature cost, and what happens if usage doubles? The honest, complete answer, in one sentence, is now available: “our monthly bill is volume times a rate, the rate is derived from <these specific hardware and lever choices>, doubling volume roughly doubles the bill on a pay-per-token API but grows more slowly if we're past our self-hosting break-even, and here is the exact break-even volume and the exact levers that would lower the rate further if we wanted to invest in them.” That sentence, not a spreadsheet nobody else can audit, is what nine chapters of derivation actually bought.

Where this lesson connects to the rest of the Inference Engineering path

This lesson deliberately treated batching, quantization, caching, and routing at the level of their cost consequences — enough arithmetic to price each one honestly, not enough to implement any of them from scratch. Three sibling lessons in this same path go the other direction, building the mechanisms this lesson's dollar figures assumed already existed: Serving Engines covers continuous batching, PagedAttention, and prefix caching (RadixAttention) as real systems, with the request-lifecycle detail this lesson's chapter 3 and chapter 5 levers only priced from the outside. Autoscaling Inference covers the fixed-cost side of chapter 4's TCO model in far more depth — cold starts, scale-to-zero economics, and the queueing theory behind picking a headroom target, rather than this lesson's flat $2,880/ month estimate. Speculative Decoding derives chapter 5's 2.77-tokens-per-round figure from first principles, including the rejection-sampling proof that makes its output-fidelity guarantee exact rather than approximate.

This lesson's sibling in the same wave, Speculative Decoding, deserves one more explicit callout: its rejection-sampling proof is exactly the kind of guarantee that let chapter 5 rank that lever ahead of distillation on the risk axis, even though it saves fewer dollars. Reading that proof in full is the natural next step for anyone who wants to verify chapter 5's claim rather than take it on faith.

Read in either order, but the two lessons teach genuinely different skills: those three build the mechanisms; this one prices what they're worth once built. A team that only reads the mechanism lessons can implement continuous batching correctly and still not know whether it was worth the engineering time at their actual traffic volume. A team that only reads this lesson can compute that batching is worth pursuing and still not know how to build it. Both halves are necessary for the decision this lesson opened with — the CFO question — to turn into a shipped, working, honestly-costed system.

One closing worked number, to leave with

Return, one last time, to the very first number this lesson derived: $21,000/month, chapter 0's naive API estimate for the baseline app. Across nine chapters, this lesson found a defensible path from that number down to $9,580/month (self-hosted, quantized, prefix-cached) — a 54.4% reduction, achieved entirely through techniques with little to no quality risk, none of which required negotiating a better API rate or accepting worse answers for users. That's the number this lesson was built to make derivable, not guessable: not “AI is expensive,” but a specific, checkable path from a specific starting bill to a specific, lower one.

“In God we trust; all others bring data.” — W. Edwards Deming. A monthly AI bill is not an act of nature. It's the output of a formula with knobs you can see, name, and turn — and every chapter of this lesson just handed you the formula.