Semantic Caching

Chapter 1: The Loop

Four steps. Embed the question, look up the nearest stored question, compare the similarity to a threshold, and either serve or generate. Everything in production semantic caching is a variation on those four steps, so it is worth walking one request through them with real shapes, real byte counts and real microseconds attached — because half the design pressure in this system comes from numbers you can only see if you write them down.

Step 1: what “embed” actually does to a string

An embedding model is a small transformer whose output you throw away except for one summary vector. Take a concrete model — bge-small-en-v1.5, 33 million parameters, 384 output dimensions, the workhorse choice for this job because it is small enough to run on the same CPU box as your web server. Feed it our question:

shapes, step by step
"how do i reset my password"
  -> tokenizer      : 9 token ids   [CLS] how do i reset my pass ##word [SEP]
  -> encoder        : (9, 384)      one hidden state per token
  -> pool           : (384,)        CLS vector, or the mean over tokens
  -> L2 normalize   : (384,)        every vector now has length exactly 1
  -> float32        : 384 x 4 = 1,536 bytes on the wire

Two of those lines are decisions, not facts, and both bite people.

Pooling. BGE-family models are trained so the [CLS] position carries the sentence meaning; E5- and GTE-family models are trained for mean pooling. Using the wrong one does not crash — it produces vectors that are merely worse, and “worse” in a semantic cache means a threshold you calibrated in Chapter 3 quietly stops meaning what you think it means. Read the model card, and write the pooling choice down next to the model name in your config.

Normalisation. After L2 normalisation every vector sits on the unit sphere, which means

— cosine similarity collapses into a plain dot product, 384 multiply-adds, no square roots at query time. That is not a micro-optimisation, it is what lets an ANN index use raw inner product as its distance and still be measuring cosine. If you forget to normalise, your index is ranking by dot product, which rewards long vectors, which means the entry that wins is systematically the one whose question was longest. This is a real bug that ships often and it looks like “the cache always returns that one rambling question.”

An aside that matters: many models want a prefix

Several strong retrieval encoders are trained asymmetrically — a question and a document get different instruction prefixes, because in retrieval those are different roles. In a semantic cache both sides are questions, so both sides must get the same treatment:

python
# WRONG: the stored side used the query prefix, the lookup side did not.
# Similarities drop by 0.02-0.05 across the board and your calibrated
# threshold silently becomes far too strict.
stored = model.encode("Represent this sentence for searching: " + q1)
lookup = model.encode(q2)

# RIGHT: one function, used on both write and read paths.
def embed(q):
    return model.encode(PREFIX + q, normalize_embeddings=True)

The general rule, and it will come back in Chapter 4: whatever function produced the stored vectors must be byte-for-byte the function producing the lookup vector. Same model, same revision, same pooling, same prefix, same normalisation. The moment those diverge, every stored vector is measured against a slightly rotated ruler.

Step 2: the index, and why it fits in RAM

Suppose the cache holds 214,000 entries. The raw vectors are

An HNSW index — the standard graph-based approximate nearest neighbour structure — adds a navigable graph on top. With the usual setting of M = 16 neighbours per node, the bottom layer stores up to 2M = 32 neighbour ids of 4 bytes each, and the sparse upper layers add roughly another 6%:

Total: about 344 MB. That fits, with enormous headroom, on the cheapest box you would ever put in front of an LLM. This is worth internalising because it sets the mental model: the index is not the expensive part of this system, and you should never trade correctness for index size until you are two orders of magnitude past this.

A search visits a small number of nodes. With ef_search = 64 the graph walk touches on the order of ef × log₂(N) candidates, and log₂(214,000) ≈ 17.7, so

Each comparison is 384 multiply-adds, so the arithmetic is 1,150 × 384 ≈ 441,600 operations — under half a megaflop, which a single core does in about 22 microseconds. The measured latency is around 1.8 milliseconds, roughly eighty times that. The gap is not computation; it is memory. The graph walk jumps to unpredictable addresses, so almost every step is a cache miss out to DRAM. That is why HNSW latency scales with the number of hops rather than the flop count, and why the standard tuning lever is ef_search and not the vector width.

Step 3 and 4: the decision, and the write-back

The search returns a neighbour and a similarity. Compare to τ. Serve or generate. The subtlety is entirely on the write path, and there are three decisions in it that people skip:

What do you store as the key text? The user’s raw question, or a cleaned version? Store the raw one for debugging and the cleaned one for the vector — and store both, because in Chapter 7 the only way you will diagnose a false hit is by reading the two questions side by side.

Do you store every answer? No. An answer that was a refusal, an error, a timeout, or an “I don’t have that information” must never be written. Cache a refusal once and you have built a machine that refuses that question forever, long after the underlying cause is fixed. This gate is five lines of code and it is the highest-value five lines in the whole system.

What else goes in the row? At minimum: the vector, the question text, the answer, the creation timestamp, the model ids that produced both the vector and the answer, and — if this is a RAG system — the ids and content hashes of every document that was retrieved. Chapter 4 is entirely about why that last field is not optional.

python
# the row you actually store
{
  "vector":        v,                          # float32[384], unit norm
  "question_raw":  "hi, i need to reset my password!!",
  "question_key":  "i need to reset my password",  # what was embedded
  "answer":        "Open Settings, choose Security, then...",
  "created_at":    1755000000,
  "embed_model":   "bge-small-en-v1.5@rev3f2c",   # see chapter 4
  "gen_model":     "assistant-v4.2",
  "prompt_hash":   "9a1c4f...",                  # see chapter 5
  "doc_hashes":    ["kb/auth/reset#a17f", "kb/auth/2fa#0c93"],
  "hits":          0,                          # for eviction and for chapter 8
}

Push the entry count to two million and the index grows to about 3.2 GB while the search latency rises by roughly one hop — log₂(2,000,000) = 21 versus 17.7, so about 18% more work. That is the property that makes this architecture pleasant: lookup cost is logarithmic in cache size while the value of the cache is linear in it. Nothing else in your serving stack behaves that well.

The latency ledger, honestly

A hit is roughly 207 times faster than a generation. A miss costs three tenths of one percent extra. We will turn that asymmetry into a break-even calculation in Chapter 6, but you can already see where it lands: the arithmetic is not close.

Cosine by hand, so the numbers in Chapter 2 mean something

Real vectors have 384 dimensions and nobody can hold that in their head. Do it in four. Let the question “how do I reset my password” be

and compare it against three candidates. First, a genuine paraphrase — “I forgot my password, how do I change it”:

Second, a same-topic-different-answer neighbour — “how do I reset my API key”:

Third, a near-duplicate — the same sentence with a typo:

0.9991, 0.9414, 0.8171. Three regimes, and a threshold is a horizontal line drawn somewhere among them. In this toy the line practically draws itself — anywhere between 0.85 and 0.93 separates the paraphrase from the impostor cleanly.

Chapter 2 is about why that never happens in real data.

Chapter 2: The Threshold

Someone on your team is going to write THRESHOLD = 0.92 and move on. This chapter is about what that line of code actually promises, and why the promise it seems to make — “the questions are 92% the same” — is not a thing cosine similarity can say.

First: 0.92 is not “92% similar”

Cosine ranges from −1 to 1, so the instinct is to read 0.92 as “92 percent of the way to identical.” Modern sentence encoders make that reading badly wrong, because they do not use the whole range. They are trained with a contrastive objective on a temperature that compresses everything into a narrow band near the top.

Measure it on your own logs and you get something like this — these are the numbers from ten thousand random pairs drawn from our support traffic, encoded with bge-small-en-v1.5:

Read the first row again. Two questions with nothing whatsoever in common score 0.681. That is your floor. The entire usable range of this model on this traffic is roughly 0.68 to 1.00 — about 0.32 wide. Setting τ = 0.92 is not asking for “92% agreement”; it is asking for the top

of the usable band. And because the density is not uniform — most pairs pile up in the middle — in practice τ = 0.92 is somewhere around the 96th percentile of all pairs. The number 0.92 has no meaning outside the specific model, the specific pooling, the specific prefix and the specific traffic that produced that table. A threshold is not portable. Copying one from a blog post is copying someone else’s percentile.

Why negation and antonyms score so high

This is not a bug in the model, it is a consequence of what the model was trained to do. Retrieval encoders are trained so that a query lands near the document that answers it. “How do I enable 2FA” and “how do I disable 2FA” are answered by the same help-centre page. From the training objective’s point of view they should be neighbours — that is the model working correctly for retrieval.

But a semantic cache is not doing retrieval. Retrieval wants “which documents are relevant?” and the answer is a ranked list that a downstream model will read and reason over. A cache wants “is this the same question?” and the answer is a hard yes/no that gets served directly to a human. You are borrowing a similarity function trained for a strictly weaker relation and using it as an equivalence test.

Concretely, the words that flip an answer are almost always short function words or single content-word swaps, and both are exactly what a sentence embedding compresses away:

Now try to draw a horizontal line through that table that keeps the two “yes” rows and drops the five “no” rows. There isn’t one. 0.967 and 0.968 sit one thousandth apart and disagree about the answer. This is the central difficulty of semantic caching and no amount of threshold tuning resolves it.

Naming the two failure modes

Fix the vocabulary now, because Chapters 3 and 8 depend on it.

The asymmetry is the design principle. In most caching problems — a CDN, a database query cache — a hit is correct by construction because the key is exact, so the only trade-off is memory against hit rate. Semantic caching breaks that guarantee. It is the first cache most engineers meet where a hit can be wrong, and the habits from every other cache are actively harmful here.

What a false hit looks like when it lands

Make it concrete, because “wrong answer” is abstract until you see one.

production trace — false hit at τ = 0.92
incoming  : "how do i disable 2FA on my account"
nearest   : "how do i enable 2FA on my account"        sim = 0.9731
decision  : 0.9731 >= 0.92  ->  SERVE
served    : "Go to Settings, choose Security, tap Two-Factor
             Authentication, and follow the prompts to add your
             phone. You'll receive a code each time you sign in."
latency   : 11 ms
logged as : cache_hit=true, status=200, user_visible_error=false

Every system-level signal says success. Fast, no error, cache working. The user, who wanted to turn two-factor off, has been told how to turn it on. If they follow the instructions they will end up more locked in, not less. And the only place this shows up in your telemetry is a support ticket three days later, filed against a completely different part of the product.

Count it by hand

Five pairs share an answer (1, 4, 6, 9, 12) and seven do not. At τ = 0.92, everything at 0.920 or above is served. Going down the table: 0.941 (yes), 0.912 (below, missed), 0.973 (no), 0.958 (yes), 0.934 (no), 0.967 (yes), 0.968 (no), 0.897 (below), 0.981 (yes), 0.986 (no), 0.681 (below), 0.929 (yes). So

Now raise it. At τ = 0.97 only 0.973, 0.981 and 0.986 survive — that is one correct (pair 9) and two wrong (pairs 3 and 10):

You tightened the threshold by five hundredths, gave up four fifths of your hits — and your precision got worse. Not marginally worse. It fell by twenty-two points.

Then what is the safety dial?

Three things, and none of them is τ. We build all three in the chapters that follow, but here is the shape so the arc is visible:

A cross-encoder is worth a sentence of explanation now, because it is the piece that does the heavy lifting. An embedding model is a bi-encoder: it reads each sentence separately and squeezes each into one vector. Once compressed, the vectors are compared — and the comparison has no access to anything the compression discarded. A cross-encoder reads both sentences in one forward pass, so its attention can put “enable” and “disable” in the same window and notice they disagree.

The reason you cannot just use a cross-encoder for everything is arithmetic: a bi-encoder lets you precompute 214,000 vectors once and compare with a dot product, while a cross-encoder would need 214,000 forward passes per query. The standard resolution is exactly the one we will build — bi-encoder to retrieve, cross-encoder to verify, so the expensive model runs on one candidate rather than all of them.

Concept → realization

python
# The naive version everybody ships first.
if sim >= 0.92:
    return nbr.answer                    # 55.6% precision on the table above

# The version this lesson builds. Note the ORDER: cheapest test first,
# and the expensive one never runs on a query that already failed.
if sim >= TAU:                             # 1.8 ms, already paid
    if not decision_tokens_agree(q, nbr.question):   # 0.1 ms
        return generate(q)
    if cross_encoder(q, nbr.question) < 0.60:        # 4 ms
        return generate(q)
    return nbr.answer                    # 99.6% precision, chapter 3

Chapter 3: Calibration

Twelve hand-picked pairs made the point. They cannot set your threshold, because twelve is not a sample and “pairs I found interesting” is not a distribution. This chapter builds the real thing: a labelled set, a sweep, and a decision rule that comes from a stated budget rather than from taste.

Step 1: build the labelled set from your own logs

The set must be drawn from the traffic the cache will actually see, because everything in Chapter 2 was model-specific and traffic-specific. Here is the procedure that takes about half a day.

python
# 1. Sample requests uniformly from a recent window.
sample = random.sample(load_requests(days=14), 4000)

# 2. For each one, find what the cache WOULD have returned.
#    Build the index from everything BEFORE that request's timestamp,
#    or you will leak the future into the past and inflate every number.
rows = []
for r in sample:
    idx = index_as_of(r.ts)
    nbr, sim = idx.search(embed(r.question), k=1)
    rows.append({"q": r.question, "nbr": nbr.question,
                 "sim": sim, "stored_answer": nbr.answer})

# 3. Label. One question, asked of a human or a strong LLM judge:
#    "Would the stored answer be a CORRECT and COMPLETE answer to q?"
#    Not "are these similar". Not "are these related". Correct and complete.

Three things about step 3 decide whether the whole exercise is worth anything.

Label the answer, not the question. “Are these questions similar?” is a question about wording and every annotator answers it differently. “Would this stored answer be correct and complete for this question?” is a question about the product and has one right answer. Change the prompt and inter-annotator agreement typically jumps from the sixties to the nineties.

Do not label pairs, label requests. Each row is one live request and the one neighbour the cache would actually have returned. That makes every downstream number directly interpretable as a rate over traffic. If you label arbitrary pairs instead, your precision number describes a population your cache never sees.

Over-sample the danger. Uniform sampling gives you very few minimal-edit near-misses, which means your estimate of the thing that hurts most has the widest error bars. So sample uniformly for the rates, then deliberately mine a second stratum — pairs with high cosine and low token overlap, or pairs differing by exactly one content word — and label those too, weighting them back down when you compute rates.

Step 2: look at the three populations you got

On our support traffic the 4,000 labelled requests split like this:

Positives are 1,600 of 4,000 = 40% — exactly the head share from Chapter 0, which is the consistency check that tells you the sampling was done right.

And look at the hard negatives: μ = 0.960, above the positives’ 0.945. That single inequality is the whole difficulty, made numerical. In a well-behaved classification problem the negatives sit below the positives and a threshold separates them. Here one population of negatives sits on top of the positives, with a tighter spread. No horizontal line separates a distribution from one that dominates it.

Step 3: sweep τ and do the arithmetic

Treat each population as approximately normal and read off what fraction clears the bar. The fraction of a normal above τ is Φ((μ − τ) ÷ σ), where Φ is the standard normal cumulative — the same table from any statistics text.

Do τ = 0.92 by hand, all three populations.

Positives. How many standard deviations is 0.92 below the mean of 0.945?

The threshold is one σ below the mean, so the fraction above it is Φ(1.00) = 0.8413.

Easy negatives.

Hard negatives.

Twenty from the two thousand easy negatives; three hundred and forty-two from the three hundred and fifty hard ones. Ninety-four percent of your wrong answers come from nine percent of your negatives. Assemble:

There is the 42.7% and the 9.1% promised at the end of Chapter 0. One request in eleven gets a wrong answer.

The full sweep

Read the precision column top to bottom: 78.1, 79.6, 78.8, 75.4, 71.2, 69.9, 70.7. It is flat, and if anything it drifts down. You can sweep τ across its entire useful range and you cannot buy precision with it. What you can buy is a smaller cache: recall falls from 99.5% to 3.6%, a factor of twenty-eight.

Step 4: add the guards, and watch precision finally move

Now put the two mechanisms from Chapter 2 in front of the decision and redo the arithmetic. Start at τ = 0.93 — deliberately looser than 0.97, because the guards, not the threshold, are going to do the safety work.

At τ = 0.93, before any guard:

So 1,161 true and 340 false pass the threshold. Now the first guard.

Guard 1 — the decision-token check. Build a set of tokens that flip answers in your domain: negations (not, no, never, isn't, don't, without), polarity verbs (enable/disable, add/remove, cancel/renew, activate/deactivate), platform words (ios, android, web, desktop), plan words (free, pro, enterprise), period words (monthly, annual, yearly, per year), and every numeral. If the symmetric difference of the two questions’ decision tokens is non-empty, refuse the hit regardless of similarity.

python
DECISION = NEGATIONS | POLARITY | PLATFORMS | PLANS | PERIODS

def decision_tokens_agree(a, b):
    ta = {t for t in tokens(a) if t in DECISION or t.isdigit()}
    tb = {t for t in tokens(b) if t in DECISION or t.isdigit()}
    return ta == tb            # symmetric difference must be empty

Measured on the labelled set, this rejects 88% of hard-negative hits (they are minimal edits by construction, so a decision token is exactly what differs), 20% of easy-negative hits, and — the cost — 3% of true hits, because some honest paraphrases do legitimately swap a listed word (“turn off 2FA” versus “disable 2FA”). Apply it:

False hits fell from 340 to 49 — a 7× reduction — for a 3% haircut on true hits. Nothing you can do with τ comes within an order of magnitude of that trade.

Guard 2 — the cross-encoder. A 22-million-parameter MiniLM cross-encoder, fine-tuned on a few thousand of your own labelled pairs, reading both questions in one pass. Score it and require 0.60. Measured: it catches 92% of the remaining false hits and costs 7% of true hits.

The final numbers:

Compare against the threshold-only configurations. At τ = 0.92 you had 42.7% hit rate and 9.1% wrong. At τ = 0.97 you had 9.1% hit rate and 2.7% wrong. With guards at τ = 0.93 you have 26.3% hit rate and 0.10% wrong — ninety-one times fewer wrong answers than the 0.92 config while keeping nearly two thirds of its hits.

Step 5: choose from a budget, not from a feeling

The last step is the one that turns this from an analysis into a decision. Write down, before you look at any curve, the sentence: “we will tolerate at most F wrong answers per thousand requests.” Then pick the configuration with the highest hit rate that satisfies it.

F is a product decision and it varies enormously by surface:

Our support assistant takes F = 1 per 1,000. The guarded config delivers 1.0 per 1,000 — exactly at budget — at a 26.3% hit rate. The τ-only configs deliver 91, 99 and 27 per 1,000 at τ = 0.92, 0.90 and 0.97. Not one of them is admissible under a stated budget, which is precisely why the budget must be stated first. Stated afterwards, it gets adjusted to whatever the system happens to do.

Slide the hard-negative count to zero and watch precision behave the way a textbook says it should — rising smoothly with τ, guards barely needed. That is the world people think they are in. Slide it to 800 and no threshold is admissible at any budget without the guards. Your job in the first week of this project is to find out which world your traffic lives in, and that is a labelling exercise, not an engineering one.

How big does the labelled set need to be?

You are estimating a rate, so the standard error of a proportion applies:

With n = 4,000 and a hit rate near p = 0.26:

Fine for hit rate. But the number that matters is the false-hit rate at p = 0.001, and it is estimated from four events. Four. Its standard error is

which is half of the quantity itself. Your point estimate of 0.10% has a two-sigma interval of roughly 0% to 0.2%. You genuinely cannot tell 0.05% from 0.15% with 4,000 labels, and no amount of care in the labelling fixes that — it is a counting limit. Two consequences, both practical:

One. Do not tune the last hundredth of τ against a number this noisy. Pick the configuration whose mechanism you believe in and whose estimate clears the budget by a comfortable margin.

Two. Get the real estimate from production, continuously, by shadow sampling — which is exactly what Chapter 8 builds, and now you know why it is not optional.

Chapter 4: Invalidation

The cache is calibrated. It answers 26.3% of requests in twelve milliseconds and it is wrong once every thousand requests. Ship it, and then, three weeks later, someone changes the price of the Pro plan.

Nothing in the system notices. The stored answer for “how much is the pro plan” still says $29/month. It will keep saying $29/month, correctly and instantly, to every one of the four hundred people a day who ask — until something removes it. A semantic cache is a frozen copy of a model’s opinion at one moment, and the world does not hold still.

There are exactly three reasons a stored answer stops being correct, and they need three different mechanisms.

Mechanism 1: time-to-live, and what it actually costs

A TTL says: delete this entry L days after it was written. It is the crudest mechanism and the one you should reach for first, because it needs no coordination with anything.

The question is how to choose L. Do it from a model rather than a vibe. Suppose the fact underlying an answer changes at random, at an average rate of once every T days — pricing pages in our product change about six times a year, so T = 365 ÷ 6 = 60.8 days. Treat changes as a Poisson process with rate λ = 1÷T. An entry written at time 0 lives until L. The probability it has already gone stale by age t is 1 − e^−λt, so the expected amount of its life spent stale is

and the fraction of served responses that are stale is that divided by L. Work L = 7 days:

So with a one-week TTL, about one in eighteen pricing answers you serve is out of date. Now L = 1 day:

A seven-fold reduction in staleness. The obvious next question is what it cost you in hit rate — and the answer is the most useful non-obvious fact in this chapter.

The TTL asymmetry: short TTLs are nearly free where the money is

An entry has to be created before it can be hit. If an intent receives r requests per day and the TTL is L days, then in each L-day cycle the first request regenerates and the rest hit:

Put four cases side by side.

Read the top row. Going from a week to a day on your busiest intent costs you two tenths of one percentage point of hit rate and cuts staleness sevenfold. On the head — which is 40% of your traffic and essentially all of your savings — short TTLs are close to free.

Read the bottom row. On a cold intent asked once every two days, a one-day TTL means the entry has always expired by the time the second request arrives. Hit rate is exactly zero and you are paying storage and lookup cost for nothing.

Better still, make L a function of what the answer is about. Volatility is a property of the content, and you usually know it:

python
TTL_BY_CLASS = {
    "pricing":      3600,        # 1 hour   - changes without warning
    "availability": 300,         # 5 min    - inventory, status, capacity
    "policy":       86400,       # 1 day    - legal review gates changes
    "how_to":       604800,      # 7 days   - UI changes are quarterly
    "conceptual":   2592000,     # 30 days  - "what is an API key" is timeless
}
# Classify once, at write time, from the retrieved documents' section
# tags - not from the question, which is exactly the ambiguous thing.
ttl = TTL_BY_CLASS.get(doc_class(retrieved_docs), 86400)

Note where the classification comes from: the documents, not the question. The question “how much does this cost” is ambiguous; the fact that the answer was built from kb/pricing/plans.md is not.

Mechanism 2: version the producers, because the ruler can change

Chapter 1 ended on a rule: the function that produced the stored vectors must be identical to the one producing the lookup vector. A TTL cannot enforce that, because the problem is not age — it is that the measuring instrument changed.

Three producers can change under you, and each breaks something different.

The embedding model. Swap bge-small (384 dims) for bge-base (768 dims) and the index will not even load — a loud, immediate, survivable failure. The dangerous version is subtler: the same model at a new revision, same dimension, slightly different geometry. Nothing errors. Similarities shift by a few hundredths across the board, and your calibrated τ — which Chapter 3 established is a percentile, not a physical constant — silently starts meaning something else. If it drifts one way you lose half your hits; if it drifts the other you double your false hits.

Hosted embedding APIs make this worse by offering unversioned endpoint names. If your config says "text-embedding-small" with no revision pin, you have handed a third party write access to your cache’s correctness.

python
# the namespace every cache row and every lookup is scoped by
NAMESPACE = hashlib.sha256("|".join([
    "bge-small-en-v1.5", "rev-3f2c9a",   # embedder + PINNED revision
    "cls", "l2",                        # pooling + normalization
    PREFIX,                             # the instruction prefix, verbatim
    "assistant-v4.2",                  # generator model
    system_prompt_hash,                 # chapter 5
    "kb-2026-08-11",                   # retrieval corpus snapshot
]).encode()).hexdigest()[:16]

Change any component and the namespace changes, which means every lookup misses, which means the cache rebuilds itself from scratch. That is not a bug — it is the correct behaviour, expressed as a default rather than as a runbook step someone might forget.

Sizing the rebuild: re-embedding 214,000 stored questions at 6 ms each is

or about 2.7 minutes on eight threads. That is cheap enough that you should prefer a full re-embed over any clever migration — and cheap enough that there is no excuse for leaving stale-geometry vectors in place.

The generator. Upgrade the LLM and every cached answer is still the old model’s work. Your evaluation now measures a blend of two models in an unknown ratio that drifts as the cache refills, and that ratio is different in every environment. Put the generator id in the namespace too and take the one-day cost of a cold cache after each model change.

The retrieval corpus. Which is the third mechanism, and it deserves its own section.

Mechanism 3: provenance, the one RAG makes mandatory

In a retrieval-augmented system a cached answer is not a function of the question. It is a function of the question and the documents that were retrieved:

A TTL invalidates on the first argument’s age. It is blind to the other k. So when someone edits kb/auth/reset.md at 09:14, every cached answer built from that document became wrong at 09:14 — and your one-day TTL will keep serving them until tomorrow morning.

The fix is to store the provenance and index it backwards. Every cache row already carries doc_hashes from Chapter 1; add the inverted map.

python
# forward, on write: which docs did this answer come from
row["doc_hashes"] = [d.id + "#" + d.content_hash for d in retrieved]

# inverted, also on write: which entries touch each doc
for d in retrieved:
    redis.sadd(f"docidx:{d.id}", row_id)

# on any document change, from your ingestion pipeline's webhook
def on_document_changed(doc_id):
    victims = redis.smembers(f"docidx:{doc_id}")
    index.delete_many(victims)          # vectors
    redis.delete(*victims)              # answers
    redis.delete(f"docidx:{doc_id}")

Note that the hash is over content, not just the id. A document that is re-indexed without changing does not invalidate anything, which matters because most re-ingestion runs touch everything.

Size the bookkeeping. With 214,000 entries retrieving 4 documents each:

As Redis sets keyed by document id, that is under 60 MB — a rounding error against the 344 MB index. Now size the churn. If a popular document is referenced by 3,000 entries and your team edits 20 documents a day:

That is an effective TTL of roughly 3.6 days imposed by document churn alone — and it is correct churn, precisely targeted at the entries that actually went bad, instead of the blind expiry a global TTL would apply. This is also the number that tells you whether a semantic cache in front of RAG is worth building at all: if your corpus turns over faster than your traffic repeats, it is not.

Eviction is not invalidation

One vocabulary distinction that trips people. Invalidation removes entries because they are wrong. Eviction removes entries because you are out of room. They have different triggers and different correct policies, and conflating them produces a cache that keeps wrong answers because they are popular.

For eviction, least-recently-used is close to optimal here for the same reason it is everywhere: the access pattern is Zipf, so recency predicts future use. But bound the cache by entry count rather than by bytes and give the head a floor — if your 200 head intents ever get evicted by a burst of tail traffic, you lose essentially all of the value while keeping all of the cost.

And one small thing worth doing: on every hit, do not refresh the TTL. A cache entry’s age should measure how long ago the answer was produced, not how recently it was popular. Sliding expiration on a semantic cache means your most-served answer is the one most likely to be stale, which is the exact inversion of what you want.

Chapter 5: Keys

Two users type the same six words, one second apart:

two requests, byte-identical questions
user 88213 (Enterprise, en-GB, iOS) : "can i export my data to csv"
user 41007 (Free,       en-US, web) : "can i export my data to csv"

Cosine similarity: 1.000. Exactly, unavoidably one — it is the same string, so it is the same vector, so no threshold and no verifier in the world will separate them. And the correct answers are different: CSV export is an Enterprise feature, and the button is in a different place on iOS than on the web.

This is a failure the entire apparatus of Chapters 2 and 3 cannot touch, because the information that makes the answers differ was never in the text. It lives in the request context. The only place to put it is the key.

The rule

Mechanically, the key becomes a namespace — a partition id computed from context — and vector search runs within a namespace rather than across the whole index. Most vector stores support this as a metadata filter applied during the graph walk; if yours does not, run physically separate indexes.

python
def namespace(req):
    return hashlib.sha256("|".join([
        MODEL_NAMESPACE,             # chapter 4: embedder, generator, corpus
        req.locale,                  # "en-GB"  - currency, spelling, legal text
        req.plan,                    # "free"   - entitlements gate the answer
        req.platform,                # "ios"    - the UI differs
        req.persona,                 # "concise" - system prompt variant
        str(temperature_bucket(req.temperature)),
    ]).encode()).hexdigest()[:16]

nbr, sim = index.search(v, k=1, filter={"ns": namespace(req)})

Walking the candidate fields

Temperature, and what caching does to randomness

If you generate at temperature 0.9 for variety and then cache the result, the second user does not get a different sample — they get a byte-identical replay of the first user’s. Your cache has silently converted a stochastic endpoint into a deterministic one for 26.3% of traffic.

Sometimes that is fine, even good: consistency across users is often a feature in support. Sometimes it is the entire product, as in a creative-writing tool where two people asking for “a name for my cat” must not receive the same name. The decision is a product decision, and the failure mode is that nobody makes it.

Bucket temperature into the key so at least the different regimes cannot mix, and add an explicit policy flag:

python
def temperature_bucket(t):
    if t <= 0.05: return "det"       # deterministic: caching is free
    if t <= 0.5:  return "low"       # caching flattens mild variation
    return "high"                      # caching removes variety - decide!

CACHEABLE = {"det": True, "low": True, "high": SETTINGS.cache_creative}

Conversation history: the field that looks impossible

“And what about the second one?” has no meaning on its own. Two obvious options and one good one.

Option A: hash the whole history into the key. Correct, and useless — every conversation is unique, so every namespace has one member, so the hit rate is zero.

Option B: ignore history and cache on the last turn. Fast, and a false-hit factory: two users asking “and the second one?” after entirely different first turns get each other’s answers.

Option C: cache the resolved question. Most RAG stacks already have a query-rewriting step that turns a follow-up into a standalone question before retrieval. Cache on that. “And what about the second one?” becomes “what are the rate limits on the Pro plan?” — which is a head intent, shared across thousands of users, and cacheable.

python
# the rewrite you already run for retrieval is also the cache key
standalone = rewrite(history, question)   # "and the second one?" ->
                                          # "what are the rate limits on the Pro plan?"
v = embed(standalone)
nbr, sim = index.search(v, k=1, filter={"ns": namespace(req)})

This is the highest-leverage single change in this chapter. It moves multi-turn traffic — which is most traffic in a chat product and is entirely uncacheable in its raw form — into the same key space as single-turn traffic. The rewrite already exists; you are only pointing the cache at its output instead of its input.

The partition tax, with numbers

Every field you add multiplies the namespace count and divides the traffic. Use the Chapter 4 formula — hit rate = 1 − 1÷(rL) — with a head intent at 400 requests/day and a 1-day TTL.

The first five rows are a survivable tax: 99.75% down to 82.0% on your hottest intent. The last row is annihilation. At 288,000 namespaces the average namespace sees one request every two years, so nothing ever hits, and you are running an index, an embedder and a verifier to achieve nothing whatsoever.

python
# the write-path gate that makes tenant-free namespaces safe
def is_cacheable(req, retrieved, answer):
    if any(d.source == "tenant_private" for d in retrieved):
        return False                     # answer contains someone's data
    if req.tools_called:
        return False                     # answer reflects live state
    if answer.is_refusal or answer.is_error:
        return False                     # chapter 7, failure 4
    if contains_pii(answer):
        return False                     # belt and braces
    return True

Templating: how to un-partition a field

Plan tier looks like it must be in the key. Often it does not have to be, and removing it is worth twelve namespaces.

Ask what actually differs between the Free and Pro answers to “what is my rate limit?” Usually one number. So cache the answer with a slot and fill the slot at serve time from the request context:

python
# stored once, shared by every plan
answer = "Your plan allows {{rate_limit}} requests per minute. " \
         "You can see current usage under Settings, API."

# filled per request, from context the cache never needed to partition on
served = render(answer, rate_limit=req.entitlements.rate_limit)

Four plan partitions collapse into one entry. Applied to the table above, dropping plan takes 72 namespaces to 18 and the head intent’s hit rate from 82.0% back to 1 − 1÷(400÷18) = 95.5%.

The catch is that templating is only safe when the structure of the answer is plan-independent. If Free users get “that feature is not available on your plan, here is how to upgrade” and Pro users get a four-step how-to, there is no shared template and the field genuinely belongs in the key. Getting this wrong produces answers that are grammatically fine and semantically absurd — the classic symptom is a Free user being told the exact rate limit of a feature they cannot access.

Chapter 6: Architecture

You know what the cache decides and how it stays honest. Now decide where it lives, what it is allowed to cost, and whether the whole thing pays for itself. The last question has a clean numeric answer and it is not close.

Two placements

Cache-aside. Your application owns the lookup. It embeds, searches, decides, and on a miss calls the model itself and writes back. The cache is a library your code calls.

Read-through (proxy). A gateway sits between your application and the model API and speaks the model API’s own protocol. Your application changes one base URL and gets caching for free.

The row that decides it in practice is the third. A semantic cache is a new component with a vector index, an embedding model, a verifier and an eviction policy — that is a lot of new failure surface to put in the hard path of every model call in the company. Cache-aside degrades to “call the model,” which is exactly what you did last week.

python
# the degradation contract, and it is the whole point of cache-aside
try:
    hit = semantic_cache.get(q, ctx, timeout_ms=25)
except (CacheTimeout, CacheDown):
    metrics.incr("cache.unavailable")
    hit = None                # a dead cache is a slow day, not an outage

Note the timeout. Twenty-five milliseconds is generous against a 12 ms hit path and tiny against a 2,600 ms generation. Without it, a cache that has become slow rather than dead is worse than no cache at all.

The latency budget

Write the budget down as a fraction of what you are replacing, because that framing prevents most bad arguments.

The miss overhead is 7.8 ÷ 2,600 = 0.30% of a generation at p50 and 19 ÷ 7,400 = 0.26% at p99. There is no meaningful latency argument against trying the cache.

One real caveat: the embedder shares a CPU with your web server. Under load its p99 climbs, and it climbs at exactly the moment you are least able to absorb it. Either give it its own small service with its own saturation metric, or batch across concurrent requests with a 5 ms window — batching eight queries costs about 11 ms total instead of 48 ms serially, because the model is memory-bandwidth-bound at batch 1.

The thundering herd, and the one line that fixes it

A new intent goes viral — a status-page incident, a launch, a bug everyone hits at once. Fifty requests per second arrive for a question with no cache entry. Generation takes 2.6 seconds, so before the first answer is written back you launch

— 130 × $0.01065 = $1.38 spent to learn one answer, plus a load spike on the model API at the exact moment your product is already having a bad day. The fix is singleflight: the first miss takes a short-lived lock on the namespace-plus-rounded-vector; everyone else waits on it.

python
def get_or_generate(q, ctx):
    hit = cache.get(q, ctx)
    if hit: return hit
    lock = f"sf:{namespace(ctx)}:{lsh_bucket(embed(q))}"
    if redis.set(lock, "1", nx=True, px=8000):        # I am the leader
        try:
            ans = llm.generate(q); cache.put(q, ctx, ans); return ans
        finally:
            redis.delete(lock)
    for _ in range(40):                              # follower: wait 40 x 100 ms
        time.sleep(0.1)
        hit = cache.get(q, ctx)
        if hit: return hit
    return llm.generate(q)                          # leader died; do it yourself

The lsh_bucket is doing quiet work: a locality-sensitive hash of the vector, so 130 differently-worded versions of the same viral question take the same lock. Locking on the exact string would give you 130 leaders and no benefit at all. Note the last line too — a follower that waits forever turns a cache miss into a hung request, which is a far worse failure than a duplicated generation.

The break-even calculation

Now the question finance will ask. Two costs per request, always paid:

plus the ANN search, which is CPU on a box you already have — call it zero at the margin and fold the box into fixed cost. One saving, paid only on a hit:

So with hit rate h, the expected cost per request is

and the cache is worth running as soon as C(h) < $0.01065, which happens when

One hit in 24,200 requests pays for every embedding you will ever compute. Do the same for latency: you spend 7.8 ms on every request and save 2,600 ms on a hit, so

One hit in 333 requests makes the cache latency-positive. Both break-evens are three to four orders of magnitude below the 26.3% you actually achieve. The marginal economics of semantic caching are not a judgement call.

Fixed costs, and the honest payback

Marginal economics are not the whole story. The cache needs a box:

The hit rate at which the cache pays for its own infrastructure:

And the actual result at h = 26.3%:

Build cost is real too. Say three engineer-days for the first version and the calibration — roughly $3,600 fully loaded. Payback:

Which is the honest version of the pitch: not “caching saves 26%,” but “this pays back in about five weeks and then returns roughly $39,000 a year, and the ongoing cost is one dashboard and a quarterly recalibration.”

Drag the request count down to 50,000 a month and watch the ledger flip: at that volume the cache saves $140 and costs $110, and three engineer-days will never pay back. Semantic caching is a volume play. Below roughly 200,000 requests a month, on these prices, the honest recommendation is an exact-match cache and a note in the backlog.

The one place the arithmetic changes: prompt caching

Most providers now offer prompt caching — a discount on repeated input prefixes, usually around 90% off cached input tokens. That is a different mechanism (exact prefix match, provider-side, no semantics) and it is nearly free to turn on, so turn it on first.

It changes your break-even by changing C_gen. With a 1,500-token shared prefix at 90% off, the input cost falls from $0.00540 to

Your bill drops to $7,920 and the semantic cache’s savings drop with it to 1,200,000 × 0.263 × $0.00660 = $2,082/month. Still a 3.5× return on the $110, still paying back the build in under two months — but the two mechanisms are multiplicative and you should size the semantic cache after enabling prompt caching, not before. Sizing it before is how projects get approved on numbers that no longer exist by the time they ship.

Chapter 7: Failure Gallery

Seven ways a semantic cache goes wrong in production. For each one: what the user sees, what is actually happening, the test that finds it, and the fix. Read this chapter before you ship, not after.

They are ordered by how much damage they do, not by how likely they are.

1 · Cross-user leakage

Mechanism. The question text is nearly identical across all users — cosine 0.99 or exactly 1.000 — while the answer is tenant-specific. If tenant is not in the key and the write path did not refuse to cache a private answer, the first user’s answer becomes everyone’s answer. Every protection built in Chapters 2 and 3 is powerless here, because there is nothing wrong with the similarity judgement: the questions really are the same question.

Why it survives review. It is invisible in staging, where there is one test tenant. It is invisible in metrics, because it looks like a fast, successful hit. And it is invisible in eyeball testing, because you only see it if you are the second user.

Test. An automated one, in CI:

python
def test_no_cross_tenant_leak():
    a = ask(tenant="A", q="summarise my last invoice")
    b = ask(tenant="B", q="summarise my last invoice")
    assert a != b                       # identical text = leak
    assert TENANT_A_SECRET not in b
    assert metrics.last("cache_hit") is False

Fix. The write-path gate from Chapter 5 — refuse to cache any answer whose retrieval touched a private index, called a tool, or contains PII. Do not rely on tenant-in-the-key, because that both destroys the hit rate and fails open the day someone forgets to pass the tenant.

Severity. This is a data-protection incident, not a quality bug. It gets disclosed. Treat the write-path gate as a security control with a test that cannot be deleted.

2 · Stale answers

Symptom. The assistant confidently quotes a price, policy or limit that changed. Nobody can reproduce it — opening the source page shows the new text.

Mechanism. Chapter 4, all three of them: TTL too long for the volatility class, or a document changed with no provenance index, or the generator was upgraded and the old model’s answers are still being served.

Test. Ship a canary. Pick five facts you control, put a version marker in the source document, and have a synthetic probe ask about each one every fifteen minutes. Alarm when the served answer lags the document by more than the class TTL. This is the only failure in the gallery that is cheap to detect continuously and automatically.

Fix. Provenance-keyed invalidation plus per-class TTLs. And log the age of every served entry so “p99 age of served answers” is a number on the dashboard rather than a shrug.

3 · Threshold drift as the query mix shifts

Symptom. The cache was calibrated in March at 26% hit rate and 0.1% false hits. In June the hit rate is 34% and nobody changed anything. Everyone is pleased.

Mechanism. τ is a percentile of a distribution, and the distribution moved. You launched a new product area; 30% of traffic is now about it; the embedder was trained on less of that vocabulary, so its similarities in that region are compressed upward; and the new domain has more minimal-edit pairs (“v1 endpoint” versus “v2 endpoint”) than the old one. Your fixed τ is now a different percentile than the one you chose.

Put numbers on it. Suppose the top-1 similarity distribution over live traffic moves its mean from 0.912 to 0.934 with unchanged spread 0.05. The fraction above τ = 0.93 goes from

to

— a hit-rate jump of seventeen points with no code change, and there is no reason to think the extra hits are correct. A rising hit rate that you did not cause is an alarm, not a win.

Test. Log the top-1 similarity of every request, hit or miss, and chart its p50 and p90 weekly. Alarm on a shift of more than 0.01 in either.

Fix. Recalibrate quarterly and after every launch, against a freshly labelled set. Or make τ adaptive — set it to a fixed percentile of the trailing similarity distribution rather than a fixed number — which keeps the operating point stable at the cost of being harder to reason about. Start with the quarterly review; it is what most teams need.

4 · Poisoned entries: refusals, errors and truncations

Symptom. One question always returns “I’m sorry, I can’t help with that” or an answer that stops mid-sentence, forever, for everyone, long after the cause is fixed.

Mechanism. The write path stored whatever came back. A rate-limit error, a safety refusal triggered by an unrelated transient, a stream truncated by a client disconnect — all of them are strings, and all of them cache beautifully.

Why it is worse than it sounds. A transient one-in-a-thousand failure becomes permanent for one intent, and semantic caching then spreads it: every paraphrase of the poisoned question also gets the refusal. One bad second becomes one bad month across an entire cluster of queries.

Fix. A quality gate before every write. This is five lines and it is the best five lines in the system.

python
def worth_caching(ans, meta):
    if meta.status != 200:              return False
    if meta.finish_reason != "stop":   return False   # length/timeout/abort
    if len(ans) < 40:                  return False
    if REFUSAL_RE.match(ans):          return False
    if "I don't have" in ans[:120]:    return False
    return True

5 · The evaluation feedback loop

Symptom. Your offline quality score is stable at 0.87 for three months while user complaints climb.

Mechanism. This one is genuinely sneaky. Your evaluation set is sampled from production traffic and scored on the served answers. As the cache grows, more of the sample is cache hits, and cache hits are drawn from the head — the easiest, most-templated questions in your product. Your eval set has silently become a measurement of your cache’s favourite questions instead of your model’s ability.

Worse, if you feed served answers back into any fine-tuning or few-shot selection, cached answers get reinforced, which makes them more likely to be served, which makes them a larger share of your eval. The loop closes.

Fix. Never sample the eval set from served traffic. Sample from requests, and run evaluation with the cache disabled so you are measuring generation. Then run a second, separate evaluation with the cache on, and report both. Chapter 8 makes this concrete.

6 · The cold-start herd

Symptom. Every deploy that changes the namespace — a prompt tweak, a model bump — is followed by a spike in model spend and API rate-limit errors.

Mechanism. Chapter 6’s thundering herd, triggered by your own release process. The namespace changed, so every request misses, and your head intents are all requested many times per second.

Fix. Singleflight handles the concurrency. For the spend spike, pre-warm: after a namespace change, replay the top 500 intents from the previous namespace through the new one in the background at a throttled rate. Five hundred generations is $5.33 and ten minutes, and it converts a cliff into a ramp.

7 · Silent ANN recall decay

Symptom. Hit rate declines slowly over months. Everyone assumes users are asking newer, more varied questions.

Mechanism. HNSW recall degrades as the graph accumulates deletions — every invalidation from Chapter 4 leaves a tombstone, and the graph’s connectivity assumptions weaken. At 30% deleted nodes, top-1 recall can fall several points. Every one of those is a hit you had and did not get.

Test. Once a week, brute-force the true top-1 for 10,000 sampled queries and compare against what the index returned. Chart the agreement rate. It should be 98% or better.

Fix. Rebuild the index on a schedule. At 214,000 entries a full rebuild is minutes, so make it nightly and stop thinking about it.

The diagnostic order

When something is wrong and you do not know what, run these in order. Each one is cheap and each one eliminates a whole class.

Chapter 8: Measuring It

There is a dashboard that every semantic cache ships with, and it is a trap. It has one big number on it — hit rate — and a line chart of dollars saved. Both go up when you lower τ. Both go up when the cache starts serving wrong answers. Neither can distinguish a working system from a broken one.

This chapter builds the dashboard that can.

The one metric that cannot be gamed

Hit rate is a rate of doing something, not a rate of doing something right. Precision is the metric you need and you cannot compute it in production, because computing it requires knowing the right answer, and if you knew the right answer you would not need the cache.

So you buy it. On a small random fraction of cache hits, serve the cached answer to the user as normal — and also, asynchronously, generate the real answer and compare. This is shadow sampling, and it is the only honest source of a production false-hit rate.

python
def serve(q, ctx):
    nbr, sim = lookup(q, ctx)
    if hit(nbr, sim, q):
        if random.random() < SHADOW_RATE:        # 2% of hits
            background(shadow_check, q, ctx, nbr)
        return nbr.answer                        # user waits 12 ms, as always
    return generate_and_store(q, ctx)

def shadow_check(q, ctx, nbr):
    truth = llm.generate(q, ctx)
    verdict = judge(question=q, served=nbr.answer, reference=truth)
    metrics.record("shadow", {
        "ns": namespace(ctx), "sim": nbr.sim,
        "agree": verdict.agree,                   # the number that matters
        "q": q, "stored_q": nbr.question,        # so a human can read it
    })

Two design points. The user is not made to wait — the shadow generation happens after the response is sent, so this costs latency to nobody. And the judge is asked a specific question: “would the served answer be correct and complete for this question?” — the same wording as your labelling prompt in Chapter 3, so the production number and the calibration number measure the same thing.

What shadow sampling costs, and what it buys

At 26.3% hit rate on 1,200,000 requests you get

Sample 2% of them:

Against savings of $3,361, that is 2.0% of the benefit, spent to find out whether the benefit is real. There is no other line item in this system with a better return.

Now be honest about what 6,312 samples can resolve. At a true false-hit rate of 0.1% you expect about six events, and the standard error is

So your monthly estimate of 0.10% carries a two-sigma band of roughly 0.02% to 0.18%. To detect a doubling from 0.1% to 0.2% with reasonable confidence you need about

which at 6,312 a month means a rolling quarter, not a monthly reading. Three practical consequences:

Chart it as a 90-day rolling window, so the noise does not generate false alarms and nobody learns to ignore it. Raise the sample rate where the budget is tight — 10% shadow sampling on the billing namespace costs almost nothing because that namespace is small, and it is where a false hit is most expensive. And use the served-similarity distribution as the fast signal: it has a hundred thousand samples a day and moves before the false-hit rate does.

The honest dashboard

Latency: report the mean, and say why not the median

A 26.3% hit rate replaces a 2,608 ms path with a 12.7 ms path for a quarter of requests. The mean:

Now the median. Sort all requests by latency: the fastest 26.3% are the hits, and the 50th percentile falls in the miss population. So p50 goes from 2,600 ms to… 2,600 ms. Unchanged.

This surprises people and it is worth stating as a rule: a cache moves the median only once its hit rate exceeds 50%. Below that, it compresses the fast tail and leaves the middle alone. If your SLO is written on p50 — and many are — a 26% hit rate will show up as zero improvement, and you will be asked why you built it. Report the mean, report the p10 (which drops from 1,900 ms to 12 ms), and explain the arithmetic before someone else has to.

The regression check nobody runs

Your offline evaluation set exists. Run it twice.

bash
# the two runs, and the number that matters is the difference
$ eval --set golden-400 --cache off
  accuracy 0.871   groundedness 0.912   refusal_rate 0.031

$ eval --set golden-400 --cache on --warm-from-production
  accuracy 0.858   groundedness 0.889   refusal_rate 0.034
  cache_hit_rate 0.31

# delta: -1.3 points accuracy, -2.3 points groundedness.
# That is the price of the cache, and now it is a number
# someone can accept or reject instead of a feeling.

Two requirements make this valid. The evaluation set must be sampled from requests, never from served answers — Chapter 7’s feedback loop. And the cache must be warmed from real production entries, because a cold cache in the eval harness has nothing to hit and will report a delta of zero, which is a very convincing way to be wrong.

Segment everything, because the average hides the damage

A single global false-hit rate of 0.10% can be made entirely of one namespace at 3%. Break every panel down by namespace and by intent family, and sort by false hits, not by volume. The output is a short list of question families that should be excluded from caching altogether — which is a much better lever than any global parameter.

Excluding those two families costs 41,600 hits — 13% of your hits, worth $443/month — and removes the large majority of your wrong answers. Recompute the global rate afterwards and you will find it has roughly halved. This is the single highest-leverage action available once the system is live, and it is invisible unless you segment.

With guards off, drag the budget down to 1.0 and there is no admissible threshold anywhere on the axis — the red curve never gets under the line. Turn the guards on and a wide admissible region opens up between roughly 0.92 and 0.96, and inside it you simply pick the leftmost point. That is what a calibrated system looks like: the budget picks the region, and you take the most generous point inside it.

What to write in the launch review

One paragraph, with these numbers in it, and every one of them is now something you can produce:

Chapter 9: Connections

You can build this now. Embed the question, search an index, apply a threshold you calibrated against a labelled set and a stated budget, run two cheap guards that catch what the threshold structurally cannot, scope everything by a namespace that includes every producer and every piece of context that changes the answer, invalidate on provenance rather than only on age, and put the false-hit rate on the same screen as the hit rate so that nobody can optimise one without seeing the other.

The whole thing on one page

python
def answer(req):
    q  = rewrite(req.history, req.question)     # ch5: resolve follow-ups
    ns = namespace(req)                         # ch4+5: producers + context
    v  = embed(PREFIX + q, normalize=True)     # ch1: one function, both paths

    nbr, sim = index.search(v, k=1, filter={"ns": ns})   # ch1
    if nbr and sim >= TAU \
       and decision_tokens_agree(q, nbr.question) \      # ch3
       and cross_encoder(q, nbr.question) >= 0.60 \       # ch3
       and nbr.age < ttl_for(nbr.doc_class):             # ch4
        if random.random() < SHADOW: background(shadow, q, req, nbr)  # ch8
        metrics.hit(ns, sim)
        return render(nbr.answer, req.entitlements)       # ch5: templating

    metrics.miss(ns, sim if nbr else None)
    ans, docs = generate(q, req)
    if is_cacheable(req, docs, ans) and worth_caching(ans):  # ch5, ch7
        index.add(v, q, ans, ns=ns, docs=docs)           # ch4: provenance
    return ans

Twenty lines. Every condition in that if was earned by a chapter, and removing any one of them puts a specific, named failure back into production.

What a semantic cache is not

The last row is worth dwelling on, because the two systems use the same index, the same embedder and the same cosine, and they are doing entirely different jobs. Retrieval hands its top-k to a model that will read it and decide. A semantic cache hands its top-1 straight to a human. Retrieval’s mistakes are filtered by the model; a cache’s mistakes are served. That is why a threshold that is perfectly sensible for retrieval is reckless for caching, and it is the single most common way this goes wrong.

Where it goes next

Three directions, all of them things teams are doing now.

Learned equivalence instead of borrowed similarity. Everything in Chapter 3 was a workaround for using a retrieval encoder as an equivalence test. Fine-tune a bi-encoder on your own labelled set with minimal-edit pairs as explicit hard negatives, and the two distributions separate: hard negatives drop from μ = 0.960 to something below the positives, and suddenly the threshold does work as a safety dial. A few thousand labelled pairs is enough. This is the highest-value follow-on project once the system is live.

Partial and compositional hits. A question that is 80% the same as a cached one is currently a miss. It could instead be a prefix — feed the cached answer to the generator as a draft and let it revise, cutting output tokens by half rather than to zero. The economics change: you capture the body of the distribution instead of only the head.

Caching intermediate steps. In an agent, the expensive repeated work is often not the final answer but a tool plan, a retrieval result, or a sub-question decomposition. Those are shorter, more structured, and far more repetitive than final answers, and every technique in this lesson applies to them — with the same warning attached, because a wrong cached tool plan fails in ways a wrong cached sentence does not.

Keep exploring

← Vector Embeddings — where the 384 numbers come from and what they encode
← Similarity Metrics — cosine, dot product, Euclidean, and why normalisation makes two of them the same
← Vector Databases — HNSW, IVF, filtering, and the recall knobs from Chapter 1
→ RAG — the system this cache usually sits in front of, and the source of Chapter 4’s provenance problem
→ Text Chunking — the other half of the provenance story: what a document id actually points at
→ Embedding Benchmarks — how to choose the encoder whose geometry your threshold depends on
→ On-Device Embeddings — running the 6 ms embedder somewhere other than a server
→ LLM Inference — where the 2,600 ms and the $0.01065 actually come from
→ AI Evaluation — the labelling, judging and shadow-sampling machinery of Chapters 3 and 8
→ Caching & CDNs — the classical caching this one deliberately breaks the rules of
→ Prompt Engineering — why the system prompt hash belongs in your key