Streaming Speech: Real-Time ASR & TTS

Chapter 1: Causality — What Makes a Model Streamable

We ended Chapter 0 with a promise: the reason the batch model waits is architectural. Time to open the box and point at the exact line of code responsible.

Take any speech encoder. It maps a sequence of input frames x₀…x_T to a sequence of output states h₀…h_T. The only question that matters for streaming is: which inputs is h_t allowed to depend on? That set is called the receptive field of the output, and its right-hand edge is the whole ballgame.

Three architectures, one question

A bidirectional encoder is the default in offline speech because it is simply better: the sound of a vowel is genuinely disambiguated by the consonant that follows it. English is full of this — the only acoustic difference between “grey tape” and “great ape” lives in timing cues that straddle the boundary. Letting the model look right is worth real accuracy.

A causal encoder gives that up entirely. Every output is computed from the past only, so the instant frame t arrives you can emit h_t. Zero algorithmic latency. You pay for it in errors, and we will measure exactly how much in Chapter 8.

A chunked encoder is the compromise that runs the industry: look right, but only a bounded, fixed distance. Chapter 2 builds it from the mask up.

Where causality actually lives in the code

Two components in a speech encoder have a right-hand edge, and both must be fixed. Miss either one and your “streaming” model still waits for the end of the clip.

1. Self-attention. The offending line is the softmax over all keys. In a bidirectional encoder, query t attends to keys 0…T — including keys that do not exist yet. Fixing it means adding a mask: a boolean matrix that sets forbidden scores to −∞ before the softmax, so their weight becomes exactly zero. That is the entire mechanism, and Chapter 2 works one out by hand.

2. Convolutions. Nearly every speech front-end starts with a convolutional stem (Whisper uses two conv layers over the log-mel input), and convolutions are quietly bidirectional too: a kernel of width 3 centred on frame t reads t−1, t, t+1. That +1 is a future frame. Stack five such layers and the peek compounds.

Worked example: the receptive field of a conv stack, by hand

Let’s compute exactly how far into the future a five-layer convolutional stem looks. The rule for composing receptive fields, for layers of kernel size k_i and stride s_i, is that each layer adds (k_i − 1) times the product of all strides beneath it:

Take five layers, all kernel 3, all stride 1. Every stride product is 1, so every layer contributes (3 − 1) · 1 = 2:

Eleven frames wide, centred on the output. So it reaches (11 − 1) / 2 = 5 frames to the left and 5 frames to the right. At a 10 ms hop, those 5 right-hand frames are 50 milliseconds of audio that must exist before this output is computable — before a single transformer layer has run.

Now add a stride. Suppose the second layer has stride 2 (Whisper’s stem does exactly this to halve the frame rate). Layers 3, 4, 5 now sit on top of a ×2 downsample, so each contributes (3 − 1) · 2 = 4:

The lesson: downsampling multiplies your latency debt. Every stride-2 layer doubles the cost, in input frames, of every kernel above it. This is why streaming encoders push their downsampling to the bottom of the stack, where fewer layers sit above it to be magnified.

Causality is only half the win — the other half is the cache

A causal model is allowed to emit early. That does not yet mean it is cheap. Recall the disaster from Chapter 0: re-running the model on the whole prefix every 200 ms cost 7.5× a single offline pass. Adding a mask does not fix that by itself — a masked model re-run from scratch does exactly the same redundant work, it just gets the same answer each time.

What makes streaming cheap is the second property from Chapter 0: work done for frame t is never done again. In a transformer that means the KV cache. When you compute attention, each frame produces a key vector and a value vector. In a causal model, frame 5’s key and value do not depend on frames 6 onward — so once computed, they are final. Store them; every future frame reads them instead of recomputing them.

Price it, because the difference is the difference between a demo and a product. Say the encoder has d = 512 dimensions and you are processing frame t of an utterance:

Let’s check the second row by hand rather than trusting the exponent. With a cache, frame 0 compares against 1 key, frame 1 against 2, and so on to frame 419 against 420. That is 1 + 2 + … + 420 = 420 · 421 / 2 = 88,410 comparisons. Without a cache, frame t redoes all t of its predecessors’ comparisons too, so the total is the sum of those partial sums: roughly 420³/6 = 74,088,000 / 6 ≈ 1.23 × 10⁷. Divide: 12,348,000 / 88,410 ≈ 140× the work, for identical output.

Chunked attention complicates this slightly and instructively. Because a chunk’s queries may look L frames past the chunk boundary, the keys for those lookahead frames get consumed by chunk c and then again by chunk c+1 as ordinary members. They are computed once and read twice — the cache handles it, but your bookkeeping must not evict a frame that a later chunk still needs. A cache that keeps the last 2C + L frames is the usual safe rule, and it also caps memory, which is the difference between a system that survives a forty-minute call and one that does not.

python
# The streaming encoder loop, with the cache made explicit.
class StreamingEncoder:
    def __init__(self, C, L, keep):
        self.C, self.L, self.keep = C, L, keep
        self.k_cache, self.v_cache = [], []      # finalized keys/values
        self.pending = []                        # frames not yet in a full chunk

    def push(self, frame):
        self.pending.append(frame)
        # a chunk is emittable once its own frames AND its L lookahead frames exist
        if len(self.pending) < self.C + self.L:
            return None                      # still waiting — this IS the latency
        chunk = self.pending[:self.C + self.L]
        k, v, out = self.attend(chunk, self.k_cache, self.v_cache)
        self.k_cache = (self.k_cache + k)[-self.keep:]   # bounded memory
        self.v_cache = (self.v_cache + v)[-self.keep:]
        self.pending = self.pending[self.C:]     # keep the lookahead frames — chunk c+1 owns them
        return out[:self.C]                    # only the chunk's own outputs are final

Two lines carry the whole design. if len(self.pending) < self.C + self.L: return None is the algorithmic latency, made of nothing but a comparison — the model is idle, waiting, exactly C + L frames’ worth. And self.pending = self.pending[self.C:] keeps the lookahead frames in the buffer rather than dropping them, because they are borrowed, not consumed: chunk c peeked at them, chunk c+1 owns them. Drop them there and your transcript loses a word every chunk, in a way that only shows up on long utterances.

The fix: causal padding

The cure for a convolution is not a mask but asymmetric padding. A normal “same” convolution pads one frame of zeros on each side. A causal convolution pads k−1 = 2 frames on the left and zero on the right, then crops the tail. The output at t now reads frames t−2, t−1, t. Same parameter count, same shapes, zero lookahead — the receptive field did not shrink, it shifted backwards.

python
import torch, torch.nn.functional as F

# (1) By hand: what does one output actually touch?
# kernel size 3, so each output reads 3 consecutive inputs.
x = torch.tensor([[1., 2., 3., 4., 5.]]).unsqueeze(0)   # (batch=1, ch=1, T=5)
w = torch.tensor([[[1., 1., 1.]]])                          # a plain moving sum

# (2) Symmetric "same" padding -> output t sees t-1, t, t+1  (PEEKS AT THE FUTURE)
y_sym = F.conv1d(F.pad(x, (1, 1)), w)
# y_sym = [3, 6, 9, 12, 9]   <- y[0] = 0+1+2 = 3 used input x[1]=2, which is the FUTURE

# (3) Causal padding -> pad k-1 = 2 on the LEFT only, nothing on the right
y_cau = F.conv1d(F.pad(x, (2, 0)), w)
# y_cau = [1, 3, 6, 9, 12]   <- y[0] = 0+0+1 = 1 uses only x[0]. Zero lookahead.

# (4) The one-liner every streaming codebase actually ships:
conv = torch.nn.Conv1d(1, 1, kernel_size=3, padding=0)
stream_step = lambda buf: conv(buf)      # keep a k-1 frame ring buffer, feed [buf, new]

Look carefully at the numbers in step (3). The causal output is the symmetric output shifted right by one, with the edge filled differently. Nothing was lost — the model still gets a 3-frame view. It just gets a view that ends at now instead of straddling it. That single idea, applied to convolutions, attention, and normalization alike, is what “making a model streamable” means in practice.

Play with the middle setting. A causal encoder’s highlighted row never crosses the query — the right edge and the query sit on the same column, which is precisely the statement “Δ = 0.” Now switch to chunked and slide the query within one chunk: the right edge stays pinned at the chunk boundary while the query walks toward it. That is the crucial and slightly odd behaviour we are about to quantify: within a chunk, different frames have different latencies, and the one that matters for your product is the worst of them.

Chapter 2: Chunked Attention, Worked Out By Hand

Causal attention is free but dumb; bidirectional attention is smart but infinitely slow. Chunked attention is the dial between them, and it is built from one object: a mask. In this chapter we will build that mask cell by cell, count its entries, convert the count into milliseconds, and then push actual numbers through a softmax to see what a mask does to a prediction.

The construction

Chop the frame sequence into fixed chunks of size C. Every query frame inside a chunk is allowed to attend to: all history (frames from previous chunks), all frames in its own chunk, and L extra frames past the chunk’s end — the lookahead. Formally, for a query at frame t sitting in chunk c = ⌊t/C⌋:

Read it in English: “you may look at everything up to the end of your own chunk, plus L frames more.” Note what the formula does not say: it does not depend on t itself, only on t’s chunk. Every frame in a chunk shares one right-hand edge. That is what makes the scheme cheap — the whole chunk is computed in a single batched pass, which is exactly the chunked-prefill trick modern inference engines already use, and exactly what Qwen2.5-Omni does when it changes its audio encoder “from full attention over the entire audio to performing attention in blocks of 2 seconds each.”

The hand-worked mask: C = 4, L = 1, T = 12

Twelve frames, chunk size four, lookahead one. Three chunks: frames 0–3, frames 4–7, frames 8–11. Let’s compute the right-hand edge for each chunk, step by step, substituting into the formula above.

Chunk 0 (c = 0, frames 0,1,2,3): edge = 4·(0+1) − 1 + 1 = 4 − 1 + 1 = 4. So frames 0–3 may each attend to keys 0,1,2,3,4 — five keys.

Chunk 1 (c = 1, frames 4,5,6,7): edge = 4·2 − 1 + 1 = 8 − 1 + 1 = 8. Frames 4–7 may attend to keys 0…8 — nine keys.

Chunk 2 (c = 2, frames 8,9,10,11): edge = 4·3 − 1 + 1 = 12 − 1 + 1 = 12, which is past the end of a 12-frame utterance, so it clips to 11. Frames 8–11 attend to keys 0…11 — twelve keys.

Counting the mask, and what the count means

Now count how many (query, key) pairs are allowed. This single number tells you how much of the bidirectional model’s context you actually kept.

Compare against the two extremes on the same 12 frames. Full bidirectional attention allows every pair: 12 × 12 = 144. Strictly causal attention allows query t to see keys 0…t, which is t+1 keys, so 1 + 2 + … + 12 = 12·13/2 = 78.

104 / 144 = 0.722, and 78 / 144 = 0.542. So a chunk size of four with one frame of lookahead recovers 72% of the full attention context while committing to a bounded wait, where pure causality gets 54%. Eighteen points of context for a handful of frames of delay: that is the bargain, and it is why chunked attention won.

Turning the mask into milliseconds

The count tells you about quality. Latency needs a different sum. For a query at frame t, the algorithmic latency is (right-hand edge) − t — how many frames of audio must arrive after t before t’s output exists. Chunk 0, frame by frame:

Average = (4 + 3 + 2 + 1) / 4 = 2.5 frames = 25 ms. Worst case = 4 frames = 40 ms. And the worst case is the one your users feel, because output is released chunk-by-chunk: nobody sees frame 3 before frame 0 is done. The general formula, which you can now read straight off the table:

Check both against our numbers: 4 + 1 − 1 = 4 ✓, and (4−1)/2 + 1 = 1.5 + 1 = 2.5 ✓.

Drag history to “limited”. The staircase turns into a diagonal band: each chunk keeps only a fixed number of past frames instead of all of them. This is not a latency optimisation — the right-hand edge does not move — it is a memory and compute optimisation, which keeps per-frame cost constant instead of growing with call duration. In a 40-minute call, unlimited history is not a luxury, it is an out-of-memory error.

What the mask does to an actual prediction

Counting cells is abstract. Let’s push real numbers through and watch the mask change a distribution. Take the query at frame t = 3 (last frame of chunk 0). Suppose the attention scores — the raw dot products, before softmax — against keys 0 through 5 are:

Softmax means: exponentiate each score, then divide by the sum of the exponentials of the allowed ones. Exponentiate first, once, and reuse the numbers three times:

Case A — causal mask (keys 0–3 allowed). Sum the first four exponentials:

Divide each by Z_A: 1.2214/15.3840 = 0.0794 · 2.7183/15.3840 = 0.1767 · 7.3891/15.3840 = 0.4803 · 4.0552/15.3840 = 0.2636. They sum to 1.0000. ✓

Case B — chunked mask, L = 1 (keys 0–4 allowed). Add key 4’s exponential:

Weights: 0.0710, 0.1580, 0.4294, 0.2357, 0.1059. Sum 1.0000. ✓

Case C — full bidirectional (keys 0–5 allowed). Add key 5:

Weights: 0.0525, 0.1169, 0.3177, 0.1744, 0.0784, 0.2601. Sum 1.0000. ✓

Now the mask is not a picture, it is a redistribution. Allowing just one future frame moved 10.6% of the probability mass off the past. Allowing the full future moved 33.9% — a third of this head’s attention was pointed forward, and the causal model simply never gets it. That third is, quite literally, the accuracy that streaming costs you, and Chapter 8 will show it as word error rate.

python
import numpy as np

# ---------- (1) the mask, built exactly as we did by hand ----------
def chunk_mask(T, C, L, hist=None):
    """True = allowed. Query t may see keys up to end-of-its-chunk + L."""
    m = np.zeros((T, T), dtype=bool)
    for t in range(T):
        c     = t // C                      # which chunk am I in?
        right = min(T - 1, C * (c + 1) - 1 + L)   # the hand-worked edge
        left  = 0 if hist is None else max(0, right - hist)
        m[t, left:right + 1] = True
    return m

M = chunk_mask(12, C=4, L=1)
print(M.sum())            # 104   <- matches our hand count exactly
print(M[0].sum(), M[4].sum(), M[8].sum())   # 5 9 12

# ---------- (2) masked softmax, step by step ----------
s     = np.array([0.2, 1.0, 2.0, 1.4, 0.6, 1.8])
allow = np.array([True]*5 + [False])        # chunked, L = 1
s_m   = np.where(allow, s, -np.inf)          # forbidden -> -inf
e     = np.exp(s_m - s_m.max())              # subtract max for stability
w     = e / e.sum()
print(np.round(w, 4))   # [0.071  0.158  0.4294 0.2357 0.1059 0.    ]

# ---------- (3) the one-liner, as it appears in real code ----------
# scores: (T, T);  mask: (T, T) bool
# attn = torch.softmax(scores.masked_fill(~mask, float('-inf')), dim=-1)

Line by line: s_m replaces forbidden scores with −∞; np.exp turns −∞ into exactly 0.0, so the forbidden key contributes nothing to the denominator; the division renormalizes what remains. Subtracting the max before exponentiating changes nothing mathematically (it cancels in the ratio) but prevents overflow when scores are large — the reason every real implementation does it. And notice the printed weights are the ones we computed by hand: 0.0710, 0.1580, 0.4294, 0.2357, 0.1059. The library is doing exactly what we did on paper.

Push the score of key 5 up to 3.5 with the mask on “full”. The future key eats the distribution — over half the attention mass points at audio that has not arrived. Then flip to “chunked”: that mass is redistributed over the past, and the model must make do. When people say a streaming model is “worse,” this bar chart is the mechanism they are describing.

Chapter 3: CTC vs Attention — Decoders That Can Commit

Chapter 2 made the encoder streamable. But an encoder produces vectors, not words. The thing that turns vectors into text is the decoder, and a decoder can be non-streamable in a way that has nothing to do with masks: it can be structurally unable to say anything until it has seen everything.

There are two classical answers, and the difference between them is one word: monotonic.

The attention decoder, and why it is stuck

An attention-based encoder-decoder (Whisper is one) generates text autoregressively. To produce the next word it runs cross-attention over the encoder’s output states and pulls in whatever acoustic evidence it wants, from anywhere in the utterance. That freedom is why these models are so accurate — and it is exactly what makes them non-streaming:

Nothing in the architecture says token n+1 must look later in the audio than token n. The alignment is free to jump backwards, and it sometimes usefully does — that is how the decoder revises a word once it hears the end of the phrase. It is also how Whisper hallucinates fluent sentences over silence: with no monotonic constraint and a strong internal language model, nothing anchors generation to elapsed time.

CTC: alignment by construction

Connectionist Temporal Classification takes the opposite bet. It refuses to model alignment as a free variable. Instead, it emits one symbol per encoder frame, from the alphabet plus one extra symbol: the blank, written _, meaning “nothing to report at this frame.” A frame-by-frame symbol string is then squashed into text by two rules, always in this order:

Order matters enormously, and the reason is the word “butter.” To write a genuine double letter you emit T _ T: the blank between them survives rule 1, so the two T’s never merge, and rule 2 then removes the separator. Without a blank symbol, CTC could not spell any word with a repeated letter. The blank is not padding — it is the doubling escape character.

And because output index only ever moves forward with the frame index, CTC is monotonic by construction. Frame 30’s decision can never depend on frame 200. That is the property that makes it streamable: once the encoder gives you frame t’s state, you can emit t’s symbol and never revisit it.

Worked example: decode “CAT” by hand, frame by frame

Alphabet {_, C, A, T}, eight encoder frames. Here is what the CTC head outputs — a probability distribution per frame. Take the argmax of each column (greedy decoding):

The greedy frame string is:

Rule 1, collapse adjacent repeats. The C C at frames 2–3 merges into one C; the T T at frames 7–8 merges into one T:

Rule 2, drop the blanks:

Three characters from eight frames, with no alignment model, no beam search, and no lookahead. Now let’s price the path. The probability of that exact frame string is the product of the eight selected numbers:

Multiply left to right, keeping every intermediate: 0.70 × 0.70 = 0.4900. × 0.60 = 0.2940. × 0.80 = 0.2352. × 0.75 = 0.1764. × 0.60 = 0.10584. × 0.80 = 0.084672. × 0.55 = 0.046570.

So the single best path has probability 4.66%. That sounds alarmingly low until you remember what CTC actually scores: the label, not the path. Many different frame strings collapse to “CAT.” Take one neighbour — make frame 4 a C instead of a blank (_ C C C A _ T T). It still collapses to CAT, and its probability is the same product with 0.80 swapped for 0.10:

Adding just these two paths already gives P(“CAT”) ≥ 0.0466 + 0.0058 = 0.0524. Summing over all collapsing paths — which is what the CTC loss does with a forward-backward recursion during training — pushes it far higher. Greedy decoding takes the best path as a proxy for the best label. It is not the same thing, and that gap is exactly the accuracy a beam search or an external language model buys back.

python
import numpy as np

VOCAB = ['_', 'C', 'A', 'T']
P = np.array([                      # (8 frames, 4 symbols) — the table above
    [.70, .10, .15, .05], [.20, .70, .05, .05],
    [.30, .60, .05, .05], [.80, .10, .05, .05],
    [.15, .05, .75, .05], [.60, .05, .30, .05],
    [.10, .05, .05, .80], [.35, .05, .05, .55]])

# ---------- (1) the two rules, written out longhand ----------
def collapse(sym):
    out, prev = [], None
    for s in sym:
        if s != prev:              # RULE 1: skip adjacent repeats
            out.append(s)
        prev = s
    return ''.join(c for c in out if c != '_')   # RULE 2: drop blanks

path = [VOCAB[i] for i in P.argmax(axis=1)]
print(path)                 # ['_','C','C','_','A','_','T','T']
print(collapse(path))       # 'CAT'
print(P.max(axis=1).prod()) # 0.04657  <- our hand-multiplied number

# ---------- (2) the streaming version: state is ONE variable ----------
class StreamingCTC:
    def __init__(self): self.prev, self.text = None, ''
    def step(self, probs):        # called once per new encoder frame
        s = VOCAB[int(probs.argmax())]
        if s != self.prev and s != '_':
            self.text += s           # emit — and NEVER take it back
        self.prev = s
        return self.text

# ---------- (3) the one-liner ----------
# torchaudio.models.decoder.ctc_decoder(...)  /  torch.unique_consecutive(ids)

Stare at StreamingCTC for a moment, because it is the whole point of this chapter. Its entire state is one previous symbol. It is O(1) memory, O(1) per frame, and it appends to the transcript without ever rewriting it. Compare that to an attention decoder, whose state is the full encoder output and whose next token may reinterpret any of it. Monotonicity is not a small architectural preference; it is the difference between a transcript that grows and a transcript that churns.

The production answer: the transducer

CTC has one real weakness — it assumes frames are conditionally independent given the audio, so it has no built-in language model and cannot learn that “flight” follows “my.” The RNN-Transducer (RNN-T) fixes this while keeping monotonicity, and it is what most production streaming ASR actually runs.

The transducer’s trick is a tiny reinterpretation of the blank symbol. Instead of “emit nothing for this frame,” blank means “I am done with this frame, advance the audio.” Emitting a real label does not advance the frame, so the model may emit several words during one frame if it wants. A separate prediction network — a small language model over the text emitted so far — is combined with the encoder state in a joint network. You get CTC’s monotonic streaming with an attention decoder’s linguistic knowledge.

And attention models are not out of the game: you can bolt monotonicity on. Triggered attention uses a CTC head purely as a trigger, letting the attention decoder run only over the audio up to the CTC spike. Monotonic chunkwise attention restricts cross-attention to a small window that is only allowed to move forward. Both are ways of buying back the one property CTC had for free.

Now drag frames revealed from 1 to 8 slowly. That is a streaming CTC decoder running: the transcript only ever grows — C, then CA, then CAT — and no earlier character is ever rewritten. Compare this with the next chapter, where an attention-style decoder gets to change its mind, and the text on screen starts to twitch.

Chapter 4: Flicker — Partial Hypotheses and How to Damp Them

Watch live captions on any video call and you will see it: a word appears, then mutates, then settles. “I’d like to prove” → “I’d like to move my”. That twitching is hypothesis flicker, and it is not a bug in the recognizer. It is the honest face of a model that has genuinely changed its mind because more audio arrived.

The problem is that flicker is only tolerable in one place: on a screen, in the last few words, in grey text. Everywhere else it is catastrophic. If your agent consumes partial transcripts, a retracted word means it started reasoning about a sentence that was never said. If your text-to-speech consumes them, it has already spoken a word that turned out to be wrong. You cannot un-say audio.

Where flicker comes from — three distinct sources

1. New acoustic evidence. Genuine and unavoidable. The frames after a word can disambiguate it, which is exactly the accuracy we bought lookahead for in Chapter 2. Bigger lookahead means less of this kind of flicker.

2. The internal language model. A decoder with linguistic knowledge re-scores its prefix as context grows. “to prove my” is less likely than “to move my,” so the word flips a step after the following word arrives. This flicker is lagged by one or two words, which is why it looks so jarring — the thing that changed is not the newest word.

3. Beam search churn. The top hypothesis and the runner-up can swap places on tiny score differences, dragging several words with them. This is the ugliest source, because the flip can be semantically enormous while the score difference is 0.01.

The damping algorithm: local agreement

The standard technique is beautifully simple and goes by LocalAgreement-n: emit the longest prefix that the last n hypotheses agree on. A word must survive n consecutive updates unchanged before it is promoted from volatile to stable. Below, the whole thing worked out by hand with n = 2 and updates arriving every 200 ms.

Let’s check the middle row, the one that earns the algorithm its keep. At t = 0.6 s we compare the previous hypothesis, [i’d, like, to, prove], with the current one, [i’d, like, to, move, my]. Compare word by word: i’d = i’d ✓, like = like ✓, to = to ✓, prove ≠ move ✗ — stop. Longest common prefix = 3 words. We had already emitted 3 words, so nothing new is emitted, and crucially nothing is retracted, because “prove” was never promoted in the first place. The model changed its mind entirely inside the volatile zone, where nobody downstream could see it.

Scoring both policies with actual numbers

Policy A — emit everything immediately. At 0.4 s it emits “prove.” At 0.6 s it must retract it. Final transcript is 7 words; 1 word was retracted.

Policy B — LocalAgreement-2. Zero retractions, by inspection of the table. But look at timing: the word “to” first appeared at 0.2 s and was emitted at 0.4 s. “move” first appeared at 0.6 s and was emitted at 0.8 s. “flight” appeared at 0.8 s, emitted at 1.0 s. Every word waited exactly one update:

There is the whole trade, in two columns. Stability is bought with delay, at an exchange rate of one update interval per unit of agreement. And notice how the exchange rate is set by something you also control: shrink the update interval to 100 ms and LA-2 costs only 100 ms — the same stability for half the delay, at twice the decoder invocations. That is a compute-for-latency trade, and it is usually a good one, because a decoder step is cheap next to an encoder pass.

python
# ---------- (1) longest common prefix, spelled out ----------
def lcp(a, b):
    i = 0
    while i < len(a) and i < len(b) and a[i] == b[i]:
        i += 1
    return a[:i]

# ---------- (2) LocalAgreement-n, the whole damper ----------
class LocalAgreement:
    def __init__(self, n=2):
        self.n, self.hist, self.emitted = n, [], []

    def update(self, words):
        """words = current partial hypothesis. Returns (stable, volatile)."""
        self.hist.append(words)
        self.hist = self.hist[-self.n:]          # keep only the last n
        if len(self.hist) < self.n:
            return self.emitted, words[len(self.emitted):]
        agree = self.hist[0]
        for h in self.hist[1:]:
            agree = lcp(agree, h)                # intersect all n hypotheses
        if len(agree) > len(self.emitted):
            self.emitted = agree                 # promote — never shrinks
        return self.emitted, words[len(self.emitted):]

# ---------- (3) run the hand-worked example ----------
la = LocalAgreement(n=2)
for h in ["i'd like to", "i'd like to prove", "i'd like to move my",
          "i'd like to move my flight", "i'd like to move my flight to"]:
    stable, volatile = la.update(h.split())
    print(f"STABLE: {' '.join(stable):32s} | volatile: {' '.join(volatile)}")

# STABLE:                                  | volatile: i'd like to
# STABLE: i'd like to                      | volatile: prove
# STABLE: i'd like to                      | volatile: move my
# STABLE: i'd like to move my              | volatile: flight
# STABLE: i'd like to move my flight       | volatile: to

The load-bearing line is if len(agree) > len(self.emitted). Without it, a hypothesis that gets shorter (which happens when the recognizer deletes a spurious word) would shrink the stable prefix — a retraction, the exact thing we are preventing. The stable prefix is a ratchet: it may only turn one way. Every correct implementation of this idea contains that ratchet, and every buggy one discovers the need for it in production.

Set agreement to 1 — that is the naive policy, and the readout will show you the retractions it causes. Set it to 3 and the retraction count goes to zero while the stable boundary falls two words behind the speaker. Then shrink the update interval and watch the delay column fall while the stability column holds. That last move is the free lunch, and it is the reason production systems run their decoder far more often than they run their encoder.

Chapter 5: Incremental TTS — Speaking Before You Know the Sentence

We now have a transcript that grows and never lies. The language model reads it and starts producing a reply — token by token, because that is what language models do. And here is where a second, mirrored version of the whole problem appears: text-to-speech wants a sentence, and you only have a fragment.

Waiting for the full sentence is the same mistake as waiting for the full utterance. If the reply is “Sure — I can move you to the 8:15 Tuesday flight, and there is no change fee on that fare,” that is around 20 words. At a typical 10 words per second of generation, the last word lands 1.6 seconds after the first. Speaking only when the sentence is complete throws all 1.6 seconds away.

Chunk the text, and make the first chunk tiny

Incremental TTS cuts the token stream into synthesis chunks and renders each into audio as soon as it closes. The scheduling rule that matters is: the first chunk should be as small as you can bear, and later chunks should grow. Small first chunk buys a fast start; large later chunks buy efficiency and better prosody — and by then you have a play-out buffer to hide behind.

Let’s put numbers on the first-chunk decision. Assume a speaking rate of 150 words per minute, which is 2.5 words per second, so one word = 0.4 seconds of audio. Assume the synthesizer runs at a real-time factor (RTF) of 0.3 — it takes 0.3 seconds of compute to produce 1 second of audio — plus 60 ms of fixed per-call overhead.

Chunking the first three words instead of waiting for the sentence saves 2,460 − 420 = 2,040 milliseconds of dead air on the TTS side alone. This is the single largest latency win available anywhere in a cascaded voice agent, and it costs nothing but a splitting rule.

The constraint you cannot violate: never run the buffer dry

Speaking early creates a new obligation. Once audio starts playing, it is consumed at exactly 1× real time, and if the next chunk is not ready when the current one ends, the caller hears a buffer underrun — a hole in the middle of a word. That is far worse than the original delay, because a delay reads as thinking and a hole reads as a broken line.

Track it explicitly. Let A_k be the audio duration of chunk k, t_k the moment its synthesis finishes, and e_k the moment its playback ends. Chunks are synthesized back-to-back, so:

Run it by hand for the growing schedule 3, 8, 16 words → A = 1.2 s, 3.2 s, 6.4 s, with RTF = 0.3 and 60 ms overhead:

Safe, and getting safer — each chunk buys more slack than the next one consumes. Now break it. Keep everything identical but set RTF = 0.9 (a heavy generative voice on a busy GPU):

A 1.74-second hole, right after the third word. And notice: RTF = 0.9 is still faster than real time. The naive test “is RTF below 1?” passes, and the system still stutters, because chunk 2 is 2.7× longer than chunk 1. The correct per-chunk condition falls straight out of the recursion:

Check it: 0.9 · 3.2 + 0.06 = 2.94 > 1.2 ✗, versus 0.3 · 3.2 + 0.06 = 1.02 ≤ 1.2 ✓. This inequality is the reason chunk growth is capped in real systems: you may double chunk size only as fast as your slack allows.

The prosody problem at the seams

Latency solved, quality broken. Human speech has declination: pitch drifts steadily downward across a phrase and drops sharply at the end, which is how listeners hear a sentence as one unit. A synthesizer given three words in isolation renders them as a complete utterance — full declination, final fall, and a little breath at the end. String three such chunks together and you get the unmistakable sound of someone. reading. a. list.

Worse, the boundaries are audible in three separate ways at once:

The fixes, in increasing order of sophistication:

Split at linguistic boundaries. Cut at commas, clause boundaries, and after function words — never mid-phrase. Prosody is already discontinuous at a comma, so the seam hides inside a real one.

Condition on text you do not synthesize. Feed the synthesizer the previous chunk’s text (and ideally the next few tokens) as context, but keep only the audio for the current chunk. The model now knows it is mid-phrase and continues the contour instead of restarting it. This costs no extra audio latency — only a slightly longer prompt.

Give the decoder a bounded window, not a fresh start. This is precisely what Qwen2.5-Omni does when turning speech codes into a waveform: it groups codes into blocks and gives its diffusion-transformer decoder a receptive field of four blocks — two blocks of lookback, the current block, and one block of lookahead. The decoder always has real acoustic context on both sides of the seam, so there is no seam. The same chunk-with-context trick is then applied to the vocoder that turns the mel-spectrogram into a waveform.

Anticipate the tone before the text exists. The subtlest fix in the literature: Qwen2.5-Omni’s Talker receives the high-level hidden representations from the text model in addition to the sampled text tokens, explicitly because “voice generation must anticipate the content’s tone and attitude before the entire text is fully generated.” The hidden state carries the sentence’s emotional shape a few tokens before the words arrive — so the synthesizer can start a rising contour for a question it has not finished reading.

Set the first chunk to 1 word and RTF to 0.3: time-to-first-audio drops to about 180 ms — and then watch the underrun appear anyway, because one word of audio (0.4 s) cannot cover the synthesis of the next chunk. That is the trade in its purest form: the smaller your first chunk, the sooner you speak and the thinner your safety margin. Production systems pick the smallest first chunk whose margin stays positive at the p95 of their RTF distribution, not the median — because a stutter at p95 is a stutter on one call in twenty.

Chapter 6: The Latency Ledger (showcase)

Every previous chapter handed you a knob. Chunk size and lookahead (Chapter 2). Decoder type and commitment policy (Chapters 3 and 4). First-chunk size and real-time factor (Chapter 5). This chapter puts all of them on one instrument panel, wires them to a single number — the gap between the moment you stop speaking and the moment you hear a reply — and lets you find out which knobs actually matter.

The ledger, and the one rule that makes it non-obvious

The naive model is “add up every stage.” That model is wrong, and being wrong about it is how teams spend three weeks optimizing a component that was never on the critical path.

Here is the correction. After the user’s last syllable, two clocks run at the same time. One is the endpoint hold: the silence timer that decides the user is finished (a fixed threshold is typically around 500 ms; a smart, model-based detector can run tighter). The other is the recognizer’s own tail: the last chunk still has to be encoded, decoded, and promoted from volatile to stable. Both start at the same instant. So:

The max, not the sum. Which means: if your endpoint hold is 250 ms and your recognizer tail is 150 ms, making the recognizer twice as fast changes your product by zero milliseconds. Conversely, if you push the endpoint hold down to 100 ms with a smart turn detector while your recognizer tail is 350 ms, you have bought nothing and added false interruptions. Find the max before you optimize; this one line of arithmetic is worth more than most model swaps.

Everything after the ASR tail is a sum, because it is a genuine serial chain: the language model cannot start before it has the transcript, the synthesizer cannot start before the first token, the network cannot carry audio that does not exist.

Reading the instrument

The showcase below computes that ledger live. Three reference lines are drawn across it, and you should know what each one means before you start dragging:

The default settings reproduce the ledger from Chapter 0 exactly: 95 ms uplink, 350 ms ASR tail, 30 ms of hops, 375 ms to the first LLM token, 100 ms to first audio, 95 ms downlink — 1,045 ms. Verify it on the canvas, then start breaking things.

One turn, traced millisecond by millisecond

Bars are useful; a trace is convincing. Here is a single turn of the default configuration, with a clock that starts at the instant the caller’s last syllable leaves their mouth. Every row is an event, and every number is derived from the sliders above.

Read the row at 250 ms, then the row at 350 ms. For a hundred milliseconds the system knew the caller had stopped and still had nothing to say — the recognizer was finishing. That hundred milliseconds is the only part of the ASR tail that a faster recognizer could recover, and it is 9.6% of the total turn. Meanwhile the two biggest single entries, 375 ms of first-token and 250 ms of endpoint hold, are both decisions rather than computations.

What the ledger does not contain

Three costs are deliberately excluded above, and you should know their shape so you recognize them when your production numbers exceed your model.

Tool calls. If the agent has to look up a booking before it can answer, the language model term is not 375 ms — it is 375 ms, plus a tool round trip, plus a second generation. This is why production systems learn to speak first: emit “let me pull that up” while the tool runs, which does not reduce latency at all but converts dead air into a turn. Perceived latency and measured latency part company here, and the perceived one is the product.

Reasoning. A model that thinks before answering pays its thinking in the first-token term, where it hurts most. The common architecture splits the difference: a small, fast conversational model drives the real-time dialogue while a heavier model runs asynchronously in the background and injects results when they are ready.

Contention. Every number in this ledger is a median under normal load. Speech models are unusually sensitive to it: when a GPU crosses a memory or compute threshold, task-switching overhead lands on every session at once, and the whole stack’s tail degrades together rather than independently. This is why audio services are the ones people put on dedicated hardware, even when the language model is happily shared — a language model’s tail can be hidden behind streaming, and a synthesizer’s cannot, because the buffer is already playing.

Four experiments to run right now

Experiment 1 — find the max. Leave everything at default and drag chunk size from 14 down to 4. Watch the total. It barely moves at the bottom of the range, because once the recognizer tail drops below the 250 ms endpoint hold, the endpoint hold is the max and the recognizer is free. Now raise the endpoint hold to 500 ms and repeat: chunk size now does nothing at all. This is the single most valuable habit in latency work — before optimizing, check whether you are on the critical path.

Experiment 2 — the cheapest 300 ms you will ever find. Reset, then drag TTS first chunk from 1 word to 12. The total climbs by roughly 440 ms and nothing else changed — no model, no infrastructure, no cost. Splitting the first synthesis chunk small is free latency, and it is the first thing to check in any voice agent that feels sluggish.

Experiment 3 — the honest cost of stability. Set agreement n to 3 with a 30-frame chunk. The recognizer tail becomes 300 ms of chunk plus 600 ms of stability delay, blowing well past the endpoint hold and pushing the total over the 1,400 ms limit. Chapter 4 promised that stability costs delay; here is the bill, denominated in the number your users feel.

Experiment 4 — why telephony is hard. Hit telephony worst case. Nothing about the models changed; the network profile, the endpoint hold, and a conservative configuration alone push you past 2.5 seconds. A voice agent that demos beautifully over WebRTC in the office can be unusable on a mobile call, and no amount of model optimization will fix a 110 ms one-way network leg.

The other axis: latency versus flicker

Total latency is not the only thing your knobs move. Every millisecond of commitment delay you refuse to spend comes back as instability. The second instrument plots the two against each other, sweeping chunk size for each agreement setting, so you can see the frontier rather than one point on it.

Look at the shape. Each curve is steep on the left and flat on the right: the first 100 ms of commitment delay removes most of the flicker, and the next 500 ms removes almost none. That elbow is where you want to sit, and it is a different point for different products. A live-captioning display can live at the top-left — flicker is visible but harmless and speed is everything. A voice agent whose synthesizer consumes the transcript must live past the elbow, because a retracted word there is a spoken word that was wrong. Same model, same code, different operating point, because the consumer of the transcript is different.

Chapter 7: Frame-Level End-to-End Streaming

Everything so far has been damage control on a pipeline whose stages were never designed to interlock. Chapter 6’s best tuned cascade landed around 700 ms — three times the human turn gap, with the floor set by a chain of independent services each waiting on the last.

There is another move available, and it is the one the frontier has taken: delete the pipeline. Make audio itself the token stream. Then “streaming” stops being a retrofit and becomes the model’s native mode of operation, because a language model over audio frames is already, by construction, a thing that emits one step at a time.

Step 1: turn sound into a slow, discrete stream

The enabling component is a neural audio codec — a learned encoder that compresses a waveform into a small number of discrete tokens per second, plus a decoder that turns them back into sound. Moshi’s codec, Mimi, is worth doing the arithmetic on, because every number in it is a streaming decision.

Mimi takes 24,000 samples per second and pushes them through four convolutional blocks with strides 4, 5, 6, and 8, then a final 1-D convolution of stride 2. The total downsampling factor is the product:

So the frame rate is:

Twelve and a half tokens per second. For comparison, the log-mel front-end from Chapter 1 runs at 100 frames per second, and a raw 24 kHz waveform at 24,000. Mimi has compressed the time axis by 1,920×, which is what makes it possible for a transformer to model minutes of dialogue.

Each frame is quantized by a residual vector quantizer with Q = 8 codebooks of 2,048 entries each. Since log₂(2,048) = 11 bits per codebook:

Full-band 24 kHz speech at the bitrate of a 1980s modem. And critically: every convolution is causal, and the transformer modules in the bottleneck use causal masking. The paper states the consequence plainly — both the initial frame size and the overall stride are 80 ms, so given a first audio frame of 80 ms, Mimi outputs a first latent timestep, which can be decoded to 80 ms of output audio. That sentence is the definition of a streaming codec.

Step 2: model frames with a hierarchy, and pay for the depth in delay

Now a problem appears that does not exist in text. Each 80 ms frame is not one token but eight, one per codebook. Flattening them into a single sequence would give 8 × 12.5 = 100 tokens per second, and the paper notes this is incompatible with streaming inference at scale.

The RQ-Transformer splits the work in two. A large temporal transformer steps once per frame, modelling the dialogue across time. A small depth transformer then runs inside that frame, predicting the 8 codebooks one after another conditioned on the temporal state. Big model, slow clock; small model, fast clock.

Within a frame there is one more choice: the delay pattern — how many timesteps each codebook lags the first. And because a frame is exactly 80 ms, delay converts to latency by simple multiplication. Here are the three patterns the Moshi authors compared, with the arithmetic:

Read the second and fourth rows together. Going from 2 steps of lag to 8 costs 480 milliseconds — more than double a human turn gap — and buys only a marginal perplexity improvement. The published result: Moshi pre-trains with an acoustic delay of 2 and fine-tunes with a delay of 1, for a theoretical latency of 160 ms, measured at 200 ms in practice once real compute and buffering are included.

Step 3: model both speakers at once, and turn-taking disappears

The deepest structural change has nothing to do with speed. Moshi models two audio streams in parallel — its own voice and the user’s — as separate token sequences predicted at every frame. There is no “whose turn is it” state machine, because the model is always predicting its own next 80 ms whether or not the user is talking.

The consequences ripple through every problem in Chapters 4 through 6. Overlap is representable, so backchannels (“mm-hm” while the other person continues) are just a normal prediction. Barge-in needs no special casing: the user’s stream simply becomes non-silent, and the model conditions on it in the very next frame. And endpointing — the 250 ms hold that dominated our ledger’s ASR tail — is not tuned down, it is gone, because nothing waits for a turn to end.

Text does not disappear either. Moshi’s Inner Monologue predicts text tokens aligned to its own speech, and the alignment delay between the text and audio streams turns out to be a single knob with a remarkable property: delay the text behind the audio by 2 seconds and you have a streaming speech recognizer (5.7% word error rate on LibriSpeech test-clean, with word alignments precise to one 80 ms frame); put the text ahead of the audio and the same architecture is a streaming synthesizer (4.7% WER, better than VALL-E’s 5.9%). One model, one loss, one hyper-parameter, both directions of the pipeline we spent five chapters building separately.

The other route: keep the modules, stream every one of them

Qwen2.5-Omni takes a different path to the same goal — it keeps recognizable components (a “Thinker” that produces text, a “Talker” that produces speech tokens) but makes every single stage block-streaming. It is the most systematic worked example of this lesson’s ideas in one system:

Notice the last three rows: that is precisely the mask we built by hand in Chapter 2 — a lookback window, the current block, and one block of lookahead — applied not to speech recognition but to waveform generation. Same object, opposite direction. The paper is explicit that this is done to preserve quality at the block seams while keeping the receptive field bounded: exactly the prosody-versus-latency trade of Chapter 5, solved with the mask of Chapter 2.

Push τ to 7 in Moshi mode and read the latency box: 640 ms, the fully staggered pattern, and you can see why — the last codebook of frame 0 is not emitted until frame 7 has begun. Then switch to Qwen mode and move the cursor: the highlighted window slides but never grows, which is the whole point. A bounded window means constant per-block cost and constant latency, no matter how long the conversation runs.

Chapter 8: Quality vs Latency — Measuring the Trade

We have asserted twice now that lookahead buys accuracy. This chapter makes that claim measurable, because “streaming is a bit worse” is not something you can budget against. What you need is a curve: word error rate as a function of how much future the model is allowed to see.

What to measure, and the trap in measuring it

Word error rate (WER) is the standard: count the substitutions, deletions, and insertions needed to turn the model’s transcript into the reference, and divide by the number of reference words. A quick worked example, because the denominator surprises people. Reference: “i would like to move my flight” (7 words). Hypothesis: “i would like to prove flight” (6 words). Align them: move→prove is 1 substitution; my is deleted, 1 deletion.

Note that WER can exceed 100% (insertions are unbounded) and that it treats “a” and “cancel” as equally important, which is why nobody who ships voice agents uses it alone.

The trap: streaming WER is not offline WER measured on a streaming system. If you run your streaming model and then score its final transcript, you have measured a batch system with extra steps — because the final transcript includes every late revision. To measure what the user actually experienced you must score the transcript as emitted, freezing each word at the moment it was promoted to stable. The gap between those two numbers is real, and it is the part of your quality that lives entirely in the commitment policy of Chapter 4.

The lookahead curve, and its elbow

The shape is always the same: a steep drop, then a knee, then a long flat tail. To reason about it, model it. A saturating exponential fits observed behaviour well:

where Δ is lookahead in milliseconds, WER₀ is the fully causal error rate, WER_∞ is the offline floor, and τ is the characteristic scale — the amount of lookahead that removes about 63% of the recoverable gap. Take WER₀ = 11.0%, WER_∞ = 5.5%, τ = 300 ms, which puts the 2-second-lookahead point near the published 5.7% that Moshi’s streaming recognizer reaches on LibriSpeech test-clean with a 2-second text delay. Now evaluate it by hand, 100 ms at a time:

Check one row so you trust the rest: at Δ = 300, e⁻¹ = 0.3679, so WER = 5.5 + 5.5 × 0.3679 = 5.5 + 2.023 = 7.52%. ✓

Now read the last column, which is the only column that should influence a decision. The first 100 ms of lookahead is worth 1.56 points of WER. The hundred milliseconds between 900 and 1,000 ms is worth 0.08 points — twenty times less. If a product manager tells you the recognizer must be 0.5 points better, the honest answer depends entirely on where you already sit: from 100 ms it costs you another 80 ms; from 600 ms it costs you a second and a half.

The metrics WER does not capture

A streaming system has failure modes that a single accuracy number is blind to. The full dashboard:

And one warning about the benchmark itself. LibriSpeech test-clean — the set every number in this chapter is quoted on — is read audiobook speech: no crosstalk, no 8 kHz phone codec, no “uhh,” no restarts. A model at 5% there can be at 20% on a real call. Use it to compare configurations of your own model, which is what a lookahead sweep does, and never as an estimate of what a caller will experience.

python
# ---------- (1) WER by hand: Levenshtein over words ----------
def wer(ref, hyp):
    r, h = ref.split(), hyp.split()
    d = [[0] * (len(h) + 1) for _ in range(len(r) + 1)]
    for i in range(len(r) + 1): d[i][0] = i     # all deletions
    for j in range(len(h) + 1): d[0][j] = j     # all insertions
    for i in range(1, len(r) + 1):
        for j in range(1, len(h) + 1):
            sub = d[i-1][j-1] + (r[i-1] != h[j-1])
            d[i][j] = min(sub, d[i-1][j] + 1, d[i][j-1] + 1)
    return d[-1][-1] / len(r)

print(wer("i would like to move my flight",
          "i would like to prove flight"))    # 0.2857  <- our 2/7

# ---------- (2) the metric that actually matters: score as EMITTED ----------
def streaming_wer(ref, emissions):
    """emissions = list of (t_ms, stable_prefix) from the LocalAgreement damper.

    The final stable prefix IS the emitted transcript, because the ratchet
    guarantees it only grew. Scoring the last *partial* instead would flatter
    the system by including revisions the user's agent never saw."""
    final_stable = emissions[-1][1]
    return wer(ref, final_stable)

# ---------- (3) the lookahead sweep, the experiment worth running ----------
for lookahead_ms in [0, 100, 200, 400, 800, 1600]:
    # retrain or re-mask with this lookahead, then:
    #   e = wer_stream(model(lookahead_ms), testset)
    #   print(lookahead_ms, e, turn_gap(lookahead_ms))
    pass   # plot e against turn_gap — the frontier, not the two numbers

Raise τ to 800 ms — a model or a language where context matters over longer spans, like a morphologically rich language whose word endings resolve late. The elbow slides right, and suddenly 400 ms of lookahead is on the steep part of the curve rather than past it. The correct operating point is not a number you can copy from a paper; it is a property of your model, your language, and your users, and the only way to find it is to sweep it.

Chapter 9: Cheat Sheet & Connections

You started with a model that could not speak until you stopped. You now have the full apparatus for building one that speaks while you are still talking. Here is everything, in one place.

The pipeline, end to end

Every symbol, defined

The formulas worth memorizing

Failure modes: symptom → cause → fix

The field guide. Every row here is something built in this lesson, seen from the outside — the way a bug actually arrives, as a complaint rather than a diagnosis.

The five numbers to know cold

Design rules, distilled

1. Streaming is a property of the whole path, not of one layer. One bidirectional component — a conv with symmetric padding, a global pool, a time-axis normalization — makes the whole system batch. Test by running on a prefix and checking the outputs are bit-identical.

2. Spend your latency where the curve is steep. The first 100 ms of lookahead is worth twenty times the tenth. Once past the elbow, move the milliseconds to the endpoint hold or the first TTS chunk.

3. Expose two strings, always. Volatile for the screen, stable for the machine. A ratchet that only grows.

4. Make the first synthesis chunk as small as your buffer margin allows, and check that margin at p95 of your RTF, not the median.

5. Before optimizing anything, find the max. Two clocks race after the user stops; the fast one is free.

6. If you need to be under 300 ms, stop tuning the cascade. That floor is structural, and the way past it is frame-level modelling where audio is the token stream.

Keep exploring

← Whisper — the batch encoder-decoder this lesson had to un-batch
← Audio Representations — where the 10 ms frames come from
→ Turn-Taking, Endpointing & Barge-In — the endpoint hold that dominated our ledger, built from zero
→ Moshi — the full-duplex 12.5 Hz model, paper-grade
→ Qwen2.5-Omni — Thinker-Talker and block-streaming everything
→ EnCodec — the streamable codec lineage Mimi builds on
→ TTS Architectures — how the synthesizer works inside
→ Neural Audio Codecs — audio as discrete tokens, from first principles