Crossing the uncanny valley: the Conversational Speech Model, read closely.

01 The assistant that got boring

The note opens with an unusual complaint. Not that voice assistants sound robotic — they largely do not, any more — but that speaking to one is exhausting. That word choice is the entire research programme in disguise.

Here is the observation, as the note states it: “A personal assistant who speaks only in a neutral tone has difficulty finding a permanent place in our daily lives after the initial novelty wears off. Over time this emotional flatness becomes more than just disappointing — it becomes exhausting.”

Read that as an engineering claim and it says something specific and testable: the cost of flat prosody is not paid on the first turn; it accumulates across turns. That is why the failure is invisible to the benchmark that the entire text-to-speech field spent a decade optimising. Sample one sentence, play it to a listener, ask “does this sound human?” — the answer is yes. Play forty turns of it and ask “would you use this daily?” — the answer changes, and no per-sentence metric registers the change.

You have felt this yourself in a non-speech setting. A colleague who replies to every message with “Sounds good.” is not being rude, and no individual message is defective. After two weeks you stop telling them things. The information content was fine; the signal about the relationship was absent, and the absence compounded.

Sesame gives the missing quality a name: voice presence — “the magical quality that makes spoken interactions feel real, understood, and valued.” It is a deliberately non-technical phrase, and the note is upfront about why: they do not have a metric for it. What they have is a decomposition.

Four components of presence

The note lists four ingredients. It is worth reading them not as marketing bullets but as a specification of what has to be conditioned on — because each one names a different piece of context the model must see in order to produce the right output.

Notice how three of the four are conditioning problems, and one (dynamics) is a structural problem. That split predicts the entire architecture: CSM will be a model that eats conversation history, and it will not be a model that handles turn-taking. Both of those are true, and both were decided by this list before a single layer was specified.

The one-to-many problem

Now the technical core of the chapter, and the reason context is not a nice-to-have but a mathematical necessity. The note states it compactly: models “struggle with the one-to-many problem: there are countless valid ways to speak a sentence, but only some fit a given setting.”

Formally, text-to-speech is asked to model a conditional distribution:

and that distribution is enormously multi-modal in the statistical sense — it has many separated peaks. The sentence “I can’t believe you did that” has a delighted reading, a devastated reading, a sarcastic reading, a flat administrative reading, a whispered reading, and a reading where the emphasis lands on you instead of did. All are correct renderings of the text. None is the right rendering of the moment.

A model trained by maximum likelihood on this distribution faces an unpleasant choice. Either it samples from the full distribution — in which case it is unpredictable, sometimes delivering condolences in a bright cheerful lilt — or it collapses toward the mode, which is the safest, most neutral, most averaged rendition available. The neutral assistant voice is not a design decision. It is what mode-seeking looks like when the conditioning is impoverished.

The fix is to condition on more:

The claim is that this second distribution is far less multi-modal than the first. Once you know the user just said “my dog died last night” in a shaking voice, the delighted reading of the next sentence has essentially zero probability. Conditioning did not make the model more expressive; it made the correct rendition identifiable, which is the thing that was missing.

How many ways, exactly? A back-of-envelope count

“Countless valid ways” is rhetorically fine and technically unsatisfying. Let us put a number on it, because the number is what makes the conditioning argument concrete.

Take the six-word sentence “I can’t believe you did that.” Enumerate a deliberately coarse set of prosodic choices — coarse enough that every option is clearly audible and clearly distinct:

And this is a six-word sentence with six coarse dimensions. Real prosody is continuous, jointly distributed, and unfolds over a whole paragraph. The count is illustrative, not measured — but the order of magnitude explains something the note only asserts: why naturalness metrics saturated while conversational quality did not. A metric that asks “is this a natural rendition?” gives full marks to all 486. A listener who heard the conversation gives full marks to 64.

Widget: the rendition fan

Below, the same sentence is drawn as a cloud of candidate renditions in a two-dimensional prosody space — energy on the horizontal axis, warmth on the vertical. Every dot is a natural way to say the line. Choose what happened immediately before, and watch the admissible set collapse. The number in the readout is the fraction of natural renditions that remain appropriate: your model’s odds of getting it right by luck.

Two behaviours are worth provoking deliberately. First, switch between “gift” and “loss” and notice the admissible regions barely overlap — the same text needs almost disjoint deliveries. Second, tick mode-seeking collapse with no context selected: the single output the model converges to sits in the dead centre, appropriate to none of the four situations and offensive in none of them either. That dot is every voice assistant you have ever used.

Why this became the frontier in 2025 and not 2020

The note makes a sharp historical claim in passing: “Even though recent models produce highly human-like speech, they struggle with the one-to-many problem.” The word doing the work is even though. Contextual expressivity became the frontier precisely because per-sentence quality stopped being one.

Trace the sequence of bottlenecks, because it explains why nobody worked on this earlier:

1. Intelligibility (through roughly 2016)

   Concatenative and parametric systems produced speech you could understand but never mistake for a person. The bottleneck was the vocoder, and the metric was word error rate on the synthesised audio. No one asks whether a diphone concatenation is emotionally appropriate.

   Fixed by neural vocoders — WaveNet and successors

2. Naturalness (roughly 2016–2022)

   Neural sequence models and neural vocoders closed the gap on single sentences. Mean opinion scores climbed until systems scored within noise of recorded speech, at which point the metric stopped discriminating between systems.

   Fixed by end-to-end neural TTS; metric saturated

3. Speaker identity and zero-shot cloning (2023)

   VALL-E reframed TTS as language modelling over codec tokens and showed that a three-second enrolment recording is enough to carry a voice into arbitrary new text — in-context learning for speech. Speaker similarity became a headline metric, and then it too saturated.

   Chapter 04 takes VALL-E’s architecture apart

4. Contextual appropriateness (2024 onward)

   With quality and identity solved per-sentence, the only remaining audible defect is choosing the wrong rendition for the moment — which requires the moment to be in the model’s input. This is where CSM sits, and why its evaluation suite had to be invented rather than adopted.

   The whole rest of this lesson

Each rung of that ladder was made visible by the rung below being finished. You cannot notice that a model chose the wrong emotional reading while you are still noticing that it sounds like a robot. Saturation is not the end of a research direction; it is the instrument that reveals the next one.

02 What CSM actually is

“We introduce the Conversational Speech Model (CSM), which frames the problem as an end-to-end multimodal learning task using transformers. It leverages the history of the conversation to produce more natural and coherent speech.” Two sentences, and every architectural decision in the note is downstream of them.

The note names its own two contributions, which is a courtesy worth honouring by quoting them exactly:

Note what is not claimed. Not a new tokenizer — CSM uses Mimi, from the Moshi paper. Not a new transformer — both components are Llama variants. Not a scaling result — three modest sizes, the largest an 8B backbone. The contribution is a factorisation plus a measuring instrument, which is a much more honest pair of claims than most model releases make.

So the chapter has one job: establish precisely what the model eats and what it emits, with shapes, before Chapter 04 asks why the internal split is where it is.

What enters the model

From the note: “Training samples are structured as alternating interleaved patterns of text and audio, with speaker identity encoded directly in the text representation.” Unpack that into an actual sequence.

A conversation is a list of turns. Each turn contributes two things to the sequence: the text of what was said, tokenised with a Llama tokenizer, and the audio of how it was said, tokenised with Mimi into $N$ codebooks per frame at 12.5 Hz. They are interleaved — text for turn 1, audio for turn 1, text for turn 2, audio for turn 2, and so on — so that by the time the model is asked to speak turn $t$, both the words and the sound of every earlier turn are in its context.

Two of those rows deserve a moment.

The 12.5 Hz frame rate is the load-bearing number of the entire design. It comes from Mimi, the split-RVQ tokenizer introduced in the Moshi paper, which the note cites as reference [5]. Twelve and a half frames per second means one frame covers 80 ms of audio — roughly the duration of a single phone in conversational speech. Compare with earlier neural codecs operating at 50 or 75 Hz: a four-times reduction in sequence length for the same audio, which is what makes a 2,048-position context hold two minutes of conversation instead of thirty seconds.

The context length is a conversation length, not a sentence length. The note reports training “with a 2048 sequence length (~2 minutes of audio) over five epochs.” Let us check that the two numbers are consistent, because doing so tells us something the note never states outright.

Speaker identity lives in the text stream

“Speaker identity encoded directly in the text representation.” Nine words, and a genuinely interesting design decision hides inside them.

There are at least three places you could put speaker identity in a multi-speaker speech model:

Choice C is the one that costs nothing and composes with everything. The model already has to attend over the text stream; putting speaker there means multi-speaker conversation is not a feature bolted on but a consequence of the sequence format. It also explains how the note’s “conversations with multiple speakers” sample works: “single generation using audio prompts from two speakers” — one forward pass produces both voices, because the marker in the text tells the model whose turn is being rendered, and the previous audio of that speaker is already in context to supply the timbre.

Not specified The note does not say what the marker looks like — a reserved token, a name string, a bracketed prefix — nor whether identity generalises to unseen speakers by name or only by acoustic context. Chapter 08 collects this with the rest of the unstated details.

The inference loop, step by step

The note’s figure caption is unusually precise, and it is the closest thing to pseudocode the document contains. Here it is, decomposed into the loop it describes:

1. Interleave and feed

   “Text (T) and audio (A) tokens are interleaved and fed sequentially into the Backbone.” The context is the whole conversation so far plus the text of the turn being spoken. One position per text token, one position per audio frame.

   Backbone context = conversation history, not just this sentence

2. Backbone predicts codebook zero

   “…which predicts the zeroth level of the codebook.” One forward pass of the large model produces one token: the semantic code for the next 80 ms frame. This is the expensive step, and it happens once per frame.

   Cost: one large-model step per 80 ms of audio

3. Depth decoder fills in the rest

   “The Decoder then samples levels 1 through N – 1 conditioned on the predicted zeroth level.” A much smaller model runs $N-1$ autoregressive steps within the frame, each conditioned on the backbone’s representation and the codes already chosen for this frame.

   Cost: N–1 small-model steps per frame

4. Feed the completed frame back

   “The reconstructed audio token (A) is then autoregressively fed back into the Backbone for the next step.” The full frame — all $N$ codes — becomes one input position for the backbone’s next step. This is the compression that keeps the sequence at 12.5 Hz.

   N codes collapse into one backbone position

5. Stop at end of turn

   “…continuing until the audio EOT symbol is emitted.” The model decides when the utterance is finished. Note the consequence: generation is bounded by a learned end-of-turn, which is a content decision, not a timing decision.

   Turn-based by construction — see Chapter 10

6. Next request, with the interim as context

   “This process begins again on the next inference request, with the interim audio (such as a user utterance) being represented by interleaved audio and text transcription tokens.” The user’s reply is transcribed and tokenised as audio, then appended. Both representations of the same utterance go into context.

   The user's actual voice is in context, not just their words

Step 6 is the one people skim, and it is where the entire “voice presence” thesis is cashed out. The user’s utterance enters the context as audio tokens as well as text. That is what makes emotional intelligence architecturally possible: the model can attend to how the user sounded, not merely to what a recogniser thought they said. Every cascaded voice pipeline in production throws this away at the first text boundary.

Step 4 is where the efficiency lives. $N$ codes per frame become one input position. The backbone’s sequence length is the number of frames, not the number of codes — which is the difference between a two-minute context and a five-second one.

Widget: the interleaved stream

Build a short conversation and watch the sequence the backbone actually sees. Text blocks are Llama tokens; audio blocks are Mimi frames at 12.5 Hz. Toggle between what CSM receives and what a classical text-to-speech model receives at the same moment, and watch the context evaporate.

The readout counts positions, so you can watch the 2,048-position budget being spent. Two things to notice. First, audio dominates — even at 12.5 Hz, frames outnumber text tokens roughly three to one, which is why the frame rate is the design’s most consequential number. Second, in cascade mode the position count collapses by about three quarters, and that looks like efficiency until you remember what was deleted: every acoustic property of every previous turn.

What “single-stage” is arguing against

The word “single-stage” is a contrast, and the thing being contrasted with is the dominant design of the preceding two years. The note describes it plainly:

Read the criticism carefully, because it is subtler than “two stages are slower.” The complaint is informational. In a two-stage system, stage one produces semantic tokens and stage two turns them into audio. Whatever prosody stage one failed to encode is unrecoverable — stage two has nothing else to condition on. The semantic tokens are a strict information bottleneck between the part of the system that knows about the conversation and the part that produces sound.

And the note names the specific difficulty: ensuring that the semantic tokens carry prosody is hard during training. Semantic tokens are usually trained to be speaker-invariant and phonetically discriminative — which is to say, trained to discard exactly the acoustic variation that prosody consists of. You are asking one representation to be simultaneously invariant to how something was said and informative about how something was said.

“End-to-end” in the note’s sense therefore means something precise: the gradient from the acoustic codebooks reaches the same backbone that read the conversation history. The decoder is not a separately trained vocoder receiving a summary; it is a head on the same model, conditioned on the same state. That is the whole argument for the word.

03 Two kinds of audio token, and the traffic jam between them

Before any architecture can be discussed, the substrate has to be clear: what a discrete audio token is, why there are two flavours of them, and why having several per frame creates a scheduling problem that three different papers solve three different ways.

The note’s background section is compact and precise, so start from its own definitions:

The division of labour is worth stating in one line: semantic tokens know what was said; acoustic tokens know what it sounded like. Neither is sufficient. A stream of semantic tokens can be decoded into intelligible speech in some generic voice; a stream of acoustic tokens without semantic structure is a high-fidelity recording of noise.

Both are discrete, and that is not incidental. Discretising audio is what allows a transformer trained on next-token prediction to model it at all — the same machinery that predicts the next word predicts the next 80 ms of sound. The whole line of work from AudioLM onward exists because of this one move.

Residual vector quantisation, by hand, every number

RVQ appears in every sentence of this literature and is rarely worked out. It is simple enough to do on paper, so let us do it on paper — because the arithmetic explains both why one codebook is never enough and why the codebooks are ordered, which is the fact the whole architecture chapter depends on.

The setup: an encoder has produced a continuous vector for one frame of audio. We want to represent it with a few discrete indices. A single vector quantiser replaces the vector with the nearest entry of a codebook. RVQ then quantises what is left over, again and again.

Take a two-dimensional vector so every step is visible:

and three tiny codebooks, each at half the scale of the one before — coarse, finer, finest:

Four facts fall directly out of that arithmetic, and every one of them matters later.

One. The codebooks are strictly ordered. Level 2 quantises the residual of level 1. You cannot compute index 2 without index 1 — not as a modelling preference but as an arithmetic fact. This is the “sequential dependency between codebooks in a frame” the note refers to, and it is the origin of every scheduling headache in this chapter.

Two. Early codebooks carry most of the information. Level 1 removed 62% of the magnitude; level 3 removed a further 4 percentage points. Truncate the code after $k$ levels and you get a valid, lower-fidelity reconstruction — which is why RVQ codecs offer a bitrate dial, and why a model that gets codebook 0 right is most of the way to intelligible speech.

Three. Diminishing returns are real but exaggerated here. Our codebooks have four entries; production codecs have 2,048. With 2,048 entries per level the improvement per level stays substantial much longer — but the shape of the curve, steep then flat, is the same.

Four. The bitrate arithmetic is trivial once you see it. Mimi, as specified in the Moshi paper: $Q = 8$ quantisers, codebook size 2,048, frame rate 12.5 Hz.

Not specified The note never states how many codebooks CSM uses — it writes $N$ throughout. Chapter 08 shows why an apparent typo in one figure caption is the only clue the document contains.

Why Mimi splits the RVQ

The note says CSM processes audio “using Mimi, a split-RVQ tokenizer, producing one semantic codebook and $N-1$ acoustic codebooks per frame at 12.5 Hz”, citing the Moshi paper. That word — split — is doing real work, and the reason is a conflict the Moshi authors measured.

The obvious design is a single RVQ stack whose first level is trained to be semantic, by distilling from a self-supervised speech model. Moshi tried exactly that: distil WavLM embeddings into the first quantiser of an eight-level RVQ. The result was a trade-off they describe plainly — distillation “significantly improves the phonetic discriminability of the first quantizer… it also affects audio quality negatively.”

The mechanism is the residual structure we just computed by hand. Levels 2 through 8 quantise the residual of level 1. If level 1 is pulled toward phonetic discriminability, its residual is no longer the well-behaved leftover that the acoustic levels were designed to encode. In their words: “as higher-order quantizers operate on the residual of the first one, the latter needs to trade audio quality for phonetic discriminability.”

The split-RVQ fix is structural rather than a loss-weighting compromise:

1. One plain vector quantiser for semantics

   A single, non-residual VQ, trained with a distillation loss against WavLM embeddings. Its job is phonetic discriminability, full stop.

   Codebook 0 — the one CSM's backbone predicts

2. A separate RVQ, in parallel, for acoustics

   Seven levels in Moshi’s configuration, operating on the input rather than on the semantic quantiser’s residual. Its job is reconstruction quality.

   Codebooks 1 to N−1 — what CSM's depth decoder predicts

3. Sum the two outputs

   “We sum their outputs, such that while both can be used for reconstruction, we remove the constraint that acoustic information should be conserved in the residual of the semantic quantizer.”

   The constraint that caused the conflict is simply deleted

The sequential dependency problem, and the delay-pattern answer

Now the traffic jam. Each frame carries $N$ codes, and (within the acoustic RVQ) they are ordered: code $k$ is only meaningful given codes $1 \dots k-1$. A transformer predicting one token per step therefore has a scheduling question: in what order do the $N \times T$ codes of a $T$-frame utterance get produced?

The note describes the incumbent answer and its cost:

The delay pattern — introduced for music generation and cited by the note as reference [3] — is an elegant trick. Lay the codes out as a grid: rows are codebooks, columns are frames. Then shift row $k$ rightward by $k-1$ columns. Now every column of the shifted grid contains codes that depend only on earlier columns, so a model can emit one whole column per step, in parallel across codebooks, and every dependency is still respected.

The cost is at the edges. The first column of the shifted grid contains only codebook 1 of frame 1; the rest of that column belongs to frames that do not exist yet. You need $N$ steps before any complete frame is available, and a frame is the unit the codec can decode.

The step costs above are illustrative — the note gives no latency figures at all, which Chapter 08 flags. The structure of the comparison, however, is exactly what the note asserts: $N$ large-model steps versus one large-model step plus $N-1$ small-model steps.

Widget: the delay pattern and its cost

The grid below is a token schedule — rows are codebooks, columns are frames. Each view is a snapshot: it shows exactly which codes exist at the moment the first frame becomes decodable, so you can see what the schedule spent its start-up on. Switch between the flattened schedule (every code gets its own backbone step), the delay pattern (one shifted column per step), and CSM’s split.

Drag $N$ from 4 to 32 with the delay pattern selected and watch time-to-first-audio grow linearly. Then switch to CSM’s split and drag it again: the line barely moves, because the growth term now multiplies the small model’s step cost. That is the whole point in one gesture — the architecture does not remove work, it moves the term that scales with $N$ onto a model that is cheap per step.

Why the note rejects both incumbent options

By this point the note has set up two rejected designs and one accepted one, and it is worth laying the argument out as a table because the reasoning is genuinely tight.

Two rejections for two different reasons — one about information, one about time — and one design that dodges both. Whether it dodges them as cleanly as claimed is a question Chapter 10 returns to. But the argument as constructed is coherent, and it is far more specific than most architecture justifications in company research notes.

04 One question, three answers

Every model in this literature faces the same arithmetic. A $T$-frame utterance with $N$ codebooks has $N \times T$ discrete values to produce, with a dependency structure that is causal in time and ordered in depth. Who predicts what, and in what order? Three landmark systems give three different answers, and the differences are entirely visible.

Set the problem up once, precisely, so the three answers can be compared on identical terms.

There are $N \times T$ values. A fully autoregressive model over the flattened sequence would need $N \times T$ forward passes and is correct but hopeless: at $N = 32$ and 12.5 frames per second, one second of audio costs 400 forward passes of a multi-billion-parameter model. Every system below is a scheme for spending fewer large-model passes without breaking the dependency structure — or for breaking it deliberately, in a place where it does not hurt much.

VALL-E: autoregressive, then parallel

from arXiv 2301.02111 VALL-E (January 2023) was the model that reframed text-to-speech as language modelling over codec tokens, and its factorisation is the cleanest possible split of the problem in two.

Stage 1 — autoregressive over codebook 1. A decoder-only transformer conditioned on the phoneme sequence and an acoustic prompt predicts the first quantiser’s codes, one frame at a time:

Stage 2 — non-autoregressive over codebooks 2 to 8. A second transformer, architecturally similar but with eight separate acoustic embedding layers, predicts an entire codebook level for the whole utterance in one pass. It attends over all positions at once — “unlike AR, the NAR model allows each token to attend to all the input tokens in the self-attention layer.” The current level $i$ is injected via adaptive layer normalisation, and the input is the summed embeddings of levels $1 \ldots i-1$ plus the phoneme and acoustic prompts.

Seven such passes complete the frame stack. Total large-model passes: $T + 7$.

The strength is efficiency: seven passes instead of $7T$. The weakness is the one that matters for conversation. You cannot emit a single millisecond of audio until the autoregressive stage has finished the entire utterance, because the NAR stage operates on the whole sequence. For an audiobook that is irrelevant. For a companion that has to answer you, it is fatal — and it is exactly the constraint the note gestures at when it says the delay problem is “suitable for offline applications like audiobooks.”

Moshi: the RQ-Transformer

from arXiv 2410.00037 Moshi (October 2024) — the note’s reference [5], and the source of both Mimi and the architecture CSM is explicitly differentiated from — takes the hierarchical route. Two transformers, but split along depth rather than along stages.

The formalism is worth reading in the original notation because CSM inherits it exactly:

The efficiency argument is stated directly in the paper: “the number of steps in the Temporal Transformer is always equal to $S$, rather than $K \cdot S$, and the number of steps in the Depth Transformer is at most $K$.” The large model runs once per timestep; the small model runs $K$ times within the timestep. The flattened sequence never exists.

Two configuration details from the paper are worth carrying forward. The depth transformer has 6 layers, dimension 1024, 16 attention heads — genuinely small. And it uses different parameters per depth index for its linear, projection and feed-forward layers, on the reasoning that “different sub-sequences might require different transformations.” Codebook 4 and codebook 20 are encoding different kinds of residual detail, so giving them separate weights is cheap and helps.

Now the part that makes the comparison interesting. In Moshi, what is sub-sequence $k = 1$? Not the semantic audio code. It is the text token: “We insert $W$ as the first sub-sequence in $V$, such that it acts as a prefix to the generation of semantic tokens.” This is Moshi’s Inner Monologue — the model writes what it is about to say, then says it, within the same timestep.

Which means the split lands here: the temporal transformer’s linear head predicts the text token, and the depth transformer predicts everything audio — the semantic codebook included.

CSM: split at the zeroth codebook

from the note Now read the note’s sentence with the Moshi formalism in hand, because every clause is now legible:

Four claims, taken one at a time:

1. “Inspired by the RQ-Transformer [4]”

   Reference [4] is Lee et al. (2022), the original RQ-Transformer for image generation — a large model along the sequence axis, a small one along the depth axis. CSM credits the pattern to its origin rather than to Moshi, which is correct scholarship: Moshi also cites Lee et al. as the source.

   The pattern is borrowed; the split point is not

2. “Different from the approach in [5], we split at the zeroth codebook”

   In Moshi the temporal transformer’s head emits the text token and the depth transformer emits all audio codebooks. In CSM the backbone emits the semantic audio code itself, and the depth decoder only ever sees acoustic residuals. The split moved down by one rung.

   The whole chapter is about this one rung

3. “A distinct linear head for each codebook”

   The same per-index parametrisation Moshi introduced, and for the same reason: the mapping from “frame state” to “residual code at level 7” is a different function from the one at level 24, and separate heads are nearly free on a 100M–300M model.

   Inherited, and the note says so

4. “Significantly smaller… while keeping the model end-to-end”

   The stated sizes make “significantly” concrete: 1B/100M, 3B/250M, 8B/300M. At the largest size the decoder is about 3.6% of the backbone’s parameter count — and note that it grows far more slowly than the backbone across the three sizes.

   Chapter 09 turns these ratios into a latency budget

Why does moving the split down one rung matter? Because of what the semantic codebook is. After Mimi’s split-RVQ design, codebook 0 is the distilled, phonetically discriminative, speaker-invariant representation — the code that decides what phone is being uttered and with what prosodic shape. Everything the conversation history should influence is concentrated there.

Inference, not stated The note does not spell out this reasoning. It says what it split and where, not why that rung rather than another. The argument above is the most coherent reconstruction available from the note plus the Moshi paper’s description of Mimi — and it is flagged as reconstruction rather than presented as the authors’ claim.

Showcase: all three factorisations on the same grid

This is the chapter’s centrepiece. One utterance, one token grid — rows are codebooks, columns are frames — and three schedules filling it. Press play and watch the fill order. The ledger tracks the two numbers that decide whether a design can hold a conversation: large-model passes, and how many steps elapse before the first frame is decodable.

Three things to provoke deliberately.

Run VALL-E to the end. Notice that the green “audio available” band does not appear until the very last passes — nothing is playable until the whole utterance is done. That is the audiobook regime, and it is a perfectly good design for the problem it was built for.

Compare Moshi and CSM at the same $N$. The fill patterns look nearly identical, because they are: one large-model step per column, then a fast vertical sweep. The difference is the colour of the top row. In Moshi the semantic code is filled by the small model; in CSM it is filled by the large one. Same throughput, different allocation of intelligence.

Raise $N$ with CSM selected. The number of large-model passes does not change at all — it is $T$, always. Only the small model’s inner loop grows. This is what “scaling depth is cheap” means concretely, and it is why an $N$ of 32 is affordable here and would not be under a delay pattern.

The one-word difference, stated as a table

All three systems, side by side, on identical terms. The columns are chosen so that no row is a matter of taste.

The last row is the one that keeps the comparison honest, and Chapter 10 develops it. Moshi is doing something CSM explicitly is not: modelling the user’s stream and its own stream simultaneously, so that overlap, interruption and backchannel fall out of the architecture. CSM is a better speaker with a richer conditioning story; Moshi is a conversation model. They are not competing on one axis.

The loop in code — three forms

Start with the shape of the thing, in the most explicit form possible: nested loops, no batching, no caching, every step visible.

# Generate one turn. `ctx` is the interleaved conversation so far:
# text tokens and audio frames from every previous turn, plus the
# text of the turn we are about to speak.
frames = []
while True:
    # --- one BACKBONE step: the expensive one, once per 80 ms ---
    h = backbone(ctx)                      # [d_model]  temporal state
    c0 = sample(backbone_head(h))          # scalar, codebook 0
    if c0 == AUDIO_EOT:
        break

    # --- N-1 DECODER steps: cheap, inside the frame ---
    codes = [c0]
    state = decoder_init(h, c0)
    for k in range(1, N):
        state = decoder_step(state, codes[-1])
        codes.append(sample(head[k](state)))  # distinct head per codebook

    # --- the whole frame becomes ONE backbone input position ---
    frames.append(codes)
    ctx = ctx + [embed_frame(codes)]

audio = mimi_decode(frames)                 # 80 ms per frame

Read the two loops as a cost statement. The outer loop runs once per 80 ms of output and costs one forward pass of an 8B model. The inner loop runs $N-1$ times per outer iteration and costs $N-1$ forward passes of a 300M model. Everything the architecture claims is contained in those two lines.

Now the same loop with the caching that any real implementation needs, since the backbone must not recompute its context every frame:

kv = backbone.new_cache()
kv = backbone.prefill(kv, ctx)             # history: paid once per turn

for t in range(max_frames):
    h, kv = backbone.step(kv, last_frame_embedding)
    c0 = sample(backbone_head(h))
    if c0 == AUDIO_EOT: break

    # the decoder's context is ONE frame long, so its cache is tiny
    dkv = decoder.new_cache()
    dkv = decoder.prefill(dkv, [h, embed0(c0)])
    codes = [c0]
    for k in range(1, N):
        s, dkv = decoder.step(dkv, embed[k-1](codes[-1]))
        codes.append(sample(head[k](s)))

    last_frame_embedding = sum(embed[k](codes[k]) for k in range(N))

Two details in that version are load-bearing. The decoder’s cache is re-created every frame and is at most $N$ positions long — it has no memory across frames, which is what keeps it cheap and what makes Chapter 05’s subsampling trick legal. And the frame embedding fed back to the backbone is a sum over per-codebook embedding tables, which is the same construction Moshi describes: “the Temporal Transformer receives at each step $s$ as input the sum of $K$ learnt embedding tables.”

Finally, the one-liner — not because you would write it this way, but because it names the operation:

frame = depth_decode(backbone_state, c0, N)   # N-1 tiny AR steps
# everything above is the body of depth_decode plus one backbone step

05 A small model with an enormous batch

The elegant architecture of Chapter 04 has an ugly consequence at training time, and the note is refreshingly direct about it. The fix is three sentences long, saves an order of magnitude, and is the single most reusable idea in the document.

Here is the problem, in the note’s words:

Read the first clause slowly, because it contains the whole difficulty. The decoder’s effective batch is $B \times S$.

Why? Because the decoder does not run once per training example. It runs once per frame. During training the backbone is teacher-forced — it processes all $S$ positions of all $B$ sequences in a single parallel pass, which is what makes transformer training efficient. But the decoder’s job is defined per frame: given this frame’s backbone state and its codebook 0, produce codebooks 1 through $N-1$. Every one of the $B \times S$ positions is an independent instance of that job.

So a batch of 32 sequences becomes a batch of 65,536 decoder problems, each of which is itself an autoregressive rollout of length $N-1$. The decoder is a 300M-parameter model being asked to run at a batch size three orders of magnitude larger than the model that fed it.

The ledger, every number

Put real values in. Take $B = 32$, $S = 2048$ (the note’s stated sequence length), and $N = 32$ codebooks. Every step below is arithmetic you can check on paper.

And note that this ratio is set by $N$, not by the model sizes. Double the backbone to 16B and the imbalance is unchanged. Double $N$ to 64 and it gets worse. The problem is structural.

What it costs in memory

Predictions are cheap to count; activations are what actually kill a training run. Backpropagation requires keeping intermediate activations alive from the forward pass until the backward pass reaches them, so the memory is proportional to the number of positions the decoder processed.

Estimated, not stated The note gives no memory figures, no layer counts, and no batch size. The $B = 32$ and 6-layer/1024-dim decoder above are stand-ins, chosen to be plausible and clearly labelled. What is not a stand-in is the structure: memory scales as $B \times S \times N$, and that is stated outright.

One frame in sixteen

The fix, in the note’s words:

Two clauses, and the asymmetry between them is the entire design.

The zeroth codebook is trained on every frame. Non-negotiable. Codebook 0 is what the backbone predicts, and it is the autoregressive spine of the model — every frame’s semantic code conditions every subsequent frame. Skip frames here and you are training a language model on a sequence with holes in it.

The decoder is trained on a random one-sixteenth of frames. Cheap, because the decoder’s task is local. It has no state that crosses frames; each frame is an independent problem of the form “given this state and this semantic code, produce the acoustic residuals.” Sampling a subset of independent problems is ordinary minibatching, applied one level down.

Redo the ledger with the trick applied:

Why the trick is legal here — and where it would not be

“We observe no perceivable difference in audio decoder losses” is an empirical claim, but it is not a surprising one, and it is worth working out why — because that reasoning tells you when you may reuse the trick and when you may not.

Reason one: the decoder is stateless across frames. Its context is one frame long. Skipping frame 7 does not corrupt frame 8’s training example, because frame 8’s decoder rollout never depended on frame 7’s decoder rollout. Contrast with the backbone, where skipping a frame breaks the causal chain for every subsequent frame.

Reason two: the sample is still enormous. Count it.

Reason three: the tasks have wildly different sample complexity. The backbone is learning conversational prosody, semantics, speaker behaviour, and turn structure — a task with essentially unbounded appetite for data. The decoder is learning “what acoustic residual accompanies this semantic code in this state” — a much narrower function. Giving them the same number of gradient samples was never the right allocation; one-sixteenth is a correction toward the right one.

The natural follow-up question — why sixteen and not eight or sixty-four — the note does not answer. Nor does it report the sweep that presumably found it. Chapter 08 files this with the rest.

Widget: the amortisation ledger

Set the batch, sequence length, codebook count and subsample rate, and watch where the work goes. The bars are prediction counts; the readout carries the memory estimate using the same conservative accounting as above. Drag the subsample slider from 1 to 16 and watch the decoder bar collapse while the backbone bar does not move at all.

Set the subsample to 1 and $N$ to 32 to see the pathology the note describes: a bar chart where the small model’s work is thirty times the large model’s. Then set it to 16. The interesting part is that no other number moved — same batch, same sequence, same codebooks, same backbone training signal. Only the redundant part of the decoder’s diet was removed.

Three forms of the same code

First, the training step written out longhand, so the shape of the tensors is unambiguous.

# text_audio: the interleaved batch, [B, S]
# codes:      ground-truth Mimi codes,  [B, S, N]
h = backbone(text_audio)                        # [B, S, d]
logits0 = head0(h)                              # [B, S, V]
loss0 = cross_entropy(logits0, codes[:, :, 0])   # every frame

# every position becomes a decoder problem — this is the explosion
h_flat = h.reshape(B * S, d)                     # [B*S, d]   65,536 rows
c_flat = codes.reshape(B * S, N)                 # [B*S, N]
loss_rest = decoder_loss(h_flat, c_flat)         # 31 AR steps each

loss = loss0 + loss_rest

Now the amortised version. The change is three lines, and only three lines:

h = backbone(text_audio)                        # [B, S, d]  unchanged
logits0 = head0(h)
loss0 = cross_entropy(logits0, codes[:, :, 0])   # STILL every frame

# --- the amortisation: pick a random 1/16 of the frames ---
keep = S // 16                                  # 2048 // 16 = 128
idx = torch.randperm(S, device=h.device)[:keep]  # fresh every step
h_sub = h[:, idx, :]                             # [B, 128, d]
c_sub = codes[:, idx, :]                         # [B, 128, N]

h_flat = h_sub.reshape(B * keep, d)              # [4096, d]  not [65536, d]
c_flat = c_sub.reshape(B * keep, N)
loss_rest = decoder_loss(h_flat, c_flat)

loss = loss0 + loss_rest                         # gradients still reach the backbone

Three details in that listing repay attention.

The mask is resampled every step. A fixed subset would train the decoder on a fixed 6.25% of frame positions, which correlates the subsample with position in the sequence — frames near a turn boundary would be systematically over- or under-represented. Fresh randomness makes the estimator unbiased over training.

The backbone loss is computed before the subsample. Order matters. Codebook 0 sees every frame; only the decoder’s branch is thinned.

Gradients still flow back into the backbone through h_sub. The decoder’s loss teaches the backbone something about producing states that are decodable — on one frame in sixteen. That is the “end-to-end” property surviving the optimisation, at a sixteenth of the density.

And the one-liner — the whole scheme, as an index expression:

idx = torch.randperm(S)[:S // 16]   # everything else is bookkeeping

It is worth sitting with how small that is. One line of indexing recovers an order of magnitude of training throughput, and the note reports no measurable cost. The reason it works is not the line; it is the architecture that made the line safe — a decoder with no cross-frame state and a task that does not need every example.

06 Two tests that a saturated benchmark cannot run

The note’s second stated contribution is an evaluation suite. Two of its tests are genuinely novel, and both are constructed on the same principle: build a task where the only way to succeed is to use a specific conditioning pathway, then see whether the model uses it.

Before the tests, look at what the note chooses to demonstrate. The sample sections are a list of capabilities, and each one names a different kind of context:

Notice that categories 3 and 4 are demonstrations of exactly the same claim from opposite directions: the model’s output depends on audio it was given, not just on text. The two objective tests formalise that claim so it can be measured rather than listened to.

Test 1: homograph disambiguation — does the model understand the text?

A homograph is a word whose spelling is fixed and whose pronunciation is not. The note’s example: “lead” as /lɛd/, the metal, versus “lead” as /liːd/, to guide.

The protocol, stated precisely in the note: “For Homograph accuracy we generated 200 speech samples covering 5 distinct homographs — lead, bass, tear, wound, row — with 2 variants for each and evaluated pronunciation consistency using wav2vec2-lv-60-espeak-cv-ft.”

The five words and their two readings each:

Now the crucial property of this test, and the reason it exists: word error rate is blind to it. Synthesise “the lead pipe” with the wrong vowel and a speech recogniser will still transcribe the word as “lead” — because English orthography maps both pronunciations to the same letters. The error is inaudible to the metric and glaringly audible to a human. A saturated benchmark is not merely uninformative here; it is actively misleading.

Hence the evaluator: wav2vec2-lv-60-espeak-cv-ft is a model that transcribes audio into phonemes rather than words. Run it on the generated sample, look at which phoneme sequence appears where the homograph should be, and compare with the reading the sentence requires. The metric changed representation — from orthography to phonetics — because the failure lives in the gap between them.

One more property makes this test specifically about text understanding. Everything needed to choose the right reading is in the sentence itself. No conversational history is required. So a failure here is a failure of the backbone’s language modelling, not of its use of audio context — which is precisely what makes it a clean complement to the second test.

Test 2: pronunciation continuation consistency — does the model listen?

The second test inverts the information channel. From the note: “Audio understanding through Pronunciation Continuation Consistency: Evaluates whether the model maintains pronunciation consistency of a specific word with multiple pronunciation variants in multi-turn speech. One example is ‘route’ (/raʊt/ or /ruːt/), which can vary based on region of the speaker and context.”

The protocol: “For Pronunciation Consistency we generated 200 speech samples covering 10 distinct words that have common pronunciation variants — aunt, data, envelope, mobile, route, vase, either, adult, often, caramel.”

Here is the key structural fact: the information required to pass is present only in the audio. Transcribe the earlier turn and you get the word “route” — identical text for both variants. The distinction lives entirely in the acoustic realisation. A pipeline that converts speech to text and then text to speech has thrown the answer away before the question is asked.

And notice why consistency rather than correctness is the metric. There is no correct pronunciation of “either.” Both variants are standard. What would be wrong is saying it one way at turn two and the other way at turn five — because real speakers do not do that, and a listener hears the inconsistency as a break in identity. The test measures whether the model maintains a persona across turns, which is the fourth of the note’s four voice-presence components, made numeric.

Why these two, and not others

Plenty of contextual capabilities could have been tested. Emotional appropriateness, turn-final intonation, emphasis placement, code-switching. Why these two? Three properties, and any evaluation you design yourself should aim for the same three.

1. Automatically scoreable

   Both reduce to comparing a phoneme sequence against an expected one, using an off-the-shelf phonetic recogniser. No human raters, no cost per evaluation, so they can be run on every checkpoint. Emotional appropriateness has no such scorer — which is why it stayed in the CMOS studies of Chapter 07.

   Cheap enough to be a training-time signal, not just a paper table

2. Binary and unambiguous

   The word was pronounced one way or the other. There is no partial credit, no rater disagreement, no scale to calibrate. Contrast with “was the emphasis in the right place,” where three annotators produce three answers.

   Low-variance metrics detect small model improvements

3. Blind to everything except the capability of interest

   A model can be beautiful, expressive, perfectly timed, and still fail both tests. A model can be flat and monotone and pass both. The tests are almost orthogonal to naturalness, which is exactly what you want when naturalness has saturated.

   Orthogonality is what makes a new benchmark informative

That last row is the whole thesis of the model expressed as a test case. A cascaded voice pipeline — recogniser, language model, synthesiser — cannot pass pronunciation continuation above chance except by luck, because the information was destroyed at the first text boundary. Not badly modelled: destroyed. The test is constructed so that architecture, not model quality, determines the ceiling.

Widget: the two tests, side by side

Left panel: a homograph sentence, its two candidate phoneme sequences, and which one each kind of system chooses. Right panel: a two-turn conversation where the earlier turn establishes a pronunciation variant, and the later turn must match it. Switch the system under test and watch which channel each one can actually read.

Select the cascade and flip the prior-turn variant. The right panel’s answer does not change — because nothing in the cascade’s input changed. Its transcript of the earlier turn says “route” either way. This is the clearest possible demonstration that the continuation test is measuring an architectural property, not a quality difference.

The scaling claim, read carefully

The note reports: “In general, we observe that performance improves with larger models, supporting our hypothesis that scaling enhances the synthesis of more realistic speech.” The accompanying figure compares Tiny, Small and Medium against Play.ht, ElevenLabs and OpenAI, “made with default settings and voices from their respective API documentation.”

Four observations, in descending order of confidence.

The trend direction is credible and unsurprising. Homograph disambiguation is a language-understanding task, and language understanding scales with model size in every other setting. That a 8B backbone beats a 1B backbone at deciding whether “lead” is a noun is about as safe a prediction as this field offers.

No numbers are given in the text. Not stated The accuracies exist only inside a figure. There is no table, no confidence interval, and no per-word breakdown. You cannot reproduce, cite, or compare against these results numerically from the note alone.

The baseline comparison is generous to CSM in a specific way. “Default settings and voices” is the honest thing to disclose, and disclosing it is to the authors’ credit. It is also the weakest configuration of each competitor: commercial systems offer pronunciation dictionaries, phoneme override tags, and SSML controls that a default API call does not use. The comparison measures out-of-the-box behaviour, which is a real thing to measure — but it is not the ceiling of those systems.

The continuation test is structurally unfair to the baselines, and that is the point. None of the compared APIs accepts multi-turn audio history as conditioning. They are not underperforming on this test; they are not playing it. The correct reading is not “CSM is better than ElevenLabs” but “this capability requires an architecture that these products do not have.” The note does not spell that out, and it should have.

07 When a metric stops telling you anything

“Traditional benchmarks, such as word error rate (WER) and speaker similarity (SIM), have become saturated — modern models, including CSM, now achieve near-human performance on these metrics.” That sentence deserves an entire chapter, because “saturated” is a statistical statement and almost nobody treats it as one.

A saturated metric is not a metric that has been maximised. It is a metric whose remaining variation is smaller than its noise floor. The difference matters enormously. A maximised metric says “this problem is solved.” A saturated metric says “this instrument can no longer distinguish the things I care about” — which is compatible with large, audible, important differences between systems.

Word error rate on read speech is now around 1.5% for good systems, and human transcribers disagree with each other at a similar rate. When your measurement error and your signal are the same size, the measurement has stopped being a measurement. And critically, the errors that remain in generated speech — wrong homograph reading, inconsistent pronunciation, prosody inappropriate to the moment — are largely invisible to word error rate by construction, since they do not change which words a recogniser outputs.

The arithmetic of a dead metric

Let us make saturation quantitative, because the number that comes out is startling and explains why the field needed new tests rather than more data.

Suppose system A has a true word error rate of 2.1% and system B of 2.0%. That is a real difference — B makes five percent fewer errors. How much speech must you evaluate to detect it reliably?

Thirty-five hours to resolve a tenth of a point. And a tenth of a point is not what separates a delightful voice from an exhausting one — two systems with identical word error rates can be worlds apart conversationally. That is saturation in one sentence: the metric’s resolution and the differences you care about have parted company.

The same logic applies to speaker similarity, which the note also reports as saturated. Cosine similarity between speaker embeddings of the generated and reference audio approaches the similarity between two genuine recordings of the same person. Once you are inside the within-speaker variation, the number has nothing left to say.

CMOS: comparing against a reference instead of scoring in isolation

Mean opinion score asks a listener to rate one sample on a five-point scale. It is famously unstable: scores drift with the rater pool, the playback equipment, the order of presentation, and what the listener heard yesterday. When systems are far apart that instability is tolerable. When they are close it is fatal.

Comparative mean opinion score fixes the frame of reference by construction. The listener hears two samples — here, one generated and one genuine human recording — and rates the generated one relative to the other on a seven-point preference scale. Absolute calibration no longer matters, because every judgement is a comparison within a pair.

Two studies, one variable

This is the part of the note worth admiring, and it is easy to skim past. The two studies are identical except for one thing.

1. Study one: no context

   “We presented the generated and human audio samples with no context and asked listeners to ‘choose which rendition feels more like human speech.’” The question is about naturalness in isolation.

   Result: no clear preference — naturalness is saturated

2. Study two: ninety seconds of context

   “We also provide the previous 90 seconds of audio and text context, and ask the listeners to ‘choose which rendition feels like a more appropriate continuation of the conversation.’” Same samples, same scale, same rater pool.

   Result: evaluators consistently favour the original recordings

One variable changed — whether the judge could hear the conversation — and the outcome flipped. That is a controlled experiment, and it licenses a conclusion neither study could support alone: the residual gap between generated and human speech is not in the signal; it is in the choice of rendition relative to the conversation.

The note states the conclusion without hedging: “These findings suggest a noticeable gap remains between generated and human prosody in conversational speech generation.” A company research note reporting that its flagship model loses to humans on the exact axis the model was designed for is unusual, and it is the reason this document is worth reading closely at all.

Widget: what a win rate can tell you

The left panel is the discrimination curve for word error rate: how many words you must evaluate to detect a given true difference. The right panel is the CMOS win rate with its confidence interval, so you can see what “no clear preference” and “consistently favoured” look like as intervals rather than adjectives.

Two exercises. Set the WER gap to 1 percentage point — a big difference — and note that a few tens of minutes suffice; then drag it down to 0.1 and watch the requirement explode past thirty hours. Saturation is that curve, not a philosophical position. Then set the win rate to 50% and shrink the rating count: watch the confidence interval widen until “no clear preference” becomes compatible with a substantial real preference. The interval is the finding; the point estimate never was.

The context gap, three readings

What actually causes the second study’s result? The note does not say, and this is where careful reading earns its keep. At least three explanations are consistent with the reported evidence, and they have very different implications.

Reading C deserves more attention than it usually gets. Give a listener ninety seconds of a recording and then two candidate continuations, one of which came from the same recording session, and you have handed them a channel-matching cue that has nothing to do with prosody. Whether that is controlled for — by matching acoustics, or by using generated audio for the context too — the note does not say.

How would you tell the readings apart? Each makes a different prediction, which is the useful thing about naming them:

None of these experiments is in the note. That is not a scandal — it is a research note, not a paper — but it does mean the honest summary of the headline result is narrower than it first appears: on Expresso, with human context, listeners preferred human continuations of CSM-Medium’s size, for reasons the note does not isolate. That is still a valuable, publishable, and unusually candid finding.

Building the next instrument: a checklist you can reuse

Generalise what Sesame did, because the procedure transfers to any field whose headline metric has flattened. Six questions, in order, and the note answers five of them well.

1. Is the metric saturated, or is the problem solved?

   Compute the sample size needed to resolve the differences you still care about. If the answer is absurd — tens of hours, millions of examples, more labelled data than exists — the metric is saturated and the problem is not solved. This is a calculation, not an opinion.

   Answered by the arithmetic two sections up

2. Where did the remaining errors go?

   They did not disappear. They moved into a dimension the metric does not represent. Ask what your metric projects away: word error rate projects away everything about how a word was said, which is precisely where prosodic failure lives.

   Leads directly to phonetic transcription as the new representation

3. Can the new failure be made binary?

   Graded judgements need many raters and produce wide intervals. Binary ones need few samples and detect small improvements. Homograph choice is binary; “was the emphasis apt” is not. Prefer the binary version even if it captures less, because it will actually move.

   Both new tests are pass/fail per sample

4. Can it be scored without humans?

   A test that needs a rater pool runs once per paper. A test scored by an off-the-shelf model runs on every checkpoint and becomes a development signal rather than a publication artefact. Sesame’s two objective tests can run nightly; the CMOS studies cannot.

   Automatic scoring is what makes a benchmark used

5. Is it orthogonal to what already saturated?

   If a model can pass your new test by being more natural, you have built a second naturalness metric. Check the corners: a flat, monotone model should be able to pass, and a beautiful, expressive model should be able to fail. Both are true of the two tests here.

   Orthogonality is the whole point of a new instrument

6. Does the test require an architecture, and did you say so?

   The one Sesame skipped. If only systems built a certain way can attempt your benchmark, publish that precondition next to the scores — otherwise a capability inventory gets read as a quality ranking, which is what happened here.

   The failure mode of every self-designed benchmark

08 What the note does not say

Every teardown reaches a point where the honest move is to stop reconstructing and start listing. This chapter is the catalogue of everything the note leaves unspecified — not as a complaint, but because the shape of what is missing is itself information about what kind of document you are reading.

A research note occupies an awkward genre. It has the technical density of a paper and none of a paper’s apparatus: no appendix, no hyperparameter table, no ablation grid, no error bars, no reproducibility statement. Read one as if it were a paper and you will silently promote plausible inferences into stated facts. Read one as if it were marketing and you will discard genuinely specific technical content. The correct posture is a third thing: treat every claim as true and every absence as significant.

Three kinds of absence appear below, and it is worth separating them:

The missing $N$, and the one typo that might leak it

The most conspicuous omission is the simplest. The note never states how many codebooks CSM uses. It writes $N$ throughout: “one semantic codebook and $N-1$ acoustic codebooks”, “models the remaining $N-1$ codebooks”, “an RVQ tokenizer with $N$ codebooks requires $N$ backbone steps”.

This matters more than a missing hyperparameter usually would, because almost every quantity in Chapters 03 through 05 is a function of $N$: time-to-first-audio, the decoder’s effective batch, the memory bound, the amortisation ratio, the inner-loop cost at inference. Without $N$, none of the note’s efficiency claims can be checked numerically.

There is exactly one clue in the document, and it is an inconsistency. The caption of the amortisation figure reads:

Everywhere else the note says $N-1$. Two readings are available, and honesty requires presenting both:

1. Reading one: a plain typographical error

   “N – 31” is a corruption of “N – 1”, perhaps from a stray character or a copy-edit. Under this reading the caption carries no information about $N$ and the mystery stands.

   Simplest explanation; the phrase is inconsistent with the body text

2. Reading two: a partial substitution

   Someone rendered the caption for a concrete configuration and half-substituted a numeral — leaving a hybrid of the symbolic form and the numeric one. If the concrete value was 32, then “31” is exactly $N-1$, and the caption is the symbol and its value collided into one string.

   Would imply N = 32

Outside the note The open-source CSM release that followed uses a Mimi configuration with 32 codebooks, which is consistent with reading two. Treat that as a pointer to check against the released code rather than as a fact established by the note, because the note itself establishes nothing of the kind. Every worked example in this lesson that uses $N = 32$ is labelled as an assumption for exactly this reason.

Architecture: what a reimplementer would have to guess

Suppose you sat down to build CSM from the note alone. Here is your decision list, and none of these are answered:

The last row is worth dwelling on, because it is the gap between the note and the product. Two minutes of context is enough to demonstrate contextual expressivity in an evaluation. It is nowhere near enough for a companion that remembers a conversation you had last week — and the note’s own framing section promises exactly that (“builds confidence and trust over time”, “memory” listed among the fronts of progress). Whatever bridges that gap is not in this document.

Data: the largest gap, in one sentence

The dataset description is a single sentence: “We use a large dataset of publicly available audio, which we transcribe, diarize, and segment. After filtering, the dataset consists of approximately one million hours of predominantly English audio.”

That sentence contains four verbs, each of which is a pipeline with its own failure modes, and none of which is described.

1. “Publicly available”

   Not defined. Public-domain, openly licensed, and publicly accessible are three very different sets, and the difference is the entire content of the licensing question. One million hours is far beyond the size of the audio corpora with unambiguous open licences.

   Load-bearing for anyone reproducing or deploying

2. “Transcribe”

   With what? A recogniser’s error rate becomes the model’s text supervision. If the transcriber mishears a homograph, the model learns the wrong text-to-pronunciation association — on the exact axis Chapter 06 measures.

   Directly upstream of the headline evaluation

3. “Diarize”

   Speaker segmentation is the mechanism that produces the turn structure and the speaker markers. Diarisation errors merge two speakers into one identity, which would teach the model that a single speaker’s voice changes mid-conversation — the precise opposite of the consistency the note wants.

   Failure mode is invisible in aggregate metrics

4. “After filtering”

   Filtered on what? Audio quality, transcription confidence, language, duration, speaker count, licence? The filter defines the dataset far more than the raw source does, and one million hours is the number after it.

   The most consequential undocumented design choice

There is one revealing detail elsewhere in the note. In the limitations section: “CSM is currently trained on primarily English data; some multilingual ability emerges due to dataset contamination, but it does not perform well yet.”

“Contamination” is a candid word. It tells you the language filter was imperfect, that non-English audio entered the corpus without being intended, and — interestingly — that the resulting multilingual ability was noticed rather than designed. It is a small admission that says a great deal about the pipeline: automatic, large-scale, and imperfectly controlled, which is what a million hours implies anyway.

Training, and the ablations that would have settled it

What the note gives: three sizes (1B/100M, 3B/250M, 8B/300M), sequence length 2048, five epochs, and the one-sixteenth amortisation ratio. What it does not give:

• Batch size, optimiser, learning rate, schedule, warm-up. Benign omissions individually — but they make the memory discussion in Chapter 05 unanchorable, since $B$ is the one free variable in $B \times S$.
• Hardware and wall-clock. The note motivates amortisation by saying the memory burden “slows down training, limits model scaling, and hinders rapid experimentation.” No before-and-after throughput figure is given for the fix that claim justifies.
• Why one-sixteenth. No sweep, no comparison against one-eighth or one-thirty-second. “No perceivable difference in audio decoder losses” is reported as an observation, with no curve.
• Whether the backbone is initialised from a pretrained language model. Actually this one is answered, in the limitations: “It also does not take advantage of the information present in the weights of pre-trained language models.” So: trained from scratch. That is a significant and easily-missed fact, and it makes the homograph results more impressive — whatever text understanding the model has, it learned from transcribed speech alone.
• Ablation of the split point. The central architectural claim — that splitting at codebook zero beats splitting where Moshi splits — is never tested in the note. No comparison, no perplexity, no listening study. It is asserted as a design choice, not demonstrated as an improvement.

That final bullet is the most important one in this chapter. The note’s first stated contribution is unevaluated. Everything measured in the evaluation section compares CSM against other products and against human recordings; nothing compares CSM against CSM-with-a-different-split. The architectural argument is coherent, well motivated, and entirely theoretical as presented.

Evaluation: figures without numbers

Every quantitative result in the note lives inside an image. There are no tables in the text, no accuracy values, no confidence intervals, no win-rate percentages. What is reported in prose is the direction of each result.

The single most useful number in the entire note — how big the context gap is — is the one you cannot extract. That is worth stating plainly, because it is what separates “we are nearly there” from “this is a different order of problem,” and the note’s own final paragraph leans toward the second while its figure could support either.

Widget: the specification map

Every substantive claim in the note, plotted by how completely it is specified. Click any bar to see what is known, what is missing, and what it would take to close the gap. Filter by category to see the shape of the document: architecture is well specified, data is not, and evaluation is directionally reported but numerically absent.

The distribution is the finding. A document that specifies its architecture in symbolic detail and its dataset in one sentence is telling you where its authors believe the contribution lies — and, inadvertently, where the risk actually lives.

09 A companion, not an assistant

Strip the technical section away and the note’s opening is a product thesis with a specific shape. It is not arguing that voice interfaces should be more pleasant. It is arguing that a particular product category — the always-on companion — is impossible without solving prosody, and that everything else follows.

The tell is in the failure mode they describe. Not “users find it frustrating” but “after the initial novelty wears off.” That is a retention claim, and retention claims are about the shape of a curve, not a single interaction.

Consider the two product categories side by side, because the engineering requirements diverge almost immediately:

Every column-two entry is a conditioning or timing requirement, and every one of them is what CSM was built to address. The architecture is not a general improvement to text-to-speech; it is the speech component of a companion, designed backwards from that product.

This also explains the note’s otherwise-odd structure: a warm, non-technical product essay, then a dense technical section, then a hiring pitch. The document is doing three jobs, and the technical section’s job is to make the product essay credible.

The real-time budget, derived from first principles

A companion has to answer in real time, so let us derive what that requires from the two numbers we have: a frame rate of 12.5 Hz and a two-model inference loop. This is the arithmetic that explains every size decision in the note.

Step 5 is the note’s architecture argument reduced to a single ratio. The delay pattern is not merely worse for time-to-first-audio; at any reasonable $N$ it cannot sustain real-time generation at all. Splitting the work between a large model that runs once per frame and a small one that runs $N-1$ times is the only structure that fits inside 80 ms.

Step costs assumed The 20 ms backbone step is a stand-in — the note reports no latency figures whatsoever, which for a real-time conversational system is a striking omission. What is not assumed is the structure: 12.5 frames per second and one backbone step per frame are both stated.

Which model actually costs more?

Here is a result that inverts the intuition the note’s language creates. “The decoder is significantly smaller than the backbone” is true of parameter counts and false of inference cost.

Use the standard estimate that a forward pass costs about $2P$ floating-point operations per token, where $P$ is the parameter count. Then compute the cost of one second of generated audio for each model size, with $N = 32$.

Three sizes, and in every one of them the “small” decoder consumes the majority of inference compute. The parameter ratio at Medium is about 27:1 in the backbone’s favour; the compute ratio is 1:1.16 in the decoder’s. The multiplier that flips it is $N-1 = 31$.

This also puts a hard limit on scaling $N$. Doubling the codebook count buys a modest gain in reconstruction fidelity and doubles the decoder’s inference cost, which is already the majority of the budget. Mimi’s aggressive 12.5 Hz frame rate is what keeps the arithmetic survivable at all: at 50 Hz, every number in this section would be four times worse.

Widget: the inference budget

Choose a model size and a codebook count, and watch the per-second budget split between the two models. The deadline bar shows whether one frame’s work fits inside 80 ms at the chosen step latencies — drag the backbone step cost up and watch the decoder’s per-step budget collapse toward impossibility.

Set $N$ to 4 and the decoder nearly disappears from the budget — at the cost of audio fidelity, since four codebooks cannot reconstruct speech well. Set it to 32 and the decoder dominates. Somewhere in the middle is a design point, and the note never tells us which one was chosen. This widget is, in effect, the search space the Sesame team explored and did not report.

Why the decoder barely grows across model sizes

Look again at the three configurations and notice the asymmetry:

The backbone grows eight-fold from Tiny to Medium; the decoder grows three-fold, and almost all of that growth happens in the first step. Why?

Because the two models are scaling against different limits. The backbone’s job — conversational understanding, prosodic planning, homograph resolution — is the kind of task that improves with capacity indefinitely, which is exactly what the note’s scaling observation reports. The decoder’s job is a local reconstruction mapping that saturates early: past a certain capacity, there is nothing more to learn about turning a semantic code and a frame state into acoustic residuals.

And because the decoder pays a 31× latency tax. Every parameter added to the decoder is multiplied by 31 invocations per frame in both the training-memory ledger and the inference budget. Adding 100M to the backbone costs 12.5 extra forward passes per second of audio; adding 100M to the decoder costs 387.5. The decoder is the expensive place to spend parameters even though it is the cheap-looking place.

What the product needs that CSM is not

Here is a distinction the demo blurred and the technical section is careful about: CSM is not a conversational agent. It generates speech, conditioned on conversation history. Everything else a companion requires sits outside it.

1. Something must decide what to say

   The note is a speech-generation paper. The text of the turn is an input. Whatever produces that text — a language model, a retrieval system, a persona — is not described anywhere in the document.

   The interesting half of a companion is out of scope

2. Something must transcribe the user

   Step 6 of the inference loop requires the user’s utterance “represented by interleaved audio and text transcription tokens.” The audio the model can tokenise itself; the transcription implies a recogniser in the loop, with its own latency and its own errors.

   A speech recogniser is a required dependency, unnamed

3. Something must decide when the user has finished

   The model starts generating when asked. Deciding when to ask — endpointing, turn detection, handling a mid-sentence pause — is a whole discipline, and it is the discipline that determines whether a voice product feels responsive or rude.

   See the turn-taking lesson

4. Something must handle interruption

   When the user talks over the model, generated audio is already in a playback buffer and already in the conversation history. Cancelling, flushing and repairing the context is an engineering problem no model architecture solves for you.

   See the voice agent stack

5. Something must remember beyond two minutes

   The training context is about two minutes. A companion’s premise is memory across days. The note lists memory as a front of progress and does not describe a mechanism.

   Named as a goal, not as a component

None of this is a criticism of the model — it is a correction of the frame. The demo people reacted to was a system; the note describes one component of that system, and the component it describes is the one that makes the others worth building. But if you are budgeting a voice product from this document, four of the five items above are yours to solve.

10 A speaking model, at the edge of a conversation model

The note ends with a paragraph that reframes everything before it. Most readers stop at the demo. The paragraph is the most important technical statement in the document.

Read that against Chapter 01’s four components of voice presence and the accounting becomes exact:

Three out of four, with the fourth named as the frontier. That is an unusually precise self-assessment, and it lines up exactly with the CMOS result: listeners with context prefer the human, and one entirely plausible reason is that human continuations carry the timing and pacing information a turn-based model structurally cannot produce.

Content versus structure, made concrete

What does “the structure of the conversation” contain, exactly? Enumerate it, because the list is the specification of the next model:

• Turn taking. Who speaks next, and how the handover is negotiated — usually through prosodic cues in the final syllables rather than through content.
• Pause length. A 200 ms gap before “yes” means something different from a 900 ms gap before the same word. The gap is a message.
• Overlap. Real conversation is full of it — simultaneous starts, collaborative completions, the speaker who keeps going for half a second after the other begins.
• Backchannels. “Mm-hm”, “right”, a short laugh — produced while the other person is speaking, and absent from any turn-based formulation by definition.
• Interruption and recovery. Being cut off, yielding, retaking the floor, repairing a half-finished sentence.
• Pacing over the whole exchange. Conversations accelerate and slow, and both parties converge on a shared rhythm.

Every one of these is a property of two streams in time, not of a single utterance. A model that emits one turn and stops cannot represent any of them, however good the turn is. And the architecture that can is the one Moshi built: two audio streams modelled simultaneously, so silence, overlap and interruption are all just token patterns in a joint sequence.

There is a real irony here, and it is worth naming. CSM’s data pipeline — “transcribe, diarize, and segment” — is the very process that destroys the structural information a duplex model would need. Diarisation and segmentation take overlapping, messy, human conversation and turn it into tidy alternating turns. The preprocessing that makes CSM trainable is the preprocessing that makes duplex untrainable on the same corpus.

Widget: turn-based against duplex

One conversation, two architectures. The top pair of lanes is a turn-based model: the user speaks, a gap, the model speaks. The bottom pair is a duplex model holding both streams at once. Trigger an interruption or a backchannel and watch what each architecture can represent.

Select “user backchannels”. In the duplex lanes the “mm-hm” overlaps the model’s speech and the model keeps going — which is what a person does. In the turn-based lanes there is nowhere to put it: either it is a turn, in which case the model must stop and respond to it, or it does not exist. Both options are wrong, and no amount of prosodic quality fixes either.

Then select “user pauses mid-thought”. The turn-based system faces a decision it has no information to make: is this the end of a turn or a pause inside one? That decision is made outside the model, by an endpointer, on a timer — which is why a turn-based architecture pushes its hardest conversational problem into a component that knows the least.

Honest critique, six items

Everything below is a criticism of the document, not of the work. Several of these are entirely reasonable things for a research note to omit; they are listed because a reader who does not notice them will over-read the evidence.

1. The central architectural claim is unevaluated

   Splitting at the zeroth codebook is the note’s first stated contribution, and nothing in the note compares it against splitting elsewhere. No perplexity, no listening test, no latency measurement of the alternative. The argument is sound; the evidence is absent.

   Severity: high — it is the headline claim

2. No latency numbers, in a paper about latency

   “Enabling low-latency generation” is the justification for the whole design, and the document contains no milliseconds anywhere. Time-to-first-audio, real-time factor, throughput at each size — all absent, all easy to measure.

   Severity: high — directly checkable, and not checked

3. A capability benchmark presented as a product comparison

   The pronunciation continuation test requires multi-turn audio conditioning, which the compared commercial APIs do not accept. Charting them side by side implies a quality ranking where the real finding is an architectural precondition.

   Severity: medium — the finding is real, the framing is not

4. All quantitative results live inside images

   No tables, no values in prose, no intervals. “Consistently favour” is compatible with 55:45 and 80:20, and the difference between those two is the difference between a nearly-solved problem and an open one.

   Severity: medium — blocks citation and comparison

5. One sentence of data description for one million hours

   “Publicly available” is undefined; the transcription, diarisation and filtering pipelines are unnamed; the licensing question is untouched. For a system whose behaviour is learned from conversational structure, the corpus deserved more than the architecture got.

   Severity: medium–high — the pipeline is the model at this scale

6. Two known weaknesses, disclosed by the authors

   English-only with “multilingual ability… due to dataset contamination”, and no use of pretrained language-model weights. Both are stated plainly in the limitations, which is to the note’s credit — and the second one means every bit of text understanding in the homograph results was learned from transcribed speech alone.

   Severity: low as a criticism — disclosed, and interesting

What the note gets right

Criticism is cheap. Four things in this document are genuinely better than the norm, and it is worth being specific about them because they are the parts worth imitating.

And one structural virtue worth calling out: the technical section reasons symbolically. Every claim is stated in terms of $N$, $B$, $S$ — which means the argument can be checked, generalised and reused even though the constants are missing. That is a sign of authors who understood their design rather than merely reporting it.

Where CSM sits in the lineage

Place the note on the timeline and its contribution becomes easy to state in one line.

In one line: CSM is the model that took conversation history seriously as conditioning, and measured whether that was enough. Its answer — necessary but not sufficient — is what points the field at duplex.

The lineage also explains why CSM is not simply superseded by Moshi despite Moshi being duplex and a year older. They optimise different things. Moshi’s dual-stream architecture models structure and pays for it with a speech-native model whose linguistic reasoning is weaker than a comparable text model’s. CSM models content extremely well, conditioned on more context than anything before it, and declines to model structure at all. The next generation is trying to have both, which is exactly the “fundamental changes across the stack” the note predicted.

Connections

This lesson sits in the middle of two ladders that meet at voice. Here is where to go next in each direction.

Prerequisites and background

• VALL-E — TTS as codec language modelling. The autoregressive-plus-non-autoregressive factorisation Chapter 04 compares against, and the origin of the three-second enrolment prompt.
• Moshi — speech-text foundation model for real-time dialogue. The RQ-Transformer formalism, the Mimi codec, and the dual-stream duplex design CSM explicitly is not.
• EnCodec — the neural audio codec. Residual vector quantisation from the bottom up, if Chapter 03’s hand-worked example left you wanting the full treatment.
• TTS architectures. The pre-token generation of speech synthesis, and why the field arrived at discrete tokens at all.
• AudioLM. The semantic-then-acoustic two-stage design whose bottleneck the note argues against.

Where to go from here

• The voice agent stack. Everything Chapter 09 said CSM is not: recognition, endpointing, barge-in, transport, the latency ledger, and the production decisions that turn a speech model into a product.
• Voice turn-taking. The endpointing and barge-in problem from zero — the “structure” half that CSM declines to model.
• Streaming speech. Chunked attention, incremental decoding, and where the milliseconds actually go.
• Qwen2.5-Omni. The thinker–talker answer to the same factorisation question, with a pretrained language model as the brain — the thing CSM’s limitations section says it does not yet do.
• PersonaPlex. Voice and role control for full-duplex models, and a benchmark for interruption handling.
• Duplex speech-language-action. Where the note’s closing paragraph points: synchronised speech, reasoning, and action in one real-time loop.