DuplexSLA: Speech, Language, and Action on One Clock

Chapter 0: A Voice Agent That Can Only Talk Is Not an Agent

You are driving. The cabin is freezing. You say, mid-conversation, without stopping the conversation: "It's too cold in here — turn up the AC, put on something relaxing, and I'm getting hungry, find me a restaurant."

A human co-driver would have already reached for the climate dial before you finished the word "cold". They would keep talking to you while they did it. They would not say "please hold" and then go quiet for two and a half seconds.

Every deployed voice assistant you have used does exactly that. And the reason is not that the language model behind it is weak. The reason is architectural: the assistant's mouth and the assistant's hands are on different clocks, and neither is on yours.

The pipeline everyone actually ships

Open any production voice-assistant architecture diagram from the last decade and you will find the same four boxes in the same order. The paper names them in its first paragraph of Section 1, and it is worth drawing before we criticize it, because most of this lesson is a rebuttal to this picture:

This is called a turn-based or cascaded pipeline: each box waits for the previous box to finish a unit of work. It is easy to build, easy to debug, and every component can be swapped independently. Those are real virtues, and it is why the architecture dominates production.

It also has two structural problems in duplex spoken interaction — and the word structural is doing heavy lifting. These are not bugs you fix with a better VAD or a better LLM. They follow from the shape of the pipeline itself.

Structural problem 1: silence is ambiguous, and energy cannot disambiguate it

An energy-based VAD sees one number over time: how loud is the microphone right now. From that number it must decide whether the floor has been yielded. But four completely different conversational events produce nearly identical energy traces:

Look at the last two rows. The first 800 milliseconds are acoustically identical. "You're right." followed by silence is a backchannel; "You're right, but the project schedule is tight, I don't really have a choice" is an interruption. Nothing in the energy envelope distinguishes them at the moment you must decide. The difference is semantic, and it only exists in meaning, not in loudness.

The industry's answer has been to bolt a second model on top: a semantic VAD — typically a small classifier or an ASR-plus-LLM turn detector that reads the partial transcript and predicts whether the turn is really over. The paper's Section 1 concedes this "recovers part of this nuance", and then makes two objections that you should hold onto for the rest of the lesson:

1. It adds latency. The detector is another chain: buffer audio, transcribe, classify, decide. Every millisecond it spends is a millisecond the user waits after already having stopped talking.
2. It cannot see the assistant's internal state. This is the deeper objection. An external detector sees the user's audio. It does not see what the assistant is currently saying, what the assistant was about to say next, or how committed the assistant is to the current sentence. But "should I yield the floor?" depends on all three. A short user utterance that contradicts what the assistant just said is an interruption; the same utterance agreeing with it is a backchannel. The evidence lives on both sides of the conversation, and only one side is visible to the bolted-on module.

Structural problem 2: there is no good place to put a tool call

Now the part this lesson is really about. Suppose the assistant, mid-conversation, needs to do something — raise the AC, start music, start navigation. Where in the turn loop does that tool call go?

There are exactly three options in a turn-based pipeline, and the paper's introduction dispatches all three in a single sentence. Sit with each one, because the fact that all three are bad is the entire motivation for the architecture:

Option C is the interesting failure. It fails not because the model cannot produce the JSON, but because one channel cannot carry two things that both need the clock. Speech is a real-time signal: every 40 milliseconds of audio requires its token, on time, or the waveform has a hole in it. Tool-call JSON is bursty and long. Multiplexing them onto one lane means the audio starves whenever the JSON is talking.

Once you see the problem this way, the solution names itself, and the paper says so directly: what is missing is "a model that can listen, speak, think, and act on one synchronized timeline." Not one channel — one timeline, with enough channels that nothing has to starve.

The teaser: watch the same request through both architectures

Before any machinery, watch the phenomenon. The simulation below runs one user request — "It's too cold in the car, turn up the AC" — through the cascaded pipeline and through DuplexSLA, on the same wall clock, and marks the moment the AC actually changes.

Do not read the code or the architecture yet. Just watch when the orange marker lands in each lane, and notice what the assistant's voice is doing at that moment.

Three things the sim is showing you, in order of how much they should bother you:

1. The cascade's tool call cannot fire before the user stops talking. Not "does not" — cannot. The ASR final hypothesis is gated on the endpoint, and the LLM reads the final hypothesis. The architecture forbids acting on a request while it is still being spoken.
2. DuplexSLA's tool call fires while the user is still mid-sentence, at the chunk where the request became semantically clear — and the assistant's voice does not pause when it happens.
3. Switch the pattern to "multi action." The cascade's delay balloons to 4.71 s (it must hear all three intents, then serialize three calls); DuplexSLA stays at 0.68 s, because each call is anchored to the chunk where its own intent became clear. That is a 6.9× gap on the pattern that matters most for real assistants.

What "Action" means here — the third letter is not decoration

The name is DuplexSLA: Speech, Language, Action. Plenty of systems claim two of those. The paper is explicit about what makes the third non-trivial, and this sentence is the one to memorize:

Notice the scope: the action lane carries both tool calls and turn-taking decisions. That is a design unification most readers miss on the first pass. "Should I stop talking because the user interrupted?" and "should I call navigate()?" are, in this architecture, the same kind of event: a discrete, time-stamped decision emitted on a dedicated lane by the same backbone that is producing the voice. Interruption handling is not a special case — it is a tool call with an empty argument list.

The scoreboard we are going to earn

Here is everything this lesson will explain and justify, with the paper's numbers attached, so you know what the destination looks like. Do not try to absorb it now — return to this table after Chapter 9 and check that every row means something to you:

Two rows in that table point in opposite directions, and that tension is the intellectual spine of the lesson. DuplexSLA is dramatically faster and it is slightly less accurate on tool calls. Chapter 10 refuses to paper over that; the interesting question is not "who wins" but "what did the speed cost, mechanically, and is the trade the right one for a voice interface?"

The paper's identity card

One editorial note on how to read a report like this. It is not a proof-heavy theory paper — there is exactly one piece of notation in the whole thing (the chunk index) and no theorems. It is a systems paper, which means the intellectual content lives in the design decisions and the numbers that justify them. Our job in the next eleven chapters is to make every decision feel inevitable and every number feel earned.

Where we are going

One habit to install before Chapter 1. Every time this paper quotes a latency, convert it into chunks by dividing by 0.16. A 0.27 second delay is 1.7 chunks. A 1.18 second delay is 7.4 chunks. Once you think in chunks, the numbers stop being abstract benchmark scores and become statements about how many grid squares of hesitation the user perceives. That conversion is the single most useful reading aid in this lesson, and it is why the next chapter is entirely about the clock.

What "full duplex" actually means — and what it does not

The term is borrowed from telephony, where a full-duplex line carries audio in both directions simultaneously, as opposed to a half-duplex walkie-talkie where one party transmits at a time. Applied to spoken dialogue models, the paper's abstract gives the operational definition: a full-duplex design is one "where the model continuously listens to the user while generating responses."

Read that carefully. Continuously listens — not "listens, then generates, then listens again." At every instant, the model is consuming user audio and producing assistant audio. There is no moment where the microphone is logically closed.

This distinction is worth a moment because it explains a result you will meet in Chapter 9 that otherwise looks like a typo. On the backchannel scenario, one commercial system scores 0.33% and another 13.00%. Those are not broken systems; they are systems whose architecture has no way to represent "acknowledged, continuing." Every short user utterance is either ignored entirely or treated as a turn. The label does not exist, so the behavior cannot exist.

Why this paper, and why 2026

DuplexSLA does not appear from nowhere. Its introduction sketches three converging lines of work, and knowing the shape of each will make Chapter 3 much easier:

DuplexSLA sits precisely at the intersection: it takes a strong audio foundation model (Step-Audio 2 mini, 7B), puts it on a duplex clock, and then adds the thing none of the three lines had — a rate-limited textual lane for actions, co-decoded with the voice. The paper's own framing of its novelty is careful and worth quoting: "We focus on a combination that existing duplex backbones and benchmarks do not jointly stress: semantic-driven turn-taking control plus in-conversation tool calling."

The word jointly matters. Turn-taking benchmarks exist. Tool-calling benchmarks exist. Nothing measured them on the same timeline, which is why Section 5 has to build a new benchmark before it can report a result.

The latency arithmetic, in chunks

Let us do the conversion habit once, together, on the headline numbers, so that the rest of the lesson can lean on it. One chunk is 0.16 s. Divide:

Now stare at that last pair. DuplexSLA answers within roughly one and a half to two chunks of the moment the user's turn ends. Since the model can only act at chunk boundaries, and a semantic anchor lands uniformly at random inside a chunk, even a hypothetical perfect model pays an expected half-chunk (80 ms) of quantization latency before it can do anything at all. DuplexSLA's 270 ms is therefore not "fast for a neural system" — it is within about 190 ms of the architectural floor imposed by its own clock.

A first look at the three lanes

Figure 1 of the paper is a chunk-level architecture diagram, and its worked example is the one we will keep returning to. The user, sweating, says: "The heat is killing me! I feel like I'm going to get heatstroke." Here is what the three lanes carry, aligned on the same clock:

Four observations that will each become a chapter:

• The assistant's spoken sentence says "I've just switched on the air conditioning" — and the action lane makes that sentence true at the moment it is spoken, not three seconds later. The speech and the side-effect are simultaneous because they are emitted in the same autoregressive step.
• The action lane carries rationale text before JSON. That planning fragment is not decoration — it is what gives the action object something to reason with, and Chapter 5 shows how its length interacts with the token budget.
• The function chosen is decrease_car_setting with argument "air-conditioner: 26 degree" — the user is hot, so the AC temperature setting goes down. A small detail that reveals the schema is about setting values, not about comfort semantics.
• Every lane terminates its chunk with an explicit marker. The grid is enforced by tokens, not by wall-clock timers. Chapter 2 makes that mechanism exact.

The car cabin as a laboratory

It is worth being blunt about the deployment target, because it explains several design choices that look arbitrary in the abstract. All 50 tool schemas in this paper are cabin and smart-home functions: climate, windows, seats, navigation, media, on-device search, phone calls. The training data is Chinese-language dialogue synthesized with 18 voice-clone speakers. The examples are people who are cold, hungry, or heading somewhere.

Why is this a good laboratory rather than a limitation to apologize for? Because the in-car setting is the environment where the failure modes of turn-based voice hurt most:

Keep this in mind, and Chapter 10's scope discussion will land properly: the cabin is where the argument is strongest, which is both why the results are convincing and why we should be careful about extrapolating them to open-domain agentic tool use.

How to read this lesson

Four notes on method, because a systems paper rewards a particular reading posture:

1. Every design choice is a budget decision. When you meet a number like "at most 10 action tokens per chunk", do not memorize it — ask what it is trading against. In this case, assistant audio smoothness. The paper says so explicitly, and Chapter 5 does the arithmetic.
2. The data section is not filler. For a model that must learn when as well as what, the format of the training data is the method. Chapter 7 is one of the most important chapters here, and it is the one readers skip.
3. Distinguish what the paper measured from what it asserted. This lesson flags the difference every time. "DuplexSLA is 4× faster" is measured. "A large native duplex model absorbs these phenomena into its core competence" is an assertion supported by, but not identical to, the measurement.
4. Keep the deployment context in view. Every one of the 50 tool schemas is a car cabin or smart-home function. This is an in-vehicle assistant paper wearing foundation-model clothes, and that shapes what generalizes. Chapter 10 says so plainly.

Objections you are probably already forming

Good. Here are the five most common ones, with where each is answered:

What you will be able to do by the end

The bar for this lesson, stated as capabilities rather than topics. After Chapter 11 you should be able to:

• Write out the per-chunk token serialization from memory, in order, and say which tokens the model produces.
• Compute the action-token budget for any accelerator from its per-token decode latency, and say what happens when the budget is exceeded.
• Explain why the chunk is 160 ms rather than 100 or 200, from the codec and encoder rates.
• Lay out an action object's tokens across chunks under the FIFO and spill rules, and compute its trigger time and its full-arrival time separately.
• Distinguish pause, interrupt, and backchannel by what each lane does, and explain why the first 800 ms of audio cannot distinguish the last two.
• Explain why 90,000 hours of assistant-side ASR is load-bearing for sub-second tool-call latency.
• Read every results table critically — including knowing why one delay column reads N/A and why a 0.36 s delay at 10% accuracy is not a good number.
• Name the ablation the paper is missing and describe the experiment that would supply it.

One last framing note before the clock. Throughout this lesson, "action" means something narrower and sharper than in agent literature generally. An action here is an object emitted on a textual lane at a specific chunk index: a name, optional planning text, optional JSON arguments, and a trigger offset. It is not a plan, not a policy, not a rollout. Its entire ambition is to be the right object at the right millisecond — and, as we will see, that ambition turns out to be enough to make voice feel agentic.

Chapter 0 in review

The problem statement, compressed. Two structural failures of the turn-based pipeline, and what each one costs:

If you remember one thing: a voice agent's mouth and hands are on different clocks, and putting them on the same one is an architectural change, not an optimization.

Three exercises before Chapter 1:

1. Explain, without using the word "slow", why a cascade cannot dispatch a tool call before the user's endpoint even with infinitely fast components.
2. A user says "you're right" over the assistant. List everything a system would need to know to decide correctly, and mark which items an external turn detector can see.
3. Convert these to chunks at 160 ms: 0.32 s, 0.95 s, 2.33 s, 4.71 s. Which pair differs by roughly fifteen chunks?

Chapter 1: The 160 Millisecond Clock

A language model has no idea what time it is. It has an order — token 1, token 2, token 3 — and nothing else. Order is not time. A transformer will happily spend a hundred milliseconds on one token and a microsecond on the next, and nothing in the architecture notices or cares.

Speech does care. Air pressure has to arrive on schedule. If the assistant's waveform needs a sample at t = 3.240 s and the model has not produced it, the user hears a hole. There is no "buffer more" escape hatch in a conversation, because buffering is latency and latency is the thing we are trying to kill.

So the first thing a full-duplex spoken language model must do is nail token order to wall-clock time. That nail is the conversational clock, and in DuplexSLA it ticks every 160 milliseconds.

That is the paper's entire notation. Time t in seconds; chunk index c as the floor of t divided by the chunk size. Everything else in this paper is a statement about what happens inside chunk c.

Concretely: chunk 0 covers [0.00, 0.16) seconds. Chunk 1 covers [0.16, 0.32). Chunk 37 covers [5.92, 6.08). An event at t = 4.31 s belongs to chunk 26, because 4.31 / 0.16 = 26.9 and the floor of that is 26. Do that conversion three times by hand right now; the rest of the lesson assumes it is automatic.

What a chunk contains

At every chunk, the model receives one user audio segment and one assistant audio segment, and produces two outputs: an assistant audio segment and an action segment, both indexed by the same c. Here is the full inventory, straight from Section 2.1:

Check the arithmetic yourself, because the number 160 is not arbitrary — it is the number that makes the three granularities line up:

The chunk is the smallest window in which every stream completes a whole number of its own units. Choose 120 ms and the user features (80 ms apart) no longer tile it. Choose 100 ms and neither stream tiles it. The clock is the least common multiple of the physical rates the model is built on, and everything else — the action budget, the latency floor, the benchmark's delay resolution — is downstream of that choice.

Rates: what all of this means per second

Convert everything to per-second quantities, because that is the unit in which hardware is specified:

Appendix D states the middle rows as a serving fact: "Per-chunk model output: 5 assistant TA4 tokens (always) plus up to 10 action text tokens." The words always and up to are the whole story. The voice is a floor; the actions are a ceiling.

Hand-worked: does it fit in real time?

Real-time full-duplex interaction requires the per-chunk decoding cost to fit inside one 160 ms chunk on the actual inference hardware. Let us make that requirement quantitative, by hand, with every intermediate step. This is the arithmetic that determines the number 10.

Setup. Let N be the number of tokens the model must autoregressively decode in one chunk, and let t_tok be the wall-clock time to decode one token on your accelerator. Define the real-time factor:

The system is real-time capable exactly when RTF ≤ 1. Now push numbers through it.

Step 1 — the floor. The TA4 unit is unconditional: 5 tokens every chunk whether the assistant is speaking or silent (silence still needs its anchor and its four silence audio codes). So N ≥ 5 always.

Step 2 — the ceiling. Add the action channel's maximum:

Step 3 — the per-token time this allows. Set RTF = 1 and solve for t_tok:

Equivalently, the decoder must sustain 1000 / 10.667 = 93.75 tokens per second of autoregressive throughput, sustained, with no gaps.

Step 4 — what happens if you miss. Suppose your accelerator decodes at 12 ms per token. Then:

Read that last line again. A 12.5% overshoot on token latency does not degrade quality by 12.5%. It makes the assistant fall progressively further behind, one fifth of a second for every second of talking, until the conversation is unusable. Real-time systems do not degrade gracefully; they diverge. This is why the budget has to be a hard cap rather than a soft preference.

Step 5 — solve for the action budget. Turn the equation around. Given a measured t_tok, how many action tokens can you afford?

That last row is the sobering one and it is worth stating as a principle: on slow enough hardware, the action channel simply cannot exist. The third letter of "SLA" is purchased with decoding throughput. This is not a metaphor — it is the literal accounting the paper does in Section 2.3, which says the throughput of a 7B backbone "leaves room for only a small number of action-channel tokens per chunk", and settles on 10 "with a safe margin against the per-chunk wall-clock budget."

python — the budget, as code
def action_budget(chunk_ms=160, tok_ms=10.0, ta_tokens=5, margin=1):
    """How many action tokens per chunk can this accelerator afford?"""
    total = int(chunk_ms / tok_ms)      # tokens we can decode in one chunk
    return max(0, total - ta_tokens - margin)

action_budget(tok_ms=10.0)   # -> 10   (the paper's setting)
action_budget(tok_ms=12.0)   # -> 7    (retune down, no retraining needed)
action_budget(tok_ms=6.0)    # -> 20   (spend it on longer planning text)

The latency floor nobody can beat

Here is a consequence of having a clock at all, which the paper does not spell out but which its numbers quietly respect.

A semantic event — the moment the user's request becomes clear, the moment an interruption really starts — happens at some continuous time t^★. The model can only act at a chunk boundary. So the earliest possible response time is the end of the chunk containing t^★:

So a perfect model on this clock averages 80 ms of unavoidable delay. Now compare with the measured numbers from Chapter 9: DuplexSLA's normal-turn delay is 270 ms and its interrupt delay 400 ms. Subtract the floor:

Two chunks to recognize a semantic interruption and switch the voice to silence. That is the actual claim hidden inside "0.40 s", and it is a much more impressive number when you strip out the clock's own contribution.

What if you shrank the chunk?

The obvious next thought: if 160 ms costs 80 ms of average quantization latency, why not use 80 ms chunks? Work it through — this is a derivation the paper does not do, so treat it as our extension of its accounting, not as its claim.

At Δ = 80 ms, the assistant channel would carry 2 audio tokens (still 40 ms each) plus its anchor, so a "TA2" unit of 3 tokens. The user channel would carry 1 feature. The per-chunk decode budget scales down with the chunk:

Notice what the last column does: action bandwidth per second falls as chunks shrink, even though the audio rate is unchanged. The reason is fixed overhead. Every chunk pays for its text anchor and its boundary markers regardless of length, so shrinking the chunk raises the fraction of the budget spent on framing rather than content. Halve the chunk and you halve the quantization latency, but you also squeeze the lane that carries the tool calls — and, at 40 ms, you can no longer fit even a short JSON fragment inside a single chunk.

160 ms is the compromise: fast enough that the quantization floor (80 ms average) is below human turn-taking sensitivity, slow enough that ten action tokens fit alongside the voice. That is the whole justification, and now you can reproduce it from first principles.

Things to try, in order. (1) Start at the paper setting and push the token time up until the bar turns red — note that it happens at 10.67 ms, exactly where the Step 3 arithmetic said it would. (2) Set the chunk to 40 ms and watch the action budget collapse to a couple of tokens while the latency floor drops to 20 ms — the trade made visible. (3) Set the chunk to 320 ms and notice you can afford twenty-three action tokens but the user now waits 160 ms on average before anything can happen. Somewhere in the middle is a product decision, and 160 ms is the paper's.

Silence costs exactly as much as speech

Here is a design decision that looks wasteful and is not. Section 2.2 says: "Whenever the chunk has nothing to say, T is predicted as a special anchor token (<vad_silence> or <tts_pad>) and the four A tokens are predicted as the corresponding silence audio codes."

So a silent chunk costs five tokens, the same as a speaking chunk. Over a ten-minute conversation:

Why pay for silence? Three reasons, each of which becomes visible later:

1. Constant compute means predictable latency. A system whose per-step cost depends on whether anyone is talking has jitter, and jitter in a real-time loop is worse than a slightly higher constant cost. Here every step costs the same, so the RTF calculation above is not an average — it is the actual per-chunk cost, always.
2. Silence becomes a decision the model makes. Predicting <vad_silence> is a choice, scored by the loss, conditioned on everything the user is currently doing. That is precisely what lets Chapter 6's pause behaviour exist: the model actively decides to keep quiet while the user hesitates, rather than passively not being invoked.
3. The clock never stops. If silent chunks were skipped, chunk index would no longer be a linear function of time, and every timestamp in the paper — every trigger offset, every delay measurement — would need a separate accounting of skipped time. The uniform grid is what makes c = ⌊t/Δ⌋ usable as a clock.

Two distinct silence anchors are worth distinguishing now, because they mean different things. <vad_silence> is "there is nothing to say here" — the assistant is genuinely not speaking. <tts_pad> is "the text has run out but the audio has not" — the text anchors for an utterance were consumed earlier than the audio that renders them, and the remaining chunks pad the anchor slot while the audio tokens finish. You will see both in the appendix traces in Chapter 2, and the difference will matter in Chapter 7 when we ask why assistant-side ASR is necessary at all.

Reading a real timeline in chunk indices

Let us practise the conversion on a concrete scenario — the multi-intent request from Figure 3b. Suppose the user says, starting at t = 0:

"It's too cold in the car [1.9 s], turn up the AC [3.1 s], play some relaxing music [4.6 s]. I'm feeling a bit hungry now, please navigate to a nearby restaurant [8.2 s]."

The bracketed times are the moments each intent becomes semantically unambiguous — the semantic trigger offsets that Chapter 7 will show are exactly what the data pipeline annotates. Convert each to a chunk index:

Now the punchline of the whole architecture, expressed purely in this table. A cascaded system cannot emit any of these tool calls before chunk 53 — the endpoint — and in practice fires several chunks after that, once ASR finalizes and the LLM plans. DuplexSLA can emit the AC call at chunk 19, the music call at chunk 28, and the navigation call at chunk 51: 34 chunks (5.4 s), 25 chunks (4.0 s), and 2 chunks (0.3 s) earlier respectively, all while the assistant's voice is doing something else entirely.

That is where the paper's 4.71 s versus 0.68 s multi-action delay comes from. It is not a better model being faster; it is an architecture that is allowed to act before the sentence ends.

Where the clock shows up later

Everything downstream of this chapter is a statement about chunks, so here is the forward index:

Five conversions to do until they are automatic

Cover the right-hand column and work each one. Every number in the rest of the lesson is one of these five operations.

If question 5 surprised you — nearly a full second to clock out one tool call — hold that reaction. It is the subject of Chapter 5, and the resolution (the trigger time is the first token, not the last) is one of the more elegant details in the design.

Two clocks that must not be confused

There are actually three notions of time floating around this system, and keeping them apart prevents most confusion later:

The RTF calculation above is precisely the statement "the token clock must be able to keep the conversational clock synchronized with the wall clock." When people say a duplex model is "real-time", that equation is what they mean, whether or not they can write it down.

One practical corollary: a single slow step — a garbage collection pause, a scheduling hiccup, a cold cache — is not recoverable by running fast afterwards, because there is no buffer to catch up into. The audio for chunk c was needed at time 0.16 c and that moment has passed. Real-time speech systems therefore care about tail latency, not mean latency, in a way that batch inference never does. The paper's phrase "with a safe margin against the per-chunk wall-clock budget" is doing exactly this work: the margin is there to absorb the tail.

What the clock does not fix

A short honesty note, so the chapter does not oversell its own idea. Having a clock guarantees that events are indexed consistently. It does not guarantee that they are indexed correctly. A model can emit a perfectly well-formed action token at chunk 40 when the intent actually became clear at chunk 26 — the format is satisfied, the timing is wrong, and nothing in the serialization will notice.

Correct timing has to come from supervision, which is why Chapter 7's dual-side ASR slices exist and why Chapter 9's benchmark scores trigger time as a correctness criterion. The clock is the coordinate system. Learning where things go in it is a separate problem, and it is the harder one.

One closing thought to carry into Chapter 2. The clock is what makes the phrase "synchronized speech, language, and action" mean something checkable rather than aspirational. Two events are synchronized here in a precise sense: they carry the same chunk index, and therefore they were produced in the same autoregressive step, conditioned on the same context, and they will be executed against the same 160 ms window of the world. That is a much stronger notion of synchrony than "both happened around the same time", and it is what a shared clock buys.

Chapter 2: Two Streams, Three Channels, One Token Sequence

We have a clock. Now we need to decide what the model actually sees and produces at each tick — and, crucially, in what order, because a transformer consumes a single flat sequence and nothing else. There are no parallel tapes inside an autoregressive decoder. Whatever "three channels on one timeline" means, it has to be expressible as one string of tokens.

This chapter builds that string, token by token, until you could write it out from memory.

Why "dual-stream three-channel" is not a contradiction

The paper's name for the design is a mouthful with a precise meaning:

Unpack the two halves.

Two physical streams. In the world, there are exactly two sounds: the user's voice and the assistant's voice. A microphone captures one; a speaker plays the other. This is the "dual-stream" formulation inherited from earlier full-duplex models — the model conditions on both simultaneously instead of alternating.

Three semantic channels. On the model's interface, however, there are three things to represent, because the assistant's timeline carries two different kinds of content: the audio it is emitting, and the decisions it is making. The action channel is text-only. It has no waveform. It does not play through a speaker. It rides on the assistant's timeline as a parallel annotation.

That last row deserves attention. The action channel's consumer is not the user. Nobody hears it. It exists to be read by the software around the model — the thing that actually turns the AC dial — and, as Chapter 9 shows, by the benchmark harness, which reads backchannel labels directly off the lane even when there is no audible change at all. A channel whose output no human perceives is a strange object in a speech model, and it is the single most important structural idea in this paper.

The user channel is continuous, and that is deliberate

Note an asymmetry that is easy to skim past. The assistant's audio is discrete tokens; the user's audio is continuous features. Why not tokenize both?

Because they are used for different things. The assistant's audio must be generated, and generation from an autoregressive language model requires a finite vocabulary — you cannot sample a real-valued waveform from a softmax. The user's audio only needs to be understood, and for understanding, discretization is pure loss: quantizing to a codebook throws away prosody, hesitation cues, and the fine acoustic detail that distinguishes "you're right." (agreeing) from "you're right," (about to object).

So the design keeps the user side continuous. Appendix D adds the constraint that matters for latency: "User audio is encoded by a causal speech front end; no future user audio is required to advance one chunk." Causal means the encoder for chunk c looks only at audio up to the end of chunk c. No lookahead, no bidirectional attention over the utterance, no waiting for the next 200 ms to disambiguate.

The assistant channel: what TA4 actually is

TA4 is the paper's name for the per-chunk assistant unit: one text anchor T followed by four audio tokens A. Written out, the assistant stream looks like this across chunks:

Three questions, answered in order.

What is the text anchor for? It carries the word the assistant is saying — a text token, interleaved with the audio tokens that render it. This is the "text-guided speech generation" pattern that runs through modern speech LMs: generating text alongside audio keeps the model's language ability engaged and gives the audio something semantically stable to hang on. Without it, an audio-only decoder drifts into fluent nonsense.

What are the audio tokens? Discrete speech units, four per chunk at 40 ms each, produced by the backbone and rendered to a waveform by the codec's decoder. Four tokens at 40 ms is 25 tokens per second of audio — the rate the whole clock was built around.

Why one anchor and four audio tokens, rather than a matched number? Because text and speech run at different natural rates. A Chinese character or an English word takes rather more than 40 ms to say. One anchor per 160 ms is roughly a comfortable speaking rate of about six units per second; the audio needs 25 tokens per second to reconstruct. The ratio 1:4 is the paper's chosen alignment between those two rates, and it is exactly why the timing of the text is imprecise — a point that will detonate in Chapter 7.

The serialization, exactly

Now the string. Within a chunk, the three channels are interleaved into a single token stream consumed by the backbone. Section 2.2 gives it as:

the per-chunk serialization
<|user_audio_begin|>       U U        <|user_audio_end|>
<|assistant_audio_begin|>  T A A A A  <|assistant_audio_end|>
<action text>                          <|action_end|>

Read it as one flat sequence, left to right, top to bottom. That is literally the order the tokens arrive in. Walk through it:

Two properties of this ordering are load-bearing, and both are easy to miss.

The user segment comes first. Within chunk c, the model sees this chunk's user audio before it must produce this chunk's assistant audio and action text. That is what makes the reaction latency as low as it is: the response to what you just said is conditioned on what you just said, in the same tick, not one tick later.

The action text comes last. The assistant's TA4 unit is decoded before the action segment. So the model has already committed to this chunk's audio when it writes the action tokens. This ordering is why a long action burst cannot starve the voice: the voice is already paid for. Reverse the order and the failure mode of option C from Chapter 0 comes right back.

The terminator that keeps the grid honest

One token deserves its own section: <|action_end|>. The paper is emphatic: it "terminates the chunk regardless of whether any action text was emitted, which keeps every chunk strictly aligned to the 160 ms clock."

Think about what would happen without it. The action segment is variable length — sometimes zero tokens, sometimes ten. If nothing marked its end, the decoder would have no way to know whether the next token it sees is more action text or the start of the next chunk's user segment. The sequence would become ambiguous, and the alignment between token position and wall-clock time — the entire premise of Chapter 1 — would dissolve.

So the terminator is what turns a variable-length text lane into a fixed-rate lane. Every chunk emits exactly one <|action_end|>, at 6.25 Hz, forever. It is a heartbeat.

This also means the empty case is explicit. A chunk with nothing to do looks like:

a completely idle chunk
<|user_audio_begin|> U U <|user_audio_end|>
<|assistant_audio_begin|> T(<vad_silence>) A A A A <|assistant_audio_end|>
<|action_end|>

Eleven tokens, no content, still fully specified. The model is silent, listening, and has decided not to act — and every one of those decisions was made explicitly, scored by the loss, and time-stamped. In a cascaded pipeline, the equivalent moment is nothing happening, which is not a decision anyone can supervise or measure.

What the action channel may contain

Section 2.2 lists a small set of structured markers alongside free text. The full vocabulary of the lane:

The canonical abstract form for a chunk carrying planning plus one call, exactly as the paper writes it:

one action segment, abstract form
planning<|toolcall_begin|>{"function": "function_name", "arguments": "arguments"}<|toolcall_end|>

And the concrete instance from Figure 1, which we met in Chapter 0:

the heatstroke example, action lane
The user is feeling hot, so I should switch on the air conditioning
and set it to a nice temperature.
<|toolcall_begin|>{"function": "decrease_car_setting",
                  "arguments": "air-conditioner: 26 degree"}<|toolcall_end|>
<action_end>

Count the tokens in that block and you will immediately see the problem Chapter 5 solves: it is far more than ten. The action lane's per-chunk cap means this single object cannot fit in the chunk that triggered it. That is not a bug; it is the reason the FIFO spill rule exists.

Timestamps for free

Here is the payoff of putting the action lane on the same grid, and it is the sentence to underline in Section 2.2:

No separate timing model. No alignment step. No timestamp prediction head. The chunk index is the timestamp, because the serialization guarantees that the n-th <|action_end|> in the stream occurs at time 0.16 n. Reading a time off this model costs a count.

Section 2.6 explains why this matters twice over: the timestamp is "needed both for downstream execution and for the latency-oriented evaluation in Section 5." Downstream execution needs it because a car that receives "turn up the AC" wants to know whether that instruction is current or three seconds stale. Evaluation needs it because the entire benchmark in Chapter 9 is built on comparing realized action times to annotated semantic anchors — and you cannot compute that difference without a clock on both sides.

Two things to check in the sim. First, cycle from "idle" to "speaking" and confirm that the token count of the assistant channel does not change — silence is not cheaper. Second, select "tool call" and watch the action lane blow past the 10-token cap and turn red: the surplus is what spills into the next chunk under the rule we derive in Chapter 5.

Reading a real trace, line by line

Appendix A of the paper gives chunk-by-chunk traces of actual training samples, and reading one is the fastest way to make the format concrete. Here is the user-channel ASR trace from Appendix A.1, transcribed into a table. The user says 今天天气很好 ("the weather is nice today"); the assistant is silent for most of it and then begins to reply.

Four things this trace teaches that no prose description can:

1. The transcript on the action lane lags the audio by exactly two chunks. 今天 is heard in chunk 0 and written in chunk 2; 天气很 is heard in chunk 1 and written in chunk 3. The paper states the rule: "The user-channel transcript is emitted on the action channel with a fixed lag of 2 chunks (320 ms). Tokens that fall in the same chunk are merged."
2. The lag is not a limitation, it is supervision. Two chunks of delay gives the model enough acoustic context to commit to a transcription without lookahead in the encoder. It is a training-time convention that teaches the model a stable, causal relationship between audio time and action time.
3. Silence anchors dominate. Six of the nine chunks carry <vad_silence>. This is what most of a conversation looks like from the assistant's side, and the model is being explicitly supervised on all of it.
4. The two silence anchors appear in different roles. Chunk 6 is <tts_pad> — the text has run out mid-utterance while the audio finishes rendering — while chunks 7 and 8 are <vad_silence>, genuine "not speaking". The model must learn the difference, because one means "I am still talking" and the other means "I have stopped."

Do the same conversion in reverse as an exercise. If you saw an action segment appear at chunk 17, what user audio produced it? Chunk 15, at t = 2.40 s to 2.56 s. That inference — from lane position to source time — is exactly what the evaluation harness in Chapter 9 does, and it is only possible because every lane shares one index.

Token anatomy of a single tool call

To feel the budget pressure that motivates Chapter 5, count what a tool call actually costs. Take the Figure 1 example and break it into rough token groups (exact counts are tokenizer-dependent; the shape is what matters):

So a single complete action object takes roughly five chunks — nearly a second of clock — to transmit through a 10-token pipe. And yet the paper reports a tool-call delay of 0.64 seconds on average. How can the delay be shorter than the transmission?

Because the trigger time is the time of the first token, not the last. Appendix D states it exactly: "tool-call closing markers can land in a later chunk than the opening marker, but the trigger time of the action object is always anchored to the chunk where the planning text starts." The action is timestamped at its birth. Its body arrives over the following chunks like a message over a serial line — and, critically, the assistant's voice keeps flowing the entire time, because the TA4 unit was already paid for in every one of those chunks.

Two streams versus three channels, side by side

The paper's comparison sentence deserves a table, because "compared with backbones that work with two streams only" is doing a lot of work in one clause:

The fourth row is quietly the most consequential and reappears in Chapter 9's Table 6, where the backchannel delay column reads N/A for every closed-source baseline. Not "worse" — not measurable. If a system's only output is audio, then a decision that produces no audible change produces no observable event, and the metric has nothing to time. Adding a channel did not just improve the model; it made a category of behaviour legible.

Why not just widen the assistant text channel?

Before moving on, kill the obvious objection. The assistant channel already carries text (the anchors). Why not let tool calls ride there and skip the third channel entirely?

Section 2.6 answers directly: "Embedding planning and tool calls into the same channel as assistant text would force that channel to alternate between TA4 audio tokens and tool-call JSON, which breaks the smoothness of the assistant audio."

The mechanism, spelled out: the anchor slot T occurs once per chunk and is consumed by the word currently being spoken. To insert JSON there you must either (a) displace the word — the audio then has no text to hang on, and pronunciation degrades — or (b) insert extra T slots — which breaks the fixed TA4 rate and hence the clock. There is no third option. The channel is rate-locked by construction, and a rate-locked channel has no spare capacity by definition.

The cost of the extra lane, meanwhile, is bounded and small: "at most 10 tokens per chunk", which Chapter 1's arithmetic showed is affordable at typical decoding speeds. A dedicated channel that costs at most two-thirds of the token budget and never touches the audio path is a much better deal than a shared channel that occasionally destroys the voice.

Ten questions about the format, answered

Everything readers reliably ask at this point, in one place. If any answer surprises you, that is the part of the chapter to re-read.

1. Does the user ever hear the action channel? No. It is text, consumed by the host application, never rendered to audio.
2. Is the action channel fed back as model input? Yes — it is part of the autoregressive sequence, so the model conditions on what it decided in earlier chunks. That is how a multi-chunk tool call stays coherent across the spill.
3. Can the model produce user audio? No. Section 2.1 is explicit: "the user audio side is kept causal and is never produced by the model." It is injected, not predicted, and never supervised.
4. What if the assistant has nothing to say for two minutes? It emits 750 chunks of silence anchors and silence audio codes, at full cost. Silence is never free.
5. Does the text anchor have to correspond to the audio in the same chunk? Not exactly — it is left-aligned and may run ahead. That imprecision is the whole subject of Chapter 7.
6. Can two tool calls be emitted in one chunk? Yes, if both are triggered there and both fit — Rule 1 in Chapter 5 concatenates them head to tail. In practice the budget usually forces a spill.
7. What happens if the model wants to emit more than ten action tokens? The surplus spills into following chunks. At training time this is enforced by the data layout; at inference time it is a learned regularity.
8. Why is the action channel text rather than a structured head? Because it must carry free-form planning prose and structured JSON and control labels and transcripts. A classification head could only do the third. Text is the only representation general enough for all four.
9. Does the model know what time it is? Only through position in the sequence. There is no explicit timestamp input — the chunk index is implicit in how many <|action_end|> tokens have gone by. Which is exactly why the alignment supervision in Chapter 7 matters so much.
10. Could you add a fourth channel? Nothing in the design forbids it — a vision lane, a result lane, a state lane. You would pay for it in the per-chunk token budget, and Chapter 1's arithmetic tells you exactly how much.

That last question is the interesting one to sit with. The three-channel design is not a fundamental truth about speech; it is the smallest set of lanes that covers listening, speaking, and acting. The framework generalizes, and its currency is tokens per chunk.

Chapter 2 in review

The format, as a checklist you should be able to reproduce blind:

1. Two physical audio streams, three semantic channels; the third is text-only and rides the assistant's timeline.
2. User audio: continuous, causal, two features per chunk, never generated.
3. Assistant audio: TA4 — one text anchor plus four discrete audio tokens — always produced, silence included.
4. Action: up to ten text tokens carrying planning, labels, JSON, or delayed transcripts.
5. Order within a chunk: listen, speak, act. The voice is committed before any action token exists.
6. Every chunk ends with an unconditional terminator, which is what keeps token position proportional to wall-clock time.
7. The chunk index is the timestamp — free, exact, and needed by both the host application and the benchmark.

If you remember one thing: an ordering is an architecture. Putting the action segment last in the chunk is what makes the priority scheduler work, and it is a one-line design decision with the entire real-time behaviour of the system riding on it.

Three exercises:

1. Write out a completely idle chunk, token by token, and count the tokens. How many did the model produce?
2. In the Appendix A.1 trace, an action segment appears at chunk 3. Which chunk's user audio produced it, and at what wall-clock time was that audio heard?
3. Suppose you moved the action segment before the TA4 unit. Describe the failure mode precisely, in terms of what the audio does when a 40-token tool call is emitted.

Chapter 3: The Full-Duplex Foundation It Stands On

DuplexSLA did not invent full duplex. It inherited it. The dual-stream idea — one backbone modelling both sides of a conversation simultaneously — was established by a line of work the paper cites fourteen references deep, and understanding what was already solved is the only way to see clearly what this paper adds.

This chapter is the lineage. It is also the chapter where this lesson connects to its siblings: if you have read the Moshi veanor or the Qwen2.5-Omni veanor, you already own most of the substrate and can read this chapter as a diff. If you have not, everything you need is built here from zero.

Generation 0: the cascade (and why it survives)

We dismantled the turn-based pipeline in Chapter 0, so here we only need its epitaph — and a fair one, because the cascade is not stupid. It is the reference system in this paper's own tool-call evaluation, and it wins on accuracy (91.33% vs 85.56%). Its virtues are real:

Hold that first column when you reach Chapter 10. The accuracy gap this paper reports is not a mystery — it is the price of collapsing four specialized stages into one 7B model that must also stay on a metronome.

Generation 1: dual-stream native duplex

The key move, made by the models the paper cites as [1–14], is to stop treating the conversation as an alternation and start treating it as two simultaneous time series that one model predicts jointly. Concretely: at every step, the model is conditioned on the user's audio so far and its own audio so far, and it predicts its own next audio.

Three capabilities fall out of this that a cascade cannot express, and each is worth naming because DuplexSLA assumes all three:

1. Overlap is representable. Both streams can be non-silent at once. That is what a real interruption looks like, and a model that cannot represent overlap cannot learn from examples of it.
2. Silence is a prediction. Choosing to stay quiet becomes a scored decision, conditioned on what the user is doing. Chapter 6's pause behaviour is impossible without this.
3. Reaction latency is bounded by the frame rate, not by an endpointer. The model can change what it emits at any frame. Nothing has to "fire" first.

The paper's own bibliography is a decent map of this generation: a full-duplex scheme based on an LLM (2405.19487), synchronous LLMs as full-duplex agents (2409.15594), Moshi (2410.00037), Freeze-Omni (2411.00774), SALMONN-omni (2411.18138), Mini-Omni and Mini-Omni2, LLaMA-Omni, OmniFlatten, SpiRit-LM, Voila, and — from 2026 — PersonaPlex (2602.06053) and Covo-Audio (2602.09823).

Two of these appear as measured baselines in Chapter 9's Table 7, so they are not just citations: Freeze-Omni and PersonaPlex are evaluated head to head on the turn-taking benchmark, and their results are among the most instructive numbers in the paper.

Worth naming the cost of the dual-stream move as well, since nothing is free: you now train one model on a joint distribution over two speakers, which is strictly harder than modelling one, and you pay tokens for silence forever. Generation 1 accepted both costs because the capabilities on the list above are unreachable otherwise.

What Moshi established

Moshi is the anchor reference for this generation and the direct architectural ancestor of the dual-stream half of DuplexSLA. Its contribution, in the terms this lesson needs:

Read the middle row again with Chapter 2's TA4 in hand. The text anchor is not a DuplexSLA invention; it is the inner-monologue pattern, adopted and given a fixed rate. What DuplexSLA adds is the observation that this text channel is already fully occupied by the words being spoken — which is precisely why planning and tool calls need somewhere else to live.

What Qwen2.5-Omni established

The other parent is the omni-modal streaming line, of which Qwen2.5-Omni is the cleanest example, and which the paper cites through its "audio-aware foundation models" group [18–32] — Qwen2-Audio, Qwen3-Omni, GLM-4-Voice, VITA-1.5, VITA-Audio, MiniCPM-o, and the Step-Audio family.

MiniCPM-o also appears as a measured baseline in Table 7, scoring 82.00% average accuracy at 0.61 s delay in the no-prefill setting — respectable accuracy, roughly double DuplexSLA's latency.

A caution about lineage tables generally: they compress a messy literature into a tidy sequence, and the tidiness is partly an illusion. Several of these systems were developed concurrently, solve overlapping problems, and cite each other in both directions. Treat the three generations as a way of organizing capabilities, not as a chronology — the useful question is never "who was first" but "which capability does this system assume, and which does it add."

Generation 1.5: thinking on the clock

There is a third line, and it is the one closest to DuplexSLA's actual contribution. Three papers by overlapping authors — Chronological Thinking in Full-Duplex Spoken Dialogue Language Models (2510.05150), Mind-Paced Speaking: A Dual-Brain Approach to Real-Time Reasoning in Spoken Language Models (2510.09592), and The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning (2603.17837) — ask how a duplex model can reason without stopping to think.

This matters because it isolates the exact gap DuplexSLA fills. Thinking on the clock gives you an internal rationale that does not interrupt the voice. But a rationale is not an action:

The last row is the interesting one. Once your model can act, a new kind of error exists: the temporally wrong action. Calling navigate() correctly but four seconds late is a failure that has no analogue in text agents, and it is why Chapter 9's evaluation protocol has to treat trigger time as a correctness criterion rather than a performance metric.

Put differently: the thinking line asks "can the model reason without going silent?" and this paper asks "can the model act without going silent?" Those turn out to be the same architectural question with different payloads — which is why the action lane carries planning text as well as JSON, and why a system that solved one is most of the way to solving the other.

The backbone: Step-Audio 2 mini

DuplexSLA is not trained from scratch. Section 4 states: "DuplexSLA is initialized from Step-Audio 2 mini, a 7B-scale audio language model." Table 1 repeats it: "7B speech-LM, initialized from Step-Audio 2 mini."

Why does this matter enough to state twice? Because it determines what continued pretraining has to teach versus what it can assume:

That right-hand column is Chapter 8's syllabus, in order. And the left-hand column explains a design pressure that is otherwise invisible: the training mixture includes ~1.92M general text samples explicitly "to preserve world knowledge and reasoning ability" while the speech format changes underneath. You are performing surgery on a model's input format without letting it forget how to think.

The family tree, interactively

The row to dwell on is "user backchannels" against "dual-stream duplex". A dual-stream model can keep talking through a backchannel — overlap is representable, so the behaviour is available. What it cannot do is say that it did so. There is no label, no timestamp, no object for the host application or the evaluator to read. The behaviour exists; the report does not. Adding the action channel converts an implicit behaviour into an explicit, addressable event, and that conversion is most of what "SLA" means.

A short history of turn-taking machinery

The three generations above are about models. It is worth tracing the parallel history of the component they replace, because each step in that history was a reasonable response to the previous one's failure:

Notice the pattern: each generation buys accuracy by adding a stage, and each added stage costs time. The native approach is the first one that buys accuracy by removing a stage — which is only possible because the information the detector needed was already inside the model that was going to run anyway.

What the existing benchmarks measured

The paper cites eleven evaluation suites [33–43]. Knowing roughly what each covers explains why a new one was necessary:

The gap is clear once tabulated. Turn-taking is measured. Understanding is measured. Tool calling is measured elsewhere, in text. Nobody measured tool calling on a duplex timeline, because until this paper there was no system whose tool calls had timestamps to score.

The codec-free contrast: SALMONN-omni

One cited system is worth a paragraph because it takes a different road at a fork DuplexSLA does not even mark. SALMONN-omni (2411.18138) is described in its own title as "a codec-free LLM for full-duplex speech understanding and generation."

Codec-free means the assistant's speech is not represented as discrete codec tokens at all. That choice removes the quantization loss of a codec and the need for a fixed audio-token rate — and it also removes the thing DuplexSLA's whole clock is built on. If your assistant audio is not a fixed number of discrete tokens per chunk, you do not have a TA4 unit, and the tidy "5 tokens always, 10 tokens at most" budget arithmetic of Chapter 1 has no meaning.

Neither road is obviously right. But it is worth noticing that DuplexSLA's clean budget story is downstream of a representational choice, not a universal law of duplex systems. When you read "at most 10 action tokens per chunk", the reason that sentence can even be written is that everything else in the chunk is discrete and counted.

What this paper adds, stated precisely

With the lineage in place, the contribution reduces to two bullets, which are the paper's own:

And the two capabilities those contributions are meant to deliver, which will structure the rest of the lesson:

1. Semantic-driven turn-taking control — interruption, pause, and backchannel handled inside the same backbone, removing the external semantic VAD. (Chapter 6, measured in Chapter 9.)
2. In-conversation planning and tool calling — planning text and JSON tool calls emitted on the action channel without halting assistant audio, under a strict per-chunk budget. (Chapters 4 and 5, measured in Chapter 9.)

A note on how quickly this moved. The dual-stream duplex line begins in earnest in 2024; the chunk-aligned reasoning line lands in late 2025; this paper is mid-2026. Three years from "a model can listen while it speaks" to "a model can act while it speaks," on a shared clock, with a benchmark to score it. If you are reading this lesson well after publication, assume the neighbourhood has moved again — and use the framework rather than the numbers.

The framework, restated for that purpose: identify the physical rates, derive the clock, count the per-chunk token budget, decide what each lane carries and in what order, then ask what the supervision has to look like for the model to learn the timing. Those five steps outlive any particular set of results, and they apply to any modality that has to run in real time next to a language model.

One honest note about lineage papers before we move on. It is tempting to read a contributions list as "everything before this was inadequate." That is not the right reading here. Every element DuplexSLA uses — discrete audio tokens, text-audio interleaving, dual-stream conditioning, streaming encoders, a strong audio backbone — was built by someone else and works. The contribution is an addition that happens to be cheap: one more lane, ten tokens per chunk, and a data pipeline that knows how to fill it. The best systems papers are usually shaped like this, and recognizing the shape makes them easier to read.

Vocabulary you will meet in this literature

Duplex speech papers use a compact jargon. Decoding it makes the citations above readable:

One term deliberately absent from this paper's vocabulary is worth noting: TTFA. Product teams optimize it obsessively, and DuplexSLA's normal delay is essentially that metric under another name. The paper's framing — delay against a semantic anchor — is stricter, because a system can improve TTFA by starting to speak before it has understood, and the anchor-based metric does not reward that.

The 2026 neighbourhood

DuplexSLA is a 2026 paper in a fast-moving area, and several of its neighbours are worth placing because you will meet them in the results tables:

Two things this table makes visible. First, the StepFun lineage is doing something deliberate: an audio foundation model, then a reasoning branch, then a duplex-action branch, sharing authors and a backbone. DuplexSLA is a limb of a programme, not a one-off.

Second, the 2026 duplex frontier has split into at least three axes — who the model is (persona and voice control), how it thinks (latent or chunk-aligned reasoning), and what it does (this paper). They are compatible, and nobody has yet published all three in one system. That is a fairly clear map of where the next paper comes from.

Chapter 3 in review

The inheritance, itemized. For each element, where it came from and where it appears in this lesson:

One more framing that helps when reading any paper in this area: ask what the model is allowed to say. A cascade's turn detector may say "turn over" or "not yet". A dual-stream duplex model may say anything it can pronounce. DuplexSLA may additionally say "I am interrupting", "I heard you and I am continuing", and "call this function with these arguments, now". The expressible vocabulary of a system is a surprisingly good predictor of what it can be evaluated on — and, as Table 6's N/A column shows, of what it can be seen to do at all.

If you remember one thing from this chapter: the difference between "the model behaved correctly" and "the model emitted a legible, time-stamped decision." Dual-stream duplex gives you the first. The action channel gives you the second — and the second is what makes a behaviour dispatchable, auditable, and scoreable.

Three exercises, answers in the chapter above:

1. Name two capabilities that a dual-stream duplex model has and a cascade cannot have at all, and explain each in terms of what silence represents in the two designs.
2. The paper cites eleven benchmarks and still builds a twelfth. Give the structural reason, in one sentence, that does not involve the word "better."
3. SALMONN-omni is codec-free. Which single sentence of Chapter 1 becomes unwriteable for a codec-free system, and why?

Chapter 4: Showcase — Three Lanes, One Clock, Live

Everything so far has been static: a clock, a serialization, a lineage. This chapter runs it. By the end you will have spoken into a live conversation, watched a tool call fire while the assistant kept talking, and seen the side-effect fold back into the dialogue — and you will be able to explain, chunk by chunk, why each of those things happened when it did.

This is the chapter to linger in. If you take one thing from this lesson, take the feeling of three lanes moving together.

Before the controls: the sim below is not a video. Every cell you see is computed from the same rules we derived in Chapters 1 and 2 — chunk indices, fixed TA4 allocation, a ten-token action cap — so if you disagree with something it draws, you can check it with arithmetic rather than opinion. That is the standard this lesson holds its simulations to.

The three patterns of in-conversation tool calling

Section 2.5 identifies the patterns that a duplex action channel makes possible, and Section 5 turns each into 300 benchmark cases. Learn the three by name, because every result table is organized around them:

That third row is the one that has no turn-based equivalent at all. Read the paper's description carefully: "A short user utterance that is topically unrelated to the current dialogue (e.g., 'play some Beatles songs' uttered while the assistant is talking about something else) is treated as a backchannel: the action channel emits a planning fragment plus a tool call without interrupting the assistant's spoken thread. The assistant audio thus stays coherent while the side-effect is dispatched."

Multi-action: why the order comes for free

The second pattern deserves its own paragraph because of a subtle claim. Section 2.5: "Because each tool call is anchored to its own chunk on the action channel, the calls are emitted in semantic order along the user's request, and the assistant audio runs in parallel with each call's planning text."

The ordering is not something the model has to plan. It falls out of the clock. Intent A becomes clear before intent B, so A's anchor chunk precedes B's anchor chunk, so A is emitted first. A turn-based agent has to construct the order at planning time from a completed transcript; a duplex agent inherits it from the passage of time.

That is a genuinely elegant property, and it is worth naming the price: because the model commits to A before hearing B, it cannot revise. If the user says "navigate to the airport — no wait, the train station," the anchor for the first intent has already passed. We will return to this in Chapter 10, and it is one plausible mechanism behind the multi-action accuracy drop.

The showcase

Here is the instrument. Three lanes on the 160 ms grid: the user's audio on top, the assistant's TA4 stream in the middle, the action channel at the bottom. A dispatch panel on the right shows what has actually been executed and when.

Controls, and what each teaches:

• Scenario — the five behaviours the paper evaluates. Start with single action, then multi action, then backchannel action.
• Play / pause / step — step one chunk at a time to read the ledger. The ledger shows chunk index, wall-clock time, and what each lane is emitting right now.
• 🎤 Speak now — press it mid-playback to inject your request into the running conversation ("play some Beatles songs"). Watch what the assistant lane does. It should do nothing at all, which is the entire point.
• Cascade ghost — overlays where a turn-based system's tool call would have landed, using the paper's measured delays. The gap between the two diamonds is the contribution of this paper, drawn to scale.

A word on why this particular visualization and not a nicer one. The obvious alternative — an animated cartoon of a car and a talking assistant — would be prettier and would teach nothing, because the thing being taught is precisely the grid: discrete chunks, fixed allocations, events with indices. Anything that smooths the timeline away hides the mechanism. Three lanes of cells on a shared clock is not a stylistic choice; it is the paper's Figure 1 rendered honestly.

What to look for, scenario by scenario

Single action. The user says "It's freezing in here, turn up the AC." The semantic anchor — the moment the request is unambiguous — arrives several chunks before the user stops talking. Watch the action lane light up there, not at the end. Then watch the assistant lane: it is producing silence anchors while the user speaks (it is listening, correctly), and it begins its reply after the user's turn ends. The tool fired before the reply began.

Multi action. Three intents, three anchors, three dispatches, and the assistant's single fluent sentence "Sure, I've turned up the AC temperature and started the music. I'm now navigating to a nearby restaurant" running across all of them. Note the ordering: the calls come out in the order the intents were spoken, without any planning step deciding that.

Backchannel action. The assistant is mid-sentence about the May Day holiday. The user says "play some Beatles songs." Look at the assistant lane during and after that utterance: unbroken. Look at the action lane: planning text plus search_music. The paper's Figure 3a in motion.

Interrupt. Contrast with the previous one. Same short user utterance shape, different meaning — and now the action lane emits interrupt and the assistant lane switches to silence anchors within a couple of chunks. Same acoustics, opposite behaviour, because the decision is semantic. This is Chapter 6's subject, previewed.

Pause. The user hesitates mid-thought. The assistant lane stays silent through the gap. An energy VAD would have declared the turn over and started talking; the model keeps listening because nothing semantically complete has been said. The absence of a barge-in is the behaviour.

The backchannel-action scenario, chunk by chunk

Because this is the pattern with no turn-based equivalent, it is worth writing out as a script — the way you would read a trace in a debugger. This is Figure 3a of the paper, expanded onto the grid with plausible chunk indices.

Read the assistant column across chunks 14–16. The user interjects, and the assistant finishes its question — "Do you have any travel plans?" — without a hitch. No pause, no restart, no acknowledgment of the interjection in speech at all. Meanwhile the action lane spends five chunks emitting a rationale and a call.

Two design consequences visible only in the trace:

• The action lane keeps working after the assistant has stopped speaking. Chunks 18–20 have silence anchors and a draining tool call. The lanes are genuinely independent; neither waits for the other.
• The trigger time is chunk 16, two chunks after the user's request began — the same two-chunk evidence-accumulation delay we will meet in Chapter 6. Then five chunks of transmission. Trigger and arrival, separated, exactly as Chapter 5 will formalize.

What the simulation simplifies

Being explicit about the model's idealizations, so you do not carry a false picture forward:

The one idealization that matters most is the first. The simulation shows you the mechanism assuming perfect intent detection; the benchmark measures how often detection is right. Keep them separate: this chapter teaches the machine, Chapter 9 grades it.

Three questions the showcase should have provoked

If you played with the sim properly, three questions should be nagging. Each is answered in a later chapter, and knowing where to look is part of learning to read a systems paper.

1. "What if the tool call is longer than the chunk allows?" It is, always. Chapter 5 derives the spill rule and the FIFO queue, and shows why the trigger time does not drift even when the body takes five chunks to arrive.
2. "How does the model know the difference between the backchannel-action and the interrupt scenario? The user audio looks the same." It does look the same, for the first several chunks. Chapter 6 is entirely about that decision, with the paper's own chunk-level traces of both cases.
3. "Where does training data for any of this come from? Nobody has ever recorded a conversation annotated with semantic trigger offsets." Correct — it has to be manufactured. Chapter 7 shows the pipeline that fabricates it, and why a boring-looking ASR data slice turns out to be load-bearing.

Latency masking, and why it is free here

Voice-agent engineering has a standard trick for hiding tool latency: acknowledgment filler. The assistant says "Sure, let me check that for you…" while the tool runs, so the user hears something instead of dead air. Every production voice stack does some version of this.

Look closely at what DuplexSLA does instead, in the Figure 3b transcript: the assistant says "Sure, I've turned up the AC temperature and started the music. I'm now navigating to a nearby restaurant."

That is not filler. It is a report of something already done. The difference is worth making precise:

Predict before you play

A short pre-registration exercise, which is worth doing honestly before you touch the controls again. Write down your answers, then check them in the sim:

The fourth is the one worth being certain about. The answer is no, by construction — and if you predicted otherwise, re-read Chapter 2's ordering argument before continuing, because everything in Chapter 5 depends on it.

The return leg — an honest gap

The chapter brief for this sim says the tool's result "folds back into the dialogue," and the simulation shows exactly that: the call dispatches, the world changes, and the assistant's later speech reflects it. You should know precisely how much of that is in the paper.

What the paper specifies in detail: the emission path. How an action object is represented, when it is emitted, how it is timestamped, how it queues, how it is scored.

What the paper does not specify: the return path. There is no described channel for a tool result to re-enter the model. Section 2.6 says the timestamp is "needed both for downstream execution and for the latency-oriented evaluation", and Section 2.5 says "the side-effect is dispatched" — the language throughout is of dispatch, not of round trip. Appendix E's action-object schema has name, planning, parameters, and offset. There is no result field.

If you were building this system, the natural completion is nearly free: results arrive as text injected into the action channel's input side at the chunk they return, exactly like user audio features are injected on the user channel. The model is already trained to condition on action-channel tokens. Nothing in the architecture forbids it. But "nothing forbids it" is not the same as "the paper did it", and this lesson holds that line.

Reading the ledger like an engineer

The ledger in the simulation reports four numbers. Each corresponds to something in Chapter 9's evaluation protocol, so learn to read them now:

One deliberate detail in the sim: the delay never goes below the quantization floor from Chapter 1, because it cannot. If you find a configuration where the ledger reports a delay under 80 ms on average, you have found a bug in the simulation, not a faster model.

One more reading of the ledger worth doing: run the single-action scenario twice, once with the cascade ghost on and once off, and watch only the assistant lane both times. It is identical. The entire difference between the two architectures lives on a lane the user never hears — which is exactly why this capability was invisible until somebody gave it a channel.

Try to break it

The best way to internalize an architecture is to push it past its assumptions. Three experiments, in increasing order of interest:

1. Press "Speak now" twice in quick succession. The second request's action object cannot start emitting until the first has fully drained — that is the FIFO rule from Chapter 5, visible as a queued object waiting its turn. Notice the second call's delay grows, while the first's does not. Actions do not preempt each other.
2. Press "Speak now" during the interrupt scenario. Now you have a floor-taking event and a side request competing. The lane order tells you which the model committed to first.
3. Turn off the cascade ghost and watch only the DuplexSLA lanes. Then turn it back on. The ghost is the version of this interaction you have experienced your whole life; the lanes are the version this paper is arguing for. Seeing them on the same axis is the argument.

Notice what never happens in any configuration: the assistant lane never has a hole in it. Not while planning text drains, not while JSON clocks out, not while a second action waits in the queue. Five tokens per chunk, every chunk, no exceptions. That invariant is the whole design, and once you have watched it hold under pressure, the rest of the paper reads as a set of consequences.

One full run of the ledger

The single-action scenario, written out as the ledger reports it, so you can check the sim against arithmetic you did yourself. The user says "It's freezing in here, turn up the AC" from t = 0 to t = 2.4 s; the intent is clear at t = 1.5 s; the action object is 38 tokens.

Three numbers to extract and check against earlier chapters. Trigger delay: 1.76 − 1.50 = 0.26 s — about 1.6 chunks, of which roughly 0.06 s is quantization. Transmission: chunks 11–14, four chunks, 640 ms — matching ⌈38/10⌉ from Chapter 5. Actuation: the AC changes at 2.24 s, while the user is still speaking, and 0.16 s before the assistant says a word.

Compare with the cascade on the same utterance: endpoint at 2.4 s plus hangover plus ASR plus planning puts its dispatch somewhere past 4.5 s. The two systems do the same thing; one does it before the sentence ends and the other after the silence.

Four acknowledgment strategies, ranked

Since the chapter is about what the assistant does while acting, it is worth laying out the full space of options a voice system has. Only the last is available to DuplexSLA, and only because of the action lane:

Strategy 4 is what the Figure 3b transcript shows, and it is qualitatively different from the other three. The assistant is not managing the user's perception of latency; there is no latency to manage. It is narrating.

That distinction has a practical consequence worth naming: strategies 2 and 3 both degrade badly when tools are slow or fail, because speech has been committed on a promise. Strategy 4 degrades badly too — but only for the subset of tools whose results the assistant must report, which is exactly the gap Chapter 10 discusses. For pure side-effect tools, it is strictly better than the alternatives.

The same interaction, four ways

To fix the showcase in memory, here is one request — "It's freezing, turn up the AC" — traced through four architectures on the same wall clock. Assume the user speaks from t = 0 to t = 2.4 s, with the intent clear at t = 1.5 s.

The third row is the one to dwell on, because it is the state of the art this paper is arguing against — not the clumsy cascade, but the good duplex model that talks beautifully and cannot do anything. Speed of speech was already solved. Speed of action was not.

Chapter 4 in review

One habit to carry out of this chapter: whenever you meet a claim about a voice system's speed, ask what it is fast relative to. Faster than a slow cascade is a low bar. Faster than the semantic anchor is impossible. The interesting quantity is the gap between the measured delay and the architectural floor — and Chapter 1 gave you the tools to compute both halves of it.

If you remember one thing: the assistant lane never has a hole in it. Five tokens per chunk, every chunk, while the action lane does whatever it needs to. Every other property in this chapter — latency masking for free, backchannel-action, multi-action ordering — is a consequence of that invariant plus a shared clock.

Three exercises:

1. In the backchannel-action trace, the assistant finishes "Do you have any travel plans?" while a tool call drains. Which two design decisions from Chapter 2 make that possible? (One is about ordering; one is about lanes.)
2. Why is the cascade faster on backchannel-action (1.27 s) than on single-action (2.33 s), even though the backchannel case is conceptually harder?
3. The paper describes dispatch but not return. Name three of the fifty tool schemas for which that gap is immediately felt, and say what the assistant would have to do while waiting.

Chapter 5: Ten Tokens a Chunk — the FIFO Queue

We ended Chapter 2 with an uncomfortable count: one complete action object — planning text plus a JSON tool call — runs to roughly forty tokens. The pipe carries ten per chunk. Something has to give.

What gives is the assumption that an action fits in a chunk. It does not, and the paper builds a small, strict protocol around that fact. This chapter derives that protocol, works a full numerical example by hand, then writes it three ways in code.

The problem, stated cleanly

Section 3.3 sets it up: "Because every chunk has a hard ≤ 10-token action-channel budget, short bursts of actions cannot always fit into the chunk where they are triggered. The data-construction format therefore turns the action stream into a single FIFO queue keyed by trigger time."

Three words in that sentence carry the design. Single: one queue, not one per action type. FIFO: first in, first out, no priorities. Keyed by trigger time: the ordering is temporal, inherited from when each intent became clear.

And two rules govern it. Here they are, close to the paper's own words, then unpacked:

Why atomicity is not a nicety

Both rules protect the same invariant: a <|toolcall_begin|>…<|toolcall_end|> block is never split by a chunk terminator, and never interleaved with another action's tokens.

Think about what a violation would produce. Suppose action A's JSON is halfway out when action B starts emitting. The action lane would read:

what atomicity prevents
<|toolcall_begin|>{"function": "navigate", "argu
  <|toolcall_begin|>{"function": "search_music"…
    ments": "nearby restaurant"}<|toolcall_end|>

That is not a parsing inconvenience — it is irrecoverable. The host application receiving this stream cannot know which fragment belongs to which call. Worse, the model would have to learn to produce and re-consume interleaved JSON, which is a strictly harder language-modelling problem for no benefit. Atomicity turns the lane into a well-formed message stream: read tokens, accumulate until you see <|toolcall_end|>, parse, execute.

Note the cost, though, and be honest about it: atomicity plus FIFO means an urgent action cannot jump the queue behind a long one. If the assistant is halfway through emitting a verbose search_food call and the user says "stop the car's fan, it's too loud", the fan call waits. Chapter 10 lists this among the design's real limits.

Hand-worked: two actions, every intermediate step

Time to push numbers. This is the arithmetic the data pipeline performs on every training sample, and it is worth doing once by hand until it is boring.

Setup. Budget B = 10 tokens per chunk. Two actions, both from the multi-intent car scenario:

Step 1 — how many chunks does each action need, ignoring interference?

Step 2 — lay A₁ out chunk by chunk. It starts at its trigger chunk, 11, and takes ten tokens per chunk until it runs out:

Step 3 — where does A₂ go? Its trigger chunk is 13. But chunk 13 is fully occupied by A₁'s tokens 21–30, and Rule 2 forbids preemption. So A₂ waits. A₁ fully drains in chunk 14, which has four free slots.

Here the paper's phrase "starts emitting from the next available chunk" admits two readings, so we compute both and label them honestly:

Step 4 — compute the realised delays. The evaluation protocol timestamps an event at the start of its chunk, t_k = 0.16 k, and takes the absolute difference from the annotated anchor. So:

Step 5 — sanity-check against the paper. DuplexSLA's measured multi-action delay is 0.67 s, comfortably above both of our computed values. Good: our arithmetic captures the mechanical component (quantization plus queueing) and the measured number includes everything else — the model's own semantic latency in recognizing the intent, which Chapter 1 estimated at roughly two chunks. Mechanism plus model, and the mechanism is the small part.

The trigger time does not drift

The most important consequence of the spill rule is stated in Appendix A.3, and it is easy to misread:

"The strict time-alignment prior installed by dual-side ASR during CPT lets the model treat this multi-chunk spillover as a bounded, budget-induced transmission delay rather than as drift in the offset itself: the first action token still emerges at the chunk that owns the semantic anchor, so the realised trigger time remains aligned with the annotated offset."

Two separate quantities, and the paper insists on keeping them apart:

That fourth row is a small honest caveat on the headline numbers. The benchmark scores the moment the call was born, but the car cannot act until it has fully arrived. For a 40-token object at 10 tokens per chunk, that is roughly 640 ms of additional transmission before execution can begin. The paper's 0.64 s average delay is a measure of the model's timing intelligence, not of end-to-end actuation latency. Both are legitimate quantities; they are just not the same quantity.

Three forms of the same computation

Now write the scheduler down. Same computation, three ways — hand arithmetic above, explicit loop, then the vectorized one-liner.

Form 2: the explicit loop. This is the greedy reading (Reading A), written so every rule from Section 3.3 is visible as a line of code:

python — the FIFO action scheduler, step by step
def schedule(actions, budget=10):
    """actions: list of (trigger_chunk, n_tokens), sorted by trigger time.
       Returns per-chunk fill and the realised first-token chunk per action."""
    fill = {}            # chunk -> tokens already placed there
    first = []           # realised trigger chunk per action
    c = 0                # the queue's write head

    for (trig, n) in actions:
        c = max(c, trig)                 # never emit before the anchor (Rule 2)
        while fill.get(c, 0) >= budget:      # this chunk is full: wait
            c += 1
        first.append(c)                    # the timestamp that gets evaluated

        remaining = n
        while remaining > 0:
            room = budget - fill.get(c, 0)
            put  = min(room, remaining)
            fill[c] = fill.get(c, 0) + put
            remaining -= put
            if remaining > 0:
                c += 1                     # spill into the next chunk (Rule 2)
    return fill, first

schedule([(11, 36), (13, 30)])
# fill  = {11:10, 12:10, 13:10, 14:10, 15:10, 16:10, 17:6}
# first = [11, 14]         <- matches the hand-worked table exactly

Trace it against Step 2 and Step 3 above and confirm every number. The max(c, trig) line is "never emit before the anchor". The while … >= budget line is "never preempt". The inner loop is the spill. Three rules, three lines.

Form 3: the vectorized version. Once you see that the scheduler is really just "lay tokens end to end and cut every B", the whole thing collapses. For the common case where actions are separated enough that nobody waits — or, more usefully, for a single action — the chunk index of every token is a division:

python / numpy — the same layout, vectorized
import numpy as np

n_tokens, budget, start = 36, 10, 11

# which chunk does each of the 36 tokens land in?
chunk_of_token = start + np.arange(n_tokens) // budget
# -> [11 x10, 12 x10, 13 x10, 14 x6]

# how many tokens per chunk?
np.bincount(chunk_of_token - start)      # -> [10, 10, 10, 6]

# how many chunks does a whole batch of actions occupy?
lens = np.array([36, 30])
chunks_each = -(-lens // budget)             # ceil division -> [4, 3]

# and the one-liner: cumulative start chunk of each action, back to back
starts = start + np.concatenate(([0], np.cumsum(chunks_each)[:-1]))
# -> [11, 15]   (strict Reading B: each action starts a fresh chunk)

Compare that last output with the hand-worked table: [11, 15] is exactly Reading B. The greedy Reading A needs the loop, because packing partial chunks is a stateful operation that cumsum cannot express. That is a useful lesson in itself — vectorization is free when your allocation is aligned, and expensive when it is not.

Form 4, for completeness: the library one-liner. Once you recognize the shape, the standard-library answer is a single call, because "cut a stream into fixed-size frames" is a solved problem:

python — the whole spill rule in one line
from itertools import batched   # Python 3.12+

chunks = list(batched(action_tokens, 10))    # [(t1..t10), (t11..t20), ..., (t31..t36)]
# len(chunks) == 4; the trigger time is the chunk of chunks[0], the rest is transmission.

That is the entire across-chunk rule: batched(tokens, budget). Everything else in Section 3.3 is bookkeeping about what happens when two of these streams meet.

Experiments worth running in the drainer, in order:

1. Budget 10, three actions, gap 2. The realistic multi-intent case. Note that the first action's delay is zero and the later ones inherit the queue — and that the total is still well under a second.
2. Budget 4. This is the Chapter 1 "slow accelerator" regime. Watch the third action's first-token chunk drift far from its anchor: on constrained hardware, multi-action requests degrade first, and they degrade in timing before they degrade in accuracy.
3. Action size 60. Verbose planning text. The queue lengthens proportionally. This is the concrete reason Appendix E specifies that planning text is "kept short so that it fits within a few chunks" — verbosity on this lane is a latency tax on every action behind it.
4. Trigger gap 0. Two intents in the same chunk. Now Rule 1 applies instead of Rule 2: they concatenate head to tail in that chunk's segment, and the terminator waits for both.

Second worked example: three actions, tight budget

Now the multi-intent car request, on a slower accelerator where the budget has been retuned to B = 6 (Chapter 1: this corresponds to roughly 14.5 ms per token). Three actions, sizes 30, 26, and 34 tokens, triggered at chunks 19, 28, and 51 — the anchors we computed in Chapter 1's timeline exercise.

Step 1 — chunks needed each, ignoring interference:

Step 2 — lay them out under FIFO:

Step 3 — the conclusion, which is the interesting part. Nobody waits. Even at a budget of 6, the three actions never collide, because their anchors are 9 and 23 chunks apart while each object needs only 5–6 chunks to drain.

That is the general case for real speech. Human intents in a multi-intent utterance are separated by the time it takes to say a clause — typically 1–3 seconds, which is 6–19 chunks — while an action object drains in 3–6 chunks. The queue is usually empty when the next action arrives.

Step 4 — find where it breaks. Set up the collision deliberately. Two intents 2 chunks apart, budget 6, 30 tokens each:

So even a deliberate collision on a degraded budget produces a delay the benchmark tolerates. The FIFO design is more robust than it first appears, and now you can say why rather than taking it on faith.

Why the queue never blocks the voice

One more sentence from Section 3.3 closes the loop with Chapter 2's ordering argument: "Because the assistant TA4 channel has its own per-chunk token budget, this FIFO queue on the action channel never blocks assistant speech: while the queue is draining over several chunks, the TA4 stream keeps producing audio in lockstep."

This is why the two-lane split was worth the ten tokens. In a single-lane design, a four-chunk action burst would be four chunks of missing audio — 640 milliseconds of silence in the middle of a sentence, exactly when the user is most attentive. With separate lanes and a fixed TA4 allocation, the burst is invisible to the listener. The queue can be arbitrarily backed up and the voice does not notice.

What the model actually has to learn from this

A closing observation that matters for Chapter 7. All of the above is a data-construction policy: it describes how supervision targets are laid out. The model is never told the rules. It sees millions of examples in which action tokens happen to arrive ten at a time, in trigger order, with atomic JSON blocks, and it learns the distribution.

Which means the rules are learned as statistical regularities, not enforced as constraints. At inference time nothing prevents the model from emitting an eleventh token, or from opening a second tool call before closing the first. The paper does not report how often that happens. Practical deployments would clamp the lane with a decoding constraint — a grammar or a hard cap — and the paper's framing of the cap as a "deployment budget" that can be "re-tuned per accelerator without retraining" is consistent with that reading.

Chapter 5 in review

If you remember one thing: this is a real-time scheduler written in token order. A periodic hard-deadline task (audio) gets a guaranteed reservation; an aperiodic soft-deadline task (actions) runs in the slack. Every rule in the chapter is a standard answer to a standard scheduling problem.

Chapter 6: Pause, Interrupt, Backchannel — Without a VAD

Play these two clips in your head. Both are the assistant mid-sentence, giving advice about overwork. Both have the user saying exactly the same three words at exactly the same moment.

These are the paper's own Figure 2 examples, and they are the cleanest statement of why turn-taking is a semantic problem. At the moment you must decide, the acoustic evidence is identical. The difference has not been spoken yet.

An energy VAD cannot solve this. Neither, quite, can a semantic VAD that only sees the user's side — because "you are right" is agreement in both clips; what differs is whether the user is done, and that depends on prosody, on the assistant's own state, and on what happens 300 ms later. This chapter is about what it takes to make that decision inside the model.

The three phenomena, defined

Section 2.4 names three behaviours and, crucially, describes each in terms of what the two lanes do. Learn them as lane behaviours, not as concepts:

The phrase "without resetting its current speech plan" in the last row is doing quiet work. The assistant has committed to a multi-sentence answer. A backchannel must not restart it, must not cause a repetition, must not derail it into responding to "you are right." The model must notice, log the notice, and continue the plan — which is a harder behaviour than either ignoring or stopping.

Reading the interrupt trace

Appendix A.4 gives an actual chunk-by-chunk trace of the interrupt case. Below it is, with the Chinese transcribed and glossed. The assistant is finishing "…health is the most important thing…"; the user starts with an acknowledgment and rolls into a real objection.

Read the timing off the table:

• The acknowledgment ("you are right") occupies chunks 0–1. The assistant keeps speaking through it. No label, no reaction — at that point the utterance is still consistent with a backchannel.
• The new content ("but the project…") begins at chunk 3. The interrupt label fires at chunk 5.
• Latency from the start of the new content to the label: 2 chunks = 320 ms. From the label to silence: 1 chunk = 160 ms.
• Total from "the user has started a real new thought" to "the assistant is quiet": 3 chunks = 480 ms.

Compare with the benchmark: DuplexSLA's measured interrupt delay is 0.40 s. Our trace-derived 0.48 s is the same order, and the difference is exactly the kind of variation you would expect between one illustrative sample and a 300-case average. The trace is not a cartoon; it is what the numbers look like from the inside.

Reading the backchannel trace

Now Appendix A.5, the same shape with the opposite outcome. The assistant is mid-answer; the user says "没错" ("that's right") and then stays silent.

The critical column is the assistant anchor. Compare it with the interrupt trace: there, the anchors turn into <vad_silence> one chunk after the label. Here, the anchors march on — 比 直接 撒糖 有意思 多了 — completing the sentence the assistant had planned. The label was emitted; the plan was not disturbed.

The label fires at chunk 2, inside the user's short utterance (chunks 1–2). The paper's accuracy window for backchannel requires the event to land in [t_bc-s − 0.2, t_bc-e + 2], and the delay is measured against the end of the backchannel utterance. Measured average: 0.32 s. So the model typically labels a backchannel about two chunks after it ends, which is roughly when the crucial disambiguating evidence — the silence that follows — has arrived.

The label vocabulary, and why it has synonyms

Appendix B lists the canonical control labels used on the action channel — and a design detail that is easy to overlook but is genuinely clever:

Why paraphrase? The paper explains: "During data construction, the same name field is paraphrased by several near-synonyms so that the model is not over-fit to a single surface form."

Think about what over-fitting to one surface form would mean here. The label is emitted as text on a text lane by a language model. If every interruption in training produced the identical eight-character string, the model would learn a brittle template: a high-probability lexical reflex triggered by superficial cues, decoupled from the semantics that should drive it. By varying the surface, the training signal forces the decision to be the invariant and the wording to be the noise.

Pause: the behaviour that is an absence

Pause deserves separate attention because it is the only one of the three whose correct behaviour is nothing happening.

The user says "I want to go to…" and stops. Half a second of silence. An energy VAD, tuned to a typical 500–700 ms endpoint threshold, declares the turn over and the assistant barges in over the user's next word. Everyone has experienced this; it is the single most common failure of deployed voice assistants.

DuplexSLA's response is to keep emitting silence anchors and response-style continue-listening labels. Nothing observable happens. And yet this is a decision the model makes 6.25 times a second, scored by the loss, supervised by a 36,000-hour data slice.

The benchmark reflects the difficulty. In the no-prefill setting, the pause scenario is where the open-source duplex backbones collapse: Freeze-Omni at 11.00% and PersonaPlex at 22.00%, versus DuplexSLA's 93.00%. The paper's caption is blunt about why: "Open-source duplex backbones without targeted post-training collapse on the pause subset, illustrating that pause robustness has to be supervised explicitly."

Read that as the chapter's engineering lesson. Being full-duplex does not give you pause robustness for free. The architecture makes the behaviour expressible; only data makes it reliable.

The lab's point is the one the chapter opened with: slide the continuation from 0 to 4 chunks and the energy envelope for the first several chunks does not change at all, while the correct behaviour inverts. Any decision rule that reads only the envelope is making a coin flip. Any decision rule that waits long enough for the envelope to disambiguate has already talked over the user.

What "removing the external semantic VAD" actually buys

Let us be precise about the win, because "no external VAD" can sound like a purity argument rather than an engineering one. Three concrete benefits, in order of how measurable they are:

And the honest counter-column, which the paper does not write but which Chapter 10 will: you also lose the ability to tune turn-taking independently of the voice, to swap in a better detector next quarter, or to explain a specific failure without retraining a 7B model. The cascade's modularity was a real asset, and integration spends it.

The pause problem, in numbers

Make the endpointing dilemma quantitative, because it explains why every voice assistant you have used feels wrong in one of two ways.

An energy VAD declares the turn over after H milliseconds of silence — the hangover. Choosing H is a forced trade:

There is no good row. The hangover must exceed the longest hesitation you want to survive, and it is paid on every turn including the ones with no hesitation at all. That single trade explains the entire commercial baseline column in Chapter 9: delays of 0.95–1.68 s are what buying pause robustness with a timer costs.

Now compare with DuplexSLA's measured numbers: normal 0.27 s and pause 0.27 s, at 96.00% and 93.33% accuracy. Identical latency on both scenarios. That is the signature of a system that is not using a timer at all — a threshold-based system must show a latency floor equal to its hangover, and this one does not.

Barge-in engineering, before and after

Handling an interruption in a cascaded stack is a genuinely fiddly piece of engineering. Enumerating it makes clear how much the duplex design absorbs:

That last row is a quiet gift from the dual-side ASR design. Context repair after barge-in is one of the most annoying bugs in production voice agents, and here it falls out of the timing supervision that was added for a different reason.

What this chapter does not cover

Human turn-taking is richer than three labels. For honesty, here is what a conversation analyst would say is missing:

• Turn-yielding cues. Humans signal that they are about to finish — falling pitch, slowing rate, syntactic completion, gaze. The model may well use these implicitly, but nothing in the paper's label set represents "the user is about to finish", only "the user has finished."
• Floor competition. When both parties start at once, humans negotiate — one drops out, often within 200 ms. Here, interruption is one-directional: the user takes, the assistant yields.
• Latching and overlap-in-agreement. Enthusiastic overlap that is neither interruption nor backchannel.
• Prosodic backchannel invitation. Speakers actively invite "mm-hmm" at certain points. An assistant that never invites feedback feels subtly monological.
• Repair. "I mean…", "sorry, what I meant was…" — self-correction mid-turn, which as Chapter 10 notes interacts badly with acting on early fragments.

None of these are failings of the paper — three labels covering pause, interrupt, and backchannel is already more than any deployed system expresses. They are the map of what a fourth, fifth, and sixth label could be, and the architecture has room for all of them at ten tokens a chunk.

An implementation note worth stealing

One detail from the evaluation protocol (Section 5.1) reveals something about how this system is meant to be operated: "the assistant audio output is post-processed by an external VAD to obtain speak and stop transitions; for DuplexSLA the action channel is also read directly."

So a VAD does still appear — on the output side, as an instrument for measuring when the assistant starts and stops speaking. That is a fundamentally different role from the input-side VAD the architecture eliminates. One is a decision-maker inside the loop; the other is a measurement device outside it. Removing the first does not mean the second is useless, and conflating them is a common misreading of "no VAD" claims in this literature.

Where the three behaviours meet the tool channel

Chapter 4's backchannel-action pattern is the point where this chapter and the tool-calling chapters intersect, and it is worth stating the composition rule explicitly:

Rows two and three are the interesting pair: the same label, opposite consequences on the action lane; and different labels, similar user utterances. What determines the split is whether the utterance requires the floor, not whether it requires an action. A request can be dispatched without taking the floor — that is precisely the insight backchannel-action encodes.

Row four is our own construction and the paper does not cover it: an utterance that both takes the floor and implies an action. The architecture can obviously express it (emit interrupt and a tool call in the same chunk), but it is not among the three trained tool-call patterns, so its behaviour is unspecified. A small, concrete gap in the benchmark.

The echo problem, and an unexpected advantage

One practical obstacle sits under every duplex system and the paper does not discuss it: the assistant's own voice comes out of a speaker in the same cabin as the microphone. Without treatment, the model hears itself.

Cascaded systems handle this with acoustic echo cancellation — a filter that subtracts an estimate of the played signal from the captured one. It works, imperfectly, and residual echo is a classic source of spurious barge-in: the assistant interrupts itself.

A duplex model has an interesting structural advantage here, which follows from the architecture rather than from any signal processing:

The paper reports nothing on this — its training audio is synthesized and merged, so there is no acoustic echo in it at all. Which is worth flagging in both directions: the architecture has a natural advantage on echo, and the evaluation contains no evidence that it materializes. A duplex model trained purely on cleanly-merged tracks may never have learned to expect its own voice on the user channel.

If you were designing the label set

Three labels is a small vocabulary for something as rich as turn-taking, and the paper says why: "The label set is kept compact so that turn-taking decisions are decoupled from spoken content." That is a real design principle, and it suggests criteria for adding a fourth:

Run a candidate through it. "Turn-yielding imminent" — the assistant signalling that it is about to finish — is decidable (the model knows its own plan), actionable (the user could be invited in), observable (anchor at the actual end of the utterance), distinct, and cheap. It passes all five, and it does not exist yet in any system. That is a small research proposal you can now write down.

Now run a bad one. "The user seems frustrated" — decidable, arguably; actionable, only if something downstream changes; observable, poorly (what is the anchor for an emotion?); distinct, no (it can co-occur with all three existing labels). It fails on observability and distinctness, which is precisely why it belongs in the planning text rather than in the label vocabulary.

Chapter 6 in review

The three behaviours, as a decision table you could implement — except that no rule engine could evaluate the middle column, which is the point:

If you remember one thing: the first 800 milliseconds of a backchannel and an interruption are acoustically identical, so the decision cannot be made from the audio alone at the moment it must be made. Everything DuplexSLA does here follows from putting that decision inside the model that already knows what it is saying and what it planned to say next.

Three exercises:

1. DuplexSLA's normal and pause delays are both 0.27 s. Explain why a threshold-based endpointer cannot produce equal numbers on those two scenarios, whatever its threshold.
2. Using the Appendix A.4 trace, compute the three latencies (new content to label, label to silence, total) and compare each with the benchmark's 0.40 s average.
3. Why does the training data paraphrase the labels, and what would you expect to break if it did not? Name the specific failure mode.

Chapter 7: Manufacturing Data That Does Not Exist

Here is the problem that would stop most teams before they started. There is no corpus of chunked, dual-track, three-channel spoken dialogue with time-stamped tool calls. There is no such recording anywhere in the world, because the format is an invention of this paper.

Section 3 opens by saying exactly that: "The chunked, dual-stream three-channel format described in Section 2 does not match the format of conventional dialogue corpora, so building DuplexSLA required a dedicated data-construction effort."

This chapter is that effort. It is the chapter readers skip and the chapter that determines whether the model works — because for a model that must learn when as well as what, the layout of the supervision is the method.

The shape of one training sample

Section 3.1 gives the schema. Every training sample is a chunked dual-track session containing:

And then the sentence that explains why the whole thing is trainable at all: "The same schema covers all task families: they differ only in which channels carry information."

Stage 1: an LLM writes the actions into the dialogue

Figure 4a shows the annotation stage. Start with an ordinary text dialogue — the kind that does exist in quantity:

raw dialogue (before annotation)
USER:      I'm off work, what do you think I should eat tonight?
ASSISTANT: That's a world-class dilemma. Are [It's so cold in the car.]
           you planning to cook or eat out?
   ...
USER:      Let's have a light food salad, it's healthier. Help me navigate
           to a nearby Wagas, and play some light music.
ASSISTANT: Okay, navigating to a nearby Wagas for you now, and the music
           has been turned on too. Let's go.

An LLM, given the tool schemas, annotates each dialogue with tool-call objects. For each one it produces a planning rationale and a structured call:

what the annotator emits
💭 The user feels cold, I should turn on the air conditioning and set it
   to heating mode.
🛠 {"func": "open_car_setting", "args": "air-conditioner: heat mode"}

💭 The user wants to go to Wagas for a light salad, I need to navigate there.
🛠 {"func": "navigate", "args": "destination: wagas"}

💭 The user wants to listen to light music, I need to open the music player.
🛠 {"func": "search_music", "args": "play: light music"}

The result is a tool-calling augmented dialogue in which each call is inserted at the position in the text where its intent occurs:

tool-calling augmented dialogue (after annotation)
USER: Let's have a light food salad, it's healthier. Help me navigate to a
      nearby Wagas [{"function": "navigate", "arguments": "destination: wagas"}],
      and play some light music [{"function": "search_music",
      "arguments": "play: light music"}].

Look at where the brackets sit. They are inside the user's utterance, immediately after the words that make each intent clear. This is the crucial move: the annotation is not "this turn requires these two calls" — it is "this call belongs here, at this point in the speech." Position in text will become position in time.

The same figure shows a backchannel-action example being planted mid-assistant-utterance: "That's a world-class dilemma. Are [It's so cold in the car.[{"function": "open_car_setting", "arguments":"air-conditioner: heat mode"}]] you planning to cook or eat out?" — the user's off-topic remark and its tool call are literally spliced into the middle of the assistant's sentence. That is how you fabricate a training example for a behaviour nobody has ever recorded.

Stage 2: from text to two aligned audio tracks

Figure 4b turns the annotated text into audio, and the pipeline has four steps:

Notice how much of this pipeline is timing machinery and how little is language machinery. That ratio is the honest signature of the problem: the hard part of teaching a model to act while speaking is not deciding what to do, it is deciding exactly when the doing should be visible.

The system prompts

Table 9 lists the training-time system prompt per task family. They are short, and reading them tells you exactly how the model is told what game it is playing:

Two observations. First, the three turn-taking families share a prompt that says nothing about turn-taking — it only sets the voice. The behaviours are taught by the data, not announced by the prompt, which is exactly what you want if the goal is "absorbed into core conversational competence" rather than "activated by an instruction."

Second, the {·} placeholder is filled with one of the 18 canonical speakers at sample-build time. So voice identity is a prompt-level variable throughout training. That is a small hint about how a system like this gets voice control — and a natural connection point to the persona-and-voice-control line of work the paper cites.

Why dual-side ASR is required — the deepest point in the paper

Section 3.4 is titled "Dual-side ASR is required for time alignment", and it contains the argument that most repays careful reading. Set it up properly.

The observation. Inside the TA4 layout, the text anchor T is left-aligned within its chunk. The assistant's text stream looks like this, with the trailing chunks padded:

Words get packed into anchor slots as they come, four audio tokens at a time. But a word's audio may not finish in the chunk whose anchor slot holds it. The paper: "a single Chinese word can be packed into the first T slot of a chunk while the corresponding audio is actually played in the next chunk."

The naive fix, and why it fails. If we want the model to learn when its own speech is happening, why not just delay-copy the anchors onto the action channel with a fixed lag, exactly as the user-side ASR does with its 2-chunk lag? Because the anchors are not on time in the first place. Copying an inaccurate timestamp with a constant offset gives you an inaccurate timestamp.

The proof is in the Appendix A.2 trace. Watch the lag vary:

There it is. Two consecutive characters, two different lags. If the anchors carried true timing, the lag would be constant. It is not, because the action-channel emission is placed "at the chunk where each character is actually being spoken" — and the anchors were packed earlier than that, at different amounts of earliness.

The consequence. Section 3.4's closing sentence is the payoff for the whole architecture: "By forcing the action channel to emit the assistant transcript at the chunk where each character is actually being spoken, we explicitly tie assistant audio to action time. As a result, the model's internal time clock stays consistent across user audio, assistant audio, and action emission, which is what makes sub-second tool-call latency feasible."

Switch between "naive delay-copy" and "assistant-side ASR" and compare the error bars. The delay-copy transcript inherits the anchors' packing error; the ASR transcript is anchored to the audio itself. The gap between those two rows is what 90,000 hours of training data buys.

The mixture

Figure 5 and Table 2 give the proportions. Two stages, seven task families:

Three ratios worth committing to memory, because each encodes a design belief:

1. 36% of CPT is ASR. More than a third of continued pretraining is spent teaching the model what time it is, not what to say. For a duplex model, timing is not a finishing touch.
2. Post-training is 10% the size of CPT. 50k hours against 500k. The capability-critical behaviours are taught with an order of magnitude less data than the format itself — because by then the model already has the clock.
3. Tool calling is the smallest slice in the whole recipe. ~14k hours, 2.5% of the total audio. The headline capability of a paper named "Speech, Language, and Action" is trained on the least data. That is only possible because everything underneath it — format, timing, turn-taking — was already installed.

The action object, field by field

Appendix E gives the abstract schema shared by every data family. Four fields, and each one is a design decision:

Look at what is not there: no priority, no dependency on another action, no result, no confirmation flag, no expiry. The object is deliberately flat. That flatness is what makes the single FIFO queue sufficient — and it is also the boundary of what this design can currently express.

What you would need to rebuild this dataset

A practical inventory, because "we constructed the data" hides a great deal of work. To reproduce Chapter 7's pipeline you need:

1. A large corpus of ordinary text dialogue in your target domain and language — the raw material the annotator augments.
2. A tool schema, written out with descriptions, so an annotator LLM can select functions and fill arguments. Fifty here; the descriptions in Appendix C are one sentence each.
3. An annotator LLM plus a prompt that inserts calls at the right position inside the utterance, not at the end of the turn.
4. A multi-speaker TTS system with voice cloning — 18 assistant voices here, plus whatever variety you want on the user side.
5. A forced aligner for your language, accurate to well under a chunk (160 ms), or your offsets will be noise.
6. A timeline merger that can place two synthesized tracks with controlled overlap — and that knows what a realistic backchannel or interruption offset looks like.
7. A codec to turn the assistant track into discrete units at your chosen rate, plus the TA4 packer that interleaves the text anchors.
8. The serializer implementing Chapter 5's FIFO and spill rules, which is the only genuinely novel piece of code in the list.
9. Dual-side ASR alignment, re-emitting both transcripts at the chunk where each token is actually being spoken.
10. Per-task system prompts and a sampler that mixes the seven families in the right proportions.

Of those ten, exactly one — item 8 — is specific to this paper. The other nine are standard speech-pipeline components, which is why the data section reads as an engineering inventory rather than a research contribution, and why it is nonetheless the section that determines whether the model works.

Note how much of that is timing infrastructure and how little is modelling. This is the shape of most speech-systems work, and it is why data sections deserve to be read as carefully as method sections.

The tool schema: 50 functions, and what they reveal

Appendix C lists the full schema exercised in training and evaluation: 50 functions across four intent families. The distribution is itself informative:

Count the right-hand column. Roughly twenty of the fifty functions produce information the assistant would have to speak back. That is a large fraction of the schema for which the paper's fire-and-forget action lane, as described, does not close the loop. It is not a fatal gap — the host application can inject the answer as ordinary dialogue context — but it is a real one, and Chapter 10 counts it honestly.

Also note what is absent: no web search, no code execution, no email, no calendar writes, no multi-step workflows with dependencies. Every function is a single flat call to a local device. The paper's own conclusion names this as future work: "broader open-domain spoken tool use."

What varies from sample to sample

A useful way to see a synthetic pipeline is to ask which knobs are randomized. Here, as far as the paper describes:

And what is not varied, as far as the report says: acoustic conditions, language, speaking style beyond the 18 voices, microphone characteristics, and the absence of echo. Those are the axes on which a synthetic corpus is most likely to leave a gap, and they line up exactly with the robustness questions Chapter 10 raises.

Chapter 7 in review

The pipeline, end to end, with what each stage contributes to the final model:

If you remember one thing: the format of the supervision is the method. A model that must learn when cannot be taught by data that only records what, and no existing corpus records when. Everything DuplexSLA can do that other systems cannot traces back to a decision made in this pipeline.

Chapter 8: The Two-Stage Recipe — and Why the Order Matters

We have a format and a corpus. Now the question every practitioner actually asks: in what order do you feed it, and what does the loss look like?

Section 4 answers in three parts — two stages, a modified loss, and a short, unusually candid paragraph about why the stages are divided the way they are. That last paragraph is the most useful thing in the section, so we will build up to it.

Read this chapter with one question in mind throughout: which of these decisions would you have gotten wrong? Most of them look obvious in retrospect and are not — particularly the loss reweighting, whose absence would have produced a training run that looked healthy and a model that was quietly worse at everything that mattered.

Where the model starts

"DuplexSLA is initialized from Step-Audio 2 mini, a 7B-scale audio language model." Before a single duplex sample is seen, the model already has world knowledge, language ability, audio understanding, speech generation from discrete units, and instruction following.

So the training run is not "learn to talk." It is "learn to talk on a clock, in a format you have never seen, without forgetting anything." That framing explains several choices that would otherwise look odd — particularly the 1.92M general text samples mixed into an audio training stage.

Stage 1: continued pretraining (~500k hours)

The goal, stated in Section 4.1, is to "make the backbone fluent in the new serialization." The model must learn three things simultaneously:

And the honest report of what CPT does not achieve: "After CPT, the model becomes comfortable with the duplex serialization, but does not yet exhibit the targeted real-time interaction behaviours, especially when the user pauses, interrupts, or issues short backchannel feedback."

Read that as a strong empirical claim about what generic duplex dialogue data contains. Three hundred and twenty thousand hours of conversation apparently do not teach reliable pause handling. Why not? Because natural dialogue is dominated by the easy case — clean alternating turns — and the interesting cases are rare, unlabelled, and drowned. This is a general lesson about scale: more of the same distribution does not fix a tail behaviour if the tail is what you care about.

Stage 2: capability-oriented post-training (~50k hours)

Post-training "shifts the data distribution from generic duplex dialogue toward the behaviours we want to evaluate. The mixture is deliberately small, but each slice is highly informative."

Two details in the first row are worth separating, because they are different skills. "Emit the right label at the right time" is a detection problem on the action channel. "Switch the assistant TA4 to silence within a small chunk-level latency" is an execution problem on the audio channel. A model could learn to detect interruptions perfectly and still keep talking through them. Post-training has to couple the two, which is only possible because both lanes are outputs of the same backbone in the same step.

And note the composition of the tool-call slice: it is not one tool-calling distribution but three, matched exactly to the three benchmark patterns. The training set and the evaluation set are structured identically. That is standard practice and it is also worth flagging — Chapter 10 returns to what it means for generalization.

The loss, and the tokens it refuses to treat equally

Section 4.3 describes a base objective and then a modification. The base:

Standard next-token cross-entropy on the two supervised channels, plus a general text-modelling term. The user audio side is never a target — it is observed only.

Then: "On top of this base loss, we apply additional loss masks and per-token weights to selected state tokens and to specific positions in the chunked dual-stream three-channel sequence, so that the optimisation is better matched to the full-duplex training setting (e.g., silence anchors, channel-boundary markers, and task-conditioned segments are not trained as ordinary content tokens)."

The paper does not give the weights. But it names the three categories, and each one has an obvious motivation you can reconstruct. Let us do that, with arithmetic.

Category 1: silence anchors. Suppose the assistant speaks 40% of the time in a typical duplex session (generous — in a conversation with a talkative user it is much less). Then:

Train those as ordinary content tokens and the majority of your gradient signal teaches the model to predict silence. It will get exceptionally good at that — it is an easy, highly predictable target — and the loss will look wonderful while the speech quality does not improve. Down-weighting silence rebalances the objective toward the tokens that carry information.

Category 2: channel-boundary markers. Count them per chunk: <|user_audio_begin|>, <|user_audio_end|>, <|assistant_audio_begin|>, <|assistant_audio_end|>, <|action_end|> — five markers, all fully deterministic given the position in the chunk.

A perfectly-predictable token contributes near-zero loss once learned, so the argument is not that markers dominate the objective forever — it is that they occupy attention, context, and early-training gradient for no benefit. Masking them is cheap hygiene.

Category 3: task-conditioned segments. The system prompt tells the model which game it is playing. Training the model to generate its own system prompt is not merely useless; it actively encourages the model to model the distribution of tasks rather than to condition on the given one.

A note on what "post-training" means here

The term is overloaded in 2026, so pin it down. In text-LLM practice, post-training usually means instruction tuning followed by preference optimization — a small, alignment-shaped stage measured in millions of tokens.

Here it means 50,000 hours of audio. By Chapter 8's own conversion that is over a billion chunk-decisions, all supervised, all next-token cross-entropy. It is not alignment; it is a second curriculum with a different data distribution.

Keeping the distinction straight matters when reading the recipe: nothing here is alignment, and nothing here uses preferences. The behaviours are taught the same way the format was, just with a distribution chosen to concentrate on rare events.

Stage division by capability — the argument for the order

Section 4.4 is short and it is the most transferable paragraph in the paper. Here it is, unpacked:

"A turn-based agent can be improved by adding more text or more tool examples. A duplex spoken agent carries the additional burden of timing."

That is the framing. Turn-based agents have one axis of difficulty: content. Duplex agents have two: content and time. And the two are learned differently — content from examples, timing from a prior that must be stable before anything else can be layered on.

Hence the division: "The CPT stage therefore establishes the timing prior using ordinary duplex dialogue plus dual-side ASR, and the post-training stage sharpens it for pause, interrupt, backchannel, and tool calling."

And then the sentence that reports an actual experiment, compressed to a clause: "This division was the most data-efficient setup in our experiments: pure duplex dialogue alone teaches turn taking but not interaction control, while starting with capability-heavy data without first stabilizing the duplex serialization leads to noticeably worse speech smoothness on the assistant audio."

Three orderings, three outcomes:

The fourth button is our extrapolation rather than the paper's: remove the dual-side ASR slice and watch the timing meter stall while every other meter fills normally. Section 3.4's argument says this is what should happen — the model would speak beautifully and act at the wrong moments — and it is the single most instructive counterfactual in the recipe.

The loss, term by term

Write the objective out fully, with the shape of each term, so nothing is hand-waved:

Add up the "structural" rows against the "content" rows for an idle chunk and the imbalance is stark: 5 markers plus 5 silence tokens against 0 content tokens. For a speaking chunk with an active action lane: 5 markers against up to 15 content tokens. The mix swings by a factor of several depending on what is happening, which is another argument for masking — without it, the effective learning rate on content varies with how talkative the sample is.

A capacity argument for the 7B backbone

Worth doing roughly, because it explains why the paper cares so much about not wasting supervision. The model has one set of weights and four jobs:

The cascade splits these across four specialized components, one of which can be a frontier-scale LLM doing nothing but planning. DuplexSLA runs all four in a 7B model in a 160 ms window. Framed that way, an average tool-call accuracy of 85.56% against a cascade's 91.33% is a remarkably small gap, and Chapter 10's trade analysis becomes easier to reason about.

Why no reinforcement learning?

A natural question given the era: the recipe is entirely supervised — continued pretraining plus supervised post-training, with no preference optimization or RL stage. Some plausible reasons, which the paper does not state:

• The supervision signal is already dense and exact. Every chunk has a target on two channels. There is little of the "we know the outcome but not the trajectory" structure that motivates RL.
• The reward would be timing, and timing is measurable directly. If you can supervise the exact chunk an action should be emitted in, you do not need to search for it.
• Rollouts are expensive. A duplex episode is a real-time stream; sampling many trajectories per example costs far more than in text.
• Preference data would be hard to collect. Asking humans to compare two duplex conversations differing by 200 ms of yielding latency is not a practical annotation task.

That said, the obvious application exists: the benchmark's own scoring rule — correct function, correct arguments, legal trigger time — is a ready-made reward. It is a natural next paper, and its risk is equally obvious: optimizing directly for the timing window would push the model toward firing as early as the rule allows, which is exactly the speculation problem Chapter 10 raises.

What the recipe does not tell us

Being precise about the gaps is part of reading a technical report well. Section 4 does not report:

• The actual loss weights. "Additional loss masks and per-token weights" names the categories but gives no values. Reproducing this would require a search.
• Optimizer, schedule, batch size, sequence length, or hardware. None appear. This is a capability report, not a reproduction recipe.
• Ablation numbers for the curriculum. Section 4.4 reports directional outcomes ("noticeably worse speech smoothness") without a table. We are asked to take the ordering on the authors' word.
• How the 7B backbone's original abilities are affected. There is no before-and-after on general audio understanding or language benchmarks, so the cost of the format change to the base model's other skills is unmeasured.
• Anything about inference-time constraint enforcement. Chapter 5 noted that the token budget and JSON atomicity are learned regularities; whether decoding constrains them is not stated.

None of these are damning — industrial technical reports routinely omit them — but a careful reader should know which claims rest on published numbers and which rest on the authors' report of their own experience. In this section, the curriculum argument rests on the latter.

If you were reproducing this

A practical ordering, derived from everything above, for anyone building a duplex action model on a different backbone:

1. Measure t_tok on your target accelerator first. Chapter 1's arithmetic sets your action budget, and the budget shapes your data construction. Doing this last means rebuilding the corpus.
2. Build the ASR slices before the interesting ones. Both sides. This is the timing prior, and every later capability depends on it.
3. Get the serialization fluent before teaching behaviours. Watch speech smoothness as the canary; it degrades first when the format is not yet second nature.
4. Reweight the loss before your first long run. Compute the silence and marker fractions for your format. If more than half your targets are structural, uniform cross-entropy is mismeasuring your progress.
5. Only then add capability slices, sized at roughly a tenth of pretraining, matched to the behaviours you intend to evaluate.

Three things to watch while this trains

If you were running this, the loss curve would tell you very little — it is dominated by structural tokens that are learned in the first hour. Three better instruments, each derived from something in this lesson:

The middle one deserves emphasis because it is the metric that would have made Section 3.4's argument quantitative. A signed offset error tells you not just that timing is wrong but which way, which distinguishes a mis-aligned corpus (constant bias) from an under-trained model (high variance). Neither appears in the paper.

What 500,000 hours means in tokens

Audio corpora are quoted in hours, but a language model consumes tokens. Convert, because the number reframes the whole training run:

Roughly a hundred billion sequence positions, of which about seventy billion are supervised. That is a pretraining-scale run by any measure — comparable in token count to training a mid-size language model from scratch — being spent on reformatting a model that already knew how to speak.

Two implications. First, "continued pretraining" is not a fine-tune; it is a second pretraining, which is why it can install something as fundamental as a clock. Second, the ~1.92M general text samples that preserve language ability are a tiny fraction of that stream, which makes their placement and weighting a delicate business the paper does not detail.

The post-training stage, by the same arithmetic, is about 1.1 × 10⁹ chunks — still over a billion supervised chunk-decisions for behaviours as narrow as "keep talking through a backchannel." Timing behaviours are apparently expensive to install even when everything underneath them is already in place.

Reading the recipe as a dependency graph

The curriculum makes most sense drawn as dependencies rather than as a timeline. Each capability requires the ones below it:

Read upward and the data proportions stop looking arbitrary: the recipe spends the most on the widest layer and the least on the narrowest, which is what you would do for any skill that composes.

Chapter 8 in review

If you remember one thing: a duplex spoken agent carries the additional burden of timing, and timing is a prior that has to be stable before behaviours are layered on. That single sentence explains the stage division, the size of the ASR slices, and the surprising failure mode of capability-first training — which degrades the voice, not the behaviour.

Three exercises:

1. Assume the assistant speaks 25% of the time rather than 40%. Redo the silence-fraction arithmetic and say whether the case for reweighting gets stronger or weaker.
2. You have a fixed budget of 100k hours total. Allocate it across the five slices and justify each number using the dependency graph above.
3. Design the ablation that would isolate the dual-side ASR contribution. What do you train, what do you measure, and what result would falsify Section 3.4's claim?

Chapter 9: DuplexSLA-Bench — Scoring When, Not Just What

Every claim in this lesson now has to survive contact with numbers. And the first problem the authors faced is that the numbers did not exist: no benchmark measured what they built.

Section 5.1: "Existing duplex benchmarks measure pause, interruption, and backchannel behaviour, but none of them jointly stress sub-second yielding under semantic interruption, backchannel detection inside the action channel, backchannel-triggered tool calling, and multi-action tool calling on a duplex timeline."

So they built one. This chapter reads the protocol carefully — because a benchmark authored by the system's authors deserves careful reading — and then reads the results.

Composition: 2,100 cases

Each test case is a duplex audio session with semantic anchor times annotated. That annotation is the benchmark's real asset: it is what makes "did it act at the right moment?" a scoreable question rather than a vibe.

The protocol, in three deterministic stages

Table 4 gives the evaluation as pseudocode in set-theoretic form. Translated:

Four things to notice about this design:

1. Everything is streamed at 160 ms. Even systems with a different internal clock are fed in this granularity, which makes the comparison uniform and slightly favours a model whose native clock is 160 ms.
2. Events are stamped at chunk starts. t_k = 0.16 k, not at the middle or end. So the measured delay includes a quantization term of magnitude up to one chunk, averaging 80 ms — the floor we derived in Chapter 1, now visible in the metric definition itself.
3. Accuracy comes first, delay second. Delay is "computed only on hits." A system that misses a scenario contributes nothing to its delay average, so a low delay on a low accuracy is not the achievement it looks like — a subtlety worth carrying into the tables.
4. The earliest qualifying event wins. argmin t inside the window. A system that emits several candidate events is scored on the first one that lands in bounds.

Windows and anchors

Table 3 defines, per scenario, what counts as correct and what the delay is measured against. This is where the benchmark's judgment calls live:

The backchannel row needs unpacking, because it looks inverted at first read. The accuracy criterion for baselines is relaxed to "any stop-or-restart event inside the window", since closed-source systems emit no backchannel label. But a stop is exactly the wrong behaviour on a backchannel! So what is being scored?

Read the protocol note: "for s = backchannel, accuracy is relaxed to any {stop, speak} event inside W_s, since closed-source baselines emit no backchannel label and DELAY is therefore reported only when one is present." The relaxation exists so that baselines can score at all, by looking for any observable reaction to the backchannel in the audio. DuplexSLA, which does emit a label, is scored on the label directly. That asymmetry is worth flagging, and it is one reason the backchannel column deserves the most scrutiny in the results.

The tool-call correctness rule

Section 5.2 defines a triple condition. A predicted tool call counts as correct when all three hold:

Condition 3 is the one to study. It is asymmetric and both bounds are interesting:

• Not more than 1.0 s early. Firing before the intent is clear is a failure, not a triumph. A system that guessed "AC" from the word "cold" and fired immediately would be right often and catastrophic occasionally. This bound penalizes speculation.
• Not more than 3.0 s after the end of the audio. Note the reference point: the end of the audio, not the anchor. So the late bound is generous to slow systems — a cascade that plans after the endpoint has three full seconds of grace. The 2.33–4.71 s delays in Table 5 are measured against the anchors, but they remain legal because they land within three seconds of the audio ending.

And then: "Accuracy is the fraction of cases in which all ground-truth actions are matched, and delay is the average gap on matched actions." All-or-nothing per case. Get two of three functions in a multi-action request and you score zero for that case.

Result 1: tool calling

Table 5, the 900-case tool-call subset. The baseline is a cascade: "a streaming ASR module whose transcript is fed to an LLM that emits tool calls."

Four readings, in increasing order of interest:

1. The average delay ratio is 2.77 / 0.64 = 4.3×, which is the paper's "∼4x lower tool-call delay" claim. The conclusion states the range more carefully as "3−4x lower latency."
2. The multi-action delay ratio is 4.71 / 0.68 = 6.9×. The cascade degrades badly as intents multiply — it must hear all of them — while DuplexSLA does not degrade at all (0.68 vs 0.67 for single action). That flatness is the architecture's signature.
3. DuplexSLA wins outright on backchannel-action — both faster (0.57 vs 1.27) and more accurate (96.00 vs 95.33). This is the pattern with no turn-based equivalent, and it is the one where the duplex model has no handicap. Note also that the cascade is fastest here among its three patterns, because a short off-topic request ends quickly and the endpointer fires sooner.
4. The accuracy gap is real and concentrated. Single action: −3.66 points. Multi action: −14.33 points. Backchannel action: +0.67. Almost the entire average gap comes from one pattern.

Result 2: turn-taking with context prefill

Table 6, all four scenarios, systems that can preload the dialogue history:

This is the paper's strongest table, and it rewards column-by-column reading:

• normal / pause: everyone is roughly competent on accuracy; DuplexSLA is 4–6× faster. gemini even edges it on pause accuracy (94.33 vs 93.33). Latency is where the architecture shows, not accuracy.
• interrupt: now accuracy separates sharply — 99.33 against 63.67–79.00. Semantic yielding is the capability the baselines lack, and it is exactly what an energy or externally-attached semantic VAD cannot do well.
• backchannel: 98.33 against 40.00, 13.00, and 0.33. This is not a gap, it is a categorical difference. As Chapter 3 argued, a system with no way to represent "acknowledged, continuing" cannot score here at all.
• The N/A column: baselines expose no backchannel label, so only audio-derived events exist and there is nothing to time. The metric is undefined, not bad.

The two gpt-realtime configurations are also an instructive within-system comparison: semantic-vad-high buys accuracy on normal and pause (91.33/90.33 vs 82.33/71.00) at the cost of latency (1.67 vs 0.95 s). That is the semantic-VAD trade the paper predicted in its introduction, measured on a commercial system: the extra detector chain helps the decision and costs the clock.

Result 3: turn-taking without prefill

Table 7 reduces to the two scenarios every system supports, and adds open-source duplex backbones:

Three observations, one of which should make you cautious:

1. DuplexSLA is not the most accurate system here. gpt-realtime with semantic VAD is, at 96.50 against 94.34. The paper's own claim is carefully worded: "the only sub-second model with competitive accuracy." Read the claim as a Pareto argument, not a domination argument — and note that the delay difference is 5×.
2. The delay column has almost no spread among the fast systems (0.30, 0.36, 0.47) but an enormous spread in accuracy (94.34, 10.67, 22.34). Being fast is easy if you are willing to be wrong. This is exactly the "delay computed only on hits" subtlety from the protocol: Freeze-Omni's 0.36 s is averaged over the ~11% of cases it got right.
3. Freeze-Omni at 10.67% and PersonaPlex at 22.34% deserve scepticism. These are published duplex systems; scores that low usually indicate a protocol mismatch as much as a capability gap — a different native clock, a different expected input contract, a missing prefill path. The paper's own framing is about post-training ("open-source duplex backbones without targeted post-training collapse on the pause subset"), but their normal-scenario scores are equally low, which that explanation does not cover. Treat these two rows as weak evidence.

What the protocol chose not to measure

Before the numbers, one more look at the design. Three quantities a duplex evaluation could have scored and this one does not:

• Full-arrival time of an action. Only the trigger is timed. For a host application, the arrival is what gates execution — and it is several chunks later (Chapter 5).
• Speech quality during action emission. The central architectural claim is that the voice does not degrade while the action lane is busy. No metric in the paper measures that directly; it is verified only by the absence of complaint.
• False-positive actions. Accuracy is "the fraction of cases in which all ground-truth actions are matched." A system that also emitted three extra unrequested calls per case would not obviously be penalized by that definition.

The third is the most surprising omission. In a benchmark built around acting early, spurious action is the natural failure mode, and precision — not just recall — is what a deployment cares about. Nothing in the stated rule appears to count extra calls against a system.

Scoring one case by hand

Run the protocol yourself on a single interrupt case, so the tables stop being abstractions.

The case. The assistant is speaking. The user's semantic interruption anchor is annotated at t_int = 3.40 s. The audio is 8.0 s long, so K = ⌈8.0 / 0.16⌉ = 50 chunks.

Stage 1 — Init. Reset the model, prefill the history, empty log E.

Stage 2 — Stream. Feed 50 chunks. Suppose the model emits assistant speech through chunk 24 and silence from chunk 25 onward, and emits an interrupt label in chunk 24. The log gains, among others:

Stage 3 — Score. For s = interrupt, the required event type is τ = stop, the anchor is t^★ = 3.40, and the window is [t_int − 1, t_int + 2] = [2.40, 5.40].

Sanity checks. Would a slower system still hit? A stop at 5.30 s is inside the window — accurate, with a delay of 1.90 s. A stop at 5.50 s is outside — scored as a miss, and its delay is not counted at all. That is the accuracy-versus-delay asymmetry from the protocol, now concrete: late enough and you stop hurting your delay average and start hurting your accuracy instead.

Now do the arithmetic that makes DuplexSLA's 0.40 s average striking. To average 0.40 s, its stop events land, on average, 2.5 chunks after the semantic anchor — and Chapter 6's trace showed why roughly two of those chunks are evidence accumulation rather than lag.

Benchmark hygiene: what to check on any duplex evaluation

A transferable checklist, derived from reading this protocol carefully. Apply it to the next duplex paper you read:

1. Is delay computed on hits only? If so, a low delay at low accuracy is meaningless. Always read the two columns together.
2. What is the streaming granularity, and whose native clock does it match? Feeding every system 160 ms chunks is uniform, and it is also the home team's grid.
3. Where are events timestamped — chunk start, middle, or end? This sets a systematic offset of up to one chunk in every reported number.
4. Is the accuracy criterion the same for all systems? Here it is not: baselines get a relaxed audio-only backchannel criterion because they cannot emit labels.
5. Who wrote the benchmark? The system's authors, here. That is normal for a new capability and it is still a reason to weight cross-system comparisons carefully.
6. Are the anchors annotated by humans, by an LLM, or by construction? Here, by construction — the data is synthesized, so the anchors are known exactly. That is a strength for precision and a weakness for realism.
7. Is there an early bound as well as a late bound? Without one, a benchmark rewards speculation.
8. Do baselines get their intended deployment configuration? Two of the seven systems in Table 7 score under 25%, which usually means something about the harness, not only about the model.

The take-aways, and what they support

Section 5.4 states two patterns: "(1) On turn taking, DuplexSLA delivers sub-second responses in all four scenarios and is the only system that cleanly handles backchannel detection. (2) On tool calling, DuplexSLA matches the cascade in accuracy at ~4x lower delay, because the action channel emits planning and tool calls without waiting for a turn boundary or interrupting assistant audio."

Claim (1) is well supported by Table 6. Claim (2) needs a caveat that Chapter 10 supplies: "matches in accuracy" is 85.56 against 91.33 on the average, and 75.00 against 89.33 on multi-action. "Competitive" is the fairer word, and the paper uses it in the abstract ("while remaining competitive on tool-call accuracy") — the summary in Section 5.4 is the looser phrasing.

What both claims do jointly support is the design argument: "Together they validate the central design choice — an explicit action channel on top of a duplex backbone, supervised by the data recipe in Section 3." That is the claim the numbers actually license, and it is a good one.

Why the prefill split exists

Tables 6 and 7 differ by one flag: whether the system may preload the dialogue history H before streaming begins. The paper's reason is practical — "Many duplex systems cannot cheaply preload long histories" — but the split is more interesting than a logistics note.

The third row is the substantive one. You cannot test interruption without an utterance to interrupt, and you cannot put the system mid-utterance without giving it the context that produced the utterance. So the no-prefill setting is not a harder version of the same test — it is a strictly smaller test, covering only the two scenarios that need no history.

Which means the headline capabilities of this paper — semantic interruption and backchannel — are demonstrated only in the prefill setting, against three commercial baselines. That is a reasonable evaluation, and it is narrower than a quick reading of "1,200 turn-taking cases" suggests.

Chapter 9 in review

Every headline number, with the claim it supports and the caveat it carries:

If you remember one thing: this benchmark's contribution is treating when as a correctness criterion, with both an early bound and a late bound. Speed without an early bound rewards guessing; that single asymmetry is what makes the tool-call numbers meaningful.

Chapter 10: What It Cost, and What Is Missing

A lesson that ends at the results table has taught you to be impressed. This chapter is where you learn to be useful.

DuplexSLA makes a genuine architectural contribution and it is not free. Some of the costs are stated in the paper, some are visible in its numbers, and some are absences you have to notice yourself. All three kinds are below.

A note on tone before starting. Everything below applies to a paper whose central results this lesson has spent ten chapters taking seriously. Criticism at this level of detail is a compliment: vague papers cannot be criticized precisely, because there is nothing specific enough to disagree with. Every item here is possible to state only because the paper stated its own design clearly enough to be checked.

Cost 1: the accuracy trade

Put the numbers next to each other with no rounding and no framing:

Whether that is a good trade depends entirely on the product. In a car, where a two-second gap is the difference between a usable assistant and an abandoned one, giving up six points of accuracy for two seconds is obviously right. In a banking assistant that executes transfers, it obviously is not. There is no architecture-level answer.

Why is multi-action the weak point? The paper does not analyze it, so here are four mechanisms, ordered from most to least likely, each grounded in something we established earlier:

Hypothesis 2 deserves a moment. If per-call accuracy were the same as the single-action rate of 85.67%, the cubed value would be 62.9% — too low. If per-call accuracy is around 91%, the cube is 75.4%, essentially the observed number. So the data are consistent with "each individual call in a multi-action request is about as good as a single-action call, and the case-level score is just the compounding." That is a much less alarming story than "the model gets confused by multiple intents", and it is testable: report per-action accuracy alongside per-case accuracy. The paper does not.

Two crossing points worth finding by hand. On the tool-call average, the cascade's 5.77-point lead is worth 2.13 seconds, so the systems tie when one second costs 2.7 accuracy points. On multi-action, the 14.33-point lead is worth 4.03 seconds, so the tie is at 3.6 points per second. In other words: if you believe a second of voice latency is worth more than about three points of tool accuracy, DuplexSLA wins everywhere. Most voice-product intuition says a second is worth far more than that — but now you can argue about a number instead of a vibe.

One nuance on the table above before moving on: the "seconds saved per accuracy point lost" column is not a physical constant, it is a ratio between two things measured on this benchmark with this baseline. Swap in a faster cascade — better endpointing, a smaller planner — and the ratio moves. What does not move is the structural bound: the cascade cannot dispatch before the endpoint, so however fast its components get, the multi-action gap stays large. Optimize the constants and the shape of the trade survives.

Cost 2: the missing return channel

Introduced in Chapter 4, now counted properly. The action channel is emit-only, as described. Appendix E's action-object schema has four fields — name, planning, parameters, offset — and no result. Nothing in the paper describes how a tool's output re-enters the model.

Chapter 7's schema audit showed roughly twenty of the fifty functions are queries whose answers the assistant must speak: query_arrival_time, query_weather, query_stock, search_food, search_hotel, query_road_conditions. For all of these, the paper's contribution covers half the round trip.

The consequences, in increasing severity:

• Benign case: a control action (set_car_setting) always succeeds; the assistant can safely say "done" concurrently. The paper's showcase examples are all of this kind.
• Awkward case: a query action returns after some latency; the host application must inject the answer as new dialogue context, and the model must then work it into speech it is already producing. Nothing in the training data teaches that splice.
• Bad case: the action fails — navigation cannot find the destination, the media service is offline. The assistant has already said "I'm navigating there now." There is no described mechanism for retraction, and speech is not undoable.

One more consequence of the missing return path, easy to miss: it also means there is no latency for tool execution anywhere in the reported numbers. Every delay figure in Chapter 9 measures when the model spoke a call into the void. A real assistant's felt latency is dispatch plus execution plus, for query tools, the time to work the answer into speech. The paper's 0.64 s is a lower bound on an end-to-end quantity nobody has measured yet.

Cost 3: scope

The context-length point is worth its own arithmetic, because it is a real deployment concern that the paper does not raise. At 11 tokens per idle chunk and 6.25 chunks per second:

A duplex model burns context at a rate no text chat approaches, because it pays tokens for silence. Whatever context-management strategy makes a long drive work — sliding windows, summarization, state carry-over — is unaddressed here, and it interacts badly with the "context prefill" setting that Table 6 depends on.

Worth stating what is not a scope problem, since lists like the one above can leave a false impression. The 160 ms clock, the TA4 layout, the ordering argument, the FIFO queue, and the timestamp-for-free property are all language-independent and domain-independent. They would work identically in English, in a kitchen, or on a phone call. What is scoped is the evidence, not the mechanism.

Cost 4: the ablation that is not there

This is the most important gap for a careful reader, and it takes a moment to see.

The paper's central claim is that a dedicated action channel is the right design. The evidence offered is: DuplexSLA (which has one) beats a cascade and several commercial systems (which do not). But those systems differ from DuplexSLA in every other way too — backbone, training data, language, clock, objective.

The experiment that would isolate the claim is absent: train the same backbone, on the same data, with tool calls embedded in the assistant text channel instead of a separate lane, and measure the speech smoothness and tool accuracy that result. Section 2.6 asserts the outcome — "would force that channel to alternate between TA4 audio tokens and tool-call JSON, which breaks the smoothness of the assistant audio" — but asserts it without a number.

None of this makes the paper wrong. The architectural argument in Section 2.6 is a good argument, and it is the kind of thing that is genuinely expensive to ablate at 7B scale on 550k hours. But "obviously right" and "demonstrated" are different epistemic states, and a reader who conflates them will over-generalize the result to settings where the argument does not hold.

Cost 4b: the benchmark is the training distribution

A structural point that follows from Chapters 7 and 9 together, and that no single section of the paper states.

The post-training tool-call data has three families: single-action, multi-action, and backchannel-action. The benchmark's tool-call subset has three families: single-action, multi-action, and backchannel-action, 300 cases each. Both are built by the same team, presumably with the same annotation pipeline and the same 50-function schema.

This is entirely normal for a capability report and it is still worth naming. Matched training and evaluation taxonomies measure whether the capability was successfully installed. They do not measure whether it generalizes to a request pattern nobody enumerated, a function outside the schema, or an anchor annotated by a different process.

The cascade baseline is, ironically, less advantaged here: its LLM was not trained on this schema or these patterns, and it still scores higher on accuracy. That comparison is more favourable to the cascade than the headline framing suggests — and more impressive for DuplexSLA's latency claim, which no amount of distribution matching can fake.

Cost 5: acting on unfinished sentences

One more absence, and it is a safety-shaped one. The entire value proposition is acting before the user finishes speaking. The benchmark's only guard against acting too early is the legality rule: not more than 1.0 s ahead of the annotated offset.

But the annotated offset is defined as the moment the intent became clear, by an annotator who saw the whole sentence. At inference time, nobody knows the future. Consider:

• "Turn up the AC — actually no, open the window instead."
• "Navigate to the airport, wait, which terminal? Never mind, cancel that."
• "Don't play any music right now."

Each has an early fragment that reads as a clear intent and a later fragment that reverses it. A model rewarded for acting at the earliest defensible moment is a model biased toward acting on the fragment. The paper reports no results on corrections, negations, or retractions, and its 50 functions include no confirmation or undo primitives. Nothing in the design forbids adding them — but a system that turns your car's heat on because it heard "cold" in "I'm not cold" is a product problem, and the benchmark as constituted would not catch it.

The safety section this paper does not have

Technical reports on capability rarely include one, and this is not a criticism of the authors so much as a note about what a deployment review would have to add. A duplex action model raises questions that neither a speech model nor a text agent raises alone:

The fifth row is a small bright spot: the architecture happens to produce exactly the artifact an auditor would want. Every action ever taken has a name, arguments, a rationale in natural language, and a millisecond-resolution timestamp, emitted on a dedicated lane. Very few agent architectures give you that for free.

The fourth row is the darkest. A system whose selling point is acting on fragments is, definitionally, easier to trigger with fragments. The mitigations all cost the latency the system exists to save, which makes it a genuine design tension rather than an oversight to patch.

Reading the paper's own hedges

A useful exercise on any technical report: line up how the same result is phrased in the abstract, the body, and the summary. Careful authors hedge in one place and relax in another, and the difference tells you where the evidence is thin.

None of this is misconduct — the abstract is the careful version in every case, which is the right way round. But if you are going to cite one sentence of this paper in a design document, cite the abstract's.

What a version 2 of the benchmark should contain

Given the failure taxonomy below and the gaps above, a concrete wish list, ordered by how much each would change our understanding:

1. Correction and negation cases. "Turn up the AC — no, open the window." Anchor on the final intent; score an early call to the superseded function as a failure. This directly tests the cost of incremental commitment.
2. Per-action scoring alongside per-case. Resolves whether multi-action's 75% is compounding or confusion — a reporting change, not a new experiment.
3. Query tools with returns. Score the round trip: dispatch time, result arrival, and whether the spoken answer is correct and timely.
4. Failure injection. Make 10% of calls fail and measure what the assistant says next.
5. Natural audio. A subset recorded rather than synthesized, even at the cost of noisier anchors.
6. Long sessions. Ten minutes of continuous duplex, measuring whether timing degrades as context grows.
7. A second language. Ideally one with different backchannel conventions, to test whether the labels are learned or memorized.

A failure taxonomy for duplex action models

One reason this architecture is worth studying is that it creates error categories that do not exist in text agents. Naming them is useful whether or not you ever build one:

Five of the nine are unmeasured. That is not an indictment — it is a map of what a follow-up evaluation suite should cover, and several of them (revision, orphaned claims) are the ones a real user would notice first.

A fair reading of that taxonomy: the four failures the benchmark measures are the four a model can be blamed for, and the five it does not measure are the ones a system can be blamed for. That split is exactly the boundary between a foundation-model report and a product evaluation, and it is worth knowing which document you are holding.

Ten questions to ask before deploying something like this

1. What is my accelerator's per-token latency at the concurrency I actually need, at the tail, not the mean?
2. What happens to the conversation when RTF exceeds 1 for two seconds — is there a degradation path, or does it just fall behind forever?
3. Which of my tools are irreversible, and what is the confirmation flow for those?
4. How does a tool result get back into the model, and what does the assistant say while it waits?
5. What does the assistant do when a call it already announced fails?
6. How do I handle "no wait, cancel that" — is there a retraction primitive in the schema?
7. What is my context strategy for a 30-minute session at ~69 tokens per second?
8. How do I audit what was dispatched and when — and is the action lane logged with its chunk indices?
9. What is the fallback when the model's output is malformed on the action lane?
10. In which language and acoustic conditions has any of this been validated?

Question 3 deserves emphasis. The paper's fifty functions are almost all recoverable: a wrong AC temperature is fixed by saying so. Add make_call to a stranger, a purchase, or a message send, and "act at the earliest defensible moment" becomes a different risk profile entirely. The architecture makes early action possible; it does not tell you which actions deserve it.

What would change the picture

Concretely, the experiments that would most raise or lower confidence in this design:

Chapter 10 in review

The five costs, with a one-line statement of each and its severity for a real deployment:

A closing observation about how to hold all of this. Nothing above argues that DuplexSLA is a weak result — the latency numbers are large, clean, and hard to explain away, and the backchannel result is categorical. What the chapter argues is that the scope of the demonstration is narrower than the framing suggests, in five specific and individually fixable ways. That is the normal condition of a good systems paper, and being able to state the five is what turns reading into engineering judgement.

If you remember one thing: this paper demonstrates that timing is architectural. It does not demonstrate that the specific channel design is uniquely right, that the result transfers outside Chinese in-cabin assistants trained on synthetic audio, or that acting on half-finished sentences is safe. Those three are the next three papers, and knowing which is which is what separates reading a result from using one.

Three exercises:

1. Compute the crossing point for the single-action pattern: at what cost-per-second does DuplexSLA overtake the cascade? (Answer: 3.66 points over 1.66 s, so 2.2 points per second.) Compare with the multi-action crossing and explain why they differ.
2. Take three functions from Appendix C that are irreversible or externally visible, and design a confirmation flow that does not destroy the latency advantage. Where does the confirmation live — the action lane, the voice, or both?
3. Write the abstract of the follow-up paper that would resolve cost 4. What is trained, what is measured, and what is the headline number if the design is right?

Chapter 11: Voice as the Interface

Step back far enough and this paper is the last move in a sequence that took about four years.

Machines learned to hear meaning when contrastive language-audio pretraining put sound and text in one embedding space, so that a model could recognize a sound it had never been given a label for. They learned to transcribe anything when weak supervision at the scale of hundreds of thousands of hours made robust speech recognition into infrastructure. They learned to treat audio as language when neural codecs turned waveforms into discrete tokens and language models started generating them. They learned to converse when dual-stream duplex models put both voices on one clock and made overlap representable.

And now, with DuplexSLA, they learn to act — on the same clock as the voice.

Each row of that table is a lesson in this series or a chapter in one, and the column that matters is the third. Progress in this field has not been a march toward one goal; it has been a sequence of specific impossibilities being removed, one at a time, each by a system that assumed everything before it.

Why "voice as interface" is a real claim and not a slogan

Every interface has a cost of use, and the cost is not measured in features. It is measured in what the interface demands of your attention, your hands, and your eyes.

Notice the second row. Turn-based voice assistants trained us. People genuinely learned to speak differently to them — short, complete, unhesitating command sentences, delivered in one breath. That is the tell of a bad interface: when the human adapts to the machine's model of conversation rather than the reverse.

The reason DuplexSLA matters beyond its benchmark numbers is that it removes the two adaptations we were forced into. You may hesitate, because pause is handled. You may say two things at once, because multi-action is handled. You may add a side request in the middle, because backchannel-action is handled. And the assistant may act while it talks, because the action lane exists.

The interface thesis, tested against history

Claims that a new interface has arrived are cheap. Test this one against the transitions that actually stuck, and ask what each of them had in common:

The common thread in the first three is not capability — each of those systems could already do the task. It is the collapse of the gap between intent and effect. That is precisely the quantity DuplexSLA reduces from 2.77 seconds to 0.64, and it is why the latency result matters more than the accuracy result even though the accuracy result is the one that got worse.

The honest counter-argument, which you should hold alongside it: none of the three historical transitions required the interface to guess what you meant before you finished saying it. Direct manipulation is unambiguous by construction; a finger on a button is not a prediction. Acting on a partial utterance is, and Chapter 10's fifth cost is the price of that difference. Whether the bet pays off may depend less on latency numbers than on whether early action can be made safe.

The cheat sheet

Everything worth remembering, on one screen.

The clock and the channels

The serialization

memorize this
<|user_audio_begin|>      U U        <|user_audio_end|>
<|assistant_audio_begin|> T A A A A  <|assistant_audio_end|>
<action text>                         <|action_end|>

listen → speak → act, all inside one autoregressive step
the terminator fires every chunk whether or not anything was emitted

The rules

The labels

The numbers

Three predictions worth writing down

Since this lesson ends at the frontier, a small act of forecasting — each falsifiable, each following from something established above:

1. The return channel arrives before anything else. Twenty of fifty schemas need it, and the design change is small: results injected on the action lane's input side, supervised like everything else. This is the cheapest large improvement available.
2. The action-token budget rises rather than the chunk shrinking. Chapter 1's table shows shrinking the chunk costs action bandwidth per second because fixed framing overhead grows. Faster decoding buys budget; it does not buy a smaller clock without a penalty. Expect richer planning before expect lower quantization latency.
3. Turn-taking labels multiply. Three is a starting vocabulary, and the five criteria above admit several more candidates that cost only a few tokens each on a lane with spare capacity.

If all three land, the resulting system is recognizably this architecture with a wider lane and a return path. If none do — if the field instead abandons discrete audio tokens, or collapses the lanes again — then the framework in this lesson was the local optimum of one representational choice, and the interesting lesson will be why it did not hold.

If you are building one

A condensed checklist, assembled from every chapter:

1. Measure your decoder's per-token latency first. It sets your chunk size and your action budget, and both shape the data you must build.
2. Pick the chunk as the least common window of your codec frame rate and encoder stride. Do not pick a round number.
3. Make the user side continuous and causal. Tokenizing the input throws away the prosody your turn-taking decisions depend on; lookahead is latency in disguise.
4. Give the model a dedicated action lane and decode it last in the chunk, so the audio is always paid for first.
5. Terminate every chunk unconditionally. The heartbeat is what keeps token position equal to wall-clock time.
6. Build dual-side ASR data before anything interesting. This is the timing prior. Without it you have a model that acts at the wrong moments and you will not know why.
7. Reweight the loss for structural tokens — silence, markers, prompts — before your first long run. Compute the fractions for your own format.
8. Stabilize the serialization before teaching behaviours. Watch speech smoothness as the canary.
9. Annotate trigger offsets inside utterances, not at turn boundaries. Position in the sentence is what becomes position in time.
10. Score timing as correctness, with an early bound as well as a late one. A benchmark that only measures what was called will reward speculation.
11. Then close the loop the paper leaves open: a result path, failure handling, and confirmation for irreversible actions.

Test the cheat sheet the right way round: cover the values column and reconstruct each number from the design. Four audio tokens at 40 ms forces 25 tokens per second. Five TA4 tokens plus ten action tokens at 160 ms forces 93.75 tokens per second. A 160 ms clock forces an 80 ms expected quantization floor. Numbers you can re-derive are numbers you actually understand; numbers you memorized will be wrong within a year anyway, when the next paper changes the constants.

Glossary

Every term this lesson introduced, in one place, for the whiteboard test:

Explaining this paper at three lengths

A test of understanding: can you compress it without lying? Here are three versions to compare against your own.

Sixty seconds. Voice assistants can talk but not act — tool calls either delay the speech, arrive a turn late, or break the voice. DuplexSLA puts the assistant's audio and a separate text-only "action" lane on the same 160 millisecond clock, decoded by one backbone in one step. Tool calls and turn-taking decisions ride the action lane with a ten-token-per-chunk budget, so the model can call a function mid-sentence without the voice pausing. Result: sub-second tool dispatch against a cascade's two to five seconds, and the only system that can express "I heard you and I'm continuing."

Five minutes. Add: the chunk is 160 ms because four 40 ms audio tokens and two 80 ms user features both tile it. Each chunk carries user features, a TA4 unit (one text anchor, four audio tokens, always paid), and up to ten action tokens, in that order — so the voice is committed before any action tokens are written and can never starve. Longer actions spill across chunks under a FIFO queue with atomic JSON blocks, and the trigger time is the first token, not the last. Training is two stages from a 7B Step-Audio 2 mini: 500k hours of continued pretraining, a third of which is dual-side ASR that teaches the model what time it is, then 50k hours of capability post-training for pause, interrupt, backchannel, and three tool-call patterns. Evaluated on a purpose-built 2,100-case benchmark that scores when as well as what.

On a whiteboard. Draw the clock. Draw three lanes. Write the serialization in order and circle <|action_end|>. Draw a 40-token action object spilling across four chunks while the TA4 lane keeps ticking. Write two rows of numbers: 0.64 versus 2.77, and 98.33 versus 40.00. Then say the one sentence everything reduces to: the chunk index is the timestamp, and that is what makes "synchronized" a measurable claim.

A note on the compression exercise above. Each version drops something real: the sixty-second version omits the training recipe entirely, and the five-minute version omits the honest limits. Neither is dishonest, but both are incomplete in a way worth being conscious of — the parts that get cut first are always the data section and the caveats, which are also the two parts that determine whether a result transfers. If you find yourself giving the sixty-second version to someone who is about to build something, give them the five-minute version instead, and then Chapter 10.

Five open problems

1. The return channel. How do tool results, latencies, and failures re-enter a model running at 6.25 Hz — and what does the assistant say in the meantime?
2. Revision. Acting early means acting on fragments. What is the primitive for "I already called that; cancel it", and how is it supervised?
3. Budget versus thought. Ten tokens per chunk caps how much planning can accompany an action. Richer reasoning on the action lane runs straight into Chapter 1's arithmetic — is the answer a faster decoder, a smaller clock, or a compressed planning representation?
4. Generalization beyond the cabin. Fifty flat local functions become thousands of remote ones with latency, authentication, and failure. Which parts of this design survive?
5. Turn-taking beyond three labels. Yielding cues, floor competition, repair, invited feedback. The lane has room; the label set does not yet.

Where to go from here

The paper's own forward look, from its conclusion: "We view DuplexSLA as a step toward duplex spoken agents that combine fluent speech with timely action, and we expect the action-channel design to extend naturally to richer planning signals, multi-turn agentic workflows, and broader open-domain spoken tool use."

Three phrases, three research programmes. Richer planning signals means the lane carries more than a terse rationale — which runs straight into the 10-token budget and Chapter 1's arithmetic. Multi-turn agentic workflows means the return channel, dependencies between calls, and state that survives across turns. Broader open-domain spoken tool use means leaving the cabin, where latency is measured in seconds rather than milliseconds and where a wrong call costs more than a cold seat.

A reading list, in order

If this lesson made you want to go deeper, the papers in the order that builds the least confusion:

And a note on where this lesson sits in the audio series. It is the terminus of a long arc — hearing, transcribing, tokenizing, conversing, acting — but a terminus only in the sense that a station is: the line continues. The next stop is not a better speech model. It is the engineering around one: the latency ledger of a production voice agent, endpointing and barge-in as product decisions, telephony and transport, evaluation suites that measure conversations rather than utterances, and the safety machinery that acting-while-listening demands.

One last thought to leave with. The most striking thing about this paper is not any single number — it is that "when" turned out to be an architectural property. Not a scheduling problem, not a post-processing problem, not something you fix with a faster VAD. You get correct timing by giving the model a clock, putting every decision on it, and supervising the alignment for ninety thousand hours. Everything else — the labels, the queue, the budget — is bookkeeping around that one idea.

The whole lesson on one page

Twelve chapters, twelve sentences. If you can expand each of these into a paragraph, you have the paper.