The voice agent stack: every millisecond between a mouth and an ear, itemized.

01 The demo works. The deployment does not.

Every voice agent team lives the same two weeks: a prototype that feels magical on Tuesday and a pilot that gets escalated to the VP on the following Thursday.

The prototype is real. Someone wires a streaming speech recognizer to a fast model to a good text-to-speech voice, opens a browser tab, speaks into a laptop microphone in a quiet room, and the thing answers. It answers well. The voice is warm. The reasoning is correct. It calls a function and reads back an appointment slot. Everybody in the room feels the future arriving.

Then it goes onto a phone number. And the following things happen, in roughly this order, over the following two weeks:

• The agent starts talking while the caller is still mid-sentence, because the caller paused to remember an order number and the endpoint detector called it a turn boundary.
• The caller tries to interrupt the agent's long-winded answer. The agent keeps talking for another second and a half, then stops mid-word, then answers a question the caller had abandoned.
• A caller with a Glaswegian accent gets a transcript that is confidently wrong, and the agent politely confirms an appointment for the wrong day.
• A tool call to the scheduling API takes 2.1 seconds. The line goes completely silent. The caller says "hello?" The agent, which was waiting on the API, now has "hello" in its input queue and answers it.
• Somebody calls from a car. The road noise keeps the voice activity detector permanently in the "speaking" state and the agent never gets a turn at all.
• The whole thing sounds slightly worse than it did in the browser, because a phone call is eight kilohertz and the recognizer was trained at sixteen.

Not one of those failures is a model failure. Every single one is a timing or transport failure, and every single one is invisible in the demo because a demo is one person, in a quiet room, on a good network, asking a question they prepared, and not interrupting.

01.1 The conversational floor: where two hundred milliseconds comes from

The budget every design is scored against is not an engineering target. It is a measurement of human beings.

Conversation analysts have measured turn-taking gaps across languages for decades, and the result is remarkably stable: the modal gap between one speaker finishing and the next starting is close to zero, and the mean sits around two hundred milliseconds. The Moshi paper cites a figure of 230 ms as the average response time in natural conversation and designs its architecture explicitly to get under it. The Softcery survey uses a round 200 ms for the same quantity.

Two hundred milliseconds is a shockingly small number, and it is worth understanding why it is possible at all, because the reason has direct engineering consequences.

Producing a spoken word takes a human roughly 600 ms of planning. If a listener waited until the speaker finished before beginning to plan a reply, the minimum gap would be around 600 ms, not 200. The fact that real gaps are shorter proves something specific: humans predict the end of your turn and start planning their reply before you have finished speaking. They use syntax, prosody, and semantics to project the turn boundary, and they launch their own production process early enough that the words are ready when the gap arrives.

Now, an important piece of honesty: no deployed voice agent hits 200 ms, and most products do not need to. Callers tolerate longer gaps from a machine than from a person, particularly if the gap is filled or the agent's speech is expressive. The Twilio guide's launch target for a straightforward cascaded agent is a mouth-to-ear turn gap of 1,115 ms with an upper limit of 1,400 ms — five to seven times the human figure — and agents at that latency are shipping and working.

So why does the human number matter? Because it tells you what you are trading against. Every millisecond you spend is a millisecond further from the conversational feel that makes the product worth building, and because the tolerance is not linear. Somewhere around a second, the caller stops experiencing a conversation and starts experiencing a system that is processing their request. Both can be acceptable products. They are not the same product.

01.2 Silence is a paralinguistic signal, not an absence

The reason latency is not merely annoying: people read meaning into gaps, and your system is emitting meaning it did not intend.

The Twilio guide makes an observation that is easy to skim past and worth stopping on: latency is critical to voice agents because silence functions as a paralinguistic signal. People attribute meaning to the sound of silence, and the rhythm of turn-taking shapes how the words are interpreted.

Consider what a 1,200 ms gap means when a human produces it. Ask a friend "can you cover my shift Saturday?" and count the silence before they answer. Zero to 200 ms reads as yes. Around a second reads as reluctance. Two seconds reads as a no that is being softened. The words that follow are interpreted through the gap.

Your voice agent is emitting these signals constantly and accidentally. A 1.4-second pause before "yes, that slot is available" is heard as hesitation about the slot. A pause that lands after the caller asks about a refund is heard as evasiveness. This is why latency is not merely a performance metric to be optimized when there is time: the latency profile of your agent is part of what it appears to be saying.

01.3 Six conditions a demo never sees

Each of these has a chapter later. Here is the inventory, and what each one costs.

Notice the shape of this list. Only one entry — accents — is about model quality. The other five are about time, and specifically about what happens to work that is already in flight when the world changes. That is a distributed-systems problem wearing a speech-technology costume, and teams staffed entirely with machine-learning engineers are systematically surprised by it.

01.4 Widget: run the gauntlet

The same single conversational turn, sent through the demo conditions and then through the production conditions.

Below, one turn travels left to right through six gates. In demo mode every gate is in its friendly configuration: 16 kHz web audio, a quiet room, no pause, no interruption, a cached tool, a native-accent speaker. Press Production and the same turn runs through the gates in their realistic configuration.

Watch where the turn stops. The failure is annotated with the specific mechanism, and the counter tallies how many turns out of a hundred survive each gate under the current setting. The survival numbers are illustrative rather than measured — they are there to make the compounding visible, because the compounding is the actual lesson: six gates at 95% each is 74% end to end.

The compounding is the point, and it is worth writing down explicitly because it explains a phenomenon every voice team observes and few can articulate: why the pilot feels so much worse than the demo even though every individual component tests fine.

01.5 Why these six and not others

A short argument about which failures are structural and which are just bugs.

There are hundreds of ways a voice deployment can go wrong. The six above are singled out because each of them is structural: it follows from the architecture rather than from a defect, so no amount of careful coding removes it. You can only trade it against something else.

Take endpointing. There is no implementation of a turn detector that never cuts a caller off and never adds dead air, because the information required to distinguish "pausing to think" from "finished speaking" is sometimes genuinely not present in the audio at the moment you must decide. This is not an engineering gap; it is an information-theoretic one. You are forced to pick an operating point on a tradeoff curve, and Chapter 06 is about picking it well.

Or take barge-in. The caller's interruption and your outgoing audio are two events separated by network transit in both directions. Between the moment the caller's first syllable leaves their mouth and the moment your playback actually stops, some amount of audio necessarily plays. You can shrink that window — the Twilio guide's advice to colocate the recognizer, the synthesizer and the audio buffer is exactly about shrinking it — but you cannot make it zero, because information does not travel faster than the network.

One last observation before we go to the architectures. Every one of the six conditions is worse on a phone line than in a browser, and the enterprise use cases — support, scheduling, intake, collections — are overwhelmingly phone. Meanwhile the impressive demos are overwhelmingly browser-based, because the browser gives you 48 kHz audio, hardware echo cancellation and a WebRTC stack that someone else debugged. The gap between the demo and the deployment is, to a first approximation, the gap between those two channels. Chapter 09 is about what actually lives in that gap.

01.6 Concept check

Next chapter: the showcase. One conversation, two architectures, running side by side with the clock visible on both.

02 One conversation, two machines, one clock

The showcase of this lesson. The same eleven words go into two architectures and we watch every stage of both.

The caller says: "Hi, can I move my Thursday appointment to next week?"

That is eleven words, about 2.8 seconds of speech. What happens next depends entirely on which of two machines is listening, and the difference is not a matter of degree. The two architectures do not do the same thing faster or slower; they represent the caller's utterance in fundamentally different objects, and everything downstream — what you can log, what you can filter, what you can swap, what you can hear — follows from that representational choice.

Before the animation, both machines get described precisely, with the actual data at every boundary. Architecture diagrams without shapes are decoration; the shapes are where the engineering lives.

02.1 The cascade, with every shape written down

Three models, two text boundaries, ten network traversals, one sentence buffer holding the whole thing together.

Here is the caller's utterance as it actually moves through a streaming cascaded pipeline, using the concrete configuration from the Salesforce tutorial — Deepgram for recognition, a vLLM-served model for reasoning, ElevenLabs for synthesis.

1. Capture: microphone to PCM frames

   The browser's AudioWorklet captures at the device's native rate and buffers into 20 ms chunks. At 16 kHz mono with 16-bit samples that is 16000 × 0.020 × 2 = 640 bytes per chunk, sent as a binary WebSocket frame. Fifty frames per second, forever, in both directions.

   20 ms is the universal audio frame quantum: small enough to be inaudible, large enough that per-packet overhead does not dominate

2. Streaming recognition: PCM to partial transcripts

   The recognizer holds a persistent WebSocket and emits two kinds of result. Partial transcripts (is_final=false) update as more audio arrives and are useful for on-screen feedback but must never be sent to the language model, because they get revised. Final transcripts (is_final=true) are stable. A third signal, speech_final=true, is the recognizer's opinion that the caller has stopped.

   The partial-final split is the first place teams lose a hundred milliseconds by waiting for finals they did not need

3. The turn decision

   Something must convert "the recognizer thinks there was silence" into "the caller is done, start generating." That is the endpointer, and it is the single largest controllable line in the ledger. Chapter 06 is entirely about it.

   This stage has no model of its own in naive builds — it is a timer, and the timer is usually set at 500 ms

4. Language model: string in, token stream out

   The conversation history plus the tool definitions go to an OpenAI-compatible endpoint with stream=true. Tokens come back one at a time over server-sent events. The number that matters is time-to-first-token; everything after the first token is hidden behind speech that is already playing.

   Measured on Qwen2.5-7B-Instruct via vLLM 0.8.5: TTFT 337 ms P50, 17.5 tokens/s, 34 ms between tokens

5. The sentence buffer: the critical bridge

   Tokens cannot be sent to the synthesizer one at a time — a text-to-speech model needs a phrase to produce sensible prosody. So a buffer accumulates tokens and flushes on sentence boundaries: terminal punctuation followed by whitespace, excluding abbreviations and decimals, with a minimum length so fragments never go out, and a final flush when the stream ends.

   This is Pipecat's SentenceAggregator and LiveKit's text pipeline under different names; the Salesforce build measures the detection itself at 143 ms

6. Streaming synthesis: text to audio chunks

   The first sentence goes to the synthesizer, which streams audio back as it generates. Time-to-first-byte 219–236 ms P50 in the Salesforce measurements; real-time factor 0.05–0.10, meaning it generates ten to twenty seconds of speech per second of compute. The second sentence is synthesized while the first is still playing.

   Because RTF is far below 1, synthesis never becomes the bottleneck after the first chunk — only the first chunk matters

7. Playout: audio chunks to the caller's ear

   Server-to-client frames at 24 kHz go into a queue-based worklet that plays them smoothly, resampling as needed. This buffer is a jitter buffer: it trades a few tens of milliseconds of latency for immunity to network variance, and Chapter 09 shows the exact exchange rate.

   Every buffer in the chain is a latency purchase, and most of them were bought by default settings you did not choose

Two properties of this pipeline deserve emphasis because they are the source of nearly every misconception about cascaded latency.

First, the stages overlap. The naive mental model — recognize, then think, then speak — gives a turn-based total. The Salesforce tutorial writes it out: 400 + 800 + 400 = 1,600 ms if you wait for each stage to finish. Streaming changes the sum to recognition plus first sentence of generation plus first byte of synthesis: 400 + 300 + 200 = 900 ms. Same components, same models, 700 ms saved by never waiting for a stage to complete.

Second, the boundaries are strings. Between the recognizer and the model there is a UTF-8 string. Between the model and the synthesizer there is another. Both are inspectable, loggable, rewritable, and filterable by anything you care to run. Chapter 04 argues that this is the single most commercially important property of the architecture, and Chapter 05 argues about what it costs.

02.2 End-to-end speech-to-speech, with every shape written down

One model. Audio tokens in, audio tokens out, and no string anywhere on the critical path.

Now the same utterance through a native speech-to-speech model, using Moshi as the concrete reference because it is the one with a published architecture and a released serving stack.

1. Capture: identical

   The same 20 ms PCM frames. Nothing at the microphone knows or cares which architecture is downstream.

   The divergence begins one layer up, not at the device

2. Neural codec: waveform to discrete tokens

   A streaming neural codec turns the waveform into a small grid of integers. Moshi's Mimi runs at 12.5 frames per second with a residual stack of codebooks, and its first codebook is distilled to carry semantic content while the rest carry acoustic detail. So one second of speech becomes roughly 12.5 frames × K codebooks integers rather than 16,000 samples.

   12.5 Hz is chosen so an audio frame costs about the same as a text token — it makes speech affordable inside a transformer

3. The model consumes both streams at once

   The defining property of a duplex model: it models the caller's stream and its own stream simultaneously, at every frame. There is no state in which it is "not listening." It is also predicting its own next audio token while the caller talks, which is what makes sub-200 ms response possible — the reply is already forming.

   This is the architectural version of human turn projection from Chapter 01

4. Inner monologue: text as a scaffold, not a boundary

   Moshi predicts text tokens time-aligned with its own speech, ahead of the audio. This improves the linguistic quality of what it says, and it produces a transcript — but the transcript is a by-product of generation, not an interface the audio has to pass through.

   Crucial distinction for Chapter 04: a transcript that exists is not the same as a seam you can intervene at

5. Decode: tokens back to waveform, streaming

   The codec decoder is streaming, so audio comes out frame by frame as tokens are produced. One frame is 80 ms at 12.5 Hz, which sets the floor: Moshi reports a theoretical latency of 160 ms and about 200 ms in practice.

   The floor is a property of the frame rate and the acoustic delay pattern, not of the GPU

Count what is missing from that list compared with the cascade: no endpoint decision, no sentence buffer, no inter-service network hops, no text-to-speech first-byte latency, no recognizer finalization delay. Four of the six largest line items in the cascaded ledger simply do not exist, which is why the architecture's floor is so much lower.

02.3 The half-cascade, which is what most "realtime" products actually are

Native audio in, text reasoning in the middle, synthesis out. Neither pure architecture, and commercially the most common one.

The clean two-way split is pedagogically useful and empirically wrong. The Softcery survey separates three species, and the middle one is where most shipping products live:

The half-cascade is a genuinely interesting engineering compromise: it recovers the input-side paralinguistics (the model hears the sarcasm) while keeping text reasoning (the model can be a good agent, and its plan is inspectable). What it gives up is output-side expressivity, because the voice is produced by a synthesizer conditioned on text, and the Softcery survey notes bluntly that integrated synthesis in these products is generally less natural than a specialized text-to-speech model.

The Salesforce tutorial's taxonomy is the same idea sorted by how natively speech is processed, and it adds a sharp practical observation. Until recently, no Level 1 or Level 2 model supported function calling at all — which for an enterprise voice agent is not a limitation but a disqualification. An agent that cannot check the schedule cannot book the appointment. Qwen3‑Omni is called out as the first speech-native model with tool calling via XML tags, and that single capability is what moved the architecture from "research demo" to "candidate."

02.4 Showcase: the race, with the clock visible

Both architectures, the same utterance, the same wall clock. Change the deployment preset and watch which one wins — the answer flips.

The widget below runs both pipelines against the same 2.8-second utterance. The top track is the cascade; each stage is a labelled bar with its own duration, and overlapping stages are drawn overlapping because they genuinely are. The bottom track is the speech-to-speech model. The vertical playhead is the shared clock, and the two big counters are the running time-to-first-audio for each architecture.

The preset selector is where the real teaching happens. It swaps in the measured numbers from different sources and deployments — and the winner changes. That is not a bug in the widget; it is the actual state of the field in 2026.

Three things to try, in order.

One. Turn streaming overlap off on the first preset. The cascade balloons from roughly three quarters of a second to over one and a half, because now each stage waits for the previous one to finish. This single toggle is the difference between the two Salesforce equations — 1,600 ms turn-based versus 900 ms streamed — and it is worth more than any model swap on the menu.

Two. Switch to the telephony preset. Both architectures get slower, but the end-to-end model gets slower for a reason that has nothing to do with the model: the public network leg and the codec transcode are charged to both, and they are a larger fraction of the smaller total.

Three. Switch to the research floor. Now the duplex model is at 200 ms and no amount of cascade tuning gets within three times of it. This is the honest picture: when a duplex model is running locally with no network in the path, the cascade is not in the same league.

02.5 Reading the race honestly

Four sources, four different claims about which architecture is faster, and all four are defensible.

Here is where the field actually stands, with each claim attributed:

There is one asymmetry the sources agree on completely, and it decides more architecture choices than latency does: self-hosting. The Salesforce evaluation is the cleanest evidence in the literature. Qwen3‑Omni's optimized talker exists only behind a cloud API; served locally with vLLM you can run the thinker (audio to text, 516 ms) but not the talker; run the full model on the standard transformers backend and time-to-first-audio is about 146 seconds. Not milliseconds. If your requirement is "runs entirely on our own GPUs," the end-to-end option is currently not available at any latency, and that is why the tutorial's conclusion is what it is.

02.6 Concept check

Next chapter: the ledger. Twelve line items, summed by hand, then turned into a calculator — and the moment where two published numbers that look contradictory turn out to be measuring different intervals.

03 Twelve lines, one number, no mysteries

If you cannot itemize your turn gap, you cannot optimize it, and you will spend three weeks swapping models to recover forty milliseconds that were sitting in a jitter buffer.

Every conversation about voice latency that goes badly goes badly for the same reason: the participants are quoting single numbers without saying what interval those numbers cover. The fix is boring and complete. Write down every line item between the caller's last syllable and the first sample of your reply reaching their ear, put a number on each, and add them up.

Twelve line items. Most are not the model. Several are not even software.

03.1 The line items

What each one is, who controls it, and how big it typically is.

Look at the shape of the ledger before we sum it. Lines 05, 07 and 09 — the three models — total about 825 ms of the roughly 1,100. Lines 01, 02, 03, 06, 10, 11 and 12 — the plumbing — total about 270 ms. And line 04, the endpoint decision, can be anywhere from zero to 700 depending on one configuration value.

That last point deserves its own sentence. A single timer setting can be the largest line in your ledger. A team that spends a quarter migrating to a faster inference provider to save 100 ms of time-to-first-token, while running a 700 ms fixed silence threshold, has optimized the wrong thing by a factor of seven.

03.2 Hand-worked: summing the ledger, one line at a time

No shortcuts. Every partial sum written out, then two cross-checks against published totals.

The Twilio guide gives a figure for each stage and a total of roughly 1.1 seconds, but it does not show the addition. We will do the addition, because the residual is informative.

Now the second, more interesting piece of arithmetic. Twilio publishes two totals: mouth-to-ear 1,115 ms and platform 885 ms. The difference is defined to be everything outside the platform — the audio's journey across the public internet or the phone network, in both directions.

This is what number discipline buys you. We now have a defensible model of where a second of latency goes, assembled from a vendor diagram, a vendor target table, and a consultancy's throwaway sentence — and the three agree to within tens of milliseconds once you are careful about what each is measuring.

03.3 Three numbers for one quantity

Reconciling the 755 ms that Salesforce measured with the 1,115 ms that Twilio targets. Neither is wrong.

The Salesforce tutorial reports a measured time-to-first-audio of 755 ms, with a best case of 729. Twilio targets 1,115. These describe the same product category and differ by nearly 50%. Here is the reconciliation, and it is entirely mechanical.

Read the Salesforce pipeline test line by line. It lists model time-to-first-token 296 ms, sentence detection 143 ms, synthesis 316 ms, measured time-to-first-audio 755 ms. Add the three components: $296 + 143 + 316 = 755$. Exactly. The reported total is the sum of those three stages and nothing else.

Keep this table when you read anybody's latency claim:

03.4 The overlap formula, derived

Why streaming changes a sum of durations into a sum of first-response times, and what that is worth.

In a turn-based pipeline, every stage runs to completion before the next begins, so the total is the sum of full durations:

With the Salesforce illustrative values, $400 + 800 + 400 = 1600$ ms. In a streaming pipeline each stage begins as soon as its input has enough content to act on, so what matters is not each stage's duration but its latency to first useful output:

With the same values, $400 + 300 + 200 = 900$ ms. The 700 ms saved comes from two places: the model's remaining tokens are generated while the first sentence is being spoken, and the remaining sentences are synthesized while earlier audio plays.

Twilio makes the same architectural recommendation from the product side: real-time dialogue is driven by a lightweight conversational model optimized for latency and speaking style, while deeper reasoning is handled asynchronously by background systems. The derivation above is why that pattern exists, expressed in words per second.

03.5 Widget: the budget calculator

Drag any line item and watch the total move against four reference marks. This is the whole chapter in one control panel.

The stacked bar is your turn gap, drawn to scale, segment by segment. The vertical rules are the marks that matter: the human conversational gap at 230 ms, the point around 800 ms where callers still experience a conversation, Twilio's launch target at 1,115 ms, and their upper limit at 1,400 ms beyond which the pilot gets escalated.

Presets load the measured configurations from the sources. Toggling telephony adds the public-network legs and the codec transcode. Toggling streaming off replaces the three first-response latencies with full durations, so you can see the 1,600 ms sequential world.

One reconciliation first, because the widget's default total is higher than the 1,095 ms we computed in 03.2 and that is deliberate. Twilio folds the endpoint wait into its 350 ms recognition figure and never itemizes the sentence buffer. This widget separates both, so a default configuration lands near 1,740 ms — which is what a naively-configured agent actually does, and is exactly the 1,738 ms the budget function below computes with its default arguments.

The experiment worth running before you read on: load the Twilio components preset, then drag the endpoint wait from 500 down to 200 and watch 300 ms disappear in one gesture — more than any other control on the panel can buy you. Then read Chapter 06 to find out what that 300 ms actually costs, because it is not free and the bill arrives in a different currency.

03.6 The ledger in code: three versions

Manual sequential, explicit streaming, and the one-liner that expresses the whole pipeline.

Version 1 — the naive pipeline. Correct, readable, and 1,600 ms per turn. Almost every first implementation looks like this, because it is what the SDK examples show.

async def handle_turn_naive(audio_frames):
    # 1. Wait for the WHOLE transcript.
    transcript = await asr.transcribe(audio_frames)      # ~400 ms

    # 2. Wait for the WHOLE response. This is the killer:
    #    we block on the last token before speaking the first word.
    reply = await llm.complete(history + [transcript])   # ~800 ms

    # 3. Wait for the WHOLE audio file.
    wav = await tts.synthesize(reply)                   # ~400 ms

    await playout.write(wav)
    # Total: 400 + 800 + 400 = 1600 ms before the caller hears anything.

Version 2 — explicit streaming with the sentence buffer written out. Every step named, including the one that everybody forgets: the flush at the end of the stream.

ABBREV = {"dr.", "mr.", "mrs.", "ms.", "st.", "no.", "p.m.", "a.m."}

def is_boundary(buf: str) -> bool:
    # Rule 1: terminal punctuation followed by whitespace.
    if not re.search(r"[.!?]\s$", buf):
        return False
    # Rule 2: reject abbreviations -- "Dr. Chen" is not two sentences.
    tail = buf.strip().split()[-1].lower()
    if tail in ABBREV:
        return False
    # Rule 3: reject decimals -- "4.5 minutes" is not a boundary.
    if re.search(r"\d\.\s$", buf):
        return False
    # Rule 4: minimum length, so "Ok." does not become its own TTS request.
    return len(buf.strip()) >= 10

async def sentences(token_stream):
    buf = ""
    async for tok in token_stream:
        if not tok:                     # guard: empty SSE chunks are real
            continue
        buf += tok
        if is_boundary(buf):
            yield buf.strip()
            buf = ""
    if buf.strip():                  # the forgotten flush: last sentence
        yield buf.strip()          # has no trailing punctuation + space

async def handle_turn_streaming(transcript, history):
    # The LLM streams. We do not await it; we iterate it.
    tokens = llm.stream(history + [transcript])       # first token ~337 ms

    async for sentence in sentences(tokens):        # +143 ms to close one
        # TTS streams too. First chunk leaves in ~220 ms;
        # the rest arrives while the caller is already listening.
        async for chunk in tts.stream(sentence):
            await playout.write(chunk)
            if barge_in.is_set():                    # Chapter 07
                await playout.flush(); return

Version 3 — the one-liner. Once the two generators exist, the entire streaming pipeline is a composition, and the whole architecture fits on one line:

await playout.pipe(tts.stream_all(sentences(llm.stream(history))))

And the ledger itself, which is the function you should actually write first, before any of the above, so that you have a target to measure against:

def turn_gap_ms(*, endpoint=500, asr=350, ttft=375, sent=143,
                ttfb=100, hop=10, hops=8, telephony=True):
    plumbing = (30 + 25) * 2 + hop * hops           # buffers, codecs, hops
    public   = 40 * 2 if telephony else 0            # the two legs you do not own
    return plumbing + public + endpoint + asr + ttft + sent + ttfb

turn_gap_ms()                       # 1738 ms -- a default-configured agent
turn_gap_ms(endpoint=200, ttfb=100)   # 1438 ms -- one timer changed
turn_gap_ms(endpoint=200, ttft=180,
            asr=200, telephony=False)  # 1013 ms -- browser, tuned stack

03.7 Concept check

Next chapter: the strongest argument for the architecture that loses on paper — what the text boundary is actually worth.

04 The lossy boundary is the product

Every critique of the cascade names the same flaw: it destroys information at the text boundary. Every enterprise buying decision names the same feature: it exposes a text boundary.

Here is a sentence that sounds like a joke and is not: the reason enterprises buy the architecturally inferior system is that its inferiority is load-bearing.

The cascade's defining property is that at two points per turn, the entire state of the conversation is a UTF-8 string. That string is lossy — Chapter 05 is a full accounting of what it loses — but it is also inspectable, storable, searchable, diffable, redactable, injectable, and testable with tooling that already exists and that your compliance team already understands.

Inworld, who sell a cascaded product and are therefore worth reading adversarially, put the argument in one line: because every boundary is text, you can intercept it — run moderation, inject retrieved context, call tools, log the exact words both sides said. And then the sharper claim: end-to-end models offer no equivalent seam.

04.1 What actually lives in the seam

Six things enterprises put between hearing and speaking, and what each costs in milliseconds.

This list is the reason the architecture wins procurement, so it is worth being specific rather than gesturing at "compliance."

Notice the asymmetry between the input-side and output-side guardrails. Input filtering is a convenience: if you miss an abusive turn, the model probably handles it acceptably. Output filtering is a hard requirement in regulated domains, and it has a property that makes the architecture choice nearly forced: it must run after generation and before sound.

04.2 Widget: the interception seam

Switch interceptors on and watch two counters move in opposite directions: control coverage up, latency headroom down.

The widget draws one turn through both architectures. The cascade lane has six insertion points; toggle each interceptor and watch it appear in the lane, adding its cost to the turn gap. The end-to-end lane is drawn alongside with the same interceptors greyed out, because there is nowhere to put them.

The readout tracks three quantities: total turn gap, control coverage as a fraction of the six, and the remaining headroom before the 1,400 ms upper limit from Chapter 03. The design question this widget makes concrete is not "should we have guardrails" but "which guardrails fit in our budget, and what did we give up to afford them."

Press prefetch and watch the retrieval cost drop out of the critical path entirely. That is the general shape of every good latency fix in this chapter: not "remove the feature," but "move the feature off the turn." Retrieval that runs when the call connects, guardrail classifiers that run concurrently with the first sentence's synthesis, transcripts written asynchronously. Chapter 08 generalizes the trick to tool calls.

04.3 Debuggability: what a failed call looks like in each architecture

A concrete incident, traced twice.

Take a real-shaped incident. A caller reports that the agent confirmed the wrong appointment date. You have the call recording and whatever your platform logged. Here is the investigation in each architecture.

In a cascade

You open the turn log. You see the exact strings:

[14:02:11.204] asr.final    "move my thursday appointment to next week"
[14:02:11.560] tool.call    check_availability("2026-08-20")
[14:02:11.712] tool.result  {"slots": ["09:00", "14:30"]}
[14:02:12.030] llm.text     "I can move that to Tuesday the 20th at 9am."
[14:02:12.244] tts.spoken   "I can move that to Tuesday the 20th at 9am."

The bug is visible in four seconds: "next week" was resolved to the 20th, and the 20th is a Tuesday, but the caller's appointment was Thursday and they meant next Thursday. The defect is in date resolution, in the tool arguments, on the input side of the tool. You write a test with that exact transcript string, fix the resolver, and the test now guards the fix forever. Total cost: an afternoon.

In an end-to-end model

You have the audio. If the vendor emits a transcript you have an approximation of what was said. What you do not have is the intermediate decision — there is no place in the system where "the model resolved next week to the 20th" existed as an object. The model heard audio and produced audio, and the date arithmetic happened inside a forward pass.

Your options: prompt-engineer defensively and hope; add an explicit tool call for date resolution so at least that step becomes visible (which is you rebuilding a seam by hand, in the one place you could); or collect failing calls and fine-tune, which requires a data pipeline, a labelling process and a training budget to fix what was a four-line patch in the other architecture.

04.4 The economics, computed rather than asserted

Per-component pricing and bundled audio-token pricing behave differently as the call gets longer. Here is the arithmetic.

The cost story is where the sources are most useful, because they publish list prices with dates. Two structural facts drive everything:

• Cascade cost is roughly linear in call duration. You pay recognition per minute of audio, synthesis per character spoken, and the model per token — and with prompt caching the repeated system prompt collapses to roughly a tenth of its price. Softcery quotes a chained pipeline at approximately $0.15 per minute with no context accumulation.
• Some end-to-end pricing is super-linear. Where the model re-charges for accumulated context on every turn, cost per minute rises through the call. Softcery reports developer accounts of one realtime API at roughly $0.30 per minute baseline climbing past $1.50 per minute in long sessions, and a fifteen-minute session where audio input alone cost several dollars.

And the lever that only the cascade has: you can move one component. If the model is 60% of your cost, you swap in a smaller one for the routine 80% of turns and keep the expensive one for escalation, without touching the voice, the recognizer, or a single line of transport code. In a bundled architecture the price is a package, and the only lever is switching vendors.

04.5 The self-hosting argument, which currently ends the debate

For a large class of buyers the decision is made before any of the above, by a constraint that no amount of engineering removes.

Health, finance, government and defence buyers frequently have a hard requirement: the audio does not leave our perimeter. This is not preference. It is contractual, and sometimes statutory.

The Salesforce evaluation is the clearest evidence available on what that requirement does to the architecture menu. They took the strongest open-weights candidate and tried three deployments:

Their conclusion is blunt and, as of the paper's date, correct: until an optimized self-hostable talker serving solution exists, the cascaded pipeline remains the only viable architecture for fully self-hosted realtime voice agents.

Which is the last practical note of the chapter, and the one Inworld's checklist ends on: build the cascade behind a clean interface. Migrating cascade to end-to-end later means replacing a pipeline with a connection. Migrating the other way means rebuilding turn-taking, voices and observability from nothing. The asymmetry says: when uncertain, choose the architecture that is cheaper to leave.

04.6 Concept check

Next chapter: the other side of the same coin, argued just as hard — everything the string throws away, and the products that cannot be built without it.

05 Four different sentences that produce one identical string

The previous chapter argued that the text boundary is the product. This one argues that the text boundary is a hole. Both are true, which is why the field has two architectures instead of one.

A caller says four words: "I said Thursday."

Here are four situations in which a person produces exactly those four words:

• Flat, falling pitch, quiet. A neutral restatement. They are supplying information they assume you missed.
• Rising pitch on "Thursday," slight elongation. Uncertainty. They are checking their own memory, and they would welcome being corrected.
• Sharp stress on "said," clipped, faster, higher energy. Frustration. This is the second or third time, and the next thing they say may be "let me talk to a human."
• Slow, low, elongated, breathy. Resignation or fatigue. They have given up on being understood and are going through the motions.

The recognizer transcribes all four as i said thursday. Not approximately — identically, character for character. And that is not a defect in the recognizer; it is the recognizer's design goal. A recognizer is trained to be invariant to exactly the acoustic variation that distinguishes these four situations, because otherwise it would fail whenever somebody had a cold.

Now the consequence for the agent's behaviour, which is what makes this a product argument rather than a signal-processing observation. In variant three the correct action is to stop confirming details and offer a human. In variant two the correct action is to gently re-state the date so the caller can confirm. In variant one the correct action is to proceed. A text-only agent must respond identically to all four, because it received identical input. It will pick whichever behaviour its prompt favours, and it will be right about a quarter of the time.

05.1 Hand-worked: how much information does the transcript keep?

Not a metaphor. A rate comparison in bits per second, computed three ways, with every step shown.

We can put a number on "how much is thrown away," and it is worth doing because the number is more extreme than people expect. Three representations of the same one second of speech.

05.2 Widget: what survives the boundary

Pick a rendition, see its acoustic shape, then watch three architectures each report what they received.

The top panel draws the four renditions of "I said Thursday" — pitch contour in amber, energy envelope in moss, pauses marked. The panels below show what each architecture actually has available at the moment the reply is planned.

The cascade lane shows the transcript, which is identical in all four cases. The half-cascade lane shows the transcript plus retained acoustic features on the input side. The native audio lane shows the token stream, in which the four renditions are genuinely different sequences. Switch renditions and watch which lanes change and which do not.

Turn on correct agent action and the argument becomes uncomfortable. Four situations, four different right answers, and one of the three lanes has the information needed to tell them apart on the input side while a second has it on both sides. The cascade lane is choosing blind, and its accuracy ceiling on this decision is set by the prior, not by the model.

One more observation the widget is built to make. The four contours differ most in the places where the words are identical, which is the same structural fact that makes prosody hard to evaluate: the difference between an accusation and a question can live entirely in the last two hundred milliseconds of the final word.

05.3 The sidecar workaround, and the two reasons it disappoints

Every cascade team eventually builds this. It helps. It does not close the gap, and it is worth understanding exactly why.

The obvious fix: keep the cascade, and run a small audio classifier alongside the recognizer that emits a label — emotion=frustrated, confidence=0.71 — which you splice into the prompt. This is a real technique, it is cheap, and it genuinely improves behaviour on the frustration case. It also has two structural limitations that no amount of implementation quality removes.

Reason one: quantization to a label

The classifier converts a continuous, multidimensional, time-varying signal into one categorical token. Compare the rates. The pitch contour over 900 ms of speech is a trajectory; the label is roughly $\log_2 6 \approx 2.6$ bits for a six-way emotion taxonomy, delivered once per turn. You have replaced a signal with a summary statistic, and the summary was designed by someone who did not know which distinction your product needs.

This bites hardest on the distinctions that matter most and are least standard. "Frustrated" is in every taxonomy. "Reading a number off a screen and about to correct themselves" is in none of them, and it is the one that would let you not interrupt.

Reason two: it arrives on the wrong side

The sidecar tells the agent something about the caller. It does nothing for the agent's own voice. If the correct response to a frustrated caller is to slow down, soften, and shorten — and it is — then the cascade must express that through the synthesizer, using whatever style controls the synthesis API exposes. Twilio's own taxonomy is useful here: neural voices give you coarse controls like pitch and rate and are deterministic; generative voices are expressive, non-deterministic, and capable of breaths and hesitations. So the output side is not hopeless, but the control surface is a handful of parameters rather than a continuous conditioning signal.

05.4 Pricing the half-cascade: what the middle option actually buys

Half the paralinguistic channel, one of the two seams, and a synthesis quality ceiling. Worth costing out rather than hand-waving.

Chapter 02 introduced the half-cascade as a species: native audio in, text reasoning, synthesized audio out. It is worth returning to now, with the information argument in hand, because it is the option most teams should be considering and the one that gets least analysis.

Take the four properties this arc keeps trading against and score all three architectures on each:

Read the third and fourth rows together, because that combination is the interesting part and it is easy to miss. A half-cascade keeps the output seam and loses the input seam. The model's plan exists as text before it becomes sound, so you can filter it — the Chapter 04 compliance argument survives. What you lose is the ability to inspect and rewrite what the caller said before it reaches the model, which is where redaction, retrieval injection and spoof detection live.

There is one more cost that is easy to underrate. In a half-cascade you have no transcript of the caller's turn that is authoritative — there may be one produced for logging, but the model did not reason over it, so your evaluation set, your quality metrics, your analytics and your dispute records are all derived from a secondary artefact. Chapter 10's point applies with force: you cannot regression-test on a string the system never used.

And the quality ceiling is real and specific. The bundled synthesizer inside a multimodal product is optimized jointly with everything else, under latency pressure, and the sources are consistent that it lags specialist text-to-speech models on naturalness. If your brand voice is a competitive asset — and for consumer products it usually is — you are trading the thing your users actually notice for a paralinguistic channel they cannot name.

05.5 The products that cannot be built on a transcript

Four categories where the discarded 95% is the entire value proposition.

Inworld's framing is the cleanest in the sources: choose end-to-end when expressivity transfer is the product. Here is that criterion made concrete.

And the mirror-image list, from the same source and equally worth respecting: end-to-end is also simply the pragmatic choice when you want one vendor, one connection and one bill; when your turns are short and conversational so the audio-token premium is small; when you were going to use that vendor's model anyway; and when a fixed voice set costs you nothing because any pleasant voice will do.

05.6 Concept check

Next chapter: the single largest controllable line in the ledger, and the tradeoff curve it sits on. Endpointing.

06 The hardest easy problem in the stack

Deciding that somebody has stopped talking sounds like a solved problem. It is the largest controllable line in the ledger and the most common cause of a pilot being cancelled.

Chapter 03 showed that the endpoint wait can be the single biggest item in the turn gap — 500 ms of pure silence, larger than recognition, larger than the model's time to first token. It is also the only line you can change by editing one number, which makes it irresistible to optimize and dangerous to optimize carelessly.

Here is why it is hard. The caller says:

"My account number is four four two ……… sorry, let me find it ……… eight one."

At the first gap the acoustic evidence is: silence. At the end of the utterance the acoustic evidence is: silence. If you decide on silence alone, you must pick a duration, and any duration you pick will be wrong in one direction or the other on some fraction of calls. That is not an implementation gap. The information needed to distinguish those two silences is frequently not in the audio at the moment you must decide — it is in the syntax of what was just said, in the caller's intonation contour, or in nothing at all because even a human listener would have to guess.

If you want the from-zero treatment of voice activity detection, endpointing and turn-taking — the acoustics, the state machines, the psycholinguistics — the gleam Voice turn-taking: VAD, endpointing and barge-in builds all of it from first principles. This chapter assumes it and goes straight to the production tradeoff.

06.1 Four signals, four generations of endpointer

Each generation adds a signal that resolves ambiguity the previous one could not see.

Twilio compresses this into two buckets: fixed endpointing using silence timers or lightweight voice activity detection, and smart endpointing using trained models over advanced detection, acoustic features and semantics. And they add the detail that matters operationally: nearly all smart systems still include a raw-silence timeout as a fallback, because a model that decides the caller is still speaking must be overruled eventually.

That fallback is not a footnote. It is the mechanism behind the tail-latency behaviour in section 06.3, and if you do not know it is there you will misdiagnose the resulting latency histogram.

06.2 Hand-worked: choosing the threshold from data

A twenty-sample pause distribution, an explicit cost model, and the optimum computed four times.

Suppose you instrument a hundred calls and extract every within-turn pause — every silence where the caller was not finished. You observe twenty of them, in milliseconds:

Now define the two costs, because you cannot optimize until you have.

Now change the assumption and watch the answer move, because this is the part that turns a default into a decision.

One caveat worth stating: this treats the pause distribution as fixed. It is not. It shifts with the caller population, with the question you just asked (open questions produce longer pauses than yes/no ones), and with the channel. The right practice is per-prompt thresholds — short after "is that correct?", long after "what can I help you with?" — which costs nothing and is skipped by nearly everyone.

06.3 Widget: the tuning curve

The same twenty pauses, drawn. Drag the threshold and watch both failure modes move against each other.

The top strip is the pause distribution: each mark is one within-turn pause from the dataset above, and the vertical line is your threshold. Marks to the right of the line are the calls where the agent cuts the caller off.

The lower panel plots three curves against threshold: false cuts per 100 turns, dead air per turn, and the combined cost with your chosen interruption penalty. The circle marks the optimum, and it moves as you drag $K$ — which is the point of the widget.

Turn on semantic endpointing and something interesting happens to the curves: the false-cut curve drops sharply, because the pauses that follow a syntactically incomplete phrase are now held open regardless of the timer. The cost minimum moves left — you can afford a much shorter timer — which is exactly the claimed benefit. Then look at the readout's tail figure, because that is where the bill arrives.

06.4 Semantic endpointing and the discrete tail

Smart endpointing lowers the median and creates a bimodal distribution. If you only watch the median, you will not see the thing your callers hate.

A semantic endpointer classifies the transcript prefix: does this look like a finished thought? If yes, fire almost immediately; if no, hold the turn open and fall back to a long silence timeout. That gives you two operating regimes in one system:

Two operational consequences follow immediately.

First, report percentiles, not means. A mean of 305 ms conceals that one caller in seven experiences 900. And it is not a random seventh: the model's errors correlate with speaker characteristics — irregular pacing, non-standard grammar, accents, degraded call quality. Twilio says this outright. Your tail latency is unevenly distributed across your callers, and the people who get the slow experience are systematically the people who already have the worst time with automated systems. That is an accessibility issue wearing a performance-metric costume.

Second, measure the spike height as its own metric. The fraction of turns landing at exactly $T_{\text{long}}$ is a direct estimate of your semantic endpointer's false-negative rate, available for free from production telemetry, with no labelling required. It is the cheapest model-quality signal in the entire stack, and almost nobody instruments it.

06.5 Buying aggression back with graceful aborts

The clever move: make being wrong cheap, then you can afford to be wrong more often.

Everything above treats a false cut as a fixed cost. Twilio points out that it does not have to be, and this is the most useful engineering idea in their guide.

When the endpointer fires early, the pipeline does not immediately make sound. It has to generate a first token, close a sentence, synthesize the first chunk, and push the media to the edge — several hundred milliseconds during which nothing is audible yet. If, during that window, the caller resumes speaking, you can abort the whole thing and the caller never knows the endpointer made a mistake.

The abort itself has four steps, and skipping any one of them produces a distinct, recognizable bug:

1. Clear the outgoing audio buffer immediately

   Anything already queued for playout will otherwise be spoken. Skip this and the caller hears a fragment of a word after they resumed.

   Symptom: a syllable of the aborted reply, then silence

2. Abort the model stream

   Close the connection or cancel the request. Skip this and you pay for tokens nobody will hear, and the completion may later arrive and be spoken by a stale handler.

   Symptom: the agent answers a question from thirty seconds ago

3. Cancel any open tool requests

   The aborted turn may have already dispatched a database write. Cancellation must be real, and where it cannot be, the tool must be idempotent.

   Symptom: two appointments booked from one conversation

4. Clean up the conversation history

   The partial turn must not persist as though it were spoken. This is context repair, and Chapter 07 shows exactly how to get it right.

   Symptom: the agent references an answer the caller never heard

06.6 Concept check

Next chapter: what happens when the caller talks over you, and the four things that must be undone in the right order.

07 The caller talks over you. Now what?

Four things are in flight, three of them cost money, and one of them will corrupt the conversation history if you get it wrong.

The agent is thirty words into a forty-word answer. The caller says "no, wait —". At that instant your system has:

• Audio playing in the caller's ear, and more audio queued in a buffer they cannot un-hear.
• A language model still streaming tokens for a reply that has been overtaken by events.
• Possibly a tool call in flight that the abandoned reply had launched.
• A conversation history that currently claims the agent said forty words, when the caller heard about nine.

Handling this well is what separates a system that feels like a conversation from one that feels like an answering machine. Handling it badly produces the single most-reported complaint in voice pilots: "it doesn't let me interrupt" — and its subtler sibling, "it forgets what it just said."

07.1 The state machine, with the interruption path drawn in

Five states, six transitions, and one path that everybody forgets to implement.

The Salesforce tutorial gives the canonical machine, built on a small voice-activity model — a two-megabyte network that processes 32 ms chunks in under a millisecond on CPU. States:

Two properties of this machine are worth stating because implementations routinely violate them.

The microphone is never closed. A machine that stops listening while in SPEAKING cannot be interrupted at all — it is half-duplex in the worst sense. Section 07.5 explains why teams close it anyway and what the right answer is.

INTERRUPTED is a real state, not a branch. It exists because cancellation is not atomic: four things must be undone, they complete at different times, and events arriving in between must go somewhere. Implement it as a branch inside SPEAKING and you get the classic bug where a second interruption during the first interruption's cleanup leaves the machine in a state that no code path can leave.

07.2 Hand-worked: how much audio does the caller hear after they start talking?

Every millisecond of that overlap is agent speech competing with the caller's voice. Here is the exact accounting.

Let $t = 0$ be the moment the caller's first syllable leaves their mouth. We want the time at which the last agent audio stops being audible.

Twilio's colocation advice now reads as arithmetic rather than as vendor guidance. Every service hop between the recognizer, the synthesizer and the audio buffer adds directly to the overlap window, and the overlap window is audible. Colocation is not a performance nicety here; it is the difference between an agent that yields and an agent that talks over its customers.

07.3 Widget: the barge-in race

Set the detection delay, the coupling latency and the buffer depth, then trigger an interruption and watch exactly what the caller hears.

The timeline shows both audio streams. The agent's speech runs along the top; the caller's interruption enters from below. The shaded overlap is the region where both are audible at once, annotated in milliseconds, and the four cancellation events are marked where they land.

The failure-mode readout is the part to pay attention to. Depending on your settings you will see clean yield, talk-over, the stutter (where playback stops, restarts because a queued chunk arrived after the flush, then stops again), and the zombie completion — where the cancelled generation finishes and gets spoken because nothing cancelled it.

Try the combination that produces the stutter: client flush on, but a long upstream transport. The flush command and a freshly-sent audio chunk cross on the wire, so the chunk arrives at a client that has already emptied its queue, and it plays. Silence, then a fragment, then silence. Callers describe this as "it glitched." Twilio describes it as an unavoidable race condition whose probability shrinks with coupling tightness, and they are right — the widget just lets you watch the probability.

07.4 Context repair: the bug that survives the demo

The other three cancellations produce audible symptoms. This one produces a conversation that quietly stops making sense.

Your history says the assistant said this:

{"role": "assistant", "content":
 "I can move that appointment for you. I have Tuesday the twentieth
  at nine in the morning, or Thursday the twenty-second at two thirty
  in the afternoon. The Thursday slot is with Doctor Chen, who you saw
  last time, and I would recommend that one for continuity of care."}

The caller heard: "I can move that appointment for you. I have Tues—" before they interrupted with "no, next week."

If you leave the history as written, the agent now believes it has offered two slots and made a recommendation. Three turns later it says "as I mentioned, Doctor Chen would be best," and the caller has no idea what it is talking about. This is not a rare edge case; it happens on every single interruption, and it is invisible in testing because testers do not interrupt.

The better implementation does not estimate at all. Timestamp the audio chunks as you emit them, and have the playout layer report the position it actually reached:

class SpokenTracker:
    def __init__(self):
        self.chunks = []          # (word_index_end, cumulative_ms)
        self.words  = []

    def emit(self, text, audio_ms):
        # Called for every synthesized chunk BEFORE it is queued.
        self.words.extend(text.split())
        prev = self.chunks[-1][1] if self.chunks else 0
        self.chunks.append((len(self.words), prev + audio_ms))

    def spoken_prefix(self, played_ms):
        # played_ms comes from the playout layer, not from a clock:
        # it is how much audio actually left the speaker.
        n = 0
        for word_end, cum_ms in self.chunks:
            if cum_ms <= played_ms:
                n = word_end
            else:
                # Partial chunk: interpolate within it, then floor.
                prev_ms = 0 if n == 0 else self._ms_at(n)
                frac = (played_ms - prev_ms) / max(1, cum_ms - prev_ms)
                n += int((word_end - n) * frac)   # floor: never overclaim
                break
        return " ".join(self.words[:n])

# On interruption:
heard = tracker.spoken_prefix(playout.played_ms())
history[-1]["content"] = heard + " --"          # the dash marks the cut
history.append({"role": "system",
                "content": "The user interrupted you mid-sentence."})

Two details in that snippet are doing more work than they look like they are.

The trailing -- tells the model that its own turn was cut off, which changes its next turn in exactly the right way: it stops re-explaining from the top and responds to what the caller actually said. And the system note is what lets it acknowledge the interruption naturally — "sure, go ahead" rather than starting a fresh paragraph.

07.5 The echo paradox

To hear an interruption you must keep the microphone open. If you keep the microphone open, you hear yourself.

The agent's voice comes out of the caller's speaker and goes back into the caller's microphone. Without intervention, the recognizer transcribes the agent's own words, the voice detector sees continuous speech, and the system either interrupts itself or refuses to ever detect a real interruption.

Three mechanisms exist, and each one has a cost:

07.6 Concept check

Next chapter: the other source of dead air, and the three techniques for filling it without lying to the caller.

08 The two-second hole in the middle of your conversation

Everything so far assumed the agent knows what to say. A tool call is the case where it does not, and cannot, for as long as somebody else's API takes.

An enterprise voice agent is defined by its tools. The Salesforce build is a hospital receptionist with five of them — check availability, schedule, cancel, look up a patient, look up a doctor — and without them the whole thing is a very expensive way to say "I'm sorry, I can't help with that."

Tools also destroy your latency budget, and they do it in the ugliest possible way: not by adding a constant, but by adding a large variable amount on exactly the turns where the caller is most invested in the answer.

Here is the sequence, with the Chapter 03 ledger applied:

1. The caller finishes speaking

   Endpoint wait, recognition. About 850 ms gone before anything intelligent happens.

   Same as every turn

2. The model decides to call a tool

   It does not emit text; it emits a structured tool call. You pay time-to-first-token and then wait for the arguments to complete — and there is nothing speakable in a function call.

   The first place a tool turn diverges: no tokens the synthesizer can use

3. The tool runs

   A scheduling system, a CRM, a legacy database behind three layers of middleware. 200 ms if you are lucky, 2–4 s if it is a real enterprise system, occasionally 10 s.

   Entirely outside your control, and frequently outside your monitoring

4. The model runs again

   Results go back, and the model generates the actual answer. A second full time-to-first-token, plus sentence detection.

   You pay the model's start-up cost twice on a tool turn

5. Finally, synthesis

   First audio byte, transport, and the caller hears something.

   By now they have said "hello?"

08.1 The tool loop, and the re-entrancy bug hiding in it

The standard agent loop is recursive. Voice adds a concurrent input stream that the loop was never designed for.

The Salesforce agent processes each utterance through a recursive loop: send history plus tool definitions to the model; if it returns tool calls, execute them, append the results, and repeat; if it returns text, stream it through the sentence buffer to synthesis. That loop is correct for text and incomplete for voice, because in voice a new user turn can arrive in the middle of it.

async def agent_turn(history, turn_id):
    for _ in range(MAX_TOOL_HOPS):              # bound it; voice cannot wait
        resp = await llm.complete(history, tools=TOOLS)

        if turn_id != current_turn_id:            # GUARD 1: superseded
            return                                # the caller moved on

        if not resp.tool_calls:
            await speak_streaming(resp, turn_id)
            return

        # GUARD 2: say something before a slow call.
        if expected_ms(resp.tool_calls) > FILLER_THRESHOLD_MS:
            await speak_filler(resp.tool_calls, turn_id)

        results = await asyncio.gather(*[            # GUARD 3: parallel,
            run_tool(c, timeout=TOOL_TIMEOUT_MS)      # and always bounded
            for c in resp.tool_calls
        ], return_exceptions=True)

        history += as_messages(resp.tool_calls, results)

Three guards, three real bugs.

Guard 1 — the superseded turn. Every asynchronous step must check whether the turn it belongs to is still the current one. Without this, a tool that returns after the caller has interrupted causes the agent to answer a question that was abandoned thirty seconds ago. This is the zombie completion from Chapter 07, arriving via the tool path instead of the generation path.

Guard 2 — speak before you wait. Covered in the next two sections.

Guard 3 — parallel and bounded. Two independent lookups must not be serialized: 900 ms plus 700 ms is 1,600 ms sequentially and 900 ms in parallel. And every tool needs a timeout shorter than the caller's patience, with a graceful spoken fallback. An unbounded tool call is an unbounded silence, and a caller who hangs up generates a support ticket about your product, not about the vendor whose API stalled.

08.2 Hand-worked: what the caller actually experiences

The same 1,800 ms tool call, priced with and without an acknowledgment. The difference is not what you would guess.

Now the second-order observation, which is the one that separates people who have shipped this from people who have read about it.

And the cheapest trick of all, which costs nothing: fire the filler from the tool name, not from a model call. The moment you see check_availability you know a schedule lookup is happening, and a pre-recorded "let me check the schedule" is both faster and more specific than anything the model would have generated. Generating fillers with the model adds a whole time-to-first-token to the thing whose entire purpose is to eliminate a gap.

08.3 Widget: masking a slow tool

Drag the tool duration and the filler length; watch perceived silence and actual latency move independently.

The timeline shows one tool turn. The top track is real work: recognition, generation, tool execution, second generation, synthesis. The bottom track is what the caller hears: silence in terracotta, speech in amber. The two are not the same shape, and the gap between them is the entire discipline of latency masking.

The three modes are cumulative in sophistication: none, filler, and async fold-in — where the agent starts answering the part it already knows and splices the tool result in at the next sentence boundary.

Set the tool to 400 ms with a 2,400 ms filler and watch the pathology: the answer is ready almost immediately and the caller sits through two and a half seconds of "let me check that for you" before hearing it. Masking applied to a fast path is pure added latency. This is why the filler decision belongs behind a threshold on the tool's expected duration, as in the loop above, and not on every tool call.

08.4 Three techniques, ranked by how much they buy

One is nearly free, one requires care, one changes what your agent is.

1. Acknowledgment fillers — cheap, effective, slightly dishonest

   Pre-synthesized audio triggered by the tool name, behind a duration threshold. Sixty-five percent of the dead air on a slow tool turn, for a day of work and a folder of WAV files. Twilio explicitly excludes "generative interstitial fillers" from their core latency budget precisely because they are an orchestration-layer trick that hides latency rather than reducing it — which is a fair description and not a criticism.

   Buys: perceived latency. Costs: a small honesty problem, addressed in 08.5

2. Speculative dispatch — free milliseconds, at the cost of wasted calls

   You often know which tool is needed before the model tells you. If the caller's partial transcript contains "reschedule" and an account is already identified, fire the availability lookup while the caller is still speaking. When the model asks for it, the result is already warm.

   Buys: up to the full tool duration. Costs: read-only tools only, ever, and a wasted-call rate you must monitor

3. Async fold-in — the technique that makes the agent feel alive

   Do not stop talking. Answer the part you already know — "Sure, I can move that Thursday appointment" — while the lookup runs, and splice the result in at the next sentence boundary: "… and I have two slots next week." The caller experiences a continuous, thinking-aloud conversation partner instead of a request-response system.

   Buys: the entire gap on many turns. Costs: real orchestration complexity, and a hard requirement that what you say first is true regardless of the result

That last constraint is subtle and worth an example. "Sure, I can move that appointment" is unsafe to say before the lookup, because the lookup may reveal no availability. "Let me look at the schedule for next week" is safe, because it commits to nothing. Async fold-in requires a prefix that is informative but not load-bearing, and writing those prefixes is a real authoring skill.

Fold-in is also where this chapter meets the research half of the arc. A cascade fakes continuous speech-while-thinking with orchestration; a full-duplex model that synchronizes speech, language and action can do it natively, because generation never has to stop for the tool. That thesis is the subject of the Duplex‑SLA veanor, and it is the clearest example in the whole arc of a research advance that removes a product hack rather than making it faster.

08.5 The honesty constraint

Fillers are a claim about what the system is doing. Some of those claims are false, and callers eventually notice.

"Let me check that for you" asserts that a check is happening. If you fire that phrase on every turn as a latency blanket, including on turns where no tool runs, you have taught your agent to say something untrue several times per call. This matters for three reasons, in increasing order of seriousness.

It degrades. A filler that always plays stops carrying information and becomes a verbal tic. Callers learn to talk over it, which puts you back in Chapter 07 with extra steps.

It misleads about state. If the caller hears "let me check" and then the agent answers from its prompt without checking anything, they will reasonably believe the answer was verified against a system of record. In a regulated domain that is a compliance problem with a clear paper trail, because your transcript log from Chapter 04 contains the false claim in your agent's own voice.

It hides failures. The most damaging pattern is a filler that plays while a tool times out, followed by a model answer generated without the tool result. The caller heard the check happen and got an answer; nobody upstream knows the answer was unsourced. A filler must never be able to cover a failed call. If the tool fails, the agent must say so.

08.6 Concept check

Next chapter: the layer underneath all of it — packets, buffers, codecs, and a telephone network designed in the nineteen-seventies.

09 The layer nobody demos and everybody debugs

Two hundred and seventy milliseconds of the ledger, one entire class of production incident, and the reason your phone deployment sounds worse than your browser prototype.

Fifty times a second, in both directions, for the entire call, a 20 ms packet of audio has to cross a network that was not designed for it and arrive within a window narrow enough that nobody hears the seam. That is the transport layer, and it is where a surprising fraction of "the AI sounds bad" tickets actually live.

The numbers are small and the sensitivity is enormous. From Chapter 03: transport in 40, jitter buffer in 30, decode 25, and the same three on the way out. About 190 ms of a 1,100 ms budget, before any model runs. And unlike the model latencies, these lines are set by physics, geography and defaults you did not choose.

09.1 Three transports, three failure modes

What each one does when a packet goes missing, which is the only question that matters.

The pattern that most production systems converge on, and which the Softcery survey reports as the common shape: WebRTC from the client to your relay, WebSocket from your relay to the model APIs. Each protocol is used where its failure mode is acceptable — concealment where the network is hostile, reliability where the path is short.

Two implementation details in that arrangement bite people repeatedly:

Nagle's algorithm. TCP will, by default, coalesce small writes to reduce overhead. Your 20 ms frames are exactly the small writes it wants to coalesce, and the resulting delay is invisible in a throughput test and audible in a conversation. Most WebSocket libraries disable it; verify yours does rather than assuming.

Silent format conversion. Every audio model expects raw uncompressed samples at a specific rate. Recognizers typically want 16 kHz; synthesizers commonly emit 24 kHz or 48 kHz; most models are trained on mono. Provider APIs accept a variety of inputs and convert silently, which costs latency and can cost quality. Twilio's advice is not to do all conversion yourself but to know where it is happening — because the resample you did not know about is the 15 ms you cannot find.

09.2 Hand-worked: sizing a jitter buffer from real arrival times

Ten packets, one table, and the exchange rate between delay and glitches computed exactly.

Packets are sent every 20 ms. They arrive after a base network delay plus a variable extra amount — the jitter. The playout buffer's job is to absorb that variance: hold each packet for $D$ ms before playing it, so packets that arrive late still make their slot. A packet whose jitter exceeds $D$ misses its deadline and is dropped, and the caller hears a concealment artefact.

That last connection is the kind of thing that only shows up in production. A caller on a congested network experiences two degradations that look unrelated in your dashboards: slightly worse audio, and an agent that seems reluctant to let them interrupt. They are the same cause.

09.3 Widget: the jitter buffer

Packets arriving with jitter, a playout deadline you control, and the two failure modes drawn against each other.

Each mark is a packet: send time on the left, arrival after its network delay, and the playout deadline as a vertical line. Packets landing left of the line play cleanly; packets landing right are late and are drawn in terracotta with a concealment marker on the output track.

Increase the network's jitter to simulate a congested last mile, and watch the loss rate climb until you compensate with buffer depth — and watch the ledger cost at the bottom climb with it. Switch the protocol to see the difference between concealment and head-of-line blocking on the same packet trace: with reliable delivery nothing is lost, but everything after the missing packet is delayed until it is retransmitted.

09.4 Telephony: everything is worse at eight kilohertz

The enterprise use cases are phone-shaped, and the phone network undoes several of the advantages you paid for.

The standard telephone network carries speech sampled at 8 kHz through the G.711 codec. By the Nyquist limit that represents frequencies up to 4 kHz, and it discards everything above — which is where a great deal of consonant discrimination lives. The difference between "f" and "s", between "p" and "t", is largely above 4 kHz.

Every source in this lesson agrees on the consequence. Modern recognizers are trained overwhelmingly on 16 kHz or better, so 8 kHz input reduces accuracy significantly. Synthesis sounds thinner because the model's output is bandlimited on the way out. And speech-to-speech models optimized for high-quality web audio lose much of their advantage over the phone while keeping their premium pricing — which Softcery states directly and which is, on its own, a substantial argument for the cascade in telephony deployments.

09.5 Where to put things

Placement is a latency decision with an audible consequence, and it is nearly impossible to change later.

Twilio's placement guidance reduces to three rules, and each one now has a mechanism attached from earlier chapters.

1. Media services close to the media edge, and close to each other

   Audio enters through the edge nearest the caller, so what matters is the distance from that ingress point to your services, not the distance to the caller. Every hop between recognizer, synthesizer and audio buffer is a line in the ledger and a millisecond in your barge-in overlap window.

   Mechanism: Chapter 03 line 06, and Chapter 07's overlap arithmetic

2. Prioritize audio over text

   Audio is heavier, more bandwidth-hungry and more sensitive to degradation. Model latency is easier to mask — Chapter 08 is a whole chapter of techniques for masking it — and text hops are cheap. So spend the colocation budget on the recognizer and synthesizer first.

   Mechanism: you have latency-masking tools for the model and none for the media path

3. Colocate, but do not co-host, the recognizer and the synthesizer

   The subtlest advice in the guide. The two workloads are normally out of phase — one runs while the caller talks, the other while the agent talks — which makes sharing a machine tempting. But interruptions and false endpoint triggers put them in phase, so the spikes align, contention rises, latency rises across the box, which causes more collisions. A feedback loop with a bad ending.

   Mechanism: the correlation between the two loads is not stable, and it goes to one under exactly the conditions you care about

09.6 Concept check

Next chapter: how to know whether any of this is working, and why the metric you pick decides which failures you are allowed to see.

10 How do you measure something that only exists in time?

Every metric in the previous nine chapters scores a component. A conversation is not a component, and the properties that make it good are properties of the interaction.

Suppose you have two voice agents and one is better. Which number is different?

Not word error rate: both use the same recognizer. Not time-to-first-token: both use the same model class. Not synthesis quality: both use the same voice. Yet one of them is the system callers finish their business with and the other is the one they escalate. The difference is in when each one speaks, whether it yields, how it recovers from a mistake — and none of those is visible in a clip.

The corollary is uncomfortable and worth sitting with: the metrics you choose determine which failures you are permitted to notice. A team tracking word error rate, time-to-first-token and uptime has a dashboard on which every one of Chapter 01's six production failures is invisible. The dashboard will be green during the incident.

10.1 Three layers of metric, and what each is blind to

Component, conversation, product. You need all three, and most teams have only the first.

The layers are diagnostic in one direction and motivating in the other. Product metrics tell you whether; conversation metrics tell you where; component metrics tell you what. A team with only component metrics ships regressions users hate; a team with only product metrics knows it is losing and cannot say why.

Two component metrics deserve specific warnings, because both are routinely over-trusted.

Word error rate does not transfer across channels. Chapter 09 covered why. A leaderboard number measured on clean read speech tells you very little about your telephony deployment, and the spread across providers on phone-quality audio is far larger than the spread on curated benchmarks.

Real-time factor is a threshold, not a score. Below 1.0 the system keeps up; above 1.0 latency accumulates without bound until the queue collapses. An RTF of 0.1 is not "ten times better" than 0.5 in any way the caller can perceive — both keep up. What matters is the RTF under peak load with contention, which is exactly the condition Chapter 09's co-hosting warning describes.

10.2 Duplex benchmarks: measuring the interaction itself

Four categories, four metrics, and the first eval family designed for systems that listen and speak at the same time.

Full-Duplex-Bench, used as the evaluation backbone in the PersonaPlex paper, is the clearest published attempt to score conversational behaviour rather than clip quality. It probes four categories, each constructed so that a system without the corresponding capability cannot pass:

1. Pause handling

   The user's turn contains a silence that is not a turn boundary. The metric is takeover rate: how often the model starts talking during the pause. Lower is better here — this is exactly the false-cut failure from Chapter 06, measured directly.

   Impossible to pass by being a good speaker; only turn projection helps

2. Backchannel

   While the user is speaking at length, does the model produce the short acknowledgments — "mm-hm", "right" — that humans use to signal attention without taking the floor? Measured by frequency and by the distributional distance between the model's backchannel timing and human timing.

   A half-duplex system scores zero by construction: it cannot speak while listening

3. Smooth turn-taking

   The user finishes. Does the model take the floor, and how quickly? Takeover rate (higher is better here) paired with response latency. Both are needed: a model that always responds instantly but also barges into pauses is gaming one number.

   The two paired metrics make the trivial strategies fail

4. User interruption

   The user barges in mid-response. Does the model yield, and is what it says afterwards actually a response to the interruption? Scored by takeover rate, by latency, and by a judged quality score on the resulting turn.

   Chapter 07's entire cancellation chain, reduced to two numbers and a judgement

The headline results, from the PersonaPlex paper's own tables, are worth quoting with their uncertainty attached because the uncertainty is the lesson:

And from the same paper's benchmark table, the two numbers most relevant to this lesson — response latency on smooth turn-taking, and takeover rate when the user interrupts:

10.3 Widget: the eval console

Five systems, four metrics, and confidence intervals that widen as you shrink the rater pool.

Pick a metric and the bars re-sort. The whiskers are 95% intervals; on the subjective metrics they come from the paper, and on the objective ones they are computed from the rater count you set with the slider. Drag the rater count down and watch a clear ranking dissolve into a tie — which is the single most useful thing this widget does.

10.4 Hand-worked: is that difference real?

Two published means, two published intervals, and a conclusion that most write-ups of this table get wrong.

The top of the Full-Duplex-Bench opinion-score table reads 3.90 ± 0.15 for the leading system and 3.72 ± 0.14 for the runner-up. Is the leader better?

The practical rule that falls out: report intervals, and read other people's. A leaderboard that publishes only means invites you to over-read the ordering, and the top few rows of a subjective leaderboard are frequently one system wide.

10.5 The suite you should actually build

Four tiers, in the order you should build them, each one catching failures the previous tier cannot see.

1. Tier 1: the latency ledger, instrumented per line, in production

   Every line item from Chapter 03 emitted as a timing span on every turn, with percentiles. This is a week of work and it makes every subsequent question answerable. Include the endpoint-timeout-spike fraction from Chapter 06 as a first-class metric — it is a free estimate of your endpointer's error rate.

   Catches: regressions, provider degradation, the tail nobody noticed

2. Tier 2: an adversarial turn-taking suite, on your own channel

   Recorded utterances containing the hard cases: mid-sentence pauses of 400, 700 and 1,100 ms; barge-ins at 0.5 s and 2 s into the reply; overlapping speech; a caller who says "hello?" during a tool call. Replayed through your real telephony path at 8 kHz. Scored on false cuts, overlap window, and whether the transcript history matches what was audibly spoken.

   Catches: everything in Chapters 06 through 09, before your callers do

3. Tier 3: task completion on scripted scenarios with instrumented tools

   Enterprise scenario suites — the Salesforce work cites Benchforce for exactly this — define environments with function calling, log every tool invocation, and score whether the goal state was reached. This is the layer where "the agent was charming and booked the wrong day" finally shows up as a failure.

   Catches: agent-layer defects that all timing metrics call success

4. Tier 4: human listening on paired conditions

   Expensive, slow, and the only thing that measures whether the experience is good. Design it as a paired comparison where exactly one variable changes, report rater counts and intervals, and prefer many raters over many ratings each.

   Catches: the residual that objective metrics cannot represent

10.6 Concept check

Final chapter: consent, provenance, spoofing and disclosure — and then the two ladders, which is where the research half of this arc and the product half finally meet.

11 Four different problems that all get called "voice safety"

They have different threat models, different mitigations and different owners. Conflating them is how teams end up with a policy document and no defences.

The moment your product can produce a convincing human voice on demand, four distinct problems arrive at once. They are usually discussed as one topic, which guarantees that at least three of them go unaddressed.

Note the directions. Two are about what you emit, one is about what you accept, one is about the frame around the conversation. A team that has done cloning consent well may have no spoof detection at all, and the second is the one that costs money when it fails.

11.1 Consent: the enrollment sample is the artefact that matters

Modern voice cloning needs seconds of audio. That single fact rearranges the whole problem.

The research half of this arc established the capability plainly: a three-second enrollment prompt is enough for a modern model to reproduce a speaker's timbre in a new utterance. Whatever consent regime you build has to be designed for a world where the raw material is trivially available — from a voicemail, a podcast, a support call you recorded for quality purposes.

Which means the defensible artefact is not the model and not the audio. It is the record of authorization attached to the enrollment sample, and it needs four fields that most implementations do not have:

1. Who

   An identified person, not an account. "Uploaded by admin@company" is not consent from the speaker.

   The consenting party and the uploading party are frequently different people

2. For what

   Scope. Consent for a brand voice in customer support is not consent for outbound sales calls, and certainly not for a different product two years later.

   Scope creep is the normal failure, not malice

3. Until when, and revocable how

   A revocation path that actually works: which deployed artefacts must be regenerated, how long that takes, and who is paged.

   Consent you cannot withdraw is not consent; it is a licence

4. Provable how

   A verification step binding the recording to the person — a spoken challenge phrase in the enrollment, timestamped and stored with the sample.

   Otherwise anyone with a podcast can be enrolled by a stranger

And one design rule that removes most of the risk surface at a cost most products can absorb: prefer a designed voice to a cloned one. A brand voice built from a hired voice actor's licensed session, or an entirely synthetic voice that corresponds to no real person, has no consent-revocation risk, no impersonation exposure and no awkward conversation when the actor's contract ends. Cloning is necessary when the product is the person's voice. For a support line, it never is.

11.2 Watermarking and anti-spoofing: the two directions

One marks what you emit. The other inspects what you receive. Both are classifiers; only one is usually built.

Watermarking your output

An audio watermark embeds an imperceptible signal in generated speech so that a detector can later establish that the audio came from a generator. The duplex-model literature takes this seriously enough to ship detection alongside generation — the audio-language-model lineage this arc covers includes released watermark classifiers, precisely because the authors understood what they had built.

What a watermark is good for, honestly:

• Good for: establishing provenance for your own audio in a dispute; letting platforms label synthetic media; detecting your own model's output leaking into your training data, which is a real and underrated failure.
• Not good for: stopping a determined adversary. Watermarks degrade under re-encoding, telephony transcoding, and deliberate attack. Passing a watermarked clip through an 8 kHz phone codec — which your own product does on every call — is itself an aggressive transformation.

Spoof detection on your input

The inbound direction gets far less attention and is where the immediate financial loss lives. Your voice agent answers calls. Some of those calls will be synthetic audio operated by someone trying to reset a password, redirect a payment or extract account details in a voice matching your customer's.

Two engineering positions follow, and the first is not negotiable:

Voice is not an authentication factor. Voice matching is a convenience signal and a fraud-scoring input. Any flow where a matching voice alone unlocks an action is a flow that a three-second sample defeats. This was defensible in 2015 and is not now.

Spoof detection belongs in the seam. A synthetic-speech classifier on inbound audio sits exactly where the input guardrail sits in Chapter 04, costs tens of milliseconds, and produces a score that feeds risk decisions rather than a binary gate. Note the architectural implication, which is a small irony: the cascade's inspectable input boundary is where you attach the defence against synthetic speech — the same seam, doing a job nobody listed when the architecture was chosen.

11.3 Disclosure, and a regulatory surface that is not one surface

Where the same recorded call is routine in one city and a criminal offence in another.

Disclosure is the cheapest of the four and the one most often reduced to a line in a script. Two engineering details make the difference between real disclosure and theatre:

It must survive barge-in. Callers talk over greetings; that is what greetings are for. If your disclosure lives in the first sentence of a greeting the caller interrupts, it was not delivered. Either disclose in a form that cannot be skipped, or re-deliver on the first substantive turn.

It must be true about what happens next. "You're speaking with an automated assistant, and this call may be recorded" is disclosure. It becomes false the moment a human silently joins, or the recording is used to train a model, and the caller was told something specific enough to be wrong.

The jurisdictional picture is genuinely difficult, and the honest summary is that it is not one picture. The Softcery material characterizes the European situation as one rulebook on paper and twenty-seven regimes in practice, with the same recorded call being routine in one member state and a criminal offence in another; and it characterizes several Gulf jurisdictions as places where recording without consent is a crime rather than a fine, with data-residency requirements that forbid the audio leaving the country and permitting regimes around voiceprint features.

11.4 Build versus buy, layer by layer

Five levels of abstraction, what each one hands you, and what each one takes away.

11.5 The two ladders

Every research advance in this arc relaxes exactly one product constraint. Mapping the rungs tells you what ships next — and which constraints no model will ever relax.

Here is the closing argument of the whole series, and the widget is the argument.

On the left, the research ladder: the sequence of advances that took speech from a synthesis problem to an interface. On the right, the product ladder: the constraints from this lesson that decide whether a deployment survives. Click a research rung to see which product constraints it relaxes, and how strongly.

Press show only unrelaxed constraints and read what is left, because that is the actual prediction this lesson makes.

11.6 The close: the PhD ladder meets the product ladder

Where this arc has been, and the one sentence worth keeping from all of it.

The research half of this arc is a story about representation. Speech became tokens, so a transformer could model it. The first codebook was distilled toward semantics, so a small number of tokens could carry meaning. Two streams were modelled at once, so a machine could listen and speak in the same instant. Text was threaded through the audio as an inner monologue, so the thing could reason while it spoke. Voice and role were made conditionable, so it could be a specific person doing a specific job. And speech, language and action were synchronized, so it could act while talking.

Read as a ladder, each rung removes a constraint that a product engineer had previously been working around by hand. Dual-stream modelling removes the cancellation chain of Chapter 07. Speech-language-action synchronization removes the latency masking of Chapter 08. Role conditioning removes a chunk of the voice-selection and brand-consistency work that pushed teams toward the cascade in the first place.

And yet the product half of the arc is not converging on those systems, and this lesson has spent eleven chapters explaining why in specifics rather than in vibes. Not because the research is overhyped — it is not; a 160 ms duplex model is a genuinely astonishing object — but because a deployed voice agent is scored on properties the research ladder was never climbing. Debuggability. A place to stand between deciding and speaking. A price that decomposes. Audio that never leaves the building. A transcript that a regulator can read.

The practical version, for whatever you build next. Instrument the ledger before you optimize anything. Tune your endpointer against your own cost model rather than a default. Implement all four steps of the barge-in cancellation, including the one nobody sees. Mask the tools you cannot make fast, and never let a filler cover a failure. Measure the interaction and not just the components, and read your own confidence intervals. Put the seam where the compliance requirement is. And keep the interfaces boring, because the rung that is about to fall will change the calculus and you want the migration to be a project rather than a rewrite.

Then pick up the phone and call your own agent from a moving car. That is the eval that matters.

11.7 Concept check

That is the arc. The research half is in the veanors; this was the half where it meets a phone line.