Turn-Taking, Endpointing & Barge-In

Chapter 1: What Is a Turn?

We just spent a chapter arguing about when a turn ends without ever defining what a turn is. That is not pedantry — every engineering decision downstream depends on the definition, and the definition is messier than you expect.

Naively: a turn is “the stuff one person says before the other person says stuff.” Conversation analysts have been measuring real recorded conversations for fifty years, and the picture they get back has four ingredients. Learn these four words and the rest of the lesson has vocabulary.

The four primitives

An IPU — inter-pausal unit — is a continuous stretch of one person’s speech bounded by silence on both sides (the usual cutoff is 200 ms of silence). It is the atom of measurement: not a word, not a sentence, just “a run of talking.” The sentence “I’d like to order a … large pepperoni pizza” is two IPUs, not one, because the hesitation splits it.

A pause is silence between two IPUs of the same speaker. That is our villain from Chapter 0: the caller is not finished, they are just breathing, thinking, or reading a menu.

A gap is silence between IPUs of different speakers — the handoff. This is the quantity that runs about 200 ms in natural conversation and is what your agent’s response latency is competing against.

An overlap is time when both speakers have an IPU running at once. Overlap is not an error; it is normal, healthy conversation. Measurements of multi-party meetings put overlapping speech at roughly 10–20% of all spoken time. A system architecture that assumes speech is strictly alternating — a half-duplex assumption — is throwing away a fifth of reality before it starts.

The fifth ingredient: backchannels

Then there is the thing that breaks naive systems worst of all. While you are talking, the listener says “mm-hm.” “Right.” “Yeah.” “Oh wow.” These are backchannels — short vocalisations that mean keep going, I am with you. They are speech. They overlap. They are emphatically not a bid for the floor.

Now consider your agent. It is halfway through reading back an order. The caller says “mm-hm.” A naive barge-in implementation sees energy on the microphone, concludes it is being interrupted, kills the text-to-speech mid-word and goes silent, waiting. The caller, who said “mm-hm” precisely to signal continue, is met with an abrupt stop and an awkward pause. They say “…sorry, go on?” This failure is so common you have almost certainly experienced it this year.

Symmetrically, humans backchannel at agents constantly, and good agents backchannel back — the small “mm-hm” while you read out a long address is what makes the interaction feel alive. Most production systems today are half-duplex-with-barge-in: the user may interrupt the agent, the agent may not interrupt the user, except for these short acknowledgements.

A worked measurement, by hand

Let us annotate a real-shaped exchange with a stopwatch, because the numbers are the point. Timestamps in seconds from the start of the recording:

Now compute, step by step:

IPUs. The caller produces two: 0.00–1.35 and 2.05–3.40. Plus the backchannel 5.40–5.66, which is technically a third IPU. The agent produces one: 3.62–6.10.

Pause. Between caller IPU 1 and caller IPU 2: 1.35 s to 2.05 s, so the pause length is 2.05 − 1.35 = 0.70 s. Same speaker on both sides → pause, not gap. Any endpointer with a threshold below 700 ms fires here and is wrong.

Gap. Between caller IPU 2 and the agent’s IPU: 3.62 − 3.40 = 0.22 s. Different speakers → gap. And 220 ms is right in the human band, so this handoff feels human.

Overlap. The backchannel runs 5.40–5.66 while the agent runs 3.62–6.10. Overlap = min(5.66, 6.10) − max(5.40, 3.62) = 5.66 − 5.40 = 0.26 s. As a fraction of total spoken time: total speech = caller 1.35 + 1.35 + 0.26 = 2.96 s, agent 2.48 s, so 5.44 s of speech, of which 0.26 s overlaps → 0.26 / 5.44 = 4.8%. A single polite exchange is at the low end; heated multi-party meetings are where you get to 20%.

The design consequence, in one line. To handle this six-second exchange correctly, a system must tolerate a 700 ms same-speaker pause without firing, then fire within about 220 ms of a same-length silence that happens to be a real ending, and then ignore a 260 ms overlap entirely. A pure silence timer cannot do this: 700 > 220 means no single threshold separates the two silences. The information that distinguishes them is not in the silence. It is in the words on either side.

Explore the anatomy

The widget below is that exact exchange, drawn to scale. Step through the events with the slider — each stop highlights one primitive and prints its arithmetic. Notice that the pause (0.70 s) and the gap (0.22 s) look like the same thing to a microphone, and only the labels above them differ.

Flip “detector view” on and the tragedy is visible: the annotated exchange collapses into a binary strip of loud and quiet. The 0.70 s pause and the 0.22 s gap become indistinguishable rectangles of nothing, differing only in width — and the wrong one is wider. Everything we build from here is an attempt to recover the labels that this view destroys.

Chapter 2: Energy VAD, built by hand

Start at the bottom. Before any question about turns or meaning, there is a much smaller question: is there sound in this tiny slice of audio? Answering it is voice activity detection (VAD), and the oldest answer — still running in production in more places than anyone admits — is a threshold on energy.

Frames: chopping time into decisions

Audio arrives as a stream of samples: at 16 kHz, sixteen thousand numbers per second, each one the air pressure at an instant. You cannot make a decision per sample — a single sample tells you nothing, since even a loud sine wave passes through zero. So you chop the stream into frames: short blocks, typically 10, 20 or 30 milliseconds. At 16 kHz a 10 ms frame is 160 samples. One decision per frame gives you a decision every 10 ms, which is plenty — a phoneme lasts 50–150 ms.

Frame size is a real trade-off, and it is the same trade you meet everywhere in audio. Too short and the energy estimate is noisy (you might land inside a zero-crossing). Too long and your boundaries get coarse — a 100 ms frame cannot tell you the speech started 30 ms in. Ten to thirty milliseconds is the sweet spot for the same reason it is the sweet spot for spectrograms.

Energy: the one formula

For a frame of N samples x₁…x_N, the root-mean-square amplitude is exactly what it says: square every sample, take the mean, take the square root.

Why square first? Because pressure swings both ways — a loud frame has big positive and big negative values, and their plain average is near zero. Squaring makes everything positive and, conveniently, proportional to energy. The square root at the end puts us back in amplitude units so the number is comparable to the sample values themselves.

Then we convert to decibels, because loudness perception is logarithmic and because it makes thresholds behave sanely across a 100 000:1 dynamic range:

With samples normalised to the range −1 to +1, this is dBFS — decibels relative to full scale. Full-scale is 0 dB; everything real is negative. Loud speech into a phone lands around −10 to −20 dBFS; room tone lands around −45 to −60. Why 20 and not 10? Because we are taking the log of an amplitude, and amplitude squared is power — 20 log₁₀(A) = 10 log₁₀(A²). Same number, different units in.

The hand-worked example — every single step

Here is a toy utterance: eight frames. Each toy frame holds 8 samples so you can do the arithmetic on paper; treat each one as standing in for a real 10 ms frame (160 samples at 16 kHz). The clip is “quiet → speech onset → loud vowel → a stop-consonant closure → loud vowel → quiet” — the classic shape that breaks naive detectors, because the closure in frame 4 is genuinely silent inside a word.

the toy clip — 8 frames × 8 samples
f1  # room tone   [ 0.02, -0.01,  0.00,  0.01, -0.02,  0.01,  0.00, -0.01]
f2  # onset       [ 0.10, -0.14,  0.18, -0.12,  0.09, -0.15,  0.13, -0.09]
f3  # loud vowel  [ 0.30, -0.42,  0.51, -0.38,  0.27, -0.45,  0.40, -0.29]
f4  # /p/ closure [ 0.03, -0.02,  0.02, -0.03,  0.02, -0.01,  0.03, -0.02]
f5  # loud vowel  [ 0.25, -0.31,  0.36, -0.28,  0.22, -0.33,  0.30, -0.24]
f6  # room tone   [ 0.01, -0.02,  0.01,  0.00, -0.01,  0.02, -0.01,  0.00]
f7  # room tone   [ 0.01, -0.01,  0.00,  0.01, -0.01,  0.00,  0.01, -0.01]
f8  # room tone   [ 0.01, -0.01,  0.00,  0.01, -0.01,  0.00,  0.01, -0.01]

Frame 1. Square each sample: 0.02²=0.0004, 0.01²=0.0001, 0²=0, 0.01²=0.0001, 0.02²=0.0004, 0.01²=0.0001, 0²=0, 0.01²=0.0001. Sum = 0.0004+0.0001+0+0.0001+0.0004+0.0001+0+0.0001 = 0.0012. Mean = 0.0012 / 8 = 0.00015. RMS = √0.00015 = 0.012247. Level = 20 log₁₀(0.012247). Now log₁₀(1.2247) = 0.0881, and we have 1.2247×10⁻², so log₁₀ = 0.0881 − 2 = −1.9119. Times 20: −38.24 dBFS.

Frame 2. Squares: 0.0100, 0.0196, 0.0324, 0.0144, 0.0081, 0.0225, 0.0169, 0.0081. Running sum: 0.0100 → 0.0296 → 0.0620 → 0.0764 → 0.0845 → 0.1070 → 0.1239 → 0.1320. Mean = 0.1320 / 8 = 0.0165. RMS = √0.0165 = 0.12845. log₁₀(1.2845) = 0.1088 → −0.8912 → ×20 = −17.82 dBFS.

Frame 3. Squares: 0.0900, 0.1764, 0.2601, 0.1444, 0.0729, 0.2025, 0.1600, 0.0841. Running sum: 0.0900 → 0.2664 → 0.5265 → 0.6709 → 0.7438 → 0.9463 → 1.1063 → 1.1904. Mean = 0.1488. RMS = √0.1488 = 0.38575. log₁₀(3.8575) = 0.5863 → −0.4137 → −8.27 dBFS. This is a shout compared to frame 1 — 30 dB louder, which is a factor of about 31 in amplitude and 1000 in power.

Frame 4 — the dangerous one. Squares: 0.0009, 0.0004, 0.0004, 0.0009, 0.0004, 0.0001, 0.0009, 0.0004. Sum = 0.0044. Mean = 0.00055. RMS = √0.00055 = 0.023452. log₁₀(2.3452) = 0.3703 → 0.3703 − 2 = −1.6297 → −32.59 dBFS. That is only 5.6 dB above room tone. To an energy detector, the closure of a /p/ is silence. To a listener, it is the middle of a word.

Frame 5. Squares: 0.0625, 0.0961, 0.1296, 0.0784, 0.0484, 0.1089, 0.0900, 0.0576. Running sum: 0.0625 → 0.1586 → 0.2882 → 0.3666 → 0.4150 → 0.5239 → 0.6139 → 0.6715. Mean = 0.08394. RMS = 0.28972. log₁₀(2.8972) = 0.4619 → −0.5381 → −10.76 dBFS.

Frame 6. Identical squares to frame 1 (0.0001×4 and 0.0004×2, sum 0.0012) → −38.24 dBFS.

Frames 7 and 8. Squares: six values of 0.0001 and two zeros → sum = 0.0006. Mean = 0.000075. RMS = √0.000075 = 0.008660. log₁₀(8.660) = 0.9375, so log₁₀(8.660×10⁻³) = 0.9375 − 3 = −2.0625 → −41.25 dBFS.

Raw decision sequence with a −25 dBFS threshold: 0 1 1 0 1 0 0 0. The word has been chopped in half. Downstream, that reads as two IPUs with a pause between them, and if your endpointer’s threshold is short enough it will fire inside a single word. Everybody hits this bug.

Hangover: the fix that costs latency

The cure is a hangover timer (also called a hold-over or release time): once you declare speech, keep declaring speech for H more frames after the energy drops. It is exactly the “wait a beat before you conclude they stopped” instinct, made mechanical.

Trace it on our eight frames with H = 2, starting with h = 0:

Now read what the hangover bought and what it cost, because both are in the table and both matter.

Bought: frame 4 is now SPEECH. The word is one IPU again. The plosive closure no longer looks like the end of a turn.

Cost: the true end of speech was the end of frame 5. The detector does not say SILENCE until frame 8. That is a two-frame delay on every single endpoint, always, unconditionally. With 10 ms frames that is 20 ms — cheap. But real systems set hangover to 200–400 ms to survive real pauses between words, and that lands directly on your response latency. Hangover is latency you pay on every turn to avoid a chop you would suffer on some turns. That trade is the DNA of this entire field, and we will formalise it in Chapter 5.

Adaptive noise floor — the trick that makes it survive

A fixed −25 dBFS threshold is doomed: a phone call from a car has a noise floor near −30 dB, a quiet office near −55. The standard fix is to track the floor and set the threshold relative to it. Take a slow running estimate of the quietest recent level and demand speech exceed it by a fixed margin:

With α = 0.02 the floor moves at about 2% per frame — slow enough that speech does not drag it up, fast enough to follow an air conditioner switching on. A margin of 10–15 dB is typical. Work one step: if noise_k−1 = −41.0 and a silent frame arrives at −38.24 dB, then noise_k = 0.98×(−41.0) + 0.02×(−38.24) = −40.18 − 0.765 = −40.945 dB, and with a 15 dB margin the threshold becomes −25.95 dB. It crept up by 0.055 dB. Slow is the point.

Code: three forms

python — form 1: the arithmetic, exactly as we did it by hand
import math

frame = [0.10, -0.14, 0.18, -0.12, 0.09, -0.15, 0.13, -0.09]

total = 0.0
for x in frame:
    total += x * x                 # 0.0100, then 0.0296, then 0.0620 ...
mean = total / len(frame)      # 0.1320 / 8 = 0.01650
rms  = math.sqrt(mean)          # 0.12845
db   = 20 * math.log10(rms)    # -17.82 dBFS  <- matches the table

python — form 2: the streaming detector with hangover and adaptive floor
import numpy as np

class EnergyVAD:
    def __init__(self, sr=16000, frame_ms=10,
                 margin_db=15.0, hangover_ms=200, onset_frames=2, alpha=0.02):
        self.n       = int(sr * frame_ms / 1000)   # 160 samples per frame
        self.H       = int(hangover_ms / frame_ms)  # 20 frames of hold
        self.onset   = onset_frames                # consecutive loud frames to START
        self.margin  = margin_db
        self.alpha   = alpha
        self.noise   = -50.0    # dBFS, will adapt to the room
        self.h       = 0        # hangover countdown
        self.run     = 0        # consecutive-loud counter
        self.speech  = False

    def level_db(self, frame):
        rms = np.sqrt(np.mean(frame.astype(np.float64) ** 2) + 1e-12)
        return 20.0 * np.log10(rms)

    def push(self, frame):
        db  = self.level_db(frame)
        thr = self.noise + self.margin
        loud = db > thr

        if loud:
            self.run += 1
        else:
            self.run = 0
            # only learn the floor from quiet frames, or speech drags it up
            self.noise = (1 - self.alpha) * self.noise + self.alpha * db

        if self.run >= self.onset:      # ONSET: needs onset_frames in a row
            self.speech = True
            self.h = self.H
        elif self.h > 0:                 # HANGOVER: hold through short dips
            self.h -= 1
        else:
            self.speech = False

        return self.speech, db, thr

python — form 3: the library one-liner (WebRTC VAD, the industry workhorse)
import webrtcvad
vad = webrtcvad.Vad(2)                       # aggressiveness 0..3
is_speech = vad.is_speech(frame_bytes, 16000)  # 10/20/30 ms of 16-bit PCM
# Same shape of answer. Underneath: a GMM on sub-band energies plus, yes, a hangover.

Note what the library did not remove: you still choose an aggressiveness (a threshold by another name), you still choose the frame size, and you still have to add your own hangover and onset logic on top if you want stable segments. The one-liner hides the arithmetic, not the design decisions.

Play with it

The widget draws our eight-frame clip as an energy bar chart with your threshold as a line, then the raw decision strip, then the decision strip after hangover. Push the threshold up and watch the onset frame vanish; push the hangover up and watch the trailing SPEECH tail grow — that tail is your added response latency, drawn to scale.

Now crank the room noise to about 18 dB. The room-tone frames climb above −25 dB and the detector declares speech continuously. Your voice agent, in a car, now believes the user never stops talking, so it never responds at all. That is not a hypothetical: it is the number-one field failure of energy VAD, and no amount of threshold tuning fixes it, because the noise and the speech now have the same energy. They do not, however, have the same shape. Next chapter.

Chapter 3: Neural VAD — from “loud” to “speech”

We left the energy detector drowning in a car. The problem is precisely stated: energy is not speech-specific. A fan, an engine, a fridge compressor, road roar, another conversation across the room — all have energy. Turning up the threshold to reject them also rejects the quiet ends of real sentences. One number cannot separate two things that share that number.

But listen to a fan and a vowel and the difference is obvious to you instantly. Where is it? Not in the loudness. It is in the distribution of energy across frequency, and how that distribution moves over time. That is a picture, not a number — and pictures are what classifiers eat.

What makes speech look like speech

Four properties separate voiced speech from most steady noise, and every VAD since 1990 exploits some subset:

1. Harmonic structure. A voiced sound (any vowel, and the consonants that hum) is produced by vocal folds opening and closing periodically, at 85–255 Hz depending on the speaker. That periodicity puts energy at a fundamental frequency and at integer multiples of it — a comb of evenly spaced spikes in the spectrum. Fans do not do combs. This is the single strongest cue.

2. Spectral tilt. Broadband mechanical noise (engines, road, HVAC) is usually low-frequency-heavy: lots of energy under 500 Hz, falling steeply above. Speech puts substantial energy in the 500–3400 Hz band, because that is where the formants that distinguish vowels live. Measure the difference between a low band and a high band and noise gives you a steep number, speech a shallow one.

3. Modulation at the syllable rate. Speech turns on and off at roughly 4 Hz — that is the syllable rate, and it is remarkably universal. So the energy envelope of speech wiggles a few times a second. A fan’s envelope is flat. A classifier that sees a short history of frames, not just one frame, gets this for free.

4. Spectral flatness. White-ish noise spreads energy evenly across frequency; speech concentrates it in peaks. The ratio of the geometric mean to the arithmetic mean of the spectrum (near 1 for flat noise, near 0 for peaky speech) captures this in one scalar.

The hand-worked example: why the fan loses

Let us build the smallest honest version — four log-mel-ish bands into a logistic regression — and run two frames through it by hand. Bands, in dBFS:

First, notice the thing that dooms the energy detector. Total level is dominated by the loudest band, and the fan’s b₁ is −22 dB — louder than the speech frame’s loudest band (−20 dB is speech’s b₂, so they are within 2 dB). A single-number detector with any threshold either accepts both or rejects both. There is no line.

Now the features. Spectral tilt = b₁ − b₄:

Fan: (−22) − (−41) = 19 dB of tilt — steep, all the energy is low.
Speech: (−24) − (−28) = 4 dB of tilt — nearly flat across the band.

And the mid-band presence, measured as how far b₂ and b₃ sit above a nominal −34 dB / −38 dB reference floor:

Fan: b₂ is −31, so (−31) − (−34) = 3 dB above. b₃ is −36, so (−36) − (−38) = 2 dB above.
Speech: b₂: (−20) − (−34) = 14 dB. b₃: (−23) − (−38) = 15 dB.

Feed them into a logistic unit with weights we will justify in a second:

Fan frame:
0.35 × 3 = 1.05
0.30 × 2 = 0.60
tilt term: (19 − 6) = 13, times −0.25 = −3.25
bias: −1.2
z = 1.05 + 0.60 − 3.25 − 1.20 = −2.80

Squash with the logistic σ(z) = 1 / (1 + e^−z): e^2.80 = 16.4446, so σ(−2.80) = 1 / (1 + 16.4446) = 1 / 17.4446 = 0.057. Five percent probability of speech. Rejected.

Speech frame:
0.35 × 14 = 4.90
0.30 × 15 = 4.50
tilt term: (4 − 6) = −2, times −0.25 = +0.50
bias: −1.2
z = 4.90 + 4.50 + 0.50 − 1.20 = 8.70

e^−8.70 = 0.000166, so σ(8.70) = 1 / 1.000166 = 0.9998. Ninety-nine point nine eight percent. Accepted, decisively.

Look at the gap: 0.057 versus 0.9998, from two frames that an energy detector cannot tell apart at all. The positive weights on mid-band presence say “speech puts energy where formants live.” The negative weight on excess tilt says “if it is all bass, it is machinery.” The −1.2 bias says “default to not-speech,” which is the right prior for a microphone in a room.

Probability is not a decision

The network gives you p_k ∈ [0, 1] per frame. You still need a state machine, and it is worth knowing that production VADs use the same two tricks as our energy version plus one more:

Hysteresis (two thresholds). Enter SPEECH when p > 0.5; leave SPEECH only when p < 0.35. The gap prevents chattering when p hovers at the boundary — the same reason your thermostat has a deadband. One threshold gives you a detector that flickers on and off ten times a second on a marginal frame.

Min-duration constraints. Reject speech segments shorter than ~100 ms (a door click, a keyboard tap) and silences shorter than ~100 ms (a stop consonant). Two counters.

Hangover, still. Yes, even neural VADs use it. The model is per-frame; conversation is not.

python — the tiny classifier, exactly as hand-computed
import math

def features(bands_db):
    b1, b2, b3, b4 = bands_db
    tilt = b1 - b4                       # 19 for the fan, 4 for speech
    mid2 = b2 - (-34.0)                  # presence above a nominal floor
    mid3 = b3 - (-38.0)
    return mid2, mid3, tilt

def p_speech(bands_db):
    mid2, mid3, tilt = features(bands_db)
    z = 0.35*mid2 + 0.30*mid3 - 0.25*(tilt - 6.0) - 1.2
    return 1.0 / (1.0 + math.exp(-z))

print(p_speech([-22, -31, -36, -41]))   # 0.0573  fan   -> reject
print(p_speech([-24, -20, -23, -28]))   # 0.9998  voice -> accept

python — the hysteresis state machine on top of any probability stream
class VADState:
    def __init__(self, on=0.50, off=0.35, hangover_frames=7, min_speech=4):
        self.on, self.off = on, off
        self.H, self.min_speech = hangover_frames, min_speech
        self.speech, self.h, self.run = False, 0, 0

    def push(self, p):
        if not self.speech:
            self.run = self.run + 1 if p > self.on else 0
            if self.run >= self.min_speech:      # ignore 30 ms clicks
                self.speech, self.h = True, self.H
        else:
            if p > self.off:                     # note: the LOWER threshold
                self.h = self.H
            else:
                self.h -= 1
                if self.h <= 0:
                    self.speech, self.run = False, 0
        return self.speech

python — the one-liner everybody actually ships
import torch
model, utils = torch.hub.load('snakers4/silero-vad', 'silero_vad')
p = model(torch.from_numpy(chunk_512_samples), 16000).item()   # 0..1 per 32 ms
# Still yours to own: the on/off thresholds, min durations, and the hangover.

See the separation

The widget shows both frames as four-band spectra, computes the features live, and plots them in the 2-D feature space (tilt on one axis, mid-band presence on the other) with the learned decision boundary. Drag the noise slider to make the fan louder: the energy detector’s verdict flips to SPEECH and stays wrong, while the classifier’s point slides along the tilt axis and stays on the correct side, because making a fan louder does not make it look like a voice.

Try “quiet speech” at a level offset of −10 dB. The energy verdict says SILENCE — the frame is genuinely quiet — while the shape verdict still says SPEECH with high confidence, because tilt and mid-band ratios are invariant to gain. That invariance is why neural VADs pick up the soft trailing syllables at the end of a sentence, which is exactly the region where your endpoint decision gets made.

Chapter 4: Endpointing Is Not VAD

Here is the sentence that should be printed on the wall of every voice-agent team:

Endpointing (also called end-of-turn detection, or EOT) is the decision that ends the user’s turn and starts your response pipeline. It consumes VAD output but it is a different problem with different inputs and a different failure cost.

The silence-only endpointer, and exactly how it fails

The classical implementation is three lines: run a VAD, accumulate consecutive non-speech time, fire when that time exceeds a threshold T. Every voice platform ships this, and the default is nearly always around 500 ms — a number arrived at empirically by the whole industry, and one we will actually derive in the next chapter.

It fails on a specific, enumerable, and utterly common list of utterances:

Read the last row too. A one-word answer is maximally obviously complete, and the silence-only endpointer still burns its full threshold before it will admit that. You are paying the tail risk premium on the easiest turns in the conversation.

What a human uses instead

When you talk to a person, you know they are done before the silence has even started — that is how 200 ms gaps are physically possible. The cues you use, roughly in order of strength:

Syntactic completeness. “I’d like to order a …” ends on a determiner; a determiner requires a noun after it. The sentence is grammatically unfinished, and you know it without any conscious effort. Compare “… a large pepperoni pizza”: a complete noun phrase completing a complete clause.

Prosody — the melody of finality. English speakers drop pitch and lengthen the final syllable at the end of a statement (a falling final contour); they hold pitch level or rise slightly when they intend to continue. Final lengthening alone is a strong predictor. This is why endpointers that see fundamental-frequency contours beat text-only ones on the same transcripts.

Fillers. “um,” “uh,” “like” are floor-holding devices — their literal function in conversation is “I am still going, do not take the floor.” A trailing filler is nearly conclusive evidence of not-done.

Pragmatics. If the agent asked “what is your order number?” then the expected answer is a number of a certain length. “Four one seven” when the format is six digits is obviously incomplete in context, though it is a perfectly complete phrase in isolation.

Semantic endpointing, hand-worked

Modern “smart turn” detectors are models that consume the streaming ASR partial transcript (and often the audio directly) and output P(the turn is complete). In production these are small fine-tuned transformers — the open ones are in the tens-of-millions-of-parameters range and run in a few milliseconds. But the logic is legible, so let us build a bag-of-cues version we can compute by hand and get the real behaviour out of it.

“Function word” means determiners (a, the), prepositions (to, for, with), conjunctions (and, but) — the words that grammatically demand a continuation. “Content word” means nouns, verbs, adjectives, numbers. Now three prefixes, in full:

Prefix A: “I’d like to order a”
ends on “a” → function word → −2.6
contains main verb (“order”) → +1.3
content-word ending: no → 0. Question form: no → 0. Filler: no → 0.
z = −0.8 − 2.6 + 1.3 = −2.1
e^2.1 = 8.1662, so P = 1 / (1 + 8.1662) = 1 / 9.1662 = 0.109 → 11% complete. Do not fire.

Prefix B: “I’d like to order a large pepperoni pizza”
ends on “pizza” → content word → +2.1
contains main verb → +1.3
z = −0.8 + 2.1 + 1.3 = 2.6
e^−2.6 = 0.07427, so P = 1 / 1.07427 = 0.931 → 93% complete. Fire, fast.

Prefix C: “my order number is, um”
ends on filler → −1.7
contains main verb (“is”) → +1.3
z = −0.8 − 1.7 + 1.3 = −1.2
e^1.2 = 3.3201, P = 1 / 4.3201 = 0.231 → 23% complete. Do not fire.

Fusing probability with silence — the rule that ships

You do not use P alone; you combine it with elapsed silence, because confidence should be allowed to decay. The elegant formulation used by production stacks is a time-varying threshold: demand high completion confidence early, accept lower confidence as the silence stretches, and always keep a hard timeout as a floor.

Compute τ at a few silences: at s = 0 ms, τ = 0.9. At 200 ms, τ = 0.9 − 0.6×(200/1200) = 0.9 − 0.1 = 0.80. At 600 ms, τ = 0.9 − 0.3 = 0.60. At 1200 ms and beyond, τ = 0.30.

Now run our three prefixes through it:

B (“…pizza”, P = 0.931): at 200 ms of silence, τ = 0.80 and 0.931 > 0.80 → fires at 200 ms. The user gets a human-paced gap on a clearly finished sentence. The silence-only endpointer would have waited 500 ms — we just cut 300 ms off the most common case in the conversation.

A (“…order a”, P = 0.109): τ bottoms out at 0.30 and 0.109 never exceeds it. So this turn never fires on the semantic rule — it falls through to the hard timeout (typically 1.5–2 s), giving the caller all the time they need to find the word. Correct behaviour, and the exact failure the fixed threshold could not avoid.

C (“…is, um”, P = 0.231): also never crosses (0.231 < 0.30), also falls through to the timeout. Also correct.

python — the fused endpointer, complete
class Endpointer:
    """Consumes VAD frames + streaming ASR partials. Emits END_OF_TURN."""
    def __init__(self, frame_ms=20, hard_timeout_ms=1500, min_silence_ms=120):
        self.frame_ms  = frame_ms
        self.hard      = hard_timeout_ms
        self.min_sil   = min_silence_ms   # never fire on a stop consonant
        self.silence   = 0
        self.p         = 0.0              # P(turn complete) from the turn model

    def tau(self, s):
        return 0.9 - 0.6 * min(1.0, s / 1200.0)

    def on_partial(self, text):
        self.p = turn_model.complete_prob(text)   # the semantic model

    def on_frame(self, is_speech):
        if is_speech:
            self.silence = 0
            return False
        self.silence += self.frame_ms
        if self.silence < self.min_sil:
            return False
        if self.p > self.tau(self.silence):    # semantic path: can fire at 200 ms
            return True
        if self.silence >= self.hard:        # safety net: always terminates
            return True
        return False

Two details in that code are load-bearing and easy to skip. min_sil exists so a stop consonant can never end a turn no matter how confident the semantic model is. And the hard timeout exists because every production endpointer needs a path that terminates without the model’s consent — if the smart layer dies, silently returns 0, or the ASR stops emitting partials, the conversation must still proceed. Nearly all shipped end-of-turn systems keep a raw-silence fallback underneath the clever one for exactly this reason.

Drive the endpointer

Step through a transcript word by word. The widget shows the running prefix, the cue flags that fired, the computed z and P, and the τ(s) curve with the current silence marked — so you can see the exact moment the fire condition becomes true, or watch it never become true and fall to the timeout.

Chapter 5: The Trade-off Curve

We have been saying “too eager interrupts, too lazy stalls” for five chapters. Now we compute it. By the end of this chapter you will have derived the industry’s 500 ms default from a pause-length distribution, and you will know exactly what a smart endpointer buys in units of expected pain.

Step 1: the only measurement that matters

Go and log your users’ mid-turn pause lengths — the durations of silence that occur while the user is not finished. This single distribution determines your threshold, and no amount of reasoning substitutes for it, because it varies enormously by task: a chatty support call has short pauses, an intake form where people read numbers off documents has long ones.

Here is a realistic survival table — the fraction of user turns containing at least one mid-turn pause of at least the given length:

Note the shape: it falls fast and then has a long, thin tail that never quite reaches zero. That tail is why “just set it high enough” does not work — you would need a threshold of several seconds to drive interruptions near zero, and several seconds of dead air is a worse product than an occasional interruption.

Step 2: the two costs, in the same units

For a silence threshold T:

Interruption rate = P(some mid-turn pause ≥ T). Read straight off the table. Every turn containing a pause at least as long as your threshold gets cut off.

Added latency = T. You always sit through T of silence before responding, on every turn, including all the ones with no hesitation at all. This is the part teams forget: the threshold is a tax levied on the 99% of turns that did not need it.

To compare them we need an exchange rate. Ask your product: how many seconds of extra response latency would I accept, per conversation, to avoid one interruption? A common honest answer is “interruptions are about twice as bad as a second of delay,” giving weights w_int = 1.0 per interruption and w_lat = 0.5 per second. Then:

Step 3: evaluate it, every number

T = 200 ms: interruption term = 1.0 × 0.35 = 0.350. Latency term = 0.5 × 0.200 = 0.100. Cost = 0.350 + 0.100 = 0.450.

T = 400 ms: 1.0 × 0.18 = 0.180. Latency = 0.5 × 0.400 = 0.200. Cost = 0.180 + 0.200 = 0.380.

T = 500 ms: interpolate the table between 0.18 at 400 and 0.09 at 600: halfway is 0.135. Latency = 0.5 × 0.500 = 0.250. Cost = 0.135 + 0.250 = 0.385.

T = 600 ms: 0.090 + (0.5 × 0.600 = 0.300) = 0.390.

T = 800 ms: 0.040 + 0.400 = 0.440.

T = 1000 ms: 0.020 + 0.500 = 0.520.

T = 1400 ms: 0.008 + 0.700 = 0.708.

Step 4: what changing the exchange rate does

Suppose you are building an emergency-services triage line where interrupting a distressed caller is disastrous: w_int = 4.0. Redo T = 400: 4.0 × 0.18 = 0.72, plus 0.20 = 0.92. T = 800: 4.0 × 0.04 = 0.16, plus 0.40 = 0.56. T = 1000: 4.0 × 0.02 = 0.08, plus 0.50 = 0.58. The optimum has moved out to ~800–900 ms. Conversely, for a fast-paced quiz game where snappiness is the product and interruptions are forgivable (w_int = 0.3): T = 200 gives 0.105 + 0.100 = 0.205; T = 400 gives 0.054 + 0.200 = 0.254. The optimum collapses to ~200 ms. Same math, different product, threshold differing by 4×. Anyone who tells you the “correct” endpointing threshold without asking what the agent does is guessing.

Step 5: the smart endpointer, in the same units

Now quantify Chapter 4’s claim. Let the semantic model have recall 0.8 on incompleteness: of all the mid-turn pauses that would have triggered a premature fire, it correctly recognises 80% as “user is not done” and holds. And let it have a false-incomplete rate of 0.10: on 10% of genuinely finished turns it wrongly believes more is coming, so those turns fall through to the hard timeout at 1500 ms.

Interruptions. At an aggressive T = 200 ms, the raw rate was 0.35. Multiply by the 20% the model misses: 0.35 × (1 − 0.8) = 0.070. We just did better than any point on the fixed curve — the fixed curve does not reach 0.07 until T ≈ 700 ms.

Expected latency. 90% of turns fire at 200 ms; 10% fall to 1500 ms. E[latency] = 0.90 × 200 + 0.10 × 1500 = 180 + 150 = 330 ms.

Cost. 1.0 × 0.070 + 0.5 × 0.330 = 0.070 + 0.165 = 0.235 — versus 0.380 for the best possible fixed threshold. A 38% reduction in expected pain, and it came from information, not from tuning.

But look at the percentiles, not the mean. The median latency is 200 ms — excellent, human. The 90th percentile is 200 ms. The 95th percentile is 1500 ms. The distribution is bimodal: a tight spike at 200 and a second spike at the timeout, with nothing in between. Users do not experience your mean. They experience the 1 turn in 10 where the agent goes quiet for a second and a half for no reason they can perceive, which reads as the agent being confused. Report p95, always.

The latency ledger it plugs into

Endpointing does not live alone. Here is where a full cascaded response actually spends its milliseconds, end of user speech to first audio out:

Add the best case by hand: 200 + 30 + 150 + 100 + 40 = 520 ms. The worst: 800 + 80 + 300 + 200 + 150 = 1530 ms. The endpoint wait is 38% of the best case and 52% of the worst — the largest single line item in both, and the only one that is a choice rather than a physical cost. That is why this lesson exists.

There is one more lever in that table and it is delicious: if you can gracefully abort the downstream stages, you can afford to endpoint aggressively. Fire at 250 ms, start the ASR flush, start the LLM, start the TTS — and if the user resumes speaking 300 ms later, kill all of it before a single sample reaches the speaker. You bought several hundred milliseconds of “free” speculation, because the pipeline’s own latency gave you a window to be wrong in. The cost is that abort must be airtight: buffers cleared, streams cancelled, tools cancelled, context repaired. Get that wrong and the user hears a stutter — half a word of the agent, then silence. That machinery is Chapter 6.

Walk the frontier

The widget plots the pause-length survival curve on the left and the cost curve on the right. Move the threshold and watch both terms and the total. Flip on the smart endpointer to see the single point it occupies — below and to the left of the entire fixed curve — and drag the model quality slider to watch that point degrade back toward the curve as the model gets worse.

Pull the smart-model recall down to 0.3 and the advantage almost vanishes — a bad turn model is worse than no turn model, because you also inherit its false-incomplete tail. This is the honest answer to “should we use semantic endpointing?”: only if you can measure its recall on your traffic. A turn detector trained on clean read speech, dropped onto an 8 kHz telephone line with accented speakers, can easily land under 0.5 and make your product slower and ruder simultaneously.

Chapter 6: Barge-In — the state machine (SHOWCASE)

Everything so far has been about the user’s turn ending. This chapter is the other half: the agent’s turn ending, against its will, because the human started talking.

Barge-in is the ability of the user to interrupt the agent mid-sentence and be heard. It sounds like a small feature. It is the difference between a conversation and a voicemail menu, and it is one of the most bug-prone pieces of code in the entire stack, because the moment the user speaks over you, four independent asynchronous systems are all mid-flight and every one of them has to be stopped correctly, in the right order, with the right cleanup.

What is actually running when the user interrupts

Picture the instant. The agent is 1.2 seconds into an eight-second reply. At that moment:

1. Audio is buffered ahead of the ear. The text-to-speech model generated maybe 3 seconds of audio; 2.5 s of it has been handed to the transport; roughly 1.2 s has actually reached the speaker. The rest is sitting in send buffers, network buffers, and the client’s jitter buffer. If you only stop generating, the user keeps hearing the agent for hundreds of milliseconds. This is the classic “it does not stop when I talk” bug and it is a buffer problem, not a detection problem.

2. The TTS stream is still producing. It has a whole sentence queued.

3. The LLM is still generating tokens. It is 60 tokens into a 200-token reply and every token costs money and occupies a slot.

4. Tool calls may be in flight. The agent decided to look up an order; there is an HTTP request outstanding, possibly a mutating one. This is where interruption stops being a UX concern and becomes a correctness concern.

The teardown, in order, with reasons

Step 1 — flush the outbound audio buffer, first and immediately. Everything else can take a few milliseconds; this cannot, because it is the only step the user can hear. On WebRTC that means stopping the track feed and clearing your send queue; on a telephony bridge it means sending a clear-buffer control message so the media edge drops what it has queued. Twilio’s guidance is explicit about this ordering: clear the outgoing audio buffer, then abort the model stream, then cancel tools, then clean up history.

Step 2 — abort the TTS stream. Cancel the request so you stop paying for and receiving audio you will never play.

Step 3 — abort the LLM stream. Same reasoning, bigger bill. With streaming APIs this is a cancellation on the request; make sure your framework actually propagates it rather than just abandoning the iterator.

Step 4 — cancel in-flight tool calls, and know which ones cannot be cancelled. A read is safe to abandon. A write is not: if place_order already reached the server, cancelling your HTTP request does not un-place the order. The engineering answer is to make mutating tools idempotent (an idempotency key per turn) and to defer them — do not fire a mutation until the agent has actually spoken the confirmation the user is agreeing to. Design the tool boundary so that an interruption can never leave the world in a state the conversation does not reflect.

Step 5 — repair the context. This is the step everyone forgets.

Work the truncation by hand. Suppose TTS emitted these word boundaries, and playout reached 1,240 ms before the flush:

Kept text = the words whose start time is below 1,240 ms: “Your order is one large pepperoni”. Everything from “pizza” onward is discarded from history. Six words kept out of the sentence the model thought it said. Note the subtlety: “pepperoni” started at 1,090 but would not have finished until ~1,600 — the user heard a fragment. Keeping the whole word is the right call (partial words are unrepresentable in text and the user did hear its beginning), but if you want to be strict you can mark it: "...one large pepper<cut>".

The state machine

Here is the whole thing as states and transitions. Six states, and the two dashed edges are the ones that make it a conversation instead of an IVR.

python — the teardown, correct order, with context repair
async def on_user_barge_in(session, played_ms):
    # 1. AUDIBLE FIRST: nothing else matters until the speaker goes quiet.
    await session.transport.clear_output_buffer()     # local send queue
    await session.transport.send_clear_signal()      # tell the far edge to drop its queue

    # 2 & 3. Stop paying for audio and tokens nobody will hear.
    session.tts_task.cancel()
    session.llm_task.cancel()

    # 4. Tools: reads are abandoned, writes must be safe by construction.
    for call in session.inflight_tools:
        if call.mutating:
            # cannot un-send it; rely on the idempotency key and record it
            session.note_uncancellable(call)
        else:
            call.cancel()

    # 5. CONTEXT REPAIR: history must reflect what was HEARD, not what was GENERATED.
    spoken = " ".join(w.text for w in session.word_timings if w.start_ms < played_ms)
    session.history[-1] = {"role": "assistant",
                          "content": spoken + " <interrupted>"}

    session.state = "USER_SPEAKING"

One detail worth stealing: played_ms is not “time since we started sending.” It is the playout position, which you get either from the client’s audio element, from RTP timestamps, or by tracking bytes actually written to the device minus the buffer depth. Using send-time instead of playout-time over-counts by exactly the buffer depth — usually 200–400 ms — and silently corrupts every repaired transcript. Colocating your speech services and keeping buffers shallow makes this race narrower; a wide race produces the disorienting stutter where the agent stops, then emits one more fragment.

THE SHOWCASE: play the human

Now you drive. Below is a live conversation simulator. You are the caller; the agent runs whichever turn-taking policy you select. The timeline scrolls in real time: your track on top, the agent’s below, the state machine above lighting up as it walks, and a scoreboard of the three things that matter — interruptions inflicted on you, dead air you sat through, and backchannels mishandled.

The controls. Press speak (more coming) to say a chunk of a sentence you have not finished — this is “I’d like to order a…”. Press speak (done) to finish your sentence. hesitate inserts 700 ms of thinking silence mid-sentence. mm-hm fires a 250 ms backchannel — try it while the agent is talking. interrupt! is a real 600 ms takeover attempt. Or press run scenario and watch the canonical hard case play out hands-free.

The policies. naive 200 ms is a bare silence timer. fixed 500 ms + hangover is the industry default from Chapter 5. semantic uses completeness (it knows whether your last chunk left the sentence hanging) with a 1500 ms fallback. full-duplex is Chapter 8’s answer, included here so you can feel the difference before we explain it.

What to actually try

Experiment 1 — manufacture the Chapter 0 bug. Policy naive 200 ms. Press speak (more coming), then hesitate. The agent fires inside your hesitation and the interruption counter increments. Now switch to semantic and repeat: the agent sits still, because your last chunk left the sentence grammatically open. Same audio, different answer.

Experiment 2 — pay the dead-air tax. Policy fixed 500 ms. Press speak (done) alone, repeatedly. Every single turn adds ~500 ms to the dead-air counter even though you never hesitated once. Switch to semantic and watch the same sequence cost ~200 ms per turn. That difference, multiplied by 40 turns in a support call, is 12 seconds of the customer’s life.

Experiment 3 — the backchannel trap. Get the agent talking (press speak (done), wait for the orange block), then press mm-hm. Under naive and fixed, the agent stops dead — the “mishandled backchannel” counter ticks. Under semantic and full-duplex it keeps going, because those policies classify the overlap first. That classifier is Chapter 7.

Experiment 4 — a real interruption. While the agent talks, press interrupt!. Every policy stops, but watch the little teardown ladder that flashes in the CANCELLING state, and note the different stop latencies: the cascaded policies need detection plus buffer flush, while full-duplex stops in roughly one 80 ms frame.

Chapter 7: Backchannel or Interrupt?

You just felt it in the simulator: the agent is talking, the microphone lights up, and you have somewhere between 150 and 400 milliseconds to decide whether that sound means “stop, I want the floor” or “keep going, I am following.” Get it wrong in one direction and you talk over a person trying to correct you. Get it wrong in the other and you stop dead every time they say “mm.”

This is the hardest classification problem in the stack, and unlike endpointing it has almost no room to wait for more evidence — the whole point is that you must act while the sound is still happening.

The signals, and how far each gets you

Duration. The strongest single cue and the one you can only use by waiting. Backchannels are short: “mm-hm” is 200–400 ms, “yeah” 150–300 ms. Takeover attempts keep going. Waiting 500 ms to be sure destroys the whole point — that is half a second of you talking over someone who wanted the floor. Practical systems use a partial-duration signal: at 300 ms, still going and rising in energy is strong evidence of takeover.

Loudness relative to your own output. People backchannel quietly; it is a deliberate politeness signal that the sound is not a bid. People who want the floor get louder, often noticeably louder than their normal speaking level, because they are competing with your voice. Measure the user’s level relative to their own recent speaking average, not in absolute dB.

Lexical content. The backchannel lexicon is small and shockingly effective: mm-hm, uh-huh, yeah, right, okay, sure, I see, wow, oh, huh. If your streaming ASR emits a partial within ~200 ms and it matches that list, you have very strong evidence. If the first word is “wait,” “no,” “actually,” “sorry,” or “stop,” you have very strong evidence the other way — there is a takeover lexicon too and it is worth hardcoding.

Pitch and contour. Backchannels are typically flat or falling; a takeover attempt often starts high. Cheap to compute, genuinely informative, rarely used because it needs a pitch tracker in the hot path.

Where you are in your own sentence. Massively underused. Humans backchannel at phrase boundaries — the little breath points in your speech — and interrupt at points of disagreement, which are usually right after you said the wrong thing. If the user starts talking 80 ms after you said a number, the prior for “correction” is much higher than if they started during a smooth clause.

The model, hand-worked on three real cases

A logistic on three features. d = duration of the overlapping burst in seconds (capped at 1.5), e = relative loudness in [0, 1] (0 = much quieter than the user’s normal, 1 = much louder), b = 1 if the first ASR partial is in the backchannel lexicon.

Case A — “mm-hm”, soft, brief. d = 0.25 s, e = 0.35, b = 1.
3.0 × 0.25 = 0.75
1.6 × 0.35 = 0.56
−2.2 × 1 = −2.20
z = −2.4 + 0.75 + 0.56 − 2.20 = −3.29
e^3.29 = 26.84, so P = 1 / 27.84 = 0.036. Three and a half percent. Keep talking.

Case B — “wait, no—”, loud, longer. d = 0.55 s, e = 0.85, b = 0.
3.0 × 0.55 = 1.65
1.6 × 0.85 = 1.36
z = −2.4 + 1.65 + 1.36 = 0.61
e^−0.61 = 0.5434, so P = 1 / 1.5434 = 0.648. Sixty-five percent. Stop.

Case C — the genuinely hard one: “yeah…” loud and dragging. d = 0.80 s, e = 0.90, b = 1 (“yeah” is in the lexicon).
3.0 × 0.80 = 2.40
1.6 × 0.90 = 1.44
−2.2 × 1 = −2.20
z = −2.4 + 2.40 + 1.44 − 2.20 = −0.76
e^0.76 = 2.1383, so P = 1 / 3.1383 = 0.319. Thirty-two percent.

Case C is “yeah, but actually I wanted…” — a takeover that opens with a backchannel word. The lexicon says acknowledgement; the duration and loudness say takeover. The model lands at 0.32, genuinely uncertain. So what do you do at 0.32?

The decision threshold is not 0.5 — derive it

Classify by expected cost, not by probability. Two costs:

C_miss = cost of continuing to talk when it really was a takeover. You are now talking over a human who is trying to correct you. They repeat themselves louder. It is the single most rage-inducing voice-agent failure. Call it 5.

C_false = cost of stopping when it was only a backchannel. Mildly awkward: a short gap, then you resume. Recoverable, and you can even make it graceful (resume mid-phrase). Call it 1.

Expected cost of keep talking = P × C_miss = 5P.
Expected cost of stop = (1 − P) × C_false = 1 − P.
Stop when 5P > 1 − P, that is 6P > 1, that is:

Re-run the three cases against 0.167 instead of 0.5:

Case A: 0.036 < 0.167 → keep talking. Correct.
Case B: 0.648 > 0.167 → stop. Correct.
Case C: 0.319 > 0.167 → stop. And that is the right call — with a 5:1 cost asymmetry you should stop on a one-in-three chance of a takeover, because being wrong in the stopping direction is cheap and recoverable while being wrong the other way is not.

Cost matrix, as code

python — streaming overlap classifier with ducking and a decision deadline
BACKCHANNEL = {"mm", "mm-hm", "mhm", "uh-huh", "yeah", "yep",
               "right", "okay", "ok", "sure", "wow", "oh", "i see"}
TAKEOVER    = {"wait", "no", "stop", "sorry", "actually", "hold on", "but"}

C_MISS, C_FALSE = 5.0, 1.0
P_STAR = C_FALSE / (C_MISS + C_FALSE)          # 0.1667 — NOT 0.5

def p_takeover(dur_s, rel_loud, first_word):
    b = 1.0 if first_word in BACKCHANNEL else 0.0
    z = -2.4 + 3.0*min(dur_s, 1.5) + 1.6*rel_loud - 2.2*b
    if first_word in TAKEOVER:
        z += 3.0                              # hardcoded prior; cheap and very effective
    return 1.0 / (1.0 + math.exp(-z))

def on_overlap_frame(session, dur_s, rel_loud, first_word):
    p = p_takeover(dur_s, rel_loud, first_word)

    if dur_s > 0.12 and session.gain == 1.0:
        session.duck(db=-12)                   # soften while we decide

    if p > P_STAR:
        return on_user_barge_in(session, session.played_ms)   # full teardown, ch.6

    if dur_s > 0.45:                          # deadline: it was a backchannel
        session.unduck()
        session.backchannels_ignored += 1
    return None

Explore the boundary

Drag the point around the feature space. The horizontal axis is burst duration, the vertical is relative loudness; the shaded region is “stop.” Toggle whether the first word is in the backchannel lexicon and watch the whole boundary shift — that is the −2.2 weight moving the intercept. Then drag the cost ratio and watch the boundary sweep: at 1:1 the machine is stubborn and talks over people; at 10:1 it is jumpy and stops at every breath.

One last observation before we leave this chapter. Everything above is a heroic effort to recover, from acoustics and a partial transcript, a distinction the human speaker made effortlessly and communicated clearly — through timing, volume, pitch and word choice, all simultaneously. We are decoding it through a pipeline that deliberately threw most of that away when it converted audio to text. Which raises an obvious question: what if we did not throw it away?

Chapter 8: Full-Duplex — dissolving the problem

Step back and look at what we have built. A voice activity detector. A hangover timer. A semantic completeness model. A time-decaying threshold. A hard timeout. A barge-in teardown with five steps. A backchannel classifier with an asymmetric cost threshold. Seven mechanisms, thousands of lines, dozens of tunables — all of it in service of one job: reconstructing turn boundaries that the pipeline itself invented.

Because here is the thing. The turn boundary is not a property of the conversation. It is a property of the architecture. We built a system that can only be in one of two modes — listening or speaking — so we need a mechanism to switch between them, so we need to detect when to switch, so we need all of the above. Humans are not like this. You hear while you talk. You start forming a reply while the other person is still going. You say “mm-hm” without stopping your own thought.

Moshi: two streams, always on

Kyutai’s Moshi (2024) is the cleanest realisation of that idea, and it is worth understanding precisely because its design decisions all follow from this one commitment.

The architecture starts with Mimi, a neural audio codec that turns a waveform into discrete tokens at 12.5 tokens per second per codebook stream. So one “frame” of Moshi’s world is 1 / 12.5 = 80 milliseconds of audio. Mimi’s first codebook is distilled to carry semantic content (what was said) while the remaining residual codebooks carry acoustic detail (how it sounded) — so a single token stream contains both meaning and voice.

Then the key move. Moshi models two audio streams jointly: its own output, and the user’s input. Both are token sequences. Both advance every 80 ms, forever. At every frame the model predicts its own next audio tokens conditioned on both streams so far.

Read that again and notice what disappeared:

The paper is explicit that this dissolves the assumption, not just the code: pipelines “rely on a segmentation into speaker turns, which does not take into account overlapping speech, interruptions and interjections,” and overlap is 10–20% of real spoken time. Moshi always listens and can always speak.

Where the 160 ms comes from — compute it

The latency arithmetic is refreshingly small. One Mimi frame is 80 ms. Moshi generates audio tokens with a small acoustic delay: the acoustic codebooks lag the semantic one by a couple of steps so the model can condition fine detail on coarse content. With a delay of 2 steps:

That is below the ~230 ms average human turn-taking gap. For the first time the machine is not racing to catch up with human pacing; it is inside it. And compare the ledger from Chapter 5: a cascaded stack’s best case was 520 ms, of which 200–800 ms was endpoint wait. Deleting the endpoint wait is not an optimisation, it is a category change.

Contrast the delay budgets directly. If you naively delayed all eight of Mimi’s codebooks by one step each in a strict hierarchical pattern you would get 8 × 80 = 640 ms — worse than a decent cascaded pipeline. The delay pattern matters: putting the semantic codebook first with zero delay and the acoustic ones behind it at a small fixed offset is what gets you to 160. Architecture choices at the token level become product latency, directly and legibly.

Inner monologue: how it still reasons

A pure audio-to-audio model has a known weakness: audio tokens are a much less efficient carrier of meaning than text tokens, so reasoning quality suffers. Moshi’s answer is the Inner Monologue: the model also predicts a stream of text tokens, time-aligned to its own speech, slightly ahead of the audio it will produce. The text acts as a plan; the audio follows it.

This is a beautiful piece of engineering because of what it buys for free. Delay the audio tokens behind the text and you have streaming text-to-speech. Delay the text behind the audio and you have streaming speech recognition. One architecture, three products, selected by a delay parameter.

See the streams

The widget below draws Moshi’s actual data layout: user stream on top, Moshi’s audio stream below it, the inner-monologue text stream between them, all quantised into 80 ms frames. Scrub the playhead through an exchange that includes an overlap and a barge-in. Notice there is no state, no timer, no boundary marker — only tokens, some of which happen to encode silence.

So why is anyone still building cascaded stacks?

Because full duplex is not free, and in 2026 the enterprise default is still the cascade. The honest ledger:

The debuggability row is doing most of the work in real procurement decisions. When a regulated business must explain what its agent said and why, a stack with a transcript at every boundary is a much easier conversation than one where the model reasoned in audio space. This is why the dominant 2026 pattern for enterprise voice is still cascaded, with speech-to-speech winning where latency and naturalness are the product — concierge, companionship, live support with a premium feel.

It is also why the frontier is not “full duplex or not” but how much duplex you can get with a text layer intact. Inner Monologue is exactly that compromise: full-duplex audio with a readable text stream running alongside. Thinker-Talker designs make the same bet from the other direction — a text brain and a speech mouth sharing context, streaming.

Chapter 9: Cheat Sheet & Connections

You started this lesson with a caller being cut off mid-order. You can now explain exactly why it happened, compute the threshold that would have prevented it, quantify what that prevention costs on every other turn, implement the recovery when it happens anyway, and describe the architecture in which the failure is not expressible. Here is everything in one place.

The ladder

The widget places every technique on the two axes that matter — how naturally it handles the shape of real conversation, and what it costs you in response latency. Step through the rungs to see what each layer fixed and what it left broken.

Every symbol and number

The equations

The debugging table — symptom to cause

The five things worth remembering in a year

1. VAD answers “is there speech?”; endpointing answers “is this person finished?” A perfect VAD does not solve endpointing, because the evidence is in the words, not the silence.

2. Every threshold you agonise over is a point on a curve. Adding a feature moves the curve. Prefer information to tuning.

3. Report p95, not the mean. Bimodal latency — fast most turns, timeout on the rest — feels worse than a uniformly slower system.

4. Barge-in is five steps, not one, and the one everybody forgets is repairing the context to what was heard rather than what was generated.

5. When a problem needs seven mechanisms to patch, check whether it is an artifact of your architecture. Turn detection was.

Where to go next

→ Moshi (Veanor) — the full paper walkthrough: Mimi’s semantic distillation, the RQ-Transformer, the delay patterns, Inner Monologue, and the turn-taking statistics of generated dialogue.
→ PersonaPlex (Veanor) — voice and role control for full-duplex models, and the benchmark that measures whether a model actually yields the floor.
→ The Voice Agent Stack (Nuranor) — the product-level teardown: cascaded versus speech-to-speech, the full latency ledger, telephony, evals, and safety.
→ VALL-E (Veanor) — where the speech tokens that make any of this possible came from.
→ TTS Architectures — the synthesis side, including why streaming time-to-first-audio behaves the way it does in the ledger.
→ Whisper — the recognition side, and why long-form decoding has its own turn-shaped problems.
→ Audio Representations — frames, spectra and mel bands, the substrate under every VAD feature in Chapter 3.
→ Neural Audio Codecs — how a waveform becomes the 12.5 Hz token stream Chapter 8 depends on.
→ Agents & Tool Use — the cancellation and idempotency problems from Chapter 6, in their general form.