Audio LLMs

Chapter 1: Encoder + Adapter + LLM

We ended Chapter 0 with a problem that sounds impossible: an LLM eats words, and sound is not words. So how does a waveform get inside?

The answer starts with a fact about transformers that is easy to forget. An LLM does not eat words. It eats a sequence of vectors. The tokenizer chops text into token ids, and then an embedding table — a big lookup matrix — converts each id into a vector of dimension d_llm (4096 for a 7-billion-parameter model, for instance). From the transformer’s point of view, the input is just a matrix of shape (sequence length, 4096). It has no idea those rows came from a dictionary.

Which means: if you can produce rows of the same width that carry sound, you can slide them into the same sequence and the attention layers will process them exactly as they process words. That is the entire trick. Everything else is engineering to make the rows good.

Trace one clip, every shape, by hand

Abstract arrows teach nothing. Take a real thirty-second clip and follow every number. We will use the shapes of a Whisper-large-style encoder because it is the most common audio front end in this family.

Step 1 — samples. Thirty seconds at 16,000 samples per second:

Step 2 — log-mel frames. The front end takes a short window every hop of 10 milliseconds, which at 16 kHz is 160 samples. The number of frames is the number of hops:

Step 3 — the encoder’s own downsampling. Whisper’s encoder begins with two convolutions, the second with stride 2, which halves the time axis before a single transformer layer runs:

Step 4 — what one frame means in seconds. This is the number worth memorizing, because it tells you how much time each vector “covers”:

Step 5 — the adapter shortens it. Qwen2-Audio pools adjacent encoder frames with a stride of 2, so two 20 ms frames become one:

Step 6 — re-dimension. A linear layer maps each pooled vector from the encoder width to the LLM width:

So a thirty-second clip enters the language model as 750 vectors — a prefix roughly the length of a page of text. Then your instruction is tokenized and embedded normally, appended, and the model decodes.

Why the sequence length is the whole design pressure

750 vectors for 30 seconds sounds cheap until you scale it. Attention cost grows with the square of the sequence length, and every audio vector competes with your instruction and the model’s answer for a fixed context budget. Run the arithmetic for a five-minute recording at 25 Hz:

And at the raw encoder rate of 50 Hz it would be 15,000. This is why every system in Chapter 3 spends its cleverness on the same question: how few vectors can carry this sound? Systems that target long audio — Audio Flamingo 2 handles inputs up to several minutes — must compress harder or re-architect the connection entirely.

It also costs memory in a way that is easy to overlook. Those 750 vectors are stored, and every one of them gets keys and values cached for generation. In half precision the prefix alone is

which is fine for one request and decidedly not fine when you are batching hundreds. Sequence length is not an abstract elegance concern; it is the number that decides how many concurrent users a GPU serves.

There are three families of answer, and you will meet all three:

The first is the simplest thing that works and is what we will compute by hand in Chapter 2. The second is what SALMONN uses, and Chapter 4 explains why a window-level variant matters for speech. The third keeps the text pathway untouched, which is attractive when you refuse to disturb the LLM at all.

Frozen or trained? The decision that defines the system

Three components, and for each you choose: freeze it, fine-tune it, or attach low-rank adapters (LoRA). The choices are not arbitrary — each has a reason you should be able to state.

The forward pass, in code

Here is the whole system as runnable-shaped PyTorch. Read the shapes in the comments; they are the same numbers we just computed by hand.

python
import torch, torch.nn as nn

class AudioLLM(nn.Module):
    def __init__(self, encoder, llm, d_enc=1280, d_llm=4096, stride=2):
        super().__init__()
        self.encoder = encoder            # frozen Whisper-style encoder
        self.llm     = llm                # frozen (or LoRA) language model
        self.stride  = stride
        self.proj    = nn.Linear(d_enc * stride, d_llm)   # the adapter

    def encode_audio(self, mel):            # mel: (B, 128, 3000)
        with torch.no_grad():                 # encoder frozen -> no grad, no optimizer state
            h = self.encoder(mel)             # (B, 1500, 1280)
        B, T, D = h.shape
        T2 = T // self.stride                 # 1500 // 2 = 750
        h = h[:, :T2*self.stride].reshape(B, T2, D*self.stride)   # (B, 750, 2560)
        return self.proj(h)                   # (B, 750, 4096)  <- LLM-shaped

    def forward(self, mel, text_ids, audio_slot):
        a = self.encode_audio(mel)                       # (B, 750, 4096)
        e = self.llm.get_input_embeddings()(text_ids)    # (B, L, 4096)
        # splice the audio vectors in where the <audio> placeholder sits
        e = torch.cat([e[:, :audio_slot], a, e[:, audio_slot+1:]], dim=1)
        return self.llm(inputs_embeds=e)             # logits over the text vocabulary

Three lines deserve a second look. inputs_embeds=e is the door: nearly every LLM implementation lets you bypass the embedding lookup and hand it vectors directly — that is the API-level reason this whole architecture is possible. reshape(B, T2, D*stride) is the pooling: it does not average, it stacks two neighbouring frames into one wider vector so the projection can decide what to keep (Chapter 2 does this by hand). And torch.no_grad() on the encoder is the frozen decision made concrete — no gradients stored, roughly a third of the memory saved.

Where exactly do the audio vectors go in the prompt?

“Splice them in” hides a decision with real consequences. In practice you write a prompt template containing a placeholder token, tokenize it normally, and then replace that one position with the whole block of audio vectors:

python
PROMPT = "<|user|>\n<audio>\n{question}\n<|assistant|>\n"

ids  = tok(PROMPT.format(question=q)).input_ids     # <audio> is ONE token id here
slot = (ids == AUDIO_ID).nonzero()[0]                # its position in the sequence
# after splicing, that single position becomes 750 positions

Two bugs live here and both produce systems that look almost fine. If you splice after the question, the model reads the question before it has any audio, and while attention is bidirectional over the prompt in the prefill pass, generation quality drops noticeably because the instruction no longer conditions how the audio is read. If you forget to lengthen the attention mask when the one placeholder becomes 750 vectors, the extra positions are masked out and the audio is silently ignored — which is indistinguishable from an undertrained adapter unless you check the mask shape.

A third question people ask: do the audio vectors get positional encodings? Yes — they occupy real positions in the sequence, so whatever positional scheme the LLM uses (rotary embeddings, for instance) applies to them exactly as it applies to words. That is how the model can know that audio vector 412 comes after audio vector 90 and both come before the question. Time order inside the clip is carried by sequence order, not by anything the adapter adds.

What the LLM sees, from its own point of view

Put yourself inside the transformer for a moment. Positions 1 to 750 hold vectors that arrived from an adapter; positions 751 onward hold vectors that arrived from an embedding table. The self-attention mechanism treats them identically — when the model is generating the word after “the customer sounded”, its query attends over all previous positions, and nothing stops it from putting most of its weight on audio position 412, where the voice rose.

This is why the pattern generalizes so well. It is the same pattern used for images (encoder + projector + LLM) and video. The modality changes; the interface does not. What differs for audio is the sequence length problem — an image is a fixed few hundred patches, while audio grows linearly with duration — and the fact that audio carries two nearly independent kinds of information, speech and everything else. That second fact is the subject of Chapter 4, and it is where audio stops being “images with a time axis.”

Chapter 2: What the Adapter Actually Does

Chapter 1 drew the adapter as a box labelled “pool × 2 + linear.” That box is where the modality actually crosses over, so we are going to open it and compute every number by hand — a tiny example small enough to do on paper, then the same operation in NumPy, then the one-liner you would really write.

The adapter has exactly two jobs, and it is worth separating them because they fail in different ways:

Job 1 — shorten the sequence. 1,500 encoder frames is too many vectors to spend on thirty seconds. Something must reduce the count.

Job 2 — change the space. The encoder’s 1280-dimensional vectors live in a space organized around acoustics. The LLM’s 4096-dimensional space is organized around meaning-in-language. These are different coordinate systems with different widths, and no amount of reshaping bridges them. A learned linear map does.

The toy problem, by hand

Shrink everything until it fits in your head. Let the encoder emit 4 frames of 2 dimensions each. Pretend dimension 1 responds to broadband noise and dimension 2 responds to voicing — not because real encoders are that tidy, but because it lets us read the arithmetic as a story.

Reading it as a story: the clip starts noisy and unvoiced, and by frame 3 a voice has taken over.

Step 1 — stack with stride 2. Take frames two at a time and concatenate them into one wider vector. Not average — concatenate. Two frames of width 2 become one frame of width 4:

The sequence went from 4 vectors to 2. Time resolution halved; nothing was thrown away, because the width doubled to hold it. Remember that phrase — nothing was thrown away — we are about to see what happens when you average instead.

Step 2 — project. Now a linear layer maps width 4 to the toy “LLM width” of 3. It has a weight matrix W of shape 4 × 3 and a bias vector b of length 3. Here are the numbers; I chose them so each output coordinate has a readable job.

Column 1 computes “noise minus voicing, averaged over the pair.” Column 2 sums the voicing dimension of both frames. Column 3 subtracts frame 2’s noise from frame 1’s — a change detector, which is only possible because we stacked instead of averaged.

Step 3 — do the multiplication for z₁, one coordinate at a time.

Step 4 — the same for z₂ = [0.1, 0.9, 0.0, 1.0].

Read the result as the LLM would. The first vector says “noise-dominated, no voicing, slight fall across the pair.” The second says “strongly voiced, almost no noise, steady.” Four acoustic frames became two vectors in a 3-dimensional space where the coordinates mean something the downstream model can use. That is the whole adapter. The real one has bigger numbers and learns W by gradient descent instead of being handed it, but the operation is exactly this.

Why stack and not average? A two-line proof

Suppose instead of concatenating we averaged each pair — a common and tempting choice. Compare two different clips:

Identical. Averaging has erased the order of events inside the window — and “which came first” was the exact question from Chapter 0. Now stack:

Distinguishable, and the projection can learn any function of both frames — including the average, if that turns out to be what helps. Stacking is strictly more expressive at the price of a wider (and therefore larger) weight matrix. This is why frame-stacking projectors are the common choice for speech, where 20-millisecond order matters enormously.

The same thing in NumPy, step by step

python
import numpy as np

H = np.array([[1.0, 0.0],      # h1  (4 frames, 2 dims)
              [0.8, 0.2],      # h2
              [0.1, 0.9],      # h3
              [0.0, 1.0]])    # h4

# Step 1: stack pairs -> (2 frames, 4 dims). reshape IS the stacking.
Z = H.reshape(2, 4)
# Z = [[1.0, 0.0, 0.8, 0.2],
#      [0.1, 0.9, 0.0, 1.0]]

# Step 2: the learned map, width 4 -> width 3
W = np.array([[ 0.5, 0.0,  1.0],
              [-0.5, 1.0,  0.0],
              [ 0.5, 0.0, -1.0],
              [-0.5, 1.0,  0.0]])
b = np.array([0.1, -0.2, 0.0])

# Step 3: one output coordinate the long way, to check our hand work
u1_1 = (Z[0] * W[:, 0]).sum() + b[0]
print(u1_1)          # 0.9   <- matches the paper calculation

# Step 4: all of it at once
U = Z @ W + b
print(U)
# [[ 0.9  0.0  0.2]
#  [-0.8  1.7  0.1]]   <- both hand-computed vectors, exactly

And the one-liner you would actually write, where the same matrix multiply hides inside a module:

python
import torch, torch.nn as nn
proj = nn.Linear(4, 3)                       # in real life: Linear(2560, 4096)
U = proj(torch.tensor(H, dtype=torch.float32).reshape(2, 4))
# same operation; nn.Linear stores W transposed and adds bias for you.

Three views of one computation: arithmetic on paper, an explicit dot product, a library call. If you can move between them you understand the adapter completely.

How big is the real adapter?

Take the Chapter 1 shapes: stride 2 over a 1280-wide encoder means the projection input is 2 × 1280 = 2,560, and the LLM width is 4,096. The weight count is

plus 4,096 biases, so about 10.5 million parameters. Set that against an 8.2-billion-parameter system:

One eighth of one percent of the weights is the entire bridge between two modalities. That ratio is why this architecture spread so fast: if you have a good audio encoder and a good LLM, the new thing you must train is small, cheap, and trains in hours rather than weeks.

The other two adapter designs, briefly

A Q-Former (query transformer) replaces pooling with attention. You create a fixed set of learned query vectors — say 32 — and let them cross-attend to the audio frames: the queries form Q, the audio frames form K and V. The output is always 32 vectors no matter how long the audio is, because the output length is the number of queries. Beautiful for compression, and it lets the model learn what to extract rather than mechanically averaging neighbours.

But apply that globally to a minute of speech and you have destroyed word order: 32 vectors cannot carry a sentence. SALMONN’s fix is the window-level Q-Former — run the same Q-Former independently on each short window (on the order of a second) and concatenate the results, so the output length still grows with duration and temporal order survives. Chapter 4 returns to this.

Gated cross-attention, the Flamingo design, does not put audio in the token sequence at all. Instead it inserts new attention layers into the LLM that attend from text positions to the audio features, each wrapped in a gate (a scalar, often through a tanh) initialized to zero. At step zero of training, the gate outputs exactly nothing and the model is bit-for-bit the original text LLM; as training proceeds the gate opens only as far as the data justifies. It is the most conservative way to add a modality, which is why it is favoured when preserving the base model’s language behaviour is paramount.

Chapter 3: The Lineage — four systems, four lessons

The pattern from Chapter 1 is simple enough that dozens of groups built it at once. What makes the history worth studying is that each landmark system failed in a specific way, and the next one’s design is a direct answer to that failure. Read this chapter as four bug reports and four fixes.

LTU — “listen, think, and understand” (2023)

LTU asked the first question: can a model do more than tag a sound — can it explain why? Its architecture is the minimal version of Chapter 1: an audio spectrogram transformer encoder (the AST family — a ViT applied to spectrogram patches), a small projection, and a LLaMA-class LLM adapted with LoRA.

The hard part was not the model, it was the data. There was no large corpus of “audio, open question, good answer” triples. So the authors built one, OpenAQA, on the order of five million tuples, by taking existing labelled audio datasets and using a strong text LLM to turn their metadata (labels, captions, timing) into questions and answers — a technique you will see again in Chapter 5.

Their key training finding is the one to remember. Train on open-ended reasoning data from the start and the model hallucinates: it produces confident, well-formed descriptions of sounds that are not there, because the text side of the objective is so much easier to satisfy than the audio side. Their fix was a perception-to-understanding curriculum — first train on closed-ended tasks (classification, captioning) where the answer is forced to depend on the audio, and only then open the task format up.

LTU’s own follow-up made the next step visible before anyone else did. LTU-AS bolts a Whisper model alongside the sound-event encoder so that the system can handle speech content and non-speech audio in one conversation — the same recognition that leads directly to the next system in this tour. Two teams arriving at “we need a second kind of listening” independently is a good sign that the problem is structural rather than incidental.

SALMONN — two ears (2023)

LTU could describe sounds. It could not reliably transcribe speech, because an AST-style encoder trained on sound events is not a speech recognizer. And Whisper-based systems had the mirror problem: superb on words, weak on the dog barking behind them.

SALMONN’s answer was to use both: a Whisper encoder for speech, a BEATs encoder for general audio, their frame sequences concatenated along the feature dimension, then a window-level Q-Former compressing each short window into a few vectors, feeding a Vicuna LLM with LoRA. Chapter 4 is devoted to why two encoders is not the redundancy it appears to be.

SALMONN’s second contribution is a training pathology worth memorizing. After instruction tuning on a big mixture of speech and audio tasks, the model got better at the trained tasks and worse at everything else — it would answer “tell me a story about this recording” with a bare transcription, because transcription is what the fine-tuning data mostly looked like. The authors call this task over-fitting, and their remedy — activation tuning, a lightweight extra stage on long-form generative tasks with the LoRA contribution scaled down — restored the model’s willingness to follow instructions it had never been trained on.

What came back with it were the abilities they call emergent: translating speech into languages the system was never trained to translate into, telling a story grounded in the recording, and answering questions that require the words and the sounds together (“is the speaker inside a moving vehicle?”).

Qwen-Audio → Qwen2-Audio — drop the tags (2023–2024)

Qwen-Audio scaled the recipe to more than thirty audio tasks at once and hit an interference problem: different tasks want different outputs from the same clip, and training them jointly with no signal about which task is intended makes the objectives fight. Their solution was a hierarchy of task tags — special tokens naming the language, the task, the dataset — so that shared structure could be shared and conflicting structure kept apart.

It worked, and it created a new problem: the model became fluent in a control language that users do not speak. Ask it something phrased differently from the tags and performance drops.

Qwen2-Audio removed the hierarchy and trained on natural language prompts instead. Same architecture family — a Whisper-large-v3-initialized encoder, a pooling adapter with stride 2 (the exact 1500 → 750 arithmetic from Chapter 1), and a Qwen-7B language model, roughly 8.2 billion parameters in total — but the task interface is now the thing users actually type. Training runs in three stages: large-scale pretraining, supervised instruction fine-tuning, and a preference-optimization stage (DPO) that aligns the model’s answers with human judgement.

The payoff is a system with two usage modes that need no mode switch: voice chat, where your speech is the instruction, and audio analysis, where the audio is the object and a text instruction asks about it. The model infers which is meant from the content. Chapter 7 builds a simulator of exactly this decision.

Audio Flamingo 2 — long audio and deliberate skills (2024–2025)

The Flamingo lineage attaches modalities through gated cross-attention rather than prefix tokens (Chapter 2), which brings two things audio needed: in-context learning from a few examples in the prompt, and multi-turn dialogue about the same recording.

Audio Flamingo 2 pushes on two axes that the prefix-token systems find hard. First, length: because audio does not enter the token sequence, minutes-long inputs do not blow up the context, and AF2 targets clips far longer than the thirty-second window most systems assume. Second, reasoning as a data problem: rather than hoping reasoning emerges, its training mixtures deliberately include skill-targeted question sets (temporal ordering, counting, attribute comparison) and long-audio comprehension data. It also demonstrates that a carefully trained 3-billion-parameter model can match or beat much larger ones on audio understanding — capability here tracks the data recipe more than the parameter count.

Notice the shape of the whole tour. LTU fixed a data problem with a curriculum. SALMONN fixed a representation problem with a second encoder, and a second data problem with a third training stage. Qwen2-Audio fixed an interface problem by deleting a mechanism. Audio Flamingo 2 fixed a capacity problem by changing where audio enters and authoring the reasoning data by hand. Exactly one of those four is an architecture change in the usual sense — the rest are decisions about training and data wearing architectural clothes.

The four design axes, side by side

How to read the next paper in this family

New audio LLMs appear constantly and they all describe themselves as “a general-purpose audio understanding model.” Five questions extract the actual content of any of them, and you can now answer all five from the architecture diagram alone:

Question five is the one that most quickly separates careful work from marketing, and it costs the authors almost nothing to answer.

Two modes without a mode switch

Qwen2-Audio’s two interaction modes deserve unpacking, because they are the clearest illustration of what “the instruction is data” buys you. In voice chat the audio is the instruction: you speak, and the model answers what you asked. In audio analysis the audio is the object: you type a question about a recording. The same weights, the same forward pass, and no special token telling the model which situation it is in — it infers that from content.

Think about what that requires. The model must learn a rule roughly like: if there is a text instruction, the audio is evidence; if there is no text instruction, the speech in the audio is the instruction. Nothing in the architecture encodes that rule. It is entirely a property of the training mixture — which is why Chapter 5, not Chapter 1, is where systems are actually won or lost.

If you were starting today

The lineage compresses into a decision procedure. Answer four questions in order:

Notice that none of these questions is “which paper is best?” They are all questions about your inputs and your users, and the architecture is a consequence.

A number worth carrying: how little of the system is new

For a Qwen2-Audio-shaped system, count what had to be created from nothing. The encoder is initialized from a released speech model. The LLM is a released text model. Only the adapter is new:

Everything else is inherited, and the training stages exist mostly to teach the inherited pieces to cooperate. That is the honest summary of this whole architecture family: the intelligence is borrowed; the alignment is earned.

And notice what did not change across two years and four systems: the three-piece shape from Chapter 1. Encoder, connector, language model.

Every difference in this chapter is a difference in one of those three boxes, or in the data flowing through them — and more often the latter. When a field converges this hard on a structure, it usually means the structure is not the hard part — which is exactly why the next two chapters are about ears and about data rather than about layers.

Chapter 4: Two Ears — why one encoder is not enough

SALMONN runs two audio encoders side by side over the same waveform. The first time you see that, it looks like waste — surely a good encoder encodes audio, and speech is audio. This chapter argues the opposite: the two encoders are close to orthogonal, and the reason is not architecture at all. It is the training objective.

An encoder is shaped by what it was asked to predict

Whisper’s encoder was trained inside a system whose job is to output a transcript. Ask what that objective rewards. It rewards keeping every acoustic detail that determines which words were said — formants, phoneme boundaries, timing. And it actively rewards throwing away everything else, because the speaker’s identity, the room’s reverb and the dog barking outside are all nuisance variables: they vary while the correct output stays the same.

A perfect ASR encoder would map every recording of the sentence “the invoice is late” — whispered, shouted, in a car, over a barking dog — to nearly the same representation. That invariance is not a flaw. It is the definition of a good speech encoder. It is also precisely what destroys the information you need to answer “was there a dog?”

BEATs was trained differently: self-supervised masked prediction over general audio, where the model must reconstruct discrete acoustic labels for masked spectrogram patches. Nothing in that objective mentions words. What it rewards is keeping timbre, texture, and event identity — and it has no particular incentive to align its representation with phonemes.

Make it numerical: the same two clips, two ears

Take clip A — someone says “hello” while a dog barks — and clip B — the same “hello” with no dog. Suppose each encoder gives a 3-dimensional summary vector, where by construction coordinate 1 tracks phonetic content, coordinate 2 tracks “animal sound present”, and coordinate 3 tracks “clean speech texture”. Plausible values:

Now measure how distinguishable the clips are to each ear, with cosine similarity. Every step:

Speech ear, dot product:

Magnitudes:

Cosine:

To the speech ear the two clips are the same clip. No downstream adapter, LLM, or prompt can recover the dog from a representation in which the dog left no trace. Now the sound ear:

Cosine 1.00 versus 0.21 on the identical pair of clips. That gap is the argument for two encoders, and it is why “just use a bigger speech model” does not fix it — scale sharpens an invariance, it does not remove it.

How the two streams are joined

The simplest join, and SALMONN’s, is concatenation along the feature dimension, frame by frame. If the speech encoder emits 1280 numbers per frame and the sound encoder emits 768, the joined frame is 2,048 numbers wide:

This has a precondition people forget: both encoders must run at the same frame rate, or frame t of one is not the same moment as frame t of the other, and you have glued together misaligned time. In practice you configure the encoders to a common rate (50 Hz is typical) or interpolate one onto the other’s time grid before concatenating.

Then the joined 2,048-wide stream goes into the connector — for SALMONN a window-level Q-Former, which takes each short window of joined frames and emits a small fixed number of vectors for that window. The “window-level” part is load-bearing: a single global Q-Former would emit, say, 32 vectors for the whole clip, which is plenty to say “a man speaks over traffic noise” and hopelessly too few to carry a sentence verbatim. Per-window queries keep the output length proportional to duration, so word order survives.

Frame alignment, worked out

“Both encoders must run at the same frame rate” is easy to say and easy to get wrong, so do the arithmetic. Suppose your speech encoder emits 50 frames per second and your general-audio encoder emits 100. Over 10 seconds:

Concatenating naively would pair speech frame 1 (covering 0–20 ms) with sound frame 1 (covering 0–10 ms), speech frame 2 (20–40 ms) with sound frame 2 (10–20 ms), and by frame 500 the two streams are describing moments five seconds apart. The clip would appear to contain a bark long after it happened.

The ratio is 100 ÷ 50 = 2, so the fix is to fold pairs of sound-ear frames into one:

python
# speech: (B, 500, 1280) at 50 Hz   |   sound: (B, 1000, 768) at 100 Hz
assert sound.shape[1] % speech.shape[1] == 0          # integer ratio, or interpolate
r = sound.shape[1] // speech.shape[1]                  # 2
sound = sound.reshape(B, speech.shape[1], r, 768).mean(2)  # (B, 500, 768)
fused = torch.cat([speech, sound], dim=-1)                   # (B, 500, 2048)

When the ratio is not an integer — a 75 Hz encoder against a 50 Hz one, ratio 1.5 — you interpolate one stream onto the other’s time grid instead. And note the choice made in that snippet: we averaged the two sound frames rather than stacking them, because here we are matching a time grid, not compressing a sequence, and the pair covers the same 20 ms the speech frame does. Chapter 2’s “always stack” advice was about reducing sequence length; this is a different operation with a different goal. Knowing which is which is the difference between an engineer and someone copying code.

The unification alternative

Qwen2-Audio takes the other road: one encoder, initialized from Whisper, but then trained further on a broad mixture that includes sound events and music, not only transcription. That training deliberately breaks the pure-speech invariance — the encoder is no longer allowed to discard the dog, because part of its objective now requires naming it.

Two roads, one goal:

Neither dominates. If you have the data budget to retrain an encoder, unification is cleaner at serving time. If you do not — and most teams do not — two frozen ears is the cheap way to buy a capability your speech model structurally cannot have.

Why not a third ear?

If two complementary encoders help, why stop there? A music-specific encoder would keep harmonic and rhythmic structure that neither Whisper nor BEATs is optimized for; a speaker-verification encoder keeps voice identity, which both of the others try hard to discard.

Nothing forbids it, and the connector cost is modest — the joined width grows and the projection grows with it. What limits the idea is diminishing complementarity. The first two ears were chosen because their invariances barely overlap; a third encoder trained on general audio will share most of BEATs’ sensitivities, so you pay a full forward pass for a small amount of new information. The test to run before adding an ear is the one from the cosine example above: encode a set of clips that differ in the dimension you care about, and check whether the candidate encoder separates them where the ears you already have do not. If it does not, you are buying compute.

There is also a data constraint people forget. Every new ear widens the connector, and the connector is the part you must train from scratch. Wider input means more parameters means more instruction data to fit them. Two ears with 10 million connector parameters trained on a solid mixture will beat four ears with 25 million trained on the same mixture.

Applying the invariance rule to your own problem

Suppose you are not building a general assistant at all — you are building something that listens to factory machines and answers questions about them. Which encoder?

Run the rule. A speech encoder is trained to discard everything that is not words, and a bearing beginning to fail is not words, so it will be discarded. A general-audio encoder trained on human-scale event categories keeps texture and timbre, which is much closer to what you need — but its categories were things like “engine” and “machinery”, so fine distinctions between two kinds of engine fault may be exactly the variation it learned to collapse. The honest answer may be that you need to train or fine-tune an encoder on your own audio, and the value of the rule is that it tells you this before you spend a month discovering it empirically.

Chapter 5: Instruction Tuning on Audio

The architecture is a week of work. The data is the year. Nothing in this field separates a demo from a system as sharply as what went into the instruction mixture — so this chapter builds an audio instruction dataset from nothing, and then shows the arithmetic that decides what the model becomes.

Start from the hole in the world. To train “audio + instruction → answer” you need triples of exactly that form, and no such corpus exists at scale. What does exist is decades of labelled audio: transcribed speech, tagged sound events, captioned clips, timestamped detections. Every one of those is an instruction dataset wearing a disguise.

Source 1: templating — labels are answers to questions nobody wrote down

A clip tagged {Dog, Bark, Domestic animals} is already an answer. Supply the question:

python
TEMPLATES = [
    ("What sounds can you hear?",              "I hear {labels}."),
    ("Describe the audio in one sentence.",     "A recording containing {labels}."),
    ("Is there a {probe} in this recording?",   "{yes_no}"),
    ("List every sound event you notice.",      "{labels}"),
    ("Answer with one word: what is the main sound?", "{top_label}"),
]

def make_sample(clip):
    q, a = random.choice(TEMPLATES)
    probe = random.choice(ALL_LABELS)              # half the time an absent label!
    return {
        "audio":  clip.wav,
        "prompt": q.format(probe=probe),
        "answer": a.format(labels=", ".join(clip.labels),
                          top_label=clip.labels[0],
                          yes_no="Yes." if probe in clip.labels else "No."),
    }

Look at the comment on the probe line, because it is the whole craft in miniature. If you only ever ask about labels that are present, every correct answer is “Yes”, and the model learns to say yes without listening. Balanced negatives are not a nicety; they are the only thing making the question require the audio.

Source 2: LLM-assisted expansion — and its built-in hazard

Templates give you coverage but not diversity, and they never produce a question like “which happened first?” The standard move — used to build LTU’s OpenAQA and many mixtures since — is to hand a strong text LLM everything you know about a clip in text form and ask it to write varied questions and answers:

python
meta = {
  "labels":      [("Speech", 0.0, 4.2), ("Dog", 2.6, 3.1), ("Door", 7.9, 8.3)],
  "transcript":  "the invoice is late again",
  "caption":     "a man speaks indoors while a dog barks, then a door closes",
}
prompt = "Here is everything known about a 10-second clip. Write 5 question/answer " \
         "pairs. Every answer MUST be derivable from the facts given. If a question " \
         "cannot be answered from these facts, do not ask it."

The hazard is structural and you should say it out loud: the writing model never hears the audio. It works from metadata. So if it embroiders — “the dog sounds distressed, likely a small breed in the next room” — you have just written a training target that the audio does not support. Train on enough of those and you are not teaching perception, you are teaching the model to confabulate confidently. That is the origin of a large share of audio-LLM hallucination, and it is a data bug, not a model bug.

Source 3: human-authored skill data

Some abilities never fall out of repurposed labels: ordering two events, counting repetitions, comparing two segments, deciding that a question is unanswerable. Audio Flamingo 2’s approach — authoring skill-targeted question sets on purpose — is the honest acknowledgement that reasoning is a data category, not an emergent bonus. It is expensive per example, which is exactly why the mixture arithmetic below matters so much.

The mixture arithmetic that decides what your model becomes

Suppose you assemble one million supervised examples: 60% ASR, 30% captioning, 10% open question answering. Sounds reasonable. Now do the arithmetic the optimizer actually does.

Step 1 — examples per task:

Step 2 — the crucial correction: loss is computed per token, not per example. Only the answer tokens are scored (the prompt and the audio positions are masked out), so what matters is how many answer tokens each task contributes. Say a transcript averages 30 tokens, a caption 15, a QA answer 25:

Step 3 — shares of the gradient:

You designed a 60/30/10 mixture and trained a 72/18/10 model. Nearly three quarters of every gradient step says “when in doubt, transcribe.” This is the mechanism behind SALMONN’s task over-fitting: ask for a story and get a transcript, because transcription is where the loss lived.

Step 4 — rebalance by tokens, not examples. Suppose you want the three tasks to contribute equally. Fix a token budget of 25 million and give each task one third, 8,333,333 tokens:

Note how counterintuitive the result is: to give captioning an equal say you need twice as many captioning examples as ASR examples, purely because captions are short. Nobody discovers this by intuition; you discover it by multiplying.

The loss mask, in code

Since the token-share argument depends entirely on which tokens are scored, here is the construction in full. Get this wrong and every number above is meaningless.

python
IGNORE = -100          # PyTorch's cross-entropy ignore index

prompt_ids = tok(prompt).input_ids            # e.g. 24 tokens
answer_ids = tok(answer).input_ids            # e.g. 30 tokens
n_audio    = 750                              # adapter output length

input_ids = prompt_ids + answer_ids
labels    = [IGNORE]*n_audio \                # audio positions: never scored
          + [IGNORE]*len(prompt_ids) \        # the instruction: never scored
          + answer_ids                        # ONLY these contribute loss

assert len(labels) == n_audio + len(input_ids)
# this example contributes exactly 30 token-losses, no matter how long the audio was

Read the last comment again: a 30-second clip and a 3-second clip contribute the same loss weight if their answers are the same length. Long audio therefore gets no extra say in training merely by being long — another reason the mixture must be balanced deliberately rather than assumed.

Order matters inside a stage, too

One more subtlety the arithmetic hides: shuffling is not neutral. If the instruction-tuning stage happens to present all the transcription data first and the reasoning data last, the model spends most of training in a transcription regime and then gets a brief correction — which is a curriculum you did not intend. Shuffle within the stage, and if you deliberately want a curriculum (perception first, as LTU found), make it explicit as separate stages rather than an accident of file order.

Phrasing diversity is a mixture axis too

The same arithmetic applies inside a task. Suppose all 600,000 ASR examples use one instruction, “Transcribe the audio.” The model has now seen that exact string 600,000 times and every paraphrase zero times. It has effectively learned a tag — the very thing Qwen2-Audio removed — except now the tag is disguised as English, so nobody notices until a user writes “what are they saying?” and the model captions instead.

Spread the same 600,000 examples over 20 phrasings and each is seen 30,000 times — still an enormous number, more than enough to learn the task, while the invariant across them is the meaning rather than the string:

Diversity here costs nothing: no new audio, no new labels, just more templates. It is the cheapest robustness in the entire pipeline and the most commonly skipped.

The three-stage recipe, and why the order is not negotiable

Stage 3 is the one people skip, and it is the one that decides whether the system feels like an assistant or like a captioning API with a chat box on top.

Chapter 6: Audio as Tokens — the other path

Every system so far shares one limitation, and it is easy to miss because it is baked into the diagram: audio goes in, text comes out. These models can listen. They cannot speak.

If you want a voice assistant you bolt on a text-to-speech engine, and the moment you do you have a pipeline: speech → text → reasoning → text → speech. Every arrow costs latency, and two of them are lossy in a way that matters. The hesitation before “fine”, the rising pitch of a real question, the laugh in the middle of a sentence — none of that survives the trip through text, in either direction. The system hears words and speaks words, and the conversation is flattened at both ends.

The second path removes the text bottleneck entirely: make audio into tokens and let the language model predict them exactly as it predicts words. One objective, next-token prediction, over a vocabulary that contains both language and sound.

Where audio tokens come from

The machinery is a neural audio codec — a convolutional encoder that compresses the waveform into a low-rate sequence of vectors, a residual vector quantizer (RVQ) that snaps each vector to entries in a stack of learned codebooks, and a decoder that reconstructs the waveform. The quantizer’s output is what we want: integers. Integers are tokens.

“Residual” describes the stacking. The first codebook picks the nearest entry to your vector; you subtract it; the second codebook quantizes what is left over; you subtract again; and so on. Each level adds detail, so you can trade quality for bitrate by simply keeping fewer levels.

Token-rate arithmetic, by hand

The single number that decides whether this is practical is tokens per second. Compute it for a codec running at 12.5 frames per second with 8 codebooks:

And the bitrate, if each codebook holds 2,048 entries (so each index needs log₂ 2048 = 11 bits):

Now the comparison that explains every design decision in this chapter. Ordinary speech runs about 3 words per second, which is roughly 4 text tokens per second. So:

Thirty seconds of speech is about 120 text tokens, or 3,000 audio tokens. And a codec at a higher frame rate makes it far worse — at 75 frames per second with 8 codebooks you would have

which no conversational model can afford. This is why low frame rate is the headline number of codecs built for language models: Mimi, the codec inside Moshi, runs at 12.5 Hz — one frame every 80 milliseconds — specifically so that a language model can afford to think in it.

Residual quantization, by hand

“Each level quantizes the leftover” is worth doing with numbers, because it explains why quality scales smoothly with the number of codebooks. Take a 2-dimensional vector (real ones have hundreds of dimensions; the mechanics are identical):

Level 1. The first codebook holds, say, three entries:

Squared distances to x, computed one at a time:

A wins, so the first token is index A. Subtract it to get the residual — the part the first codebook could not express:

Level 2. A second, finer codebook quantizes r₁. Suppose its nearest entry is [0.10, −0.05]; the second token is that index and the new residual is

Reconstruction quality. Compare the error after each level, as squared magnitude:

A 36-fold reduction in error for one extra token per frame. That is the whole bandwidth dial: keep the first two codebooks for a low-bitrate stream, keep eight for high fidelity, and the same encoder and decoder serve both. And it is why the number of codebooks appears in the token-rate arithmetic — each level you keep multiplies your token count by one more per frame.

AudioLM: why one kind of token is not enough

The first system to make this work properly, AudioLM, discovered a split that still organizes the field.

Acoustic tokens (from a codec, as above) reconstruct audio beautifully — you can decode them back to a waveform that sounds like the original. But a language model trained on them wanders: the tokens encode how it sounds, not what it means, so long-range structure (staying on topic, finishing a sentence sensibly) is weak.

Semantic tokens come from quantizing the hidden states of a self-supervised speech model (cluster them with k-means and use the cluster index). They capture phonetic and linguistic content, so long-range structure is good — but you cannot reconstruct audio from them, because everything about timbre and speaker identity was discarded.

Neither alone works. AudioLM’s answer is a hierarchy: first model the semantic token stream (what is being said), then predict coarse acoustic tokens conditioned on it (how it broadly sounds), then fine acoustic tokens (the detail). Continuations stay coherent for seconds and still sound like the original speaker — and the same trick works on piano, where “semantics” means musical structure.

Moshi: speaking and listening at the same time

Once the model predicts audio tokens, spoken output is free — and the door opens to something a pipeline can never do. Real conversation is full-duplex: both parties are producing and receiving continuously, overlapping, interrupting, saying “mm-hm” while the other talks. Turn taking is not a protocol imposed on speech; it is an emergent outcome of both sides listening while they speak.

Moshi models this directly. Three ideas do the work:

Two streams, always on. The model predicts its own audio tokens and models the user’s stream at the same timestep, so it is never “not listening.” There is no voice-activity detector deciding whose turn it is, because there are no turns in the architecture — silence is just tokens that happen to be quiet.

Inner monologue. At each frame the model predicts a text token before the audio tokens for that frame. The text acts as a scaffold that keeps the speech linguistically coherent — it is the system thinking in words a moment before it says them. It also gives you a transcript for free.

A 12.5 Hz codec. Everything above only fits in a real-time budget because a second of audio costs 12.5 frames rather than 75.

The latency arithmetic that makes the case

Add up a conventional voice pipeline, with generous but realistic numbers:

Now the token-native version. One frame at 12.5 Hz is

and with one frame of acoustic delay in the generation scheme the theoretical floor is

Under about 200 milliseconds a response stops feeling like a reply and starts feeling like a conversation — roughly the gap between human turns. The 700-millisecond silence timeout alone puts the pipeline out of reach, and no amount of engineering on the other three stages fixes it, because that stage exists purely to answer “are they done talking?” — a question full-duplex modelling never has to ask.

One consequence deserves stating plainly: a model that generates audio can generate a voice, and a voice can be made to resemble a real person’s. This is why work in this lineage ships alongside detection — AudioLM, for instance, trained a classifier to recognize its own generated speech — and why watermarking generated audio is treated as part of the system rather than an afterthought.

Chapter 7: The Router (showcase)

Everything now goes in one picture. A clip enters, both ears encode it, the adapter fuses and shortens, the language model attends over the resulting vectors, and an answer comes out — and crucially, which vectors the model leans on changes with the prompt. That last part is the thing this simulation exists to show, because it is invisible in every architecture diagram ever drawn.

How to read the panel

Row 1, the clip. Twenty-four 20-millisecond frames, drawn as two overlaid energies: teal for speech content, purple for non-speech events. Change the input to see a spoken phrase, a dog barking, both at once, or music.

Rows 2 and 3, the two ears. Each ear’s per-frame response, using the invariance logic from Chapter 4: the speech ear is loud where words are and nearly deaf to the bark; the sound ear is the mirror image. Notice they are not the same picture — that difference is the entire justification for running two encoders.

Row 4, the adapter output. Pairs of frames are fused and projected (Chapter 2’s stride-2 stacking), so 24 frames become 12 vectors. Each vector is coloured by which ear dominates it.

Row 5, attention mass. The height of each bar is how much of the language model’s attention lands on that audio vector while it answers. This is where the routing becomes visible: switch the prompt from “transcribe” to “what sounds do you hear” without touching the audio, and watch the mass migrate from the speech-dominant vectors to the sound-dominant ones.

Row 6, the answer. Generated left to right while the animation plays.

Where those attention numbers come from, by hand

The bars are a softmax over a compatibility score, exactly like real attention. Give each audio vector a score — how well it matches what the prompt is looking for — and then normalize. Take three vectors with scores 2.0, 1.0 and 0.5:

Three numbers summing to 1.000, and the top one takes 63% of the mass off a score lead of just 1.0. That exponential sharpening is why routing looks so decisive: a modest preference for speech-dominant vectors turns into a strong concentration of attention on them.

Make it concrete with three of the panel’s own vectors. Suppose vector 1 is speech-dominant with s = 0.80, b = 0.10; vector 2 is mixed with s = 0.40, b = 0.45; vector 3 is sound-dominant with s = 0.05, b = 0.85. Under the “transcribe” prompt the weights are w_speech = 1.00, w_sound = 0.05, so the raw scores are

Multiply by the sharpening factor of 4 before the softmax — real attention divides by the square root of the head dimension, which plays the same role — and you get 3.22, 1.69 and 0.37. Exponentiate: 25.0, 5.4 and 1.4, summing to 31.8, so the attention shares are about 79%, 17% and 4%. Now switch the prompt to “what sounds” (weights 0.10 and 1.00) and vector 3’s score becomes (0.10)(0.05) + (1.00)(0.85) = 0.855, the largest of the three. Same audio, inverted attention.

In the simulation the score for vector j is

where s_j and b_j are the two ears’ energies in that vector and the weights come from the prompt. “Transcribe” sets w_speech high and w_sound near zero; “what sounds do you hear” does the reverse; the ordering question wants both, which is why its attention spreads.

A note on what to watch. The answer row is the least informative part of the panel — it is scripted, and in a real system it would be the only thing you could see. The interesting rows are 2, 3 and 5, because they are the internals a deployed model does not show you. Learning to predict what those rows will do before you move a slider is the skill this chapter is training.

Six experiments — run them, do not just read them

1. Same audio, different question. Set input to both and flip the prompt between “transcribe” and “what sounds”. The audio never changes; the attention mass moves across the panel. This is the single most important behaviour in the lesson.

2. Ask for words where there are none. Input music, prompt transcribe. Watch the speech-ear row stay flat while attention still tries to concentrate there, and read the answer. A weaker system would invent lyrics here — this is the exact configuration that produces speech hallucination in the wild.

3. Break it with noise. Push background noise to 1.0 with input spoken phrase. The sound ear lights up everywhere, the fused vectors lose their speech dominance, and the routing blurs. Noise does not merely reduce accuracy — it changes which vectors the model reads.

4. Ask an ordering question of a single event. Input dog bark, prompt which came first. There is nothing to order. Watch whether the answer says so.

5. Compare attention spread. The ordering prompt should produce the flattest attention profile of the four, because it needs evidence from both ears across the whole clip. Confirm it.

6. Step frame by frame. Use step with input both to watch the answer being generated while the attention pattern stays fixed — the audio prefix is computed once; generation reads it many times.

What the panel simplifies, and what it does not

Three honest simplifications. Real attention has many heads and many layers, each with its own pattern, so “the attention” is a summary of dozens of different distributions. Real prompt weights are not two numbers — they are learned query vectors of dimension 4096. And a real answer is generated with a key-value cache, so the audio keys are computed once and reused for every output token, which is why long answers over long audio are cheaper than the quadratic worst case suggests.

What the panel gets exactly right is the part people get wrong: the audio representation is computed once and read many times, and what is read depends on the question. If you can measure attention mass on audio positions in a real model, you have a genuine diagnostic:

python
out = model(inputs_embeds=e, output_attentions=True)
# attentions[layer]: (B, heads, query_pos, key_pos)
A = out.attentions[-8].mean(1)              # average the heads of a mid-late layer
audio_mass = A[:, -1, :n_audio].sum(-1)      # mass the last query puts on audio
print(audio_mass.item())
# healthy: varies with the question, and drops when you feed silence
# broken:  near zero, or identical for every clip -> the audio path is dead

Run that with two different clips and the same prompt. If audio_mass is identical to three decimal places, you have found the bug before your users did.

Experiment 3 deserves a sentence of theory. Noise does not add a separate “noise” signal that the model can subtract; it raises the sound ear’s response everywhere, which changes which vectors are sound-dominant, which changes the scores, which changes the attention. A degradation at the input becomes a routing change three stages later. That chain — input quality to representation to attention to answer — is the mental model to keep when a deployed system suddenly starts answering differently on noisy recordings.

One last thing to take from this panel. Every architecture diagram in Chapter 3 drew a single arrow from the adapter into the LLM, as though the audio were handed over once and consumed. Row 5 is what that arrow actually is: a distribution, recomputed for every generated token, over hundreds of audio positions. When someone says a model “attends to the audio”, this is the object they are describing — and when a model ignores the audio, this is the object that goes flat.

Chapter 8: Evaluation — grading a model that answers in sentences

We spent Chapter 0 celebrating the death of the fixed output space. Now send the bill. Every classical audio metric — accuracy, mean average precision, word error rate — assumes you know the shape of a correct answer in advance. When the output is an arbitrary sentence, what does “correct” even mean?

The field’s answer is three regimes, each buying scorability at a different price.

Regime 1: force it into a box (multiple choice)

Put the model back in a cage on purpose. Ask a question, offer options A–D, and score the letter. This is what AIR-Bench’s foundation track does across roughly nineteen tasks and on the order of nineteen thousand single-choice questions covering speech, sound and music — including a deliberately mixed-audio subset where several sources overlap, which is where single-source systems fall over. MMAU takes the same approach with about ten thousand clips and human-written questions spanning speech, environmental sound and music, organized around a large set of distinct reasoning skills.

MMAU is worth knowing for one uncomfortable fact: humans score in the low eighties, and the strong systems at release scored far below that — roughly half the questions right. Whatever “understanding” these models have, it is not yet human-level on questions humans find straightforward.

The cost of this regime is that multiple choice is easy to game. Which brings us to the control that separates a real benchmark from a press release.

The audio-blind baseline — run it before you trust any number

Take your benchmark and delete the audio. Feed the model only the question and the options. Whatever it scores now, it scored without listening. Language priors are strong: “What is the person most likely doing while a lawnmower runs?” has an obvious answer to anyone who has never heard the clip.

Do the arithmetic on a plausible result. Four options, so a coin-flipping model gets

Suppose the audio-blind model gets 52% and the full model with audio gets 58%. The naive read is “58%, a strong result.” The honest read subtracts the blind baseline and normalizes by the room that was left:

Twelve and a half percent of the available headroom came from hearing anything at all. Six points of the fifty-eight are audio; the other fifty-two are a language model doing common sense. Report that number and the benchmark starts telling the truth.

Regime 2: open-ended, judged by a model

Multiple choice cannot test the ability we actually built — free-form explanation. So AIR-Bench also has a chat track: on the order of two thousand open questions, answered in prose, graded by a strong text LLM against a reference answer.

This works well enough to be useful and has two failure modes you must hold in mind. First, the judge is deaf. It grades text against text, so it cannot detect a claim that is fluent, consistent with the reference, and untrue of the audio. Second, judges reward length and confidence; a hedged, accurate answer often loses to a verbose, wrong one. Both are the same blindness we met in Chapter 5 when a text LLM wrote the training data — and it is why open-ended scores should be read as “plausible-sounding-ness” with a large error bar.

Regime 3: keep the old metrics where they still apply

Do not throw away word error rate. If your system claims transcription, score transcription the way speech people always have; free-form output can be right in spirit and useless in practice (“the caller discussed an invoice” is not a transcript). The same goes for captioning metrics and classification accuracy on standard sets. A serious evaluation report has all three regimes, because each catches what the others miss.

Word error rate, computed by hand

Regime 3 keeps the classical metrics, so make sure you can compute the main one. Word error rate counts the edits needed to turn the hypothesis into the reference, divided by the reference’s length:

where S is substitutions, D is deletions, I is insertions, and N is the number of words in the reference. Reference: “the invoice is late again”, so N = 5.

Hypothesis 1: “the invoice was late again”. Align word by word: the=the, invoice=invoice, was replaces is (one substitution), late=late, again=again.

Hypothesis 2: “the invoice late again” — is is missing, one deletion:

Hypothesis 3: “the caller discussed an invoice”. This is a summary. Aligned against the reference it is roughly four substitutions:

And here is why an audio LLM breaks this metric. Hypothesis 3 is arguably a good answer to “what happened in this call?” and a terrible transcript. WER punishes it savagely, as it should — but only if transcription was the task. A free-form model asked an open question will often produce something that scores near 100% WER while being entirely correct, which is precisely why you cannot grade these systems with a single metric. Score transcription with WER when you asked for a transcript, and use the other two regimes otherwise.

How many questions do you need before a difference is real?

Benchmark tables are full of one- and two-point differences presented as progress. Check whether they can be. For a proportion measured on n independent questions, the standard error is

Take a model at p = 0.55 on n = 1,000 questions:

A rough 95% interval is about two standard errors either side, so this measurement is 55% ± 3.1 points. Comparing two models measured this way, the difference has a standard error of roughly √2 × 1.6 = 2.2 points, so anything under about 4.4 points is indistinguishable from noise.

Now revisit the earlier example: the audio contributed 6 points over the blind baseline. On 1,000 questions that is real but not comfortable. On a 200-question subset, SE would be √(0.2475 ÷ 200) = 0.035, i.e. 3.5 points, and the same 6-point gain would be barely outside noise. When a paper reports per-skill breakdowns over a few hundred questions each, treat the ranking of skills as a hypothesis, not a finding.

The evaluation harness you should actually write

python
def evaluate(model, items):
    """items: list of (wav, question, options, gold_letter)"""
    full, blind, swapped = 0, 0, 0
    for k, (wav, q, opts, gold) in enumerate(items):
        full    += model(wav,            q, opts) == gold   # the headline number
        blind   += model(SILENCE,        q, opts) == gold   # no audio at all
        other    = items[(k + 1) % len(items)][0]              # a DIFFERENT clip
        swapped += model(other,          q, opts) == gold   # mismatched audio
    n = len(items)
    F, B, S = 100*full/n, 100*blind/n, 100*swapped/n
    print(f"full {F:.1f}  blind {B:.1f}  swapped {S:.1f}")
    print(f"normalized audio gain {(F-B)/(100-B)*100:.1f}%")
    # S should be close to B. If S is close to F, the model is not using the audio.

That third column — the mismatched-audio score — is the one almost nobody reports and the one that catches the worst bug. A model that scores well with the wrong clip attached is not doing audio understanding at all, and no amount of headline accuracy changes that.

What “audio reasoning” actually means

The phrase is used loosely. Pin it down: reasoning is when the answer requires more than one piece of acoustic evidence combined by a rule. Recognition is one step; reasoning is at least two.

That last row deserves its own evaluation. Build a set of questions about sounds that are definitively absent and measure the false-positive rate. Many systems that look strong on standard benchmarks will cheerfully confirm a helicopter that was never there, because their training data (Chapter 5) was overwhelmingly about what is present.

The report you should publish

Put it together. A trustworthy evaluation of an audio LLM has these rows, and the last three are the ones that make the first three mean anything:

None of these requires new data collection. They are all re-runs of the benchmark you already have, with one input changed — which is what makes their absence a choice rather than an oversight.

A word on human baselines, which MMAU reports and most benchmarks do not. A human score is not a ceiling, it is a calibration: it tells you whether the questions are answerable at all from the audio provided. If humans score 60% on your benchmark, either the clips are ambiguous or the questions are unfair, and no model result on it means much. If humans score in the eighties and models score around half, you have localized a real gap rather than a measurement artifact — which is the entire value of collecting the baseline.

And one closing warning about benchmark culture. Every number in this chapter can be improved without improving the model: by choosing easier distractors, by sampling clips where the priors happen to be right, by reporting the subset where your system does well. The defence is not suspicion of other people’s results — it is publishing the controls with your own, so that a reader can compute the normalized gain themselves rather than trusting your headline.

Chapter 9: Failure Modes, Cheat Sheet & Connections

Audio LLMs fail in ways that look like success. The output is always fluent, always confident, always well-formed — so unlike a classifier, which fails by outputting a visibly wrong label, these systems fail by outputting a beautiful paragraph about a recording they did not hear. This chapter is the field guide: four failure modes, the mechanism behind each, and the test that exposes it.

Failure 1: hallucinated sounds

Symptom. The model describes events that are not in the clip — usually plausible companions of events that are there. Speech in a kitchen becomes “dishes clattering.”

Mechanism. Chapter 5. The training answers were written by a text model that never heard the audio, so they contain claims the audio does not support; and if the data lacks balanced negatives, “yes” is almost always the right answer, so the model learns to affirm.

Test. Absence probes. Ask about five sounds you know are not present and count confirmations. A healthy system says no.

Fix. Entailment-filter the generated data, balance negatives, and put closed-ended perception data first in the curriculum so the audio path becomes load-bearing before open-ended answers are allowed.

Failure 2: the audio is ignored entirely

Symptom. Answers are generic and change when you change the question but not when you change the clip.

Mechanism. The language prior is a much easier route to low loss than the audio path (Chapter 2’s untrained-adapter diagnostic, Chapter 3’s LTU curriculum finding). If the adapter is undertrained, mis-scaled, or the alignment stage was skipped, the gradient never had to travel through it.

Test. The swap test, and it is the single most valuable diagnostic in this lesson: run the same prompt against two very different clips and compare answers verbatim. Then run a mismatched pair — question about clip A, audio from clip B — and see whether the model notices. If the answers are identical, your audio path is decorative.

Fix. Retrain the alignment stage; check that audio positions actually receive attention mass; verify the splice index (a common bug is inserting the audio vectors at the wrong position, so they land after the question or get truncated away).

Failure 3: task over-fitting

Symptom. Every request is answered in the format of the dominant training task. Ask for a story, get a transcript.

Mechanism. Chapter 5’s token-share arithmetic: a 60/30/10 example mixture became a 72/18/10 gradient, and the model’s default behaviour collapsed onto the majority.

Test. Ask something no training task resembles — “write two sentences of stage directions for this recording” — and see whether the format follows the instruction.

Fix. Rebalance by tokens rather than examples, diversify phrasing, and add the third stage (activation tuning or preference optimization) whose entire purpose is restoring instruction-following.

Failure 4: time blindness on long audio

Symptom. Correct about what is in the clip, wrong about when, how many times, or in what order — and it degrades fast as the clip gets longer.

Mechanism. Compression. Every design in Chapter 1 buys context savings with time resolution: 20 ms frames become 40 ms vectors, and a global resampler can flatten a whole clip into a handful of vectors that cannot express order at all. Counting also requires segmenting repetitions, which pooled representations blur.

Test. A clip with two clearly separated events. Ask which came first, then swap the events and ask again. A model that gives the same answer both times is reading its prior, not the clip.

Fix. Per-window connectors instead of global ones (Chapter 4), skill-targeted temporal training data (Chapter 3’s Audio Flamingo 2 lesson), and honest limits on input length.

The cheat sheet

What it costs to serve

One more number before the checklist, because it decides whether your design survives contact with production. Compare the prefill work for the two ways of answering a question about a 30-second clip.

Transcribe-then-prompt: the transcript is about 120 text tokens, plus a 30-token question, so the LLM prefills roughly 150 positions.

Audio LLM at 25 Hz: 750 audio vectors plus a 30-token question, so about 780 positions.

You are paying roughly five times the context for the ability to hear tone and background. Sometimes that is obviously worth it and sometimes it obviously is not, and the decision is per product, not per architecture. The most common production shape is a hybrid: route the cheap, high-volume questions (“what did they say?”) to the recognizer, and send only the questions that need ears to the audio LLM.

The pre-ship checklist

Everything above, compressed into things you can actually run the afternoon before shipping. Each line has a chapter behind it.

Nine checks, none requiring a GPU cluster, and between them they catch every failure mode in this chapter.

Where this is going

Two directions are worth watching, and both are visible in what you have already learned.

The paths merge. There is no reason a system cannot take audio in as continuous features (deepest understanding, Chapters 1–5) and emit audio out as codec tokens (lowest latency, Chapter 6). The two designs answer different questions and compose cleanly, and systems that do both are becoming ordinary.

Audio becomes a tool-using agent’s sense. Once a model can answer arbitrary questions about sound, it can be asked those questions by another program — a monitoring loop asking “did anything unusual happen in the last minute?”, an agent deciding whether to escalate a call. The interface that made this lesson possible, f(audio, instruction) → text, is exactly the interface a tool call wants. The evaluation discipline of Chapter 8 matters far more in that setting, because nobody is reading the answers.

The two paths, one diagram

Five things to remember when everything else fades

1. An LLM eats vectors, not words — which is the only reason any of this is possible.

2. An encoder’s invariances are the negative image of its training objective, so choosing an encoder is choosing what your system will be deaf to.

3. The adapter is about 0.1% of the parameters and 100% of the alignment; nothing else in the stack is new.

4. The optimizer sees tokens, not examples — so the mixture you designed is not the mixture you trained.

5. Fluent output is not evidence of listening. Run the swap test.

Keep exploring

← Audio Representations — the log-mel front end every encoder in this lesson starts from
← Whisper — the speech encoder that became the default first ear
← BEATs — the self-supervised general-audio encoder that became the second ear
← CLAP — open-vocabulary audio understanding before generation was possible
← Self-Supervised Speech — where semantic tokens come from
→ Qwen2-Audio — the paper-grade walkthrough of the encoder+adapter+LLM design
→ AudioLM — semantic and acoustic tokens in full detail
→ Neural Audio Codecs — how a waveform becomes integers
→ TTS Architectures — the other half of a speaking system
→ Vision-Language Models — the identical encoder+projector+LLM pattern, one modality over