ACT: Action Chunking
with Transformers

Chapter 1: The Key Insight

ACT's whole contribution can be said in one sentence: instead of predicting one action at a time, predict a chunk of the next k actions at once, and treat that chunk as the output of a generative model rather than a single best guess. Everything else in this lesson — the transformer, the CVAE, the temporal ensembling — exists to make that one sentence work in practice. This chapter is about the sentence itself.

Where "chunking" comes from

The word is borrowed directly from psychology and neuroscience, not invented for robotics. Action chunking is the well-documented phenomenon that biological motor systems don't plan and execute one muscle twitch at a time — they group sequences of primitive actions into a single unit and fire that unit as a whole. Tie your shoelaces and you are not consciously sequencing forty individual finger movements; you are executing a handful of learned chunks ("make a loop," "wrap around," "pull through") that were themselves built out of finer motions long ago and compiled into one habit. Chunking is efficient to store and efficient to execute, in a nervous system exactly as in a robot policy.

The formal version

Standard behavioral cloning learns a policy that maps the current observation to a single action:

ACT instead learns a policy that maps the current observation to a whole sequence of future actions:

Every symbol, defined:

• π_θ — the policy: a neural network with learned parameters θ (in ACT's case, a Transformer encoder–decoder, Chapter 6).
• s_t — the observation at timestep t: the four camera images (480×640 each, top / front / two wrists) plus the current 14-dimensional joint state of both arms.
• a_t:t+k — the action chunk: the sequence [a_t, a_t+1, …, a_t+k−1], i.e. k target-joint-position vectors, one per future timestep, produced from a single forward pass over s_t alone.
• k — the chunk size, a fixed hyperparameter. ACT uses k = 100. At ALOHA's 50Hz control rate, 100 steps is exactly 2 seconds of committed motion per decision.

The second thing chunking buys you — for free

Reducing the decision count also fixes a completely different failure mode: non-Markovian demonstrations. A Markovian policy assumes the right action depends only on the current state — not on how you got there, and not on how long you've been there. Human demonstrators violate this constantly. They pause mid-task to reposition a hand, they hesitate before a tricky grasp, they hold still for half a second and then continue.

Here's the problem that causes for a single-step policy. Suppose four different demonstrations all pass through roughly the same visual state — gripper hovering just above the object — but each one pauses there for a different amount of time before continuing. A single-step policy sees "this exact state" many times in the training set, labeled with many different next actions: sometimes "stay put" (early in someone's pause), sometimes "move forward" (someone who had already finished pausing). The state alone can't tell these apart — the correct action depends on the timestep, on elapsed time within the pause, which a Markovian state-only policy has no way to observe. Trained on this contradiction, the policy learns something close to the average of "stay" and "go" — which in practice often means it barely moves at all, gets stuck at that state, and — because it never leaves — keeps seeing the exact same state forever.

A chunked policy sidesteps this differently. Because it predicts a whole committed trajectory segment from one observation, it doesn't have to re-decide "stay or go?" at every single tick inside the pause. If the training chunks that started near a pause were themselves generated by demonstrations that eventually resumed and finished the sub-skill, the policy can learn to produce chunks that carry the pause-then-resume pattern as a single coherent unit, rather than needing to resolve the ambiguity frame by frame. It still isn't magic — Chapter 5's CVAE is what lets the model represent "this demo's particular style of pausing" as a single latent choice instead of forcing one deterministic mean-of-everything trajectory — but chunking alone already removes most of the instant-by-instant aliasing that traps a single-step policy.

Watch what happens at confounder strength 0: there's no ambiguity, both dots glide through the "pause zone" at the same speed, because there was nothing to be confused about. Turn it up, and the single-step dot starts to visibly stall and jitter right where the training demonstrations disagreed with each other about what to do next — while the chunked dot, which never had to resolve that instant-by-instant disagreement, keeps moving at a steady pace toward the goal.

Chapter 2: ALOHA Hardware

Chapter 1 established what ACT needs from its training data: demonstrations whose chunks are genuinely coherent sub-skills, collected fast enough and cheaply enough that you can gather 50 of them per task without a research budget the size of a hospital's. Before any of the algorithm can run, somebody has to actually generate that data — and for fine bimanual manipulation, that means teleoperation, because scripting a robot to thread a zip tie by hand-coded motion primitives is not realistic. Existing bimanual research platforms — Shadow Robot hands, ABB YuMi, da Vinci surgical systems — solve this, but at a price point that puts them out of reach for most labs. ALOHA (A Low-cost Open-source Hardware system for bimanual teleoperation) is the answer this paper builds: the whole two-arm teleoperation rig, under $20k, comparable to the price of a single commercial research arm.

Five design principles

The paper is explicit that every hardware choice traces back to one of five stated goals:

Principles 1, 4, and 5 point directly at one hardware decision: build the system around ViperX 6-DoF arms with parallel-jaw grippers, rather than a dexterous multi-fingered hand. Dexterous hands are more capable in principle, but their price and their maintenance burden work directly against "low-cost" and "repairable."

Total system: under $20,000.

Joint-space mapping, not task-space

The operator doesn't wear a VR controller or move a 3D mouse whose pose maps through inverse kinematics (IK) to the follower's gripper pose — that's task-space mapping, and it has a specific, serious failure mode for this application. Instead, the operator physically backdrives a smaller WidowX arm — the leader — and its joint angles are copied directly onto the larger ViperX follower. This is joint-space mapping, and the paper gives two concrete reasons it wins here.

Two more details make this system usable for tasks that take real concentration. The stock ViperX fingers aren't built for delicate work, so ALOHA replaces them with 3D-printed see-through fingers fitted with grip tape — the operator can watch exactly what's happening between the jaws while grasping, and thin plastic films still hold without slipping. And the leader arm is retrofitted with a 3D-printed "handle and scissor" mechanism plus a rubber-band gravity-balancing rig: together they let the operator control the gripper continuously (rather than as a binary open/closed switch) while cutting the physical effort of backdriving enough to make sessions longer than 30 minutes practical.

Observation comes from four Logitech C922x webcams — top, front, and one on each follower's wrist — each streaming 480×640 RGB. Teleoperation and data recording both run at 50Hz. The wrist cameras matter more than they sound: the fixed top and front cameras simply cannot see what's happening between the gripper fingers during a delicate grasp, which is exactly the moment that needs to be visible.

Now, the detail that matters most for everything from Chapter 3 onward: when ALOHA records a demonstration, it logs the leader's joint positions as the action — not the follower's. This looks backwards at first; the follower is the arm actually doing the work. But the leader-minus-follower gap is exactly what a Dynamixel PID controller uses to compute how much torque to apply, so that gap implicitly encodes contact force — how hard the gripper is pressing, how much resistance an object is offering — without any force sensor anywhere in the system. Record the follower's position instead, and you throw that signal away: a follower joint reads almost the same whether it's pressing hard against a stuck lid or gliding through empty air, because the follower is, definitionally, always close to wherever it managed to get to. The leader's target is the actual intention; the follower's actual position is a noisy, force-blind copy of it.

Chapter 3: Action Chunking

Chapter 1 argued, in the abstract, that fewer decisions means less compounding drift. This chapter puts a number on "fewer" and shows what actually happens as you sweep the chunk size k across the range the paper tests — including the part of the story that a purely "bigger is better" intuition gets wrong.

The naive chunking rule

Fix a chunk size k. Every k timesteps, the policy receives one observation s_t, generates the next k target-joint-position vectors in a single forward pass, and the robot executes all k of them in sequence before looking again. Formally, the policy models π_θ(a_t:t+k | s_t) instead of π_θ(a_t | s_t) — exactly the object Chapter 1 defined. This reduces the effective horizon — the number of independent decisions per episode — by a factor of k.

This "naive" version — observe, commit to k actions, repeat — is fully open-loop for the duration of each chunk: nothing the robot sees during those k steps can change what it does during them, because it already committed. Chapter 4 fixes that specific weakness with temporal ensembling. This chapter is about a different question: independent of that fix, how does k itself trade off against performance?

Why bigger k helps — and then stops helping

Two effects pull in opposite directions as k grows, and neither one is optional to understand if you want to predict where the sweet spot lands.

Small k: too many replans, each one a chance to compound error. Very large k: too few replans, but each remaining one is a long, hard, uncorrectable bet. Somewhere in between is a sweet spot — and the paper finds it empirically by sweeping k with temporal ensembling turned off, isolating chunking's effect on its own.

Chunking isn't special to ACT's specific architecture, either. The same ablation grafts action chunking onto BC-ConvMLP (which starts predicting a k×action_dim block instead of one action) and onto VINN (which retrieves the next k actions from its nearest training match instead of just one) — and both baselines improve too. That's the paper's basis for calling chunking "generally useful" rather than an ACT-specific trick: it's a property of the decision-horizon math from Chapter 0, not of any one model family.

Chapter 4: Temporal Ensembling

Chunking (chapter 3) fixed the big problem: predicting k actions at once cuts the effective horizon k-fold, so errors have far less time to compound before the policy gets a fresh look at the world. But chunking on its own creates a smaller, uglier problem at the seams.

Picture the robot executing chunk after chunk, open-loop, k steps at a time. It finishes chunk #1, and now needs chunk #2. It queries the policy again — but the policy has no memory of exactly what chunk #1 was doing at its very last step. Chunk #2 starts from scratch, predicting its own k-step plan from the current observation. If that new plan disagrees even slightly with where chunk #1 left off, the follower arm's target jumps instantly at the boundary. The paper is blunt about this: “A naïve implementation of action chunking can be suboptimal: a new environment observation is incorporated abruptly every k steps and can result in jerky robot motion.”

The fix: never stop asking

The paper's answer is almost too simple: instead of querying the policy once every k steps, query it at every single timestep. Every timestep t gets its own fresh k-step chunk prediction π_θ(â_t:t+k | o_t). This means chunks now overlap — heavily. For any given real timestep, there isn't one prediction of what to do — there are up to k of them, one contributed by each of the last k queries, each one guessing what should happen right now from a slightly different vantage point in time.

That overlap is exactly the raw material temporal ensembling needs. Instead of blindly trusting whichever chunk happens to be “active” this instant (the naive, jerky version), average all of the overlapping predictions for this timestep into one number.

Unpacking every symbol: A_t is the list of all predictions currently sitting in the buffer for timestep t — one entry from every one of the last (up to) k queries whose chunk still reaches this far. i indexes that list from 0 = the OLDEST active prediction (the query made furthest in the past that still covers t) up through the newest (the query made just now, at t itself). m is a single knob that controls how fast the weight falls off as predictions get newer. w_i is that prediction's vote in the weighted average.

What ensembling costs — and buys

Querying every timestep instead of every k steps means k× more forward passes through the network at test time. The paper is explicit that this is the only price: “This procedure also incurs no additional training cost, only extra inference-time computation.” At roughly 0.01 seconds per forward pass on a single 2080 Ti, that overhead is easily absorbed at 50Hz.

Chapter 5: The CVAE

Chunking and ensembling (chapters 3-4) both assume the policy has one right answer to give at each moment. That assumption breaks the moment demonstrations come from a human. Ask the same person to repeat the exact same handover a hundred times and you will get a hundred slightly different trajectories — not because they're being sloppy, but because when precision doesn't matter, humans don't hold themselves to a single millimeter-perfect path. The paper puts it plainly: “Given the same observation, a human can use different trajectories to solve the task.”

A hand-worked collision

Picture a handover task where the gripper must pass a small block sitting on a raised stand, leaving just enough gap underneath to slip through. Some demonstrations go over the block; others go under, through the gap. Both work. Neither one is “more correct” than the other — they're two equally valid solutions the human happened to alternate between.

Measure clearance as height relative to the block's vertical center (0 = dead center, where the block itself is). The over-demonstrations clear it by +4cm. The under-demonstrations clear it by −4cm, passing through the gap below. Both demonstrations get equal weight during training. A model that must commit to one deterministic number for this observation settles somewhere between the two targets it was shown — the simplest such compromise is the plain average:

Zero centimeters of clearance is exactly the block's own height — nowhere either demonstration actually went. The “averaged” policy doesn't cautiously split the difference into a safe path; it drives straight into the obstacle both real demonstrations were carefully avoiding.

The machinery: how z gets learned

The encoder is a small, BERT-style transformer that only exists during training. Its job: look at a real demonstration (the joint observations and the k-step action chunk that followed) and compress “which behavior is this” into z. The input is built the same way BERT builds sentence-level representations — prepend a learned [CLS] token, then the embedded current joint positions, then the embedded k-step action sequence:

After the transformer processes this sequence, the feature at the [CLS] position — the one token whose whole job is to summarize everything else — gets projected into two vectors: a mean μ and a variance σ², defining a diagonal Gaussian distribution over z. Sampling z = μ + σ·ε (with ε drawn from a standard normal) lets gradients flow through the sampling step during training.

The full training loss balances two goals: reconstruct the demonstrated action chunk accurately, and keep the distribution of z close to a simple prior so it doesn't degenerate into memorizing training examples one-by-one:

L₁ reconstruction (not L₂) — chapter 6 covers exactly why the paper prefers it for actions. The KL term is the standard VAE regularizer, weighted by β: push q_φ(z|…) toward the simple prior N(0,I) so z stays well-behaved and doesn't become a lookup table for individual training examples. Higher β means less information can flow through z — squeeze the bottleneck hard enough and z stops carrying any real signal about which mode this demonstration was, which pushes the decoder right back toward the collapsed-average behavior this whole mechanism exists to avoid.

At test time: z is set to zero

Here's the part that surprises people on first read: the encoder that made all of this possible is thrown away entirely at test time. There's no real future action sequence to encode when the robot is actually running — that's the whole thing being predicted. So the paper sets z = 0, the mean of the prior distribution, and decodes deterministically. Every rollout of the policy on a given observation produces the same output; there's no randomness injected at execution time.

Scripted demonstrations come from one deterministic policy — there's no second mode to resolve, so the CVAE objective has nothing to do and success rate barely moves. Human demonstrations are exactly the multi-modal case this chapter has been building toward, and removing the CVAE objective there is catastrophic: 35.3% down to 2%. This is the single clearest piece of evidence in the whole paper that modeling multimodality isn't a nice-to-have — for learning from real human data, it's load-bearing.

Chapter 6: Architecture

Three ideas so far: chunk k actions at once (chapter 3), ensemble overlapping chunks for smoothness (chapter 4), and train as a CVAE so the policy can represent more than one valid behavior (chapter 5). Time to open the box and trace the actual network that runs all three — the exact tensor shapes, end to end, that turn four camera images and two arms' worth of joint angles into the next hundred target positions.

Reading the spine in words

Each of the 4 cameras (480×640×3, all four streaming at 50Hz per chapter 2) passes through a ResNet18 backbone, which turns the raw pixels into a 15×20×512 feature map — a coarse grid of 512-dimensional descriptors, one per spatial location. Flatten the 15×20 grid into a sequence and each camera contributes 300 tokens: 15 × 20 = 300.

Where does 15×20 actually come from? A standard ResNet backbone downsamples its input by a total stride of 32× by the time it reaches the final feature map (a stem that strides down 4× (conv + maxpool), then three more stride-2 stages). Divide the 480×640 input by 32 on each axis: 480 ÷ 32 = 15 and 640 ÷ 32 = 20, exactly the feature-map size the paper reports. This is the kind of number you should always be able to derive, not just memorize.

Flattening throws away WHERE in the image each token came from — row 3, column 8 and row 8, column 3 become indistinguishable once they're just “token #38” and “token #123” in a flat list. A fixed 2D sinusoidal position embedding gets added back in, the same trick DETR (Carion et al., 2020) uses for object detection transformers, so the network can still reason about spatial layout downstream.

Stack all four cameras' 300-token sequences together: 300 × 4 = 1200 visual tokens. Append one more token for the current joint positions (a 14-dimensional vector — 7 per arm — linearly projected up to 512 dims) and one more for the CVAE style variable z from chapter 5 (also linearly projected to 512):

That 1202×512 sequence feeds a 4-layer transformer encoder, which lets every token — every patch of every camera, the joint state, and z — attend to every other token before anything gets decided. The output is still 1202×512, just contextualized.

The 7-layer transformer decoder works differently from a typical autoregressive decoder. Its query sequence isn't generated one token at a time from previous outputs — it's a fixed, learned-once sinusoidal position embedding of shape k×512 (k=100, the chunk size from chapter 3). Those k queries cross-attend into the encoder's 1202×512 output to pull out whatever information they need, and self-attend among themselves. Output: k×512. A small MLP projects each of the k rows down from 512 to 14 dimensions — the predicted target joint positions for that one future timestep.

The four camera views aren't redundant copies of each other — they cover different failure modes. The top and front views (chapter 2) give the wide workspace context: where is the object, where is the other arm, is anything about to collide. The two wrist views ride along with the follower grippers and give the close, stable, hand-eye viewpoint that fine manipulation actually depends on — exactly the view a human uses when threading a needle by feel more than by watching from across the room. Because every one of the 1200 visual tokens attends freely to every other token in the encoder, the network is free to combine “wrist-cam says the gripper is 2mm from the lid” with “top-cam says the cup hasn't moved” in whatever way the task needs.

Design-decision ledger

Hyperparameters (Table III)

The forward pass, in pseudocode

Mirroring Algorithm 1 (training) and Algorithm 2 (inference) from the paper:

# ACT training step — mirrors Algorithm 1
def train_step(o_t, a_chunk):              # a_chunk: (k, 14) target joints
    o_bar = o_t.joints                        # proprioception only, no images
    z_mu, z_logvar = encoder(cls_tok, o_bar, a_chunk)  # BERT-style, k+2 tokens
    z = z_mu + exp(0.5 * z_logvar) * randn_like(z_mu)  # reparam trick

    tokens = []
    for cam in o_t.images:                  # 4 cameras
        feat = resnet18(cam)                  # (15, 20, 512)
        feat = flatten(feat) + pos_embed_2d    # (300, 512)
        tokens.append(feat)
    visual = concat(tokens, dim=0)             # (1200, 512)
    joints_tok = linear_joints(o_t.joints)     # (1, 512)
    z_tok = linear_z(z)                        # (1, 512)
    enc_in = concat([visual, joints_tok, z_tok]) # (1202, 512)

    memory = transformer_encoder(enc_in)       # 4 layers -> (1202, 512)
    queries = fixed_sin_posemb(k)              # (k, 512), NOT learned from data
    dec_out = transformer_decoder(queries, memory) # 7 layers, cross-attn -> (k, 512)
    a_hat = mlp_head(dec_out)                  # (k, 14)

    loss = l1_loss(a_hat, a_chunk) + BETA * kl_divergence(z_mu, z_logvar)
    loss.backward(); optimizer.step()          # Adam, lr = 1e-5

# ACT inference — mirrors Algorithm 2
buffer = [[] for _ in range(episode_len)]      # FIFO: buffer[t] = predictions for t
for t in range(episode_len):
    z = zeros(z_dim)                            # z = 0, deterministic decode
    a_hat_chunk = policy(o_t, z)                # (k, 14), same forward pass as above
    for i in range(k):
        buffer[t + i].append(a_hat_chunk[i])   # scatter into the FIFO
    preds = buffer[t]                          # every prediction covering NOW
    w = [exp(-M * i) for i in range(len(preds))] # i=0 -> oldest active prediction
    a_t = sum(w[i]*preds[i] for i in range(len(preds))) / sum(w)
    robot.execute(a_t)                          # sent to the Dynamixel PID (chapter 2)

With the network fully traced, the next two chapters put it to work: chapter 7 walks through all 8 tasks this exact architecture was trained on and how it stacked up against four prior imitation-learning baselines, and chapter 8 pulls apart every ablation — chunking, ensembling, and the CVAE — to show, numerically, how much each piece of this spine is actually pulling its weight.

Chapter 7: Experiments

Chapters 1 through 6 built the machine. This chapter is where it gets tested. A results table by itself proves very little — anyone can report one favorable number. What makes ACT's evaluation convincing is what happens right after the headline number: every task is broken into subtasks, and every method's success rate is reported at each stage. When a policy fails, you can see exactly where the wheels came off, not just that they did.

Eight tasks. Six run on the real ALOHA hardware; two run in MuJoCo simulation. Four prior imitation-learning baselines. One question: does predicting chunks as a generative model actually buy you fine manipulation, or is it just a fancier way to fail?

Eight tasks, one randomization rule

Every task starts with the target object placed randomly — on the real robot, along a 15cm white reference line taped to the table; in simulation, uniformly inside a 2D region. Same idea, two implementations: the policy can never memorize a fixed pixel location for the object, it has to actually look.

Two of these objects deserve a special mention because they're adversarial to vision, not just control: the ziploc bag is largely transparent with a thin blue sealing line, the tape and the cup's body/lid are translucent (which makes depth cameras nearly useless), and the velcro tie and the table are both black — a low-contrast pairing that will come back to bite ACT in chapter 8's failure analysis.

Task Explorer: pick a task, watch where it falls apart

Every number below is the exact subtask cascade reported in the paper. Pick a task and watch ACT's success rate step down stage by stage, against the best comparable baseline — usually BeT, the strongest of the four on real hardware. For the two simulated tasks, ACT is trained twice: once on scripted (deterministic) demonstrations, once on human (stochastic) ones — toggle between them.

The human-data drop (a preview of chapter 5)

Notice something if you flipped the sim toggle above: every method does worse when trained on human demonstrations than on scripted ones, even though it's the exact same task, the exact same policy architecture, and the exact same number of demos (50).

Baseline autopsy: four ways to fail at fine manipulation

ACT is compared against four established imitation-learning methods, each carefully tuned on the cube-transfer task before being run on everything else. Each one embodies a different, reasonable-sounding design choice for behavior cloning — and each one has exactly one weak point that fine manipulation exposes.

Aggregated across the two simulated tasks (scripted and human data each), ACT beats whichever baseline does best by 59, 49, 29, and 20 points — Cube Transfer scripted, Cube Transfer human, Bimanual Insertion scripted, Bimanual Insertion human, respectively. Every card below contributes to that gap.

What it took to collect this data

Every one of the tasks above — the ones ACT solves at 80–90% success — was learned from that little data. That's the whole point of the low-cost hardware in chapter 2: cheap enough that collecting a few dozen demonstrations per skill is a lunch break, not a research program.

Chapter 8: Ablations

Chapter 7 showed ACT winning. That's not the same as showing why it wins. ACT bundles three ideas together — chunking, temporal ensembling, and the CVAE objective — and it's fair to ask whether all three are pulling their weight, or whether one of them is doing all the work while the others are along for the ride.

This chapter turns each one off, one at a time, and measures what breaks. The paper presents this as four panels of one figure; the dashboard below rebuilds all four with the exact numbers reported.

(a) Chunk size k: how far ahead should one prediction commit?

With temporal ensembling switched off (isolating chunking's effect alone), ACT's success rate — averaged across all four settings from chapter 7 (2 simulated tasks × scripted/human) — climbs from 1% at k=1 (no chunking, i.e. plain single-step behavior cloning) to 44% at k=100, then tapers slightly at k=200 and k=400 as chunks approach the full episode length (fully open-loop control). The paper doesn't report exact numbers at k=200/400 — only that performance dips slightly, which the dashboard renders as a soft downward trend rather than a labeled value.

(b) Temporal ensembling: who benefits from smoothing?

With k fixed at each method's best setting, adding temporal ensembling on top changes success rate by:

(c) The CVAE objective: does it matter?

Removing the CVAE objective means ACT just predicts one action chunk directly from the current observation, trained with plain L1 regression — no encoder, no latent z, no KL term. Averaged across the two simulated tasks:

On human data, removing the CVAE isn't a modest regression — it's a collapse from 35.3% to 2%. This is the ablation that most directly explains chapter 5's design choice: the CVAE isn't decoration, it's the specific component that lets ACT learn from noisy, multi-modal human demonstrations at all.

(d) Is 50Hz actually necessary?

A separate user study, decoupled from ACT itself, tests whether high-frequency teleoperation is what makes fine manipulation possible in the first place — or whether a slower, more common control rate would do just as well. 6 participants (none had used ALOHA before) each performed two tasks at both 50Hz and 5Hz, order randomized:

Overall, dropping to 5Hz slowed task completion by 62%, confirmed with a repeated-measures statistical test at p < 0.001. This is a study about the human operator, not the learned policy — but it justifies the entire premise of chapter 2: if a human needs 50Hz feedback to thread a zip tie quickly, a policy trained on 5Hz demonstrations was never going to have the resolution to learn it either.

Thread Velcro failure forensics

Chapter 7's cascade for Thread Velcro was the ugliest of the six real tasks: success roughly halves at every stage, from 92% at Lift down to 20% by Insert. Two concrete failure modes explain it:

Both trace back to the same root cause: the plastic loop measures roughly 3mm × 25mm, while the tie itself measures roughly 2mm × 10–25mm depending on where you grip it. A few millimeters of error on the first grasp compounds through the mid-air handoff into more than a 10mm deviation by the insertion phase — on a target that's only 3mm wide. On top of that, the black cable tie sits against a black table (low visual contrast) and only occupies a small fraction of the frame from the top-down camera, so even localizing it precisely is hard before you get to threading it.

Chapter 9: Connections + Cheat Sheet

You've now built ACT chunk by chunk: the problem it solves, the math behind chunking, temporal ensembling, the CVAE, the full tensor spine, and the numbers that back it up. Knowing one algorithm's math cold isn't the same as knowing when to reach for it. This chapter places ACT on the map next to its neighbors, then compresses the whole lesson into one page you can return to.

Where ACT sits in the family tree

Every method in this lineage is answering the same question — what should a policy output, and how do you handle the fact that human demonstrations aren't unique? — with a different architectural bet. Plain behavior cloning ducks the question entirely: it regresses to a single action and hopes the training data doesn't disagree with itself too often. The moment you start collecting real human demonstrations — and chapter 7's numbers showed how much worse every method gets on human data versus scripted data — that hope stops holding, and the field splits into two answers.

ACT vs. Diffusion Policy, one level deeper

Both are "generative models over an action chunk," which makes them easy to conflate. The difference that matters in practice is how many forward passes it takes to turn one observation into one executable chunk:

Neither is strictly "better" — ACT's single-shot decode is what makes its ~0.01s inference time possible on an 11GB 2080 Ti, which matters when you're closing a 50Hz control loop. A denoising-based policy trades some of that speed margin for a generative process with a different, often smoother, inductive bias. Which one wins depends on how tight your real-time budget actually is.

It's worth being precise about what "same problem, different generative model" actually buys you as a practitioner, because it's easy to read that sentence and conclude the two are interchangeable. They're not: swapping ACT's CVAE for a diffusion head changes the training loss, the sampling procedure at test time, and the number of times you have to run the transformer per control cycle. What stays constant across both is the higher-level framing this whole lesson has built — predict a chunk of actions, not one action; model the distribution over valid chunks, don't regress to their mean; execute with some form of smoothing across overlapping predictions. That framing, not any one equation, is the actual transferable idea.

What came after ALOHA

ACT's own paper is the origin of the "action chunking" idea, but the ALOHA hardware platform kept evolving after this paper shipped, and both follow-ons kept the chunked imitation-learning recipe rather than replacing it.

Mobile ALOHA puts the bimanual arms on a mobile base, so the same chunked-action policy has to solve whole-body navigation-plus-manipulation tasks — walking up to a sink, opening a cabinet, reaching in — instead of working inside a fixed tabletop cell. The control problem gets harder (more degrees of freedom, a moving base to coordinate), but the underlying "predict a chunk, execute open-loop, temporally ensemble" recipe from this lesson carries over largely unchanged.

ALOHA 2 is a hardware revision of the same low-cost philosophy — refining the gripper mechanism and overall durability for more robust, longer data-collection sessions — while keeping the core idea intact: cheap, off-the-shelf hardware, joint-space teleoperation, and a policy trained on a small stack of human demonstrations. The lesson here is less about any specific successor and more about how durable the ALOHA + ACT recipe turned out to be: low-cost teleoperation hardware, paired with a chunked generative policy, has become a template the field keeps building on rather than discarding.

When to reach for which

Cheat sheet: the equations

How chapter 8's ablations pay for these equations

Every term in the equations above earned its place in the loss because removing it measurably hurt something in chapter 8. Reading the cheat sheet next to the ablation numbers makes the connection concrete:

Where to find this in the paper

For readers who want to go straight to the primary source instead of this lesson's retelling, here's the section map (arXiv 2304.13705):

Cheat sheet: symbol glossary

Cheat sheet: the numbers

Honest limitations

ACT is not magic, and the paper is upfront about where it stops working:

Rapid-fire recap

Nine chapters, a lot of claims. Click each card below — some are true, several are common misreadings of the paper worth catching before you close this lesson.