Agent Harness

Chapter 1: What AWM Induces: The Workflow

Chapter 0 ended on a requirement: somewhere to write down what worked. This chapter is about what, exactly, gets written down. Not the raw transcript of a successful run — that would just be a longer prompt with one specific example baked in, useless the moment the product name or the city changes. What AWM writes down is something more like a recipe than a transcript, and getting that distinction right is the whole chapter.

The formal shape of a workflow

A workflow w is a pair: a short natural-language description d of what it accomplishes, and a sequence of steps P^d = (p₁, p₂, …) that accomplish it.

Each step in turn has three parts, and the paper keeps all three because each does a different job during induction. A step p_i carries: a natural-language description of the environment state at that point (what the agent could see), the reasoning the agent produced to justify its choice, and the action itself — an executable program over the environment, such as click('171') or fill('158', 'Berkeley').

Why keep the reasoning and the state description at all, if only the action gets executed? Because induction is not run on a single trajectory — it is run on several, and the thing that tells the inducing model which parts are the reusable skeleton and which parts are this-task-only detail is exactly the reasoning that accompanied each action. An action alone, out of context, looks like an arbitrary click. The reasoning attached to it says why, and “why” is what transfers.

Contrast two reasoning strings the induction model might see attached to superficially similar clicks. Trajectory one: “the search box is empty, so I need to type the product name before I can search.” Trajectory two: “this particular listing has three tabs and reviews happen to be the most useful one for this specific item, so I'll check it.” The first reasoning describes a general precondition — true of any product search on this site, independent of which product. The second describes a task-specific judgment call that has no reason to generalize to the next product. An induction model reading both is picking up exactly the signal a bare list of clicks would have thrown away: which steps are “always do this first” and which are “this happened to be the right call this one time.”

How induction actually runs

Given a set of past experiences ℰ = {e_i}, each e_i a task instruction paired with the full trajectory that solved it, an induction function I turns them into a set of workflows:

The paper implements I two ways. The one that matters most is LM-based induction: prompt a language model with several trajectories at once and instruct it to “find the repetitive subset of actions across multiple tasks… each workflow should be a commonly reused sub-routine… represent the non-fixed elements with descriptive variable names.” The model reads the trajectories the way you are about to, spots what recurs, and writes the abstracted version down. The cheaper alternative is rule-based induction: deduplicate by raw action sequence and drop steps whose element references are invalid. Chapter 4 compares the two directly — the short version is that rule-based is almost free and nearly as good on WebArena, but LM-based wins clearly on Mind2Web specifically because of the variable abstraction rule-based induction cannot do.

The real example: calculating travel time in WebArena Maps

Here is an induced workflow from the paper’s WebArena Maps domain, reproduced exactly as the paper represents it — a description plus a sequence of parameterized actions:

workflow: Calculate Travel Time and Distance
fill('158', FROM_LOCATION)
fill('163', TO_LOCATION)
select_option('166', MODE_OF_TRANSPORTATION)
click('171')
send_msg_to_user('The distance between FROM_LOCATION and TO_LOCATION is DISTANCE...')

Read every token in that block, because what is fixed and what is a variable is the entire lesson. The element references — '158', '163', '166', '171' — are not variables. They stay exactly as written across every use of this workflow, because they are properties of the Maps page’s DOM: field 158 is the “from” box on this site’s this page, every time you load it. The content — FROM_LOCATION, TO_LOCATION, MODE_OF_TRANSPORTATION — is what changes task to task, and induction has replaced the literal values a specific trajectory used (say, “Berkeley” and “Oakland”) with descriptive placeholders an agent fills in at use time.

Induce one by hand

Small illustrative example, close in spirit to the Maps workflow above but simple enough to trace completely. Suppose three past trajectories on a toy shopping site, each a successful “buy this product” run:

trajectory A — task: "buy dry cat food"
1. click('search-box')
2. type('search-box', 'dry cat food')
3. click('search-btn')
4. click('result-1')
5. click('add-to-cart')

trajectory B — task: "buy a wireless mouse"
1. click('search-box')
2. type('search-box', 'wireless mouse')
3. click('search-btn')
4. click('result-1')
5. click('add-to-cart')

trajectory C — task: "buy a garden hose"
1. click('search-box')
2. type('search-box', 'garden hose')
3. click('search-btn')
4. click('reviews-tab')      # only C does this
5. click('result-1')
6. click('add-to-cart')

Line up the three by action type, ignoring the literal text typed. Steps 1, 2, 3, and the final “click result then add to cart” pair appear in all three trajectories, in the same order. Trajectory C’s detour to click('reviews-tab') appears in only one of the three — it is task-specific behavior, not a recurring pattern, so induction drops it. The type action’s literal argument (‘dry cat food’, ‘wireless mouse’, ‘garden hose’) differs in every trajectory, so it becomes a variable. Everything else is identical across all three, so it stays literal:

Five steps, one variable slot, and a description an LM would write as something like “Search for a product by name and add the first result to the cart.” That is the whole induction algorithm, done by eye on three tiny trajectories: find what recurs across more than one example, keep it in order, replace what varies with a named slot, discard what only one trajectory did.

Workflows building on workflows

One more property is worth naming before moving on, because it never shows up in a single induced example and only becomes visible once induction has been running for a while on a real stream: workflows are not only extracted from raw, primitive-action trajectories — they can be built on top of previously induced workflows too. The paper's own online run on WebArena's Maps split shows this happening concretely, and it is worth tracing because it previews exactly the online setting Chapter 3 builds in full. Early in the run, after a handful of test queries each asking some version of “where is X,” induction produces a small workflow along the lines of “find a place by its name” — a few steps that search the map and read off the result. Later in the same run, a query arrives asking for something one step further: not just where a place is, but its zip code. The successful trajectory that solves it reuses the first few steps of the already-learned “find a place” routine, essentially verbatim, then adds new steps on top to extract the zip code once the place has been found — and induction, run on that trajectory, produces a new, more complex workflow (“get the zip code of a place”) that implicitly contains the earlier, simpler one as a prefix.

Read what that means for memory over time, and notice what does not change to make it possible. The induction function I from earlier in this chapter is not modified in any way — the identical “find what recurs, keep it in order, abstract what varies” procedure applies whether the trajectory being summarized used only primitive click-and-type actions or happened to reuse an already-learned workflow's steps along the way. What changes is only what accumulates in memory over time: a library that starts out holding nothing but simple, single-purpose workflows can end up, after enough successful attempts, holding compound ones that build implicitly on the simple ones underneath them — a small, self-reinforcing snowball rather than a flat list that only ever grows wider without growing deeper. Chapter 3 measures exactly how quickly that snowball actually gets rolling once online induction runs against a real stream of test queries, including the paper's own number for how few examples it takes before the effect becomes visible.

A workflow can also become a callable action, not just injected text

Everything up to this point treats a workflow purely as guidance the agent reads, sitting in memory beside the task instruction. The paper explores one more way to use the identical induced object: instead of only adding it to the memory string M, wrap each workflow into a high-level function and add that function to the agent's available actions, alongside the primitive ones like click and type. Take the induced “find a place by its name” workflow from a moment ago, call it find_place, and it becomes a callable action exactly the way click(id) is — except calling it triggers the workflow's whole sequence of primitive steps, one after another, instead of producing a single atomic effect. A login workflow, similarly, collapses into one login(username, password) call that internally clicks the username field, types into it, clicks the password field, types into that, and clicks submit — five primitive actions compressed behind one action the agent can choose in a single step.

Does giving the agent this extra choice actually help, or is it redundant with the text already sitting in memory? The paper's own numbers on Mind2Web, gpt-4, are a useful lesson in reading a small effect honestly rather than either dismissing it or overselling it. Step success rate ticks up from AWM's 45.1% to 46.4% once workflow actions are added to the action space:

— a real but modest gain, while overall task success rate stays essentially flat either way (3.2% versus a comparable base rate). A follow-up check on the trajectories themselves found the agent actually chooses to call one of these new workflow actions in only about 18.5% of the tasks where a matching one is available — more than four times out of five, the model still prefers to work through the same steps as ordinary primitive actions, guided by the identical workflow text already sitting in memory, rather than reach for the shortcut function that would accomplish the same thing in a single call. The honest reading: turning a workflow into a callable action is not wasted effort — it modestly reinforces what memory injection already provides — but it does not replace prompt injection as AWM's primary mechanism, and a model that has never been specifically trained to prefer new, unfamiliar shortcut tools over the primitive actions it already knows well will not automatically start reaching for them just because they now exist in its action space.

Where the workflow goes once it exists

This is the detail that separates AWM from most ideas people first guess when they hear “agent memory.” There is no fine-tuning here, no gradient update, no new weights. A workflow is text, and text gets added to a prompt. Formally, the original memory M (whatever built-in documentation the agent already had — the list of available actions, say) is augmented with the induced workflow set 𝒲:

and the agent’s generation step, which used to be a function of the query and the base memory, becomes a function of the query and the augmented memory:

In code, this is almost anticlimactic:

python
base_memory = load_action_docs()                    # the tool documentation the agent always had
workflows   = induce_workflows(past_experiences)      # Chapter 2 or Chapter 3, offline or online

system_prompt = (
    base_memory
    + "\n\n# Workflows that have worked on similar tasks:\n"
    + "\n\n".join(w.description + "\n" + w.steps for w in workflows)
)
answer = llm(system_prompt, task_instruction, current_observation)

That is the entire injection mechanism. No architecture change, no retraining, nothing that touches the model’s weights at all — which is precisely why the paper reports results on both gpt-3.5 and gpt-4 with the identical technique. Compare this to the audio-LLM lesson’s adapter, which spliced continuous vectors into an embedding sequence and required training a new component from scratch. Here the memory is symbolic and textual, so it is immediately usable by any off-the-shelf model, trivially inspectable by a human (it is just readable English plus a few lines of pseudo-code), and just as trivially editable or deletable if a workflow turns out to be bad advice.

Chapter 2: Offline Induction: Build the Library Once

Chapter 1 built one workflow from three trajectories. A real deployment does not have three trajectories — it has however many training tasks the benchmark provides, each with its own successful run. The question this chapter answers is when induction runs relative to the agent actually being tested, and the first of the paper’s two answers is the simpler one: run it once, in a batch, ahead of time.

The offline algorithm

Collect a set of training experiences ℰ_train — task instructions paired with trajectories that solved them, gathered before evaluation begins. Run induction once, over the whole set, to produce a fixed workflow library:

Then, for every test query the agent will ever see — and this is the defining property of the offline setting — use that same fixed library, unchanged:

That fixed-ness is the whole tradeoff, stated plainly: offline induction is cheap to run (one pass, in batch, can be parallelized, does not slow down evaluation) and completely predictable (every test query gets the same guidance, so results are easy to reproduce). What it cannot do is adapt: if the test distribution contains a task type the training set never showed, offline memory has nothing to say about it. Chapter 3 exists because of that gap.

How big does the library actually get?

The paper reports quality metrics for the induced offline library on WebArena that are worth sitting with, because they answer the question everyone asks first: does this bloat the prompt into uselessness? These four numbers come from the paper's own Table 10, which evaluates workflow quality on WebArena and, separately, on Mind2Web — the two rows do not always agree, and the disagreement itself is informative, which the next section returns to.

Average library size: Table 10's WebArena row reports 7.4 workflows per website. (Mind2Web's own row in the same table reports a nearly identical 7.3 — two very different benchmarks landing within a tenth of a workflow of each other is itself a small data point about how much repeated structure a typical website's tasks contain, independent of which benchmark you measure it on. It is WebArena's 7.4, not Mind2Web's 7.3, that belongs in the arithmetic below, since this section is specifically about the offline library built for WebArena.) WebArena spans five domains — Shopping, a content-management admin panel, Reddit, GitLab, and Maps. Multiply WebArena's own per-site average by the number of sites:

Thirty-seven recipes, most five or six steps long, replace what would otherwise be unlimited from-scratch exploration across five different websites. That is a small, readable amount of text — you could print it and read it over coffee — not a database.

Utility rate: 0.94. Of every hundred test tasks, about 94 have at least one relevant workflow available in memory when they run. Six in a hundred get no matching guidance and fall back to plain exploration — which is exactly what you would expect for task types the training set under-sampled, and is the honest failure mode of a fixed, offline library. Scale that rate up: run 200 test tasks and you would expect

Twelve tasks in two hundred paying Chapter 0's full re-exploration price is a small, bounded cost — and it is a measured cost, not a hidden one, because the library's coverage is a number you can report alongside your accuracy metric.

Functional overlap: 0.08. Only about 8% of induced workflows do substantially the same thing as another workflow already in the library. Low overlap means the induction step is not just memorizing near-duplicate variations of the same recipe — each of the roughly 37 entries is mostly earning a distinct place in the budget, which is the property you would want to see before trusting that a batch-induced library scales sensibly rather than collapsing into repeated, redundant advice.

Read “functional overlap” concretely: two workflows overlap when using either one, for the same task, would produce essentially the same action sequence. If induction were sloppy, you would expect the library to fill up with many slightly different versions of “search and add to cart” — one per source trajectory, barely abstracted — and functional overlap would be high. An 8% rate says the opposite is happening: induction is not just replaying training trajectories with the serial numbers filed off, it is genuinely collapsing many similar examples down into a smaller number of meaningfully distinct recipes, which is the entire promise Chapter 1's induction algorithm made and the number that confirms it is actually being kept.

The full quality table, and why WebArena and Mind2Web disagree on two of four metrics

Lay the paper's own Table 10 out in full, both rows, because the disagreement between them teaches something neither row teaches alone.

Two of the four columns are close (workflow count, utility rate); two are not (coverage, functional overlap), and both gaps trace to the same underlying explanation. Coverage — what fraction of an actual solved trajectory's steps a workflow accounts for — is not reported for WebArena at all, because computing it needs a ground-truth canonical trajectory to compare against, and WebArena's evaluation is purely execution-based: it checks whether the final state of the website is correct, not whether any particular sequence of clicks produced it. Mind2Web, by contrast, is built around exactly that kind of step-by-step canonical trajectory, so coverage is measurable there, and comes out at 0.40 — meaningfully lower than WebArena's near-complete practical coverage implied by its 0.94 utility rate. Functional overlap is measurable on both, and Mind2Web's 0.20 is two and a half times WebArena's 0.08:

Both gaps trace back to the same structural fact about the two benchmarks. WebArena is five specific, deeply familiar websites, each visited by many training tasks that share a small number of underlying page layouts — induction converges quickly on a compact, low-redundancy set of routines because there simply are not that many genuinely distinct ways to accomplish a task on a site the induction model has seen dozens of examples from. Mind2Web spans hundreds of websites, with the induction model seeing at most a handful of examples from each — less evidence per site means induction is more likely to produce workflows that are almost, but not quite, duplicates of each other (hence higher overlap), and any single workflow is less likely to capture every step a genuinely novel task on that site will need (hence lower coverage). Neither number is a flaw in the method; both are exactly what you would expect from feeding the identical induction algorithm two benchmarks with very different amounts of repeated structure per site to learn from — a useful diagnostic to carry into Chapter 4, where this same WebArena-versus-Mind2Web split shows up again in a head-to-head induction-method comparison.

What actually goes in the prompt for one test task

A library of 37 workflows spanning five unrelated websites is not what gets pasted into a single test task's prompt — a Reddit-domain workflow is irrelevant noise when the current task is on the Shopping site. The per-website average (7.3–7.4, not 37) is the number that lines up with what a sensibly scoped memory would inject for one task: the handful of workflows induced from that site’s training tasks, not the whole cross-domain library. Scoping memory to the current context is the same instinct that governed the audio-LLM adapter’s sequence-length budget — every token you inject competes with the actual task instruction and the live observation for a fixed context window, so an unscoped injection would burn budget on workflows that can never apply to the task at hand.

Put real token counts on it, matching the widget below. Say a base system prompt (instructions plus tool documentation) runs about 600 tokens, and each five-step workflow costs roughly 55 tokens once rendered as text. Scoped to one site's ~7 workflows:

Injecting the entire 37-workflow cross-domain library instead:

Scoping is not a minor optimization here — it is the difference between a prompt with room left for the actual task and a prompt that is already two-thirds consumed by workflows that mostly do not apply.

Why prompt injection, and not just fine-tuning on the workflows?

A reasonable alternative design would fine-tune the model on the induced workflows instead of injecting them as text. Weigh the two against each other honestly. Fine-tuning would require GPU infrastructure, a training run per model you want workflows on, and a fresh round of training every time the library changes — and the resulting knowledge would be baked into weights, opaque to inspection, impossible to selectively remove if one workflow turns out to be bad advice. Prompt injection costs none of that: it is instant (append text, call the model), inspectable (a human can read every workflow that went in), reversible (delete the bad one, no retraining), and portable across any model that accepts a system prompt. Its one real cost is exactly what this chapter has been computing — context tokens, spent on every single call, forever, for as long as that workflow stays relevant. Chapter 5 will show a design built on the identical bet: keep configuration frozen, keep the growing part cheap to read and easy to edit, rather than folding it into weights.

Notice, too, that workflows are not the only thing in that 600-token base memory. Before AWM ever enters the picture, an agent already needs documentation of what actions exist — click(id), fill(id, text), select_option(id, value), and so on — because the model has no way to guess a valid action space on its own. Workflows are additive on top of that pre-existing base: they do not replace the action documentation, they narrow which sequence of those actions is worth trying first for a task the agent has plausibly seen before. Strip the workflows back out and the agent still functions — just as a from-scratch explorer again, back to Chapter 0's baseline.

The code, once more, with a number attached

python
# run once, offline, before any evaluation begins
train_experiences = collect_successful_trajectories(train_tasks)   # from a training split
workflows_offline  = induce(train_experiences)                     # -> ~7.4 workflows per site, ~37 total

# run once per test task, memory never changes
for task in test_tasks:
    site_workflows = [w for w in workflows_offline if w.site == task.site]  # scope to ~7
    answer = agent.run(task, memory=base_memory + site_workflows)
    # workflows_offline is READ here, never WRITTEN — that is what "offline" means

Two questions a fixed library forces you to answer operationally

How do you scope a workflow to the right task, mechanically? The simplest version is a tag match: induce each workflow with the site or domain it came from attached as metadata, then filter by that tag at retrieval time.

python
site_workflows = [w for w in workflows_offline if w.site == task.site]

That is a dictionary lookup, not a search problem, and it is enough whenever the benchmark or product already partitions tasks by site or domain. A more general system — one where task categories are not so cleanly labeled — would retrieve by similarity instead: embed the task instruction and each workflow's description, then inject the top few nearest neighbors. Either way, the retrieval step exists precisely because the library, while small in absolute terms, is still bigger than what any single task needs.

What happens when a workflow goes stale? Chapter 1 already flagged the failure mode: element references are literal, and a site redesign breaks them even though the workflow's description is still accurate English. Nothing about offline induction detects this on its own — a fixed library, by definition, does not notice the world changed. A reasonable operational practice, though the paper itself does not prescribe one specific method, is to track each workflow's success rate whenever it gets used and re-induce (or simply drop) any workflow whose usage success rate falls well below the others. That converts “is this workflow still good?” from a question you would otherwise only discover by watching overall accuracy quietly drop, into a number you can monitor per workflow.

Put a shape on how quickly staleness could plausibly bite, with illustrative round numbers matching this chapter's own style. Suppose one workflow in a 37-workflow WebArena-scale library covers a task type that fires on roughly 1 in 15 test tasks, and a site redesign quietly breaks that workflow's element references overnight. Before the break, that workflow was one of the roughly 94 in 100 tasks the utility rate says find a match; after the break, its task type falls back to plain exploration, indistinguishable from Chapter 2's honest 6% coverage gap unless something is specifically tracking per-workflow success rate the way the paragraph above describes. Multiply the failure across 200 test tasks and roughly 13 of them (200 ÷ 15) silently lose their matching guidance the moment the site changes — not a catastrophic library-wide failure, but a real, measurable drop concentrated entirely in one task type, invisible in an aggregate accuracy number until someone thinks to slice the metric by workflow rather than by benchmark total.

Chapter 3: Online Induction: Learn From the Stream

Chapter 2 ended with an honest number: 6% of test tasks — one in seventeen — get no matching workflow from a fixed, offline-induced library, because the training set that produced it did not happen to cover every task type the test set contains. Online induction is the paper’s second answer to the same question this whole session keeps circling: what do you do about the tasks the training data missed?

The online algorithm

Instead of inducing once from a separate training set, online induction learns directly from the test stream as it runs. For each test query in sequence, generate a trajectory, check whether it succeeded, and — if it did — induce a workflow from it immediately and fold that workflow into memory before the next query arrives:

In code:

python
M = base_memory                                # starts empty of induced workflows
for t, q in enumerate(test_stream):
    trajectory = agent.run(q, memory=M)         # uses whatever M has accumulated so far
    success    = evaluate(trajectory)           # the benchmark's programmatic checker, here
    if success:
        w = induce([trajectory])                # induce from this ONE fresh trajectory
        M = M + [w]                             # grows for query t+1 onward — NOT for query t

Read the loop's last line carefully, because the ordering is the entire mechanism. Workflow w^t is induced from query t after it has already been answered, so it cannot help query t — it helps every query after t. The first time a genuinely new task type shows up in the stream, the agent still has to solve it the hard way, by exploration. The second, third, and fortieth time that task type recurs, memory has something to offer. Online induction does not eliminate Chapter 0's goldfish problem on the very first encounter with anything; it eliminates it on every encounter after the first.

How fast does the online loop actually start paying off?

Chapter 0 promised the real measurements would eventually replace its illustrative “40 occurrences” example, and this is the closest the paper comes to answering that directly. Rather than reporting only a final success rate averaged over an entire test run, the paper also tracks a cumulative success rate as the online loop progresses — the running average success rate of the first k finished examples, for every k from 1 up to the full test set, on WebArena's Maps split specifically. The shape of that curve is the whole point: it rises fast over roughly the first 40 examples, as the agent acquires the handful of most essential workflows for that site, and then levels off, with later gains coming more slowly as the agent moves on to rarer, more specialized workflows. Over that same window, the gap between the online agent and a memory-less baseline running the identical model rises as high as 22.5 percentage points — not after the entire benchmark, but after “only tens of examples,” in the paper's own words.

Put that curve next to Chapter 0's illustrative arithmetic and the two now line up as intended: the placeholder numbers at the top of this session (12 steps versus 6, a $4.80 waste on one recurring task type) were invented to make a shape easy to reason about by hand before any real evidence existed. This is that same shape, measured for real, on the smallest and most tightly-scoped of WebArena's five sites — and it confirms the specific claim Chapter 0 was building toward: the benefit of workflow memory is not a slow, barely-detectable trend buried in a large final average, it is a fast, visible climb that shows up within the first few dozen attempts at a recurring task type, exactly where a goldfish agent would otherwise still be paying full exploration price on attempt number thirty-nine.

One caution worth carrying forward before Chapter 4's fuller results: this fast-learning curve is reported on one site (Maps) chosen as a clean illustration, not averaged across all five of WebArena's domains. Chapter 4's own numbers — the 35.5% total success rate and the per-domain breakdown — are the ones that account for every site, including the harder, less template-repetitive ones where a 40-example learning curve would look less dramatic. A single illustrative curve on the easiest site to learn and a rigorous benchmark-wide average are both real numbers from the same paper; treating either one as if it were the other is exactly the kind of mis-citation Chapter 4 spends its opening paragraphs teaching you to catch.

What online induction costs, not just what it buys

Nothing above is free. Every success in the test stream triggers an extra call — the induction step itself — on top of whatever calls the agent already made to solve the task. Illustrate with a round, clearly-hypothetical number: if a stream of 100 test tasks succeeds 40% of the time, that is 40 additional induction calls layered on top of the stream's own agent calls:

If each induction call costs roughly as much as one ordinary agent turn, online induction adds something in the neighborhood of a 40% tax on top of the stream's own LLM usage, for a run at that success rate — paid in exchange for the adaptivity Table 4 measures. Offline induction pays a similar cost once, up front, on the training set, and then nothing more during evaluation. Whether the online tax is worth it is exactly the question the cross-task versus cross-domain comparison below answers empirically.

There is a subtler cost too. L_eval only checks the final outcome of a trajectory, not whether the path that produced it was efficient, general, or right for defensible reasons. A trajectory can succeed by luck — a redundant click that happened not to matter, a guess that happened to land — and online induction has no way to distinguish that from a trajectory that succeeded because the agent found a genuinely good route. Offline's LM-based induction gets some protection here almost by accident: it is shown several trajectories from the same broad task type at once and asked to find what recurs across them, so a one-off lucky detour that only appears in a single example tends not to survive into the abstracted workflow. Online induction, working from one fresh trajectory at a time, does not get that same implicit filter for free — it is an open question the paper's simple per-trajectory algorithm surfaces rather than one it explicitly resolves.

One operational mitigation worth naming, though it goes beyond what either paper specifies: require a workflow to be proposed after the first success but only promoted into the injected library after it has proven useful across two or three separate occurrences of that task type. That trades a small amount of Chapter 3's adaptivity — the very first recurrence still gets no help — for real protection against enshrining a one-off lucky path as reusable guidance. It is the same recurrence-across-multiple-examples logic that made offline's LM-based induction naturally robust, deliberately reintroduced into the online setting one task type at a time instead of all at once.

Where online induction earns its keep: Mind2Web under distribution shift

The paper measures offline and online side by side on Mind2Web's three generalization splits, each a progressively bigger gap between the training distribution and the test distribution. Cross-task tests new task instances on websites the training data already covered. Cross-website tests entirely new websites within a familiar domain. Cross-domain tests websites from categories the training data never touched at all. Here is the real Step Success Rate (gpt-4) for the baseline (MindAct) and both AWM settings:

Read the last column as a story, because the numbers tell one cleanly. At cross-task — the smallest shift, where the offline library was trained on data close to what the test set actually needs — offline slightly beats online (45.1 vs 43.6). The batch-induced library already has the right recipes; there is nothing left for on-the-fly learning to add, and the online agent even pays a small cost from the first few, still-workflow-less queries in the stream. At cross-website the two are statistically tied (+0.2). At cross-domain — the largest shift, where the offline library was built from a completely different kind of website — online pulls ahead by the widest margin of the three (+2.9), because this is precisely where a frozen library has the least relevant coverage and learning from the test stream itself has the most to offer.

Notice, too, what both settings share: every single cell in the AWM columns beats the MindAct baseline by a wide margin, including offline induction's cross-domain score of 32.6 against a baseline of 18.6 — a lift of nearly 14 points even when the library was built on unrelated websites. Workflow memory of either kind captures something more general than site-specific tricks; online induction is the extra correction that matters exactly when the gap between what you trained on and what you are tested on is largest.

Offline and online are not mutually exclusive

Nothing about the algorithm above requires choosing one setting forever. A production system can start with an offline-induced library (cheap, predictable, covers the common cases you already have training data for) and run online induction on top of it during live traffic, so genuinely new, previously unseen task types still get captured the second time they recur instead of the fiftieth. The two settings compose because they operate on the same underlying object — a workflow is a workflow regardless of when it was induced; only the schedule of induction differs.

Where L_eval actually comes from, outside a benchmark

WebArena and Mind2Web can evaluate a trajectory programmatically because both were built with that evaluation in mind — a benchmark task's success criteria are known in advance and checkable by code. Production systems have to manufacture an equivalent signal, and the honest options are limited. Explicit user feedback (a thumbs-up, a “this worked”) is the cleanest but requires the user to bother providing it. An automated verifier — run the generated code's test suite, check the output against a schema — works well whenever success has a checkable structure, which is common in coding agents specifically. A downstream proxy (did the user immediately undo the action, did the ticket get reopened) is noisier but available even when nobody explicitly labels anything. Whichever source you use, the online loop from earlier in this chapter is only as trustworthy as that signal is — a noisy or gameable L_eval will happily induce workflows from trajectories that only looked successful.

A simple decision rule

Given everything this chapter and the last one measured, a defensible default: ship with offline induction from day one, because it is cheap, predictable, and Chapter 2's utility rate already told you roughly how much of your traffic it will cover. Add online induction once you have a reliable success signal in production and you observe real distribution shift — new task types, new surfaces, new user populations that your training data did not anticipate. Table 4's own pattern is the justification: online barely helps, or slightly hurts, when the test distribution already matches training; it earns its added cost specifically when they diverge. Turning it on before you have evidence of that divergence is paying Chapter 3's tax for a benefit Chapter 3's own numbers say may not be there yet.

The graceful-degradation property worth naming explicitly

One property of this whole design deserves to be stated plainly, because it is easy to miss and it is what makes adopting workflow memory low-risk. On the roughly 6% of tasks Chapter 2's utility rate says get no matching workflow — and on the very first occurrence of any task type online induction has not yet seen — the agent falls back to exactly the behavior it would have had with no memory system at all. Injecting an empty or irrelevant set of workflows does not corrupt the prompt or confuse the model; it simply adds nothing. That means the worst case for adding AWM to an existing agent is no worse than not having it, and the only real risk is the one Chapter 1 already flagged — a stale or wrong workflow actively misleading the agent on a task it otherwise would have solved by exploration. Coverage gaps degrade gracefully; bad advice does not, which is exactly why Chapter 2's staleness-tracking suggestion matters more than chasing coverage all the way to 100%.

Notice, too, that online induction inherits this same graceful-degradation property automatically, without any extra design work. A brand-new task type that has never recurred contributes w = 0 workflows to memory for that type, which is indistinguishable, from the agent's point of view, from a task type that simply is not in scope for workflow memory at all — it runs exactly as if AWM had never been added. Nothing about the online loop from earlier in this chapter can make a task type worse off than it would have been with no memory, because the loop only ever adds workflows after a success; a string of early failures on a genuinely hard, recurring task type produces no workflows and therefore no injected guidance, leaving the agent to keep exploring on its own terms rather than being steered by a workflow induced from a lucky, unrepresentative success. The floor is the same for both settings, offline and online, for the identical underlying reason: a workflow library that has nothing relevant to say adds nothing to say, and adding nothing cannot make a system worse than it already was.

Chapter 4: Reading the Results: WebArena, Mind2Web, and the Ablations

Time to check the actual claims against the actual numbers, and to understand two things every paper's results table hides unless you compute them yourself: what “51% better” is relative to, and why a benchmark reports two success rates that can differ by a factor of ten for the same run.

WebArena, in full

Table 1 of the paper reports total success rate (SR, as a percentage of tasks fully completed) and average steps to completion, all with gpt-4, across five prior methods and AWM:

And AWM's total broken out by WebArena's five domains: Shopping 30.8, the content-management admin panel 29.1, Reddit 50.9, GitLab 31.8, Maps 43.3 — the abstract's headline “35.5%” is the average across all five.

The relative-improvement claim, computed. The paper's abstract calls this a “51.1% relative improvement.” Take AWM's 35.5 against BrowserGym's 23.5 — the strongest non-AWM baseline in the table without a tree-search crutch:

That arithmetic reproduces the abstract's number exactly, which tells you which baseline “51.1% relative improvement” is measured against — a detail the one-sentence summary never states, but the table makes checkable.

The steps claim, computed two ways. AWM finishes tasks in an average of 5.9 steps. Against BrowserGym-tree's 7.9:

Against AutoEval's 46.7, the gap is enormous:

A gap that large usually means AutoEval's average is being dragged up by trajectories that ran all the way to the step budget without ever succeeding — a wandering, unresolved agent racks up dozens of steps the same way a lost driver racks up miles. AWM's 5.9-step average, by contrast, is what “follow a known route, then stop” looks like in a table: this is the direct, measured version of Chapter 0's illustrative “12 steps versus 6” example, now with the paper's real number in place of the placeholder. Chapter 0 guessed 12 versus 6 to make a point about arithmetic; the actual measured gap — 46.7 steps at the high end against 5.9 — is nearly eight times larger than that placeholder assumed, which if anything understated how expensive Failure 1 really is.

One more row deserves more than a passing glance, because misreading what it actually is would miss the single most interesting comparison in the whole table. The paper's own Table 1 does not list SteP as just another autonomous baseline — it groups it, alone, under a section header the paper writes as “with human engineered workflows”: 14 hand-written recipes, authored by a domain expert who had studied WebArena's five sites in advance, executed via structured planning. Read that description again next to Chapter 1's own definition of a workflow — a description plus parameterized steps, reused across tasks — and the resemblance is not a coincidence. SteP is not an unrelated method that happens to share a table with AWM; it is the closest conceptual relative AWM has in the entire comparison, running the identical idea (workflow-shaped memory guiding an agent) with the one variable AWM's whole contribution exists to remove: who writes the workflows. SteP's are hand-authored by an expert who read the benchmark in advance. AWM's are induced automatically, from the agent's own successful trajectories, with zero human authorship of any kind.

That reframing changes what beating SteP is actually worth. AWM's 35.5% against SteP's 33.0% is a 2.5-point absolute win, and the paper computes the same comparison as a relative gain, the identical arithmetic discipline used above:

Seven and a half percent looks modest sitting next to the 51.1% relative improvement over BrowserGym computed above — and that gap between the two percentages is exactly the point, not an inconsistency to explain away. BrowserGym gets no workflow guidance of any kind, so beating it mostly demonstrates that workflow memory in general helps over having none. SteP already has workflow guidance — expensive, expert-curated, WebArena-specific workflow guidance, written by a person who knew the test set in advance — and AWM still comes out ahead of it, using workflows nobody hand-wrote. That is a fundamentally stronger claim than beating an unguided baseline: it says automatic induction is not merely better than nothing, it is competitive with, and here slightly ahead of, a human expert who spent real effort writing site-specific recipes by hand, on a benchmark that expert got to study beforehand and AWM's induction module never did.

Now the earlier lesson about reading a leaderboard still applies, just aimed correctly: the gap between AWM and the fully autonomous, no-workflow baseline (14.9%, more than doubled) tells a very different story from the gap between AWM and the workflow-guided alternative (SteP, 2.5 points) — and a table's headline number rarely tells you which of those two comparisons you are looking at unless you check what each baseline is actually doing, not just the score it posted.

Does AWM just memorize WebArena's task templates?

There is an obvious objection to any of the numbers above, worth taking seriously before moving on: WebArena's 812 tasks are not 812 independent problems — many are generated from a smaller number of underlying task templates, with only the specific values (a product name, a city, a repository) swapped out. If AWM's workflows are just memorizing per-template tricks from training examples, its real-world generalization could be far weaker than the headline 35.5% suggests, because a live deployment will not repeat the exact same handful of templates forever.

The paper tests this directly by building a cross-template subset: group WebArena's examples by their source template, then randomly keep only one example per template, so no two tasks in the subset share an underlying generator. If AWM's gains evaporate on this subset, the objection above would be confirmed. They do not:

AWM still wins — 33.2% against SteP's 32.1%, both very close to their full-benchmark scores of 35.5% and 33.0% respectively. That closeness is the actual finding: if AWM's workflows only helped by memorizing template-specific tricks, forcing every task to come from a different template should have hurt AWM far more than it hurt a baseline with no learned workflows at all. It did not. AWM's score barely moved (35.5 → 33.2, a 2.3-point drop) while still beating every other method on the harder, template-diverse subset by the same rough margin it held on the full benchmark. The workflows induced in Chapter 1 — a description plus parameterized steps, with the site-specific element references kept literal and the task-specific content abstracted into variables — are general enough to transfer across different task templates on the same site, not just across different literal values within one template. That is a stronger generalization claim than the headline number alone would let you make, and it is only visible once you go looking for the specific failure mode a skeptical reader would expect to find.

Mind2Web, offline setting, both model sizes

Element accuracy (did it click the right element), action F1, step success rate, and task success rate — task success requires every step in the task to be correct, which is why it is always the smallest number in the row:

The 24.6% claim, computed. The abstract's other headline number, Mind2Web's “24.6% relative improvement,” is AWM's gpt-4 Step SR against MindAct's gpt-4 Step SR:

Exact match again. Two abstract claims, two table lookups, two divisions — that is the whole discipline of reading a results section: find the two cells being compared, and do the subtraction yourself before trusting the adjective attached to it.

Task success, by contrast, moved from tiny to still-small. At gpt-3.5, task SR goes from 0.8 to 2.8 — a 3.5× multiple. At gpt-4, from 2.0 to 4.8 — a 2.4× multiple:

Why AWM beats Synapse, specifically: abstraction versus retrieved examples

The Mind2Web table has a second baseline worth reading closely: Synapse, at gpt-3.5, reaches 34.0% element accuracy and 2.4% task success — already well ahead of MindAct's 20.3% and 0.8%, and the closest competitor to AWM's 39.0% and 2.8% at that model size. Synapse's mechanism is instructive precisely because it looks similar to AWM on the surface but differs in exactly the way Chapter 1 predicted would matter: instead of inducing abstracted workflows, Synapse retrieves the most relevant full, concrete training examples and augments the agent's context with them directly — the complete trajectory, literal values included, no abstraction step at all.

AWM's edge over Synapse at the same model size is concentrated in exactly one place. Element accuracy — did the agent click the right thing — goes from Synapse's 34.0% to AWM's 39.0%:

That five-point gain is the paper's own explanation for why abstraction beats retrieval here, and it is worth spelling out mechanically rather than taking on faith. Augmenting an agent's context with a full concrete example — “here is exactly how a past task on this site was solved, including which literal element IDs were clicked” — risks biasing the agent toward selecting elements that merely resemble whatever appeared in that specific retrieved example, whether or not they are the right element for the current page. AWM's induced workflows have already stripped that bias out at induction time: the example-specific content (a product name, a city) is replaced with a variable slot, so what survives into memory is the pattern — search, then filter, then select — not a specific prior instance the agent might pattern-match against too literally. Less bias toward one remembered example translates directly into more accurate element selection on a page that is superficially different from whatever the memory came from.

There is a second, related reason AWM's abstracted workflows generalize better than Synapse's retrieved examples: a full example trajectory is, by construction, unlikely to recur verbatim — the exact sequence of clicks that solved “book a flight to Chicago on this specific date” is a poor match for the next task even if the next task is also about booking a flight. A workflow's whole purpose, from Chapter 1 onward, is frequently-reused sub-routines rather than one-off full solutions, which makes it a better match for a wider range of future tasks by design, not by accident.

The one place AWM does worse: learning when to diverge from a workflow

Honesty about a method's limitations belongs in the same table as its wins, and this table has one worth naming plainly. At gpt-4, AWM's action F1 — a check that the agent chose the correct action on a correctly-selected element — is 57.3%, slightly below MindAct's 60.6%, even though AWM's element accuracy (50.6% versus 41.6%) is far ahead on the metric this chapter has spent the most time on:

The paper's own explanation is a genuinely useful caution about what injected guidance can cost, not just what it buys. A workflow's steps describe what usually needs to happen for a task of that type — but “usually” is doing real work in that sentence. An agent guided by a workflow can be pulled toward following the workflow's prescribed action even in the specific moments where the actual, current environment state calls for something different, because the workflow text sitting in memory is, in effect, a strong prior toward one particular sequence. Following the workflow generally produces more successful trajectories overall, which is exactly what the higher step and task success rates already show — but the agent still has real difficulty learning when to depart from a workflow's suggested steps rather than follow them past the point where they still apply, and that difficulty shows up as a small, specific cost on this one narrower metric even while every broader metric in the same row goes up. It is the same tension Chapter 1's callout named when a workflow was first introduced: a workflow narrows the search enormously, but it does not remove the agent's obligation to actually observe the page in front of it.

Why task success is always so much smaller than step success

Both AWM rows report a Step SR around 35–45% but a Task SR around 3–5%. That is not two different methods disagreeing — it is the same run, scored two different ways. Step SR credits each individual step as right or wrong; Task SR requires every step in the task to be right, an AND across the whole sequence. If steps failed independently with the same probability p = Step SR, a task of k steps would succeed with probability p^k. Run that Fermi estimate on the gpt-4 numbers, solving for k:

A four-step task, roughly, is enough to explain the entire order-of-magnitude gap between 45.1% and 4.8% under the simplifying assumption that steps fail independently. Real errors are not fully independent — a wrong click early in a task changes what the later steps even see — so treat 3.8 as an estimate of scale, not a literal step count. What it establishes solidly is the mechanism: Task SR is not a worse or noisier metric than Step SR, it is a much stricter one, and any multi-step agent benchmark will show this same compounding.

Two ablations that explain design choices from Chapters 1 and 2

Induction method: rule-based vs. LM-based. Recall Chapter 1's two ways to implement I — deduplicate by raw action sequence (rule-based) or prompt an LM to abstract variables (LM-based). On WebArena the two are essentially tied: 35.6% (rule) versus 35.5% (LM), a difference of one-tenth of a point. On Mind2Web, LM-based induction wins clearly: step SR 43.4% (rule) versus 45.1% (LM).

Why would the two benchmarks disagree about which induction method wins? WebArena's five domains each have a small number of task templates repeated with different content, so simple sequence deduplication already catches most of the recurring structure. Mind2Web spans a huge number of distinct websites, where the same abstract skill (“search, then add to cart”) shows up with wildly different literal element references and page text across sites — exactly the case where an LM's ability to recognize “these are the same underlying skill, just abstract the values” earns its keep. This is the same lesson Chapter 1's element-reference example taught, now visible as a benchmark-level effect rather than a single hand-worked case.

Workflow format: code-like vs. prose. Representing each induced workflow as pseudo-code steps versus as prose sentences barely moves the needle: 45.1% versus 45.4% step SR on Mind2Web, a difference well inside noise. The lesson is almost the mirror image of the previous one — how a workflow is written down matters far less than whether its content correctly abstracts what recurs. Format is a style choice; abstraction is the mechanism.

Environment representation: natural language vs. raw HTML. Describing the page state in natural language for each step outperforms including the raw HTML: 34.6% versus 33.8% step SR — and combining both representations does not split the difference, it makes things worse, degrading to 32.9%.

The whole chapter, read as one argument

Four separate numbers, one throughline. AWM wins on WebArena's headline success rate (51.1% relative) and on Mind2Web's (24.6% relative) — two different benchmarks, two different task families, the same mechanism producing a similar-sized win on both. It wins even more dramatically on efficiency (87.4% fewer steps than the weakest baseline), which is the number that maps most directly onto real deployment cost. And the two ablations explain why the mechanism works rather than just that it works: variable abstraction matters most when the same skill recurs across very different-looking surface forms (Mind2Web's many websites), and injected context is only valuable when it is compact and non-redundant (the NL-versus-HTML result). None of these four numbers would be trustworthy in isolation — a single success-rate headline could be cherry-picked, a single step-count could reflect a fluke run — but four independent measurements, on two benchmarks, all pointing the same direction, is what makes a paper's claim something you can actually rely on.

One more habit worth building before trusting any single percentage-point gap: ask how many test tasks the number was computed over. A jump from 36.2% to 45.1% on a few dozen tasks is a very different kind of evidence than the same jump on several hundred — a handful of tasks flipping outcome by chance can move a small sample's percentage by several points, while the same handful barely moves a large sample's. This session's tables report the paper's own numbers as given, but the question “over how many tasks, and how much would a few flipped outcomes move this?” is worth asking of any single benchmark run you read, this one included, before treating a reported gap as more precise than the sample size actually supports.

Chapter 5: Enter OpenHands: One Source of Truth

Chapters 1 through 4 assumed something we never questioned: that there is a well-behaved agent loop to inject a memory string into in the first place. Give AWM's workflow library to a harness with no clean way to swap models, no safe way to pause a risky action, and no way to recover after a crash, and you have made the memory smarter without making the skeleton any less fragile. This is Failure 2 from Chapter 0, and it is where we turn now.

The OpenHands Software Agent SDK is a from-scratch rebuild of the execution layer underneath a coding agent, published by Xingyao Wang and eleven co-authors — among them Graham Neubig, the same last author on the AWM paper this session started with. Two different corners of the same problem, one overlapping research group. The paper frames the goal in three requirements: flexibility in how you implement and experiment with an agent, reliable and secure execution once it is running, and clean interfaces for a human to actually watch and steer it.

What the first version got wrong

The paper is candid about its own predecessor. Version zero was monolithic: sandboxing via Docker was mandatory for every agent, which forced a second process to always be running and let the in-container state and the orchestrator's idea of that state quietly drift apart. Tool implementations were duplicated wherever a new interface needed them. Adding a capability meant touching the core loop, because there was no seam to add it at.

Version one's fix was structural, not cosmetic: split the monolith into four packages, each independently deployable, each with one job.

And sandboxing flipped from mandatory to opt-in: agents now run locally by default, with Docker or remote execution as a configuration choice rather than a forced default. Chapter 7 goes deep on what that choice means for safety. This chapter is about the rule that makes all four packages compose cleanly in the first place.

The one rule: stateless by default, one source of truth for state

Every component that represents configuration — which model to call, which tools are available, what security policy applies — is immutable, validated once at construction, and never changes for the rest of its life. Exactly one thing in the whole system is mutable: the record of what has actually happened in a specific conversation. Everything else is either a pure function of that record or a frozen configuration object that record consults.

Agent: an immutable spec, not a running thing

An Agent is best understood as a specification, not a process. It bundles LLM settings, the tool specifications available to it, its security policy, and its system-level instructions — and nothing about runtime state. Functionally it behaves as a stateless event processor: given the conversation's current state, it decides the next step and emits that decision as a structured event through a callback, rather than returning a result the caller has to thread through by hand.

Because an Agent carries no state of its own, the same Agent object can be handed to many conversations at once without any of them stepping on each other. There is nothing shared to step on.

python
# one frozen configuration, reused across two independent conversations
reviewer_agent = Agent(llm=llm, tools=[read_file, comment], instructions="You review code.")

convo_a = Conversation(agent=reviewer_agent, workspace="/repo/pr-142")
convo_b = Conversation(agent=reviewer_agent, workspace="/repo/pr-143")
# convo_a and convo_b share zero mutable state — reviewer_agent has none to share
# running convo_a to completion cannot corrupt convo_b, and vice versa

Compare that to the pattern most hand-rolled agent loops fall into: one object whose instance attributes hold both its configuration and its running conversation history. That design makes reuse across two conversations require either deep-cloning the whole object (easy to get subtly wrong) or accepting a race condition if two conversations ever touch it concurrently. Splitting configuration (Agent, immutable) from history (ConversationState, mutable, one per conversation) removes the question entirely — there is no shared mutable object left to race over.

What "immutable LLM configuration" buys beyond simplicity

The LLM object inside an Agent is worth a closer look, because its own internal richness is easy to undersell as “just a model name and an API key.” It provides a unified interface across more than a hundred providers through LiteLLM, so switching from one vendor's model to another's is a configuration change, not a rewrite of how requests are formed or responses parsed. It has native support for reasoning models — typed objects like ThinkingBlock and ReasoningItemModel that carry a model's intermediate reasoning distinctly from its final answer, rather than forcing every provider's output into one generic string. A RouterLLM subclass allows dynamic model selection based on the content of a request — sending straightforward steps to a cheaper model and only escalating to an expensive one when the situation calls for it, entirely inside the same LLM abstraction the rest of the SDK already knows how to use. And a NonNativeToolCallingMixin extends tool-calling support to models that were never trained with function-calling built in, by prompting them to produce structured output in a format the SDK can still parse into an ordinary Action.

Customizing behavior without touching the Agent's code: Skills

One more piece of the Agent's frozen configuration is worth naming here, because it is the mechanism that lets a team customize how an agent behaves without editing anything inside the SDK itself. AgentContext centralizes everything that shapes what the LLM sees beyond the raw conversation — prefixes and suffixes attached to system and user messages, and a set of Skill objects. A Skill can be written directly in code, or loaded from a plain markdown file — the SDK reads files from a project's .openhands/skills/ directory, and is deliberately compatible with the same file formats other coding tools already use for this exact purpose, such as .cursorrules and agents.md. That compatibility is not an accident: a team that has already written project-specific instructions for a different coding assistant does not have to rewrite them from scratch to get the identical customization working inside this SDK.

Two activation modes cover the two situations a team actually runs into in practice. A skill with no trigger condition is always active — it persistently augments the system prompt on every single call, the way a project-wide coding-style guide should. A skill with a keyword trigger only activates when the user's message actually matches it — a skill about, say, database migration conventions only needs to occupy context budget on the turns where the agent is actually touching a migration, not on every unrelated turn, which is exactly Chapter 2's context-budgeting instinct applied to an entirely different kind of injected text. Skills can also bundle their own MCP tools, so a single skill file carries both instructions and the tools those instructions refer to, activated together as one unit rather than configured in two separate places.

ConversationState: the one place mutation is allowed

ConversationState is the single source of truth for everything that has happened. It holds a small amount of metadata — the agent's current status, running statistics, the active confirmation policy — and, more importantly, an append-only EventLog: every action, every observation, every message, recorded in order and never edited after the fact, only ever appended to. A FIFO lock and a two-path update pattern (metadata-only changes go one way, event-appending changes go another) keep this safe under concurrent access without needing a database.

Conversation: one factory, two runtimes, identical code

Conversation is a factory that hides where the agent actually executes. Construct it with a plain path, or a LocalWorkspace, and you get a LocalConversation running in-process on your machine. Construct it with a RemoteWorkspace instead, and it transparently builds a RemoteConversation that talks to an Agent Server over HTTP and WebSocket — same Agent, same method calls, same code. Prototype locally, and the exact code you wrote moves to a sandboxed production deployment by changing one constructor argument, not by rewriting the agent.

python
# prototyping on a laptop
conversation = Conversation(agent=agent, workspace="/home/me/project")

# the SAME agent, running against a sandboxed production server instead
conversation = Conversation(agent=agent, workspace=RemoteWorkspace(host="agents.internal:8080"))

# everything below this line is identical in both cases
conversation.send_message("Fix the failing test in payments/checkout.py")
conversation.run()

One argument changes; send_message, run, and everything the Agent decides to do stay exactly as written. That is what “the factory hides where execution happens” means concretely: not a promise in a design document, but a diff you could point at.

The minimal agent, traced line by line

Here is the SDK's own minimal example, exactly as the paper gives it:

python
from openhands.sdk import LLM, Conversation
from openhands.tools.preset.default import get_default_agent

llm = LLM(model="openhands/claude-sonnet-4-5-20250929", api_key="...")
agent = get_default_agent(llm=llm)
conversation = Conversation(agent=agent, workspace="/path/to/project")
conversation.send_message("Write 3 facts about this project into FACTS.txt.")
conversation.run()

Trace what each line actually constructs. LLM(…) builds an immutable model-configuration object — nothing about the conversation yet. get_default_agent(llm=llm) returns a fully frozen Agent: model, default tool set, default instructions, validated once, done. Conversation(agent=agent, workspace=…) is the factory call — a plain path string resolves to a LocalWorkspace, so this returns a LocalConversation, and its ConversationState starts with an empty EventLog. send_message(…) appends exactly one event — a message event — to that log; nothing has executed yet. run() is the only line that actually drives the loop: the Agent reads the current state, emits an action, the workspace executes it, the result is appended as an event, and the cycle repeats until the Agent decides it is done. Four lines to configure, two to run — and every piece of that four-line configuration is reusable for the next conversation without touching it again.

What building a whole new SDK bought over an existing one

It is fair to ask why a team would build this rather than adopt an existing agent SDK from a major provider. The paper's own comparison names specific gaps: at the time of writing, OpenHands SDK offered native sandboxed execution and lifecycle control (pause and resume a conversation mid-flight) where comparable offerings from OpenAI and Claude were library-only, leaving sandboxing and remote execution to the caller to build; it offered model-agnostic multi-LLM routing where competing SDKs were built around a single provider's models; and it paired an LLM-powered security analyzer with a built-in REST and WebSocket server for multi-user deployment, neither of which the alternatives shipped out of the box. None of these differences are exotic — they are exactly the four capabilities Chapter 5 through 7 walk through in detail — but naming them against specific competitors is a useful reminder that “build a good abstraction” is not free advice in the abstract; it is a real decision made because the alternatives, at the time, made a specific set of tradeoffs this project did not want to inherit.

Did the rebuild actually work? Fifteen days of production evidence

Chapter 0 flagged this number before the vocabulary to explain it existed; this is the chapter that explains it. The team ran V0 (the monolithic predecessor from earlier in this chapter) and V1 (everything built in Chapter 5 through 7) side by side, serving real users simultaneously, for fifteen days, and counted errors that were attributable to the harness's own infrastructure or logic — deliberately excluding LLM provider errors like rate limits, which are external to both architectures and would tell you nothing about which harness design is better.

Two details underneath that headline number are worth pulling apart, because they show the reduction came from exactly the architectural change this chapter described rather than from a general, unexplained improvement. V0's 69.8-per-1,000 infrastructure errors break down into three named causes: authentication failures between the conversation manager and the execution runtime running as separate processes (43.0 per 1,000), the runtime process not yet being ready when the conversation manager tried to reach it — a startup race condition between two independently-scheduled processes (18.8 per 1,000), and plain network connection or timeout errors (3.1 per 1,000). Every one of those three causes is a direct consequence of V0's mandatory two-process design — agent and sandbox running separately, coordinating over a network boundary that had to succeed for the conversation to proceed at all. V1's co-located, single-process default execution model — opt-in sandboxing, from earlier in this chapter — does not eliminate these bugs through better error handling; it eliminates the inter-process dependency that made them possible in the first place, which is why the infrastructure-error row drops to exactly 0.0, not merely a smaller number.

V1 is not error-free, and the table is honest about that: 29.7 SDK-internal errors per 1,000 conversations appear in a category that had no equivalent in V0 at all, because V0 had no separately-versioned SDK layer to have a bug in. The paper traces the majority of these to a specific condensation bug that surfaced only after a provider introduced new constraints on its message format during the rollout of extended-thinking support — the mechanism Chapter 6's event-hierarchy discussion touches on again shortly — and reports it as already fixed by the time of publication. That is worth reading as one more instance of Chapter 0's caution about a system's own honesty: a real redesign trades one category of failure for a different, smaller, more legible one, rather than promising zero failures outright, and the measured net effect — 78.0 down to 30.0 — is what actually matters for a team deciding whether the rebuild was worth it.

Chapter 6: The Event Loop: Action, Execution, Observation

Chapter 5 said the EventLog is where all mutable state lives, and left “events” mostly abstract. This chapter opens that box: what precisely gets appended when the agent decides to run a shell command, and why recording it this way — rather than as one more line in a chat transcript — is what makes crash recovery possible at all.

The event hierarchy

Every event shares a base Event type: immutable once created, carrying an id, a timestamp, and a source. A subtype, LLMConvertibleEvent, adds one method — to_llm_message() — that knows how to render itself as a message in the format the LLM's chat API expects. Concrete events specialize from there: SystemPromptEvent for the initial instructions, ActionEvent for a tool call the agent decided to make, ObservationBaseEvent for what came back from executing it. A separate family of internal events — CondensationRequest, PauseEvent, and others Chapter 7 will use — exists purely for the SDK's own bookkeeping and is deliberately invisible to the LLM: they never produce a chat message, they just steer what happens next.

The full family tree, and why each branch earns its own type

The paper's own event hierarchy has more branches than the four named above, and walking the rest of them is worth the time, because each one exists to answer a question a plain chat-history list cannot answer on its own. On the LLM-visible side, alongside SystemPromptEvent, ActionEvent, and ObservationBaseEvent: MessageEvent carries an ordinary user or assistant text message — the parts of a conversation that are not a tool call at all. CondensationSummaryEvent is what Chapter 5's condenser actually writes to the log — the LLM-authored summary that stands in for a batch of older events once the context condenser fires. And ObservationBaseEvent itself splits further: an ordinary ObservationEvent for a tool call that ran and returned a normal result, UserRejectObservation for the specific case where Chapter 7's security gate paused an action and a human said no, and AgentErrorEvent for a tool call that failed for reasons internal to the agent or its scaffolding rather than the environment it was acting on. Three different reasons a step might not have gone smoothly — a normal execution, a human veto, an internal failure — and three distinct, individually typed events rather than one generic “something happened” entry, which matters the moment you try to answer a question like “how often does a human actually reject a proposed action” by querying the log instead of re-reading transcripts by eye.

On the internal, bookkeeping side, alongside CondensationRequest and PauseEvent: ConversationStateUpdateEvent records a change to one of ConversationState's own metadata fields — the agent's status flipping to WAITING_FOR_CONFIRMATION, say — as its own typed, timestamped entry, rather than a silent in-place mutation the log has no record of. Condensation is the sibling of CondensationRequest: the request asks for a summarization pass, and Condensation is the record of what actually got forgotten and what the resulting summary contains, kept as two separate events for the same reason Chapter 6 already separates a decision from its outcome everywhere else — an ActionEvent from its ObservationBaseEvent, a request from its result.

The tool contract: Action → Execution → Observation

Every tool in the SDK follows the same three-part contract, and the names describe exactly what happens at each stage. An Action is the tool call's input, validated as a Pydantic model — if the LLM's tool call does not match the tool's declared schema, this is where it fails, before any code runs. Execution is the actual logic, implemented by a ToolExecutor — the part that really does open the file, run the shell command, or call the API. An Observation is the structured result handed back, ready for the LLM to read on its next turn. MCP tools plug into this same contract as first-class citizens: an MCP server's JSON Schema is converted automatically into an Action model, and whatever it returns surfaces as an ordinary Observation — a locally implemented tool and a remote MCP tool are indistinguishable once they are inside this pipeline. Full OAuth support for MCP servers means this extends to tools that require an authenticated session with an external service, not just tools you can run unauthenticated on your own machine — the same SecretRegistry from Chapter 7 handles the credential, and the same Action/Execution/Observation contract handles everything else, regardless of whether “execute” means running local Python or making an authenticated call to someone else's server.

Trace one tool call, event by event

Concretely: the agent decides to run the test suite. Here is what actually gets appended to the EventLog, in order, and this is the level of detail worth being able to reproduce from memory.

Two events, not one, and that separation is deliberate. The ActionEvent records a decision; the ObservationBaseEvent records what actually happened when that decision was carried out. They are two different facts with two different timestamps, and collapsing them into a single log line would throw away exactly the information Chapter 5's replay guarantee depends on.

From event log to chat messages, concretely

The LLM never reads the EventLog directly — it reads whatever to_llm_message() renders each LLMConvertibleEvent into. Trace a short, four-event log through that method by hand:

event log (append-only, on disk)
evt_01  SystemPromptEvent   {content: "You are a coding agent..."}
evt_02  MessageEvent        {role: "user", content: "Run the test suite."}
evt_03  ActionEvent         {tool: "bash", action: {command: "pytest tests/"}}
evt_04  ObservationEvent    {stdout: "12 passed, 1 failed", exit_code: 1}

to_llm_message() rendering — what the LLM API actually receives
[ {"role": "system",    "content": "You are a coding agent..."},
  {"role": "user",      "content": "Run the test suite."},
  {"role": "assistant", "content": null, "tool_calls": [{"name": "bash", "args": {"command": "pytest tests/"}}]},
  {"role": "tool",      "content": "12 passed, 1 failed (exit 1)"} ]

Four persisted events become four chat messages, in the same order, each event owning exactly one message. The EventLog is the durable record; the message array is a disposable projection of it, rebuilt fresh on every LLM call. Nothing about the message array is ever saved — if it were, you would be back to the fragile, un-replayable design Chapter 5 warned against.

Events the LLM never sees

Not every event implements LLMConvertibleEvent. Internal bookkeeping events — CondensationRequest (Chapter 7's context condenser asking for a summarization pass) and PauseEvent (Chapter 7's security gate freezing execution) among them — are ordinary entries on the same append-only log, timestamped and replayable exactly like an ActionEvent, but they simply have no to_llm_message() method. They steer the harness's own control flow without ever becoming a line the model reads. This is the same append-only discipline Chapter 5 introduced, doing double duty: one log records both the conversation the model experiences and the operational events the harness needs to coordinate itself, cleanly separated by which events know how to render themselves as chat.

Writing a tool: the contract, once more, as code

Concretely, a new tool means implementing the same three-part shape every built-in tool follows. Sketch a minimal write_file tool to see how little ceremony the contract requires:

python
class WriteFileAction(BaseModel):        # the Action — validated LLM input
    path: str
    content: str

class WriteFileExecutor(ToolExecutor):  # the Execution — what actually runs
    def execute(self, action: WriteFileAction) -> Observation:
        Path(action.path).write_text(action.content)
        return Observation(content=f"wrote {len(action.content)} bytes to {action.path}")

That is the entire contract: a Pydantic model the LLM's tool call gets validated against, and an executor whose return value becomes the next Observation on the log. Nothing here differs from how Chapter 7's delegation tool or the built-in bash tool work — which is exactly the point the next chapter leans on.

Deterministic replay and how the state actually gets saved to disk

Because ConversationState is nothing but metadata plus an append-only log, reconstructing a conversation from scratch is just: load the metadata, then replay every event in order. Persistence follows the same split. Metadata — agent status, running stats, the confirmation policy — serializes to one small file, base_state.json, rewritten each time it changes. Events are different: each one persists as its own individual JSON file in an EventStore, and only new events get written — the ones already on disk from three hundred steps ago are never touched again.

Put a number on why that incremental design matters. Suppose a conversation is 200 events deep, each event file averaging roughly 2 KB on disk — a plausible size for a small JSON record with a command and its output (illustrative figures, to show the shape of the argument, not numbers the paper states). A naive persistence scheme that rewrites the entire conversation to one file on every single new step would write:

The SDK's incremental EventStore writes only the new files — one for the new ActionEvent, one for its ObservationBaseEvent:

This is the same shape of argument as Chapter 2's context budget and Chapter 4's environment-representation ablation, now applied to disk instead of tokens: doing the minimum necessary work per step, rather than redoing everything that already happened, is what lets a system stay cheap as it scales, whether the resource being spent is context tokens or write bandwidth.

The illustrative estimate, checked against the paper's real measurements

That 200-event, 2 KB-per-file estimate was flagged as illustrative on purpose — here is what the paper actually measured, so the shape of the argument above can be checked against real numbers rather than left as a plausible guess. The team replayed real production traffic through the exact persistence path described above: 433 completed SWE-bench Verified conversations, 39,870 events in total, an average of roughly 92 events per conversation. Four numbers from that replay are worth holding onto, reported as a median and a 95th-percentile (P95) to show both the typical case and a realistic worst case:

Read the two most load-bearing rows against what an LLM call itself costs in wall-clock time. A single LLM round trip typically takes somewhere in the range of 1 to 30 seconds. Persisting one action-observation pair costs 0.40 milliseconds at the median:

Even at the slow end — a 30-second LLM call and the worst-case P95 persist latency of 0.56 ms — the event log adds roughly 0.002% overhead to a step that was already going to take thirty seconds regardless. Crash recovery, the operation Chapter 5's whole append-only design exists to make possible, completes in under 20 milliseconds even for the single longest conversation observed in the entire 433-conversation sample. The illustrative “100× less I/O” argument above was reasoning about the right shape of the tradeoff from first principles; these numbers are the paper's own confirmation that in a real system built this way, event-sourcing overhead is not a theoretical nicety that happens to sound good — it is negligible enough, in absolute wall-clock terms, that it never has to enter the conversation about where a real deployment's latency budget is actually going.

Resume, sketched as code

Put the crash-recovery claim from earlier in this chapter into a shape you could actually implement. Loading a conversation back is: read the small metadata file, then walk the EventStore's individual event files in order, and check whether the last ActionEvent has a matching Observation after it.

python
state = ConversationState.load("base_state.json")          # small file, metadata only
events = EventStore.load_all(conversation_id)          # one file per event, replayed in order
state.event_log = events

last_action = next((e for e in reversed(events) if isinstance(e, ActionEvent)), None)
last_obs    = events[-1] if isinstance(events[-1], ObservationBaseEvent) else None

if last_action and not last_obs:
    # the exact incomplete-conversation shape from earlier in this chapter
    resume_from(last_action)      # re-check or re-run just this one step
else:
    resume_from(len(events))  # everything completed cleanly — continue from the next turn

Every line of that sketch reads directly from what has already been established: metadata and events persist separately (Chapter 5), events are typed so “is this an ActionEvent” is a real, checkable question rather than string-matching a chat transcript, and the append-only ordering means “the last thing that happened” is always unambiguous. None of this requires a database or a distributed consensus protocol — it requires the discipline of never mutating a record once it is written.

Chapter 7: Delegation, Sandboxing & Security

Three more pieces complete the harness: how a parent agent hands work to sub-agents without every sub-investigation's noise flooding the parent's own context, where the code actually executes, and what stops a risky action before it runs rather than after. All three turn out to be built from the same small vocabulary Chapter 6 already introduced.

Why swappability isn't cosmetic: two real benchmarks

Chapter 5's immutable, swappable LLM object is not an abstract nicety — it is what makes it cheap to ask “which model should actually be running this agent?” and get a real answer, and this is where that pays off in numbers rather than just architecture. Two of the rows below come directly from the paper's own Table 4, which isolates exactly the effect Chapter 5's whole redesign was for: it runs Claude Sonnet 4 and Claude Sonnet 4.5 through both V0 and V1 on SWE-bench Verified, changing nothing but which architecture is executing the identical model. Claude Sonnet 4 scores identically on both, 68.0%, confirming the rebuild preserves raw model capability rather than quietly degrading it. Claude Sonnet 4.5 gains 8.2 points under V1 (64.6% → 72.8%), which the paper attributes to extended-thinking support — the ThinkingBlock and ReasoningItemModel objects from Chapter 5's LLM abstraction — that V1's event-sourced design integrates natively but would have needed significant retrofitting work inside V0's older, multi-process design. The full four-model, two-benchmark table below extends that same evaluation harness with GAIA scores plus two more models, GPT-5 and Qwen3 Coder 480B. Those additional numbers are not printed in the paper's own tables — across all fourteen models it evaluates, the paper explicitly defers full per-model results to its companion, continuously-updated OpenHands Index (index.openhands.dev), which it cites by name rather than reproducing the entire fourteen-model table on the page itself. Same SDK, same evaluation harness, same underlying claim about swappability; the wider table below is simply where the extra rows actually live:

Compute the spread on each benchmark. SWE-bench Verified is relatively tight across all four:

GAIA is not tight at all:

The best and worst models in this table are separated by 12 points on one benchmark and 27 points on another — and which model is “best” for your agent depends entirely on which kind of task it spends most of its time doing. A harness where swapping models costs one line lets you actually run this comparison on your own workload and act on it. A harness where the model is wired into a dozen call sites turns the same question into a multi-day migration, which in practice means most teams never ask it and just keep whatever model they started with.

How the SDK itself stays reliable: three tiers of testing

“Reliable execution” from the paper's own stated goals is not just an architecture claim — the project backs it with three explicit tiers of testing, each trading cost for realism. Programmatic tests run on every commit, mock the LLM entirely, and finish in seconds. LLM-based tests run daily or on demand against real models, cost roughly $0.5 to $3 per run, and finish in under five minutes. Full benchmark evaluation — the SWE-bench and GAIA runs in the table above — runs on demand, costs $100 to $1,000, and takes hours, because it means actually solving hundreds of real tasks end to end.

Do the arithmetic on the middle tier, since it is the one a team would plausibly run continuously. Twice a day, every day, at the top of the stated range:

Cheap enough to run continuously as a matter of course, which is exactly the point of having a middle tier at all — the $100–$1,000 benchmark tier is too expensive to run on every change, and the free programmatic tier cannot catch the class of bug that only shows up when a real model is actually in the loop. Three tiers, three different price points, each catching what the others cannot afford to catch as often.

Delegation is just another tool

A sub-agent is not a new kind of object in the SDK. It is implemented as a standard tool: the Action is “spawn a sub-conversation to handle this task,” the Execution is running an independent Conversation — inheriting the parent's model configuration and workspace — through to completion, and the Observation is that sub-conversation's final result, handed back to the parent exactly like a bash command's stdout would be. The current implementation is blocking, parallel execution: a parent can spawn several sub-agents at once, each running independently, and the parent waits for all of them to finish before continuing — a fork, then a join.

Read “blocking, parallel” honestly, though: it means the parent cannot yet continue doing other useful work while a slow sub-agent finishes, and if one sub-agent in a fork hangs, the whole join waits on it. The paper names exactly these gaps — async delegation, dynamic scheduling, fault-tolerant recovery — as open directions rather than solved problems, which is a useful thing to notice about any systems paper: the list of what a design deliberately does not yet handle is as informative as the list of what it does, and it tells you where to be careful adopting it as-is.

Delegation as context management, not just parallelism

The obvious reason to delegate is speed: three sub-agents investigating three modules in parallel finish faster than one agent investigating them in sequence. The less obvious reason is context isolation, and it is worth an illustrative worked example (round numbers, to show the shape of the argument, not a reported result).

Suppose investigating one module of a codebase takes about 40 events, and each event averages roughly 300 tokens once rendered as a chat message. Done inside one shared conversation across three modules:

Delegate each module to its own sub-agent conversation instead, and each sub-agent's context only ever holds its own investigation:

The parent's own context only grows by whatever each sub-agent chooses to report back — a concise summary, not a full transcript. Three summaries at, say, 500 tokens each add 1,500 tokens to the parent, not 36,000:

Delegation, read this way, is Chapter 2's context-budgeting lesson applied one layer up the stack: instead of choosing which workflows to inject, you are choosing which sub-investigations to keep out of the parent's context entirely, letting them run their noisy exploration somewhere it cannot compete with the parent's own reasoning for space.

The context condenser: the same problem, one conversation at a time

Even without delegation, a single long-running conversation eventually approaches its context limit just from accumulating its own events. The SDK's default answer is the LLMSummarizingCondenser: when the EventLog grows past the usable context, it drops a batch of older events and inserts a single CondensationEvent — an LLM-written summary — in their place. The full, uncondensed events remain on disk in the EventStore (nothing is destroyed), but what actually gets sent to the LLM on the next call is the summary plus whatever recent events have not yet been condensed. The paper reports this reduces cost by up to 2× without measurable performance degradation.

Illustrate the mechanic with round numbers. A conversation has grown to roughly 128,000 tokens' worth of events, right at a typical context limit. The condenser drops the oldest half and replaces it with a 2,000 token summary:

The condensation event, notice, is stored on the log exactly like any other event — it is not a special exception to Chapter 6's append-only rule, it is one more entry that happens to summarize its predecessors rather than record a tool call.

Where the code actually runs: the Workspace abstraction

Version zero's mistake, recall from Chapter 5, was making Docker sandboxing mandatory for every agent. Version one replaces that with a single BaseWorkspace interface and three implementations you choose between per conversation:

The same Conversation code from Chapter 5 works unchanged against any of the three — the Workspace is a constructor argument, not a rewrite. A prototype running LocalWorkspace on a laptop moves to DockerWorkspace or RemoteWorkspace in production by changing that one argument.

Security: separating what is risky from what is allowed

Sandboxing controls the blast radius if something goes wrong. Security analysis tries to stop the wrong thing from happening in the first place, and the SDK splits this into two deliberately independent pieces. A SecurityAnalyzer looks at a proposed Action and rates it — LOW, MEDIUM, HIGH, or UNKNOWN. A separate ConfirmationPolicy decides, given that rating, whether the action needs a human's explicit approval before it is allowed to execute. The built-in default pairs an LLMSecurityAnalyzer (an LLM call that appends a security_risk field to each tool call before execution) with a ConfirmRisky policy (blocks anything above a configured threshold).

When a risky action is caught, the agent does not silently refuse or silently proceed — its agent_status transitions to WAITING_FOR_CONFIRMATION, an explicit state on ConversationState, and execution pauses there until a human responds. On rejection, the agent may retry with a safer alternative rather than simply failing. Trace it the way Chapter 6 traced a bash call:

Secrets: the case where the LLM must not see the value it is using

A last piece, easy to miss: the SecretRegistry. Credentials are stored per-session, accessed only at the moment a tool actually executes — never embedded directly in a prompt the LLM reads — and can be a static value or a callable, which matters for short-lived tokens that need refreshing mid-conversation. The registry also auto-masks: if a command's output happens to echo a secret value verbatim (a misconfigured script printing an API key, say), the Bash tool replaces every occurrence with <secret-hidden> before that output is ever wrapped into an Observation the model can read. The secret is usable by the tool executing the action, and invisible to everything downstream of it — including the model whose entire job is deciding what to do next.

python
registry.register("GITHUB_TOKEN", refresh_github_token)   # a callable, not a static string
# refresh_github_token() runs only when the bash tool actually needs it —
# never embedded in the prompt, and any literal occurrence in stdout gets
# replaced with  before the Observation is built

Registering a callable instead of a fixed string is what makes live rotation possible: a short-lived OAuth token that expires in an hour can be refreshed transparently on its next use, with no conversation restart and no code change in the agent itself — the SecretRegistry is doing for credentials exactly what the LLM object does for models in Chapter 5, hiding a swappable, time-varying detail behind one stable interface.

Reading the tradeoffs as one table, not three isolated choices

Delegation, workspace, and security are not three unrelated features bolted on next to each other — they compose along the same axis. A parent agent running locally with security off is fastest and least safe. Swap in DockerWorkspace and a strict ConfirmationPolicy and the same code, unchanged, becomes slow enough to notice and safe enough to hand to an intern's laptop. Delegate to sub-agents and each one can independently choose its own point on that same tradeoff — a sub-agent reading documentation might run locally with no gate, while a sub-agent asked to modify production configuration runs sandboxed with every action confirmed. None of that differentiation needed a special case in Agent, Conversation, or the EventLog; it is Workspace and SecurityAnalyzer being ordinary constructor arguments, chosen per conversation.

Chapter 8: Showcase — a self-improving agent inside a harness

Everything from both papers, in one running system. Picture a coding agent built on the OpenHands SDK — Agent, Conversation, EventLog, security analyzer, all of it — with an AWM-style workflow memory layered on top as ordinary text injected into its system prompt. A stream of tasks arrives. Some recur. Some steps are risky. Watch what each piece of the harness actually buys, in isolation and together.

How to read the panel

Row 1, the task queue. Each square is one task, colored by type — fix a failing test, add an endpoint, refactor a module, update docs — task types repeat across the stream, exactly like WebArena's task templates repeated across Chapter 0’s forty occurrences. The current task is outlined.

Row 2, workflow memory. One bar per task type, height proportional to how many workflows have been induced for it so far. With memory on, a bar grows every time that task type succeeds — the online induction loop from Chapter 3, running live. With memory off, every bar stays at zero: the goldfish agent from Chapter 0, exactly as introduced.

Row 3, the event loop. The current task’s Action → Execution → Observation cycle from Chapter 6, ticking step by step. When the security analyzer is on and flags a step, this row shows WAITING_FOR_CONFIRMATION instead, and the simulation pauses until you press approve.

Row 4, running success rate. A rolling record of finished tasks, colored by outcome. The number above the line is a plain running average — after i+1 tasks have finished, it is (successes so far) ÷ (i+1). Trace the first four outcomes of a hypothetical run by hand: success, fail, success, success gives running rates of 1÷1 = 100%, then 1÷2 = 50%, then 2÷3 ≈ 67%, then 3÷4 = 75%. Early in any run this number swings hard on a single outcome; watch it settle down as more tasks accumulate, which is itself a small, live lesson in why a handful of trials is not enough evidence to trust a reported success rate — the same caution Chapter 4 applied to reading a benchmark table applies here to reading four data points off a live chart.

Why these two toggles and not more? Because they are the two independent claims this whole session makes: AWM changes what an agent knows going in, OpenHands SDK changes how safely and reliably it acts once it starts. Four task types is enough to show memory accumulating unevenly across categories — exactly the pattern Chapter 2's utility rate measured, where some task types get covered and others do not — without the panel turning into an unreadable wall of bars.

One honest limitation of the toy model, worth stating outright: the simulation’s pass/fail outcome tracks task correctness only. A real security analyzer is not there to raise your success rate — it is there to catch the rare action whose failure mode a correctness label does not even capture, like leaking a credential or deleting the wrong directory. In this panel, toggling security changes how many steps a task takes and how often you are asked to approve something; toggling memory changes whether the agent succeeds and how fast. Keeping those two effects visibly separate is deliberate — in a real system they are separate concerns for exactly the reason Chapter 7’s split between SecurityAnalyzer and ConfirmationPolicy was.

Two more simplifications, named so the panel is not mistaken for a faithful reproduction of either paper. Success in this simulation is a weighted coin flip; a real L_eval is an actual check — a test suite passing, a diff matching a spec — run against an actual completed trajectory, not a probability sampled from a formula. And the security gate fires at a fixed, flat chance per task in this panel; a real LLMSecurityAnalyzer makes an actual judgment about the actual proposed command, so its trigger rate depends entirely on what kind of work the agent is doing, not on a dial you can turn. What the panel gets right is the shape of both effects — memory compounds with recurrence, security adds a bounded but real tax — and that shape is worth internalizing even though the exact numbers behind it are toy ones.

The formulas behind the panel, worked by hand

The simulation is honest about its own mechanics — here they are, in full, so nothing behind row 2 or row 4 is a black box. For a task of type t with w workflows accumulated so far, the success probability and the number of action–execute–observe cycles needed are:

Trace two tasks of the same type by hand. First occurrence, w = 0: p = 0.35 + 0 = 0.35, steps = 6 − 0 = 6. Suppose it succeeds — memory now holds w = 1 for that type. Fourth occurrence of the same type, w = 3 by then: p = 0.35 + min(0.45, 0.12×3) = 0.35 + 0.36 = 0.71, steps = 6 − min(3,3) = 3. Twice as likely to succeed and half the steps, purely from three prior successes accumulating in memory — the exact shape of Chapter 3's online loop, just with invented constants standing in for the paper's measured ones so you can watch the mechanism move in real time instead of reading it off a static table.

These round constants (0.35 base rate, 0.12 per workflow, a cap at 0.92 so no task type ever becomes a sure thing) are chosen to be legible, not fitted to any number in either paper. The real, paper-measured versions of these same two effects are Chapter 3's Mind2Web table for memory's effect on success rate, and Chapter 4's WebArena step-count table for memory's effect on steps. A static table cannot show you both moving together, turn by turn, as memory accumulates — which is the one thing this panel is for that the tables are not.

How the toy curve compares to Chapter 3's real one

It is worth checking the panel's invented formula against the real curve Chapter 3 measured, not to claim they match — they will not, exactly — but to see whether the toy version at least points the same direction as the paper's own evidence. Chapter 3's Figure 5 result showed the gap between an online agent and a memory-less baseline on WebArena's Maps split rising to roughly 22.5 percentage points within about 40 examples. Run the panel's own p(t) formula out to a comparable point — a task type that has recurred often enough to reach w = 5 accumulated workflows, well within reach of a 30-task stream where each of four types recurs seven or eight times:

Forty-five points is not the paper's 22.5 — and it should not be, since the panel's constants were chosen to make the mechanism visible on a short, four-type toy stream rather than fitted to reproduce Chapter 3's exact number. What the comparison does confirm is the shape: both curves rise fastest over the first several recurrences and then flatten as the gain per additional success shrinks (the panel enforces this with an explicit cap at 0.92; the real WebArena curve does the same thing implicitly, by leveling off once the essential workflows for a site have already been learned). A toy simulation that got the direction right but the flattening wrong would be teaching a false lesson about diminishing returns; this one does not, which is the one property worth checking before trusting round numbers to stand in for measured ones anywhere in this lesson.

Extending the hand-trace: what the running average looks like over a longer stretch

Chapter 8's “how to read the panel” section traced the running success rate by hand through four outcomes: 100%, 50%, 67%, 75%. Push that same trace eight outcomes further, alternating in a plausible pattern of mostly-successes-with-occasional-misses once memory has had a chance to accumulate, and watch what the running average actually does as the sample grows.

Compare the swing at the start of this trace to the swing at the end. Outcome 7 — the first miss in this extended sequence — drags the running rate down by 11.9 points in one step (83.3% → 71.4%). Outcome 12, also a miss, drags it down by only 6.8 points (81.8% → 75.0%), even though both are single failures. The same single data point moves the average less and less as the sample it is averaged into gets larger — which is precisely the caution Chapter 4 attached to reading any small-sample benchmark gap, now visible as arithmetic you can watch happen in real time rather than a warning to take on faith. A running success rate reported after 5 finished tasks and the identical-looking number reported after 50 are not equally trustworthy evidence, even when they happen to read the same, and this is the mechanical reason why.

The state machine, in the same vocabulary as Chapter 6

Row 3's ticking status is not decoration — it is Chapter 6's Action → Execution → Observation loop, running for real inside the simulation, with Chapter 7's security gate wired in exactly where it would sit in the actual SDK:

pseudocode — one simulated task
stage = 'action'                                    # the LLM decides a step
if security_on and risky(step) and not yet_gated_this_task:
    stage = 'gate'                                  # agent_status -> WAITING_FOR_CONFIRMATION
    # step() is now a no-op until c8Approve() fires
else:
    stage = 'execute' -> 'observe'              # Chapter 6's ActionEvent + ObservationEvent pair

if steps < needSteps(workflow_count):        # fewer needed as memory grows — Chapter 3
    stage = 'action'                                # loop for another cycle
else:
    stage = 'result'                                # roll success ~ p(workflow_count) — Chapter 1-3
    if success and memory_on: workflow_count += 1   # the online induction step, Chapter 3

Every branch in that pseudocode names the chapter it came from because none of it is new machinery invented for the showcase — it is the two papers' mechanisms, wired into the same loop, with round numbers standing in for the calls a real system would make to an actual induction function and an actual security analyzer.

What changes at the edges of the slider

Push the stream-length slider to its minimum, 5 tasks, and memory barely gets a chance to show its effect — each task type recurs at most once or twice, which is close to Chapter 2's world of a mostly-empty offline library. Push it to the maximum, 30 tasks, and each of the four types recurs seven or eight times, enough for its bar to approach the step-count floor and the success-probability cap — the regime Chapter 3's online loop is built for, where a task type has been seen often enough that memory has said everything it has to say. Watching the same four-type memory panel behave differently at 5 versus 30 tasks is a compressed, interactive version of exactly the question Chapter 0 opened with: how much does recurrence count actually matter, and the honest answer is that it matters more, not less, as the stream gets longer.

Four experiments, run them

1. Memory off, security off. The baseline goldfish agent. Every occurrence of a task type costs roughly the same number of steps and succeeds at roughly the same, unimproving rate, no matter how many times it recurs.

2. Memory on, security off. Watch one task type’s bar grow after its first few successes, and watch that same type’s later occurrences take fewer steps and succeed more often — Chapter 3’s online loop, made visible.

3. Memory off, security on. Occasional pauses, no improvement in success rate or steps otherwise — the harness is safer but not smarter.

4. Both on. The full stack: an agent that gets measurably better at what it repeats, inside a harness that still stops to ask before doing something risky. Neither property substitutes for the other, and a production system wants both simultaneously, which is the entire argument for treating “harness” as more than one problem.

One run, narrated

Set both toggles on and press play for a moment, and here is roughly what you are watching happen underneath the animation. Task 1 arrives as “fix test” — memory has zero workflows for that type, so it runs the full 6-step cycle at the base 35% success chance. Say it succeeds: the teal bar for “fix test” grows by one. Task 2 is “add endpoint,” a different type — no benefit carries over, because AWM's memory is scoped by what recurs, not a general skill boost; it runs its own full cycle from zero. Task 5 happens to be “fix test” again, and now the panel shows it plainly: fewer steps, a visibly fuller bar, and a better chance the outcome is teal rather than red. Somewhere in the run, a step gets flagged HIGH risk — the event-loop row turns red and freezes on WAITING_FOR_CONFIRMATION until you click approve, regardless of how many workflows that task type has accumulated, because Chapter 7's gate does not know or care what memory contributed to the decision it is now checking. Two independent systems, visibly doing two independent jobs, on the same stream of work.

What you would actually monitor, if row 2 and row 4 were a real dashboard

The panel draws four rows because four rows fit legibly on a screen and tell this chapter's story. A real production system watching a fleet of agents would not stop there, and it is worth naming what the next layer up looks like, since Chapter 9 returns to this exact gap. Row 2's per-type bar height is a live version of Chapter 2's utility rate — what fraction of this task type's occurrences found a matching workflow — except the panel only ever shows you the current moment, not the trend. A real dashboard would plot that rate per task type over days or weeks, so a type whose coverage is quietly declining (a website redesign going stale, in Chapter 1's sense) shows up as a downward line long before anyone notices task success dropping. Row 3's gate events are a live version of Chapter 7's SecurityAnalyzer firing — except the panel fires it at a flat, fixed rate per task, while a real LLMSecurityAnalyzer's trigger rate depends entirely on what kind of work is actually happening. A real dashboard would track that rate per task type too, and flag any type whose gate-trigger rate suddenly spikes, since a rising rate of flagged actions on a previously calm task type is itself a signal worth a human's attention, independent of whether any individual flagged action turns out to be a false alarm.

Row 4's running success rate is the one row where the panel's honesty limit is most worth restating plainly: it is a live number computed the same way any of this chapter's hand-traced examples were computed, and the “extending the hand-trace” section above showed exactly why a rate reported after five finished tasks and one reported after fifty are not equally trustworthy, even when they read the same. A real dashboard answers that by reporting a confidence interval alongside the point estimate, not just the point estimate alone — a discipline this lesson's own tables, including the paper's, do not show either, and Chapter 4 flagged that same gap explicitly when it asked how many test tasks a reported percentage was actually computed over. None of this is a criticism of the panel — a four-row toy simulation was never meant to be a production observability stack — it is a map from what the panel does show to what a real deployment would need to add on top of it, which is exactly the kind of layer Chapter 9 names as the piece neither paper in this session owns.

Chapter 9: Beyond: What a Good Harness Abstracts

Two papers, read as one argument. AWM answers “how does an agent get better at what it repeats?” OpenHands SDK answers “how does the code around the agent stay reliable while it does?” Put them side by side and a shared design instinct becomes visible — the same one, applied at two different layers of the stack.

The instinct they share

Look again at the “where it lives” and “when it changes” rows. AWM keeps the model's weights completely untouched and puts everything it learns into a small, growable, text-based memory that sits beside the frozen model. OpenHands SDK keeps every piece of configuration — Agent, LLM, tool specs — completely immutable and puts everything that happens into a small, growable, append-only event log that sits beside that frozen configuration. Both papers independently arrived at the same shape: freeze what defines the system, and let exactly one clearly-owned, ever-growing record carry everything the system has learned or done. Neither paper fine-tunes anything to hold state. Both papers instead build a disciplined place to write things down.

Wiring the two together, concretely

Neither paper requires the other, but nothing stops them composing, and seeing exactly how clarifies both. AWM's induction step (Chapter 1) needs a completed, successful trajectory to run on; OpenHands' ConversationState (Chapter 5) is precisely a completed trajectory, already structured as typed events instead of a loose chat log. A synthesis is a few lines, not a new architecture:

python
def on_conversation_end(state: ConversationState):
    if state.outcome == "success":                     # the L_eval signal from Chapter 3
        trajectory = [e for e in state.event_log        # Chapter 6's typed events ARE the
                      if isinstance(e, (ActionEvent, ObservationBaseEvent))]  # trajectory AWM's induce() expects
        workflow = induce([trajectory])                    # Chapter 1's induction function, unchanged
        workflow_library.append(workflow)                  # Chapter 2/3's memory, now durable

def get_default_agent(llm, task_site):
    workflows = [w for w in workflow_library if w.site == task_site]  # Chapter 2's scoping
    return Agent(llm=llm, instructions=base_instructions + render(workflows))     # Chapter 5's frozen spec

Read what each side contributes. AWM never needed to know anything about ConversationState, EventLog, or Workspace — it only ever needed a successful trajectory, in whatever shape it comes. OpenHands SDK never needed to know anything about workflows, induction, or memory — it only needed Agent.instructions to be an ordinary string, which it always was. Two papers, solving genuinely different problems, meet at exactly one seam: a trajectory in, a string of guidance out. That narrow seam is what “composable harness layers” means in practice, not just in the abstract.

If you can only build one of these first

A practical closing question, for anyone about to actually build something from this session rather than just read it. If your agent's biggest visible problem is that it keeps failing or floundering on tasks it has never seen before, start with the harness — Chapter 5 through 8's event sourcing, sandboxing, and security gate — because no amount of memory helps an agent that cannot survive a crash or safely attempt a risky action in the first place. If your agent is already reliable but visibly wasteful — succeeding, but re-deriving the same route every time on tasks you know recur — start with Chapter 1 through 4's workflow memory, layered on whatever harness you already have, even a simple one. The two are not sequenced by importance; they are sequenced by which failure is actually costing you the most right now, and Chapter 0's two failures were kept deliberately separate for exactly this reason — so you could recognize which one you are looking at.

A small, real coincidence worth noticing

Graham Neubig is the last author on both papers this session builds — AWM's memory-layer research and OpenHands SDK's execution-layer research, coming out of overlapping groups working on agents from two different angles at once. That is not a coincidence you should read too much into, but it is a useful data point about how this kind of research actually happens: the people building the memory layer and the people building the execution layer are often the same people, on the same project, discovering in practice that neither one alone was the whole story. This session's argument — that a harness is more than one problem — is not just a pedagogical framing device; it is close to how the field building these systems arrived at the same conclusion.

What a good harness abstracts, as a checklist

Every row on that table is a place where a hand-rolled agent loop typically special-cases something — hard-codes a provider, keeps history in whatever variable was convenient, discards a successful trajectory the moment the process exits, executes first and asks questions never. A harness is what turns each of those special cases into a designed, reusable answer.

Walk the five rows once more, each tied back to the chapter that built it. Which model is thinking is Chapter 5's LLM object — a hundred-plus providers, reasoning models, and non-function-calling models all behind one interface, so a model swap is a constructor argument, not a rewrite. What actually happened is Chapter 6's append-only EventLog, which is what let a crash mid-tool-call be diagnosed precisely instead of leaving an ambiguous gap. What was learned is Chapter 1 through 3's induced workflow library, the one piece of this checklist that AWM owns outright and OpenHands SDK does not attempt. What is safe to do is Chapter 7's split between SecurityAnalyzer and ConfirmationPolicy, assessment and enforcement kept independently swappable. Where it runs is Chapter 7's Workspace abstraction, the same Conversation code working unchanged against a laptop, a container, or a remote server.

What the redesign explicitly considered and rejected

A design decision is usually easier to trust once you know what the alternative actually was, and the OpenHands SDK paper is unusually explicit about naming the roads it did not take, in its own closing discussion. For state management, the team considered a traditional database-backed model — the default most web applications reach for. They rejected it for two reasons that trace directly back to everything Chapter 5 and 6 built: a database ties the SDK to one specific storage backend, and it makes offline replay — Chapter 5's entire crash-recovery story — harder to reason about than an append-only log any tool can read as plain JSON files on disk. Event sourcing was chosen specifically for reproducibility and for being storage-agnostic, not because it was the more familiar or conventional choice.

For execution isolation, the team weighed the two extremes against each other directly: mandatory containerization everywhere (V0's actual mistake, from Chapter 5) against fully local-only execution with no sandboxing option at all. Neither extreme won. The optional-isolation model Chapter 7's Workspace abstraction implements is a deliberate middle position — aligned with MCP's assumption that tools can run locally with direct access to credentials and files, while still preserving a sandboxed path for the situations that actually need one. And for the package structure, the team weighed one monolithic SDK (simpler to manage as a single dependency) against a full microservices split (maximum flexibility, at the cost of far more moving parts to version and deploy together). The four-package design Chapter 5 opened with — sdk, tools, workspace, agent_server — is explicitly a balance point between those two extremes, arrived at by comparison, not an obviously correct answer nobody had to argue for.

Naming all three rejected alternatives matters for a reason beyond historical interest: it shows that Chapter 5's one rule — freeze configuration, keep exactly one mutable record — was not the only design on the table, and each alternative traded something away rather than being simply worse in every respect. A database-backed ConversationState would have been easier to query with off-the-shelf tooling a team might already know. Fully local-only execution would have been simpler to reason about than an optional Workspace abstraction with three implementations to keep behaviorally identical. A monolithic package would have meant one dependency to version instead of four. Every one of these three alternatives loses to what Chapters 5 through 7 actually built, for the specific goals this paper set out with — reproducibility, MCP compatibility, independent testability — but a different project with different goals could reasonably land on a different answer at any one of these three forks, which is worth remembering before treating this session's design as the one correct way to build a harness rather than one well-justified answer to one specific set of requirements.

A caution the OpenHands paper itself demonstrates

Version zero's mistake was not having no abstraction — it was one abstraction applied too bluntly: mandatory Docker sandboxing for every single agent, whether the task needed isolation or not. There is an old line in computer science, often attributed to David Wheeler: “All problems in computer science can be solved by another level of indirection — except the problem of too many levels of indirection.” A harness is exactly that kind of indirection layer between what a model wants to do and what actually happens. Version one's fix was not to remove the layer — sandboxing, security analysis, and event sourcing are all still there — it was to make each layer optional and composable rather than mandatory and fused to everything else. The lesson generalizes past this one SDK: build the abstraction, then make sure it can be turned off exactly where it is not needed.

The layer this session leaves for someone else to build

Notice what neither paper owns: knowing whether the whole system is getting better or worse over time, across many deployed conversations, not just within one. AWM's workflows and OpenHands' event log both make a single conversation legible — you can inspect what memory a task used, or replay exactly what a crashed agent did. Neither paper is about aggregating that legibility across a fleet of agents running in production: which workflows are actually earning their context budget, which security gates fire often enough to be worth a policy change, which task types still have no matching memory six months after launch. That is the evaluation and observability layer, a natural next question once memory and execution are both solid, and it is exactly where this site's AI Evaluation and Harness Optimization lessons pick the thread back up.

The OpenHands SDK paper is explicit about two more gaps it does not close, in its own words rather than this lesson's inference, and both are worth stating precisely rather than left implicit. First, the current implementation focuses on single-agent conversations — Chapter 7's delegation tool spawns independent sub-conversations, but coordinating genuinely multiple agents working on a shared problem at once, negotiating over a shared resource or dividing labor dynamically, is named as requiring further design work the paper does not claim to have finished. Second, and more pointed: the security framework, “while substantially improved over V0, cannot guarantee complete safety” — an LLM-based SecurityAnalyzer is itself a model making a judgment call, and a model's judgment can be fooled by an adversarially-crafted prompt or simply misclassify an action inconsistently between two structurally similar calls. Chapter 7 built the mechanism that pauses on HIGH-risk actions; it did not, and could not, build a guarantee that every genuinely risky action gets rated HIGH in the first place. A comprehensive multi-tenant security audit, the paper adds, remains explicitly future work rather than something already validated. Naming these limits in the paper's own words is not a weakness of the design — it is the same honesty Chapter 0's misconception callout asked for from the start, applied to the paper's own claims about itself rather than only to what came before it.

The paper's own feature comparison, for reference

Chapter 5 named specific gaps against competing SDKs in prose; the paper backs that claim with an actual feature-by-feature comparison table, checking itself against the OpenAI Agents SDK, the Claude Agent SDK, Google's Agent Development Kit, and LangChain/LangGraph, using the versions current as of October 2025. A condensed version, limited to the rows where OpenHands SDK is the only one of the five with full support:

Two of those six rows are worth pointing at directly, because this lesson already built the exact mechanism behind them from zero. “Secrets management with auto-masking” is Chapter 7's SecretRegistry, credentials handed to a tool only at the moment it executes and never embedded in anything the model reads. “Native non-function-calling model support” is Chapter 5's NonNativeToolCallingMixin, extending tool use to models never trained to support it. Seeing both appear as the SDK's own claimed differentiators against four well-known alternatives is a useful confirmation that the specific design choices this session spent real time on were not incidental implementation details — they are, in the authors' own accounting, among the features that most distinguish this harness from the alternatives a team building an agent today would otherwise default to. “Agent stuck detection” is new, not covered anywhere in Chapters 5 through 8, and stands as an honest reminder that the four capabilities this session chose to build out in depth — state, tools, delegation, security — are not the entire feature surface of a production harness, only the four this session judged most worth teaching from first principles.

Where each paper connects on this site

← Harness Engineering — the general anatomy this session's two papers each specialize: context, memory, tools, evaluation loops
← Harness Optimization — tuning the context and tool design a harness like this one runs on
← Self-Improving Harnesses — the broader arc AWM's online induction is one concrete instance of
← Agents & Tool Use — the ReAct-style loop this session's Action/Execution/Observation contract builds on
← MCP — the protocol OpenHands SDK's tool system treats as a first-class citizen
← CS224N Lecture 10: Agents, Tools, and RAG — the foundational agent-loop lecture this session assumes
← OpenClaw: The Agent Loop — another concrete, production agent-loop implementation to compare against OpenHands' design

References

Two sentences, if that is all that survives

If everything else in this session fades, keep these two. AWM: a workflow is a description plus parameterized steps, induced from what actually recurred across more than one successful trajectory, and injected as plain text beside a model that never gets retrained. OpenHands SDK: freeze every piece of configuration, let exactly one append-only event log record everything that actually happened, and make sandboxing and security gates something you opt into per conversation, not something forced on every agent whether it needs it or not. Neither sentence is the whole paper, but each is enough to reconstruct the shape of the whole system from, which is the standard this entire lesson has tried to write to.