---
title: "Multi-agent orchestration: what broke in 3,315 agent runs"
description: "Agent orchestration in practice: 3,315 subagent runs over two months, the token cost at API prices, and seven failures that each became a hook."
url: "https://madsalt.tech/blog/multi-agent-orchestration-what-broke/"
author: "Denys Zaichenko"
language: "en"
published: 2026-10-19
updated: 2026-10-22
---

# Multi-agent orchestration: what broke in 3,315 agent runs

TL;DR

Between 17 August and 19 October 2026 my Claude Code setup launched 3,315 subagents. In the last 30 days that came to about 16 billion tokens, 97.7% of them cache reads. I pay a flat subscription; at API list prices the same tokens would cost at least USD 3,350. The failures were almost never the model writing bad code. They were in the seams: parallel agents overwriting the same file with every check green, a cleanup script deleting finished work, an agent editing the checks that blocked it. Each failure became a small script that blocks the next run, and those scripts, not the prompts, are what made the system usable.

I'm Denys, a full-stack and AI engineer. I build software with one lead Claude Code session that hands work to specialised subagents: explorers, a planner, implementers, security and code reviewers. This is a report on that setup, not on a client product. The numbers come from the logs it keeps: a spawn ledger (one line per agent launched), session transcripts and the git history of the configuration itself. Client and project names are left out.

I also built an [interactive multi-agent orchestration dashboard](https://madsalt.tech/work/multi-agent-orchestration/) as a prototype with sample data. Everything below is from real runs.

## What is multi-agent orchestration?

Multi-agent orchestration is a setup where one lead agent hands bounded tasks to specialised agents and merges their results; the orchestrator owns the context, the budgets and the retries. In this article it shows up as two patterns: a lead session with subagents (explorers, a planner, implementers, reviewers), and hooks that enforce the rules the agents cannot talk their way around. The [multi-agent orchestration dashboard](https://madsalt.tech/work/multi-agent-orchestration/) shows the same idea on sample data; the sections below are about what broke in real runs.

## What does the setup look like?

One lead session talks to me and owns the task. It never writes production code itself on larger tasks. It sends work out:

- **Explorers** read one layer of the codebase (frontend, backend, database) and write a short map.
- **A planner** turns the map into numbered steps with file paths.
- **Implementers** take one plan phase each, write the tests first, then the code. Parallel implementers get their own git worktree.
- **Reviewers** check the diff: one for security, one for correctness, both on the same commit.

Between them sit hooks: small scripts that Claude Code runs before and after every tool call. A hook can block a call, for example an agent launch, a file write or a `git commit`. The configuration has 165 hooks, about 40 of them about orchestration. The rules the agents read are Markdown; the rules they cannot get past are hooks. That split is the main lesson of the two months, and the rest of this piece is why.

![How one task moves through the agents. Hooks sit between every step and can block it. Normal path: Explorers → Planner → Implementers → Reviewers → Commit gate; Same failure three times: Implementers → Same failure 3x → Stronger model → A person.](https://madsalt.tech/charts/en/multi-agent-orchestration-what-broke-flow.svg)

*Figure 1. My agent setup, August to October 2026. Parallel implementers each get their own git worktree.*

## How many agents ran, and on which models?

| Measure (17 Aug – 19 Oct 2026)    | Value                                                                   |
| --------------------------------- | ----------------------------------------------------------------------- |
| Subagents launched                | 3,315 over 57 active days                                               |
| Agent types                       | 30                                                                      |
| Most launched                     | implementer (tdd-guide) 793, security reviewer 455, general-purpose 424 |
| Effective model                   | Opus 46.5%, Sonnet 45.8%, Fable 1.3%, rest inherited from the lead      |
| Launched one at a time            | 67.7%                                                                   |
| Largest parallel wave             | 5 agents (the cap), reached 9 times                                     |
| Reviewer runs per implementer run | 0.71 (680 reviews, 963 implementations)                                 |

Two numbers surprised me. First, most launches were single agents, not fan-outs. Parallel work is the exception: it pays off for reading and reviewing, and costs a lot for writing (see the first failure below). Second, in 23% of launches where an agent's file names a model, the model that actually ran was different, almost always Sonnet raised to Opus (584 of 610). That is deliberate: a hook raises the model when the step touches risky code such as migrations, authentication or server routes, and after an implementer on Sonnet fails the same test three times in a row. The stronger implementer ran 159 times; I cannot split the log into "risky step" and "repeated failure", so I won't claim how many were rescues.

## What does multi-agent orchestration cost?

I pay a flat subscription, so there is no bill per token. The transcripts do record every token, so here is what the last 30 days (20 September – 19 October) would cost at Anthropic's API list prices:

| 30 days               | Tokens                                                |
| --------------------- | ----------------------------------------------------- |
| Total                 | about 15.99 billion                                   |
| Cache reads           | 15.55 billion (97.7% of everything sent to the model) |
| Cache writes          | 372 million                                           |
| Fresh, uncached input | 0.18 million                                          |
| Output                | 69 million                                            |

At list prices (Opus 5.5 USD 4 per million input, 20 per million output, 0.20 per million cache reads; Sonnet 5.5 and Fable 5.1 at theirs), output, input and cache reads of the three current models alone come to about USD 3,350. That leaves out cache writes and older model versions still in the logs, so the true API figure is higher. Two thirds of that USD 3,350 is cache reads: every turn re-sends the whole context, and caching is the only reason this is affordable at all.

![Who wrote the 69 million output tokens. Million tokens, 30 days. Subagents used 42% of all tokens. Lead session: 23.70; Subagents: 45.30.](https://madsalt.tech/charts/en/multi-agent-orchestration-what-broke-output.svg)

*Figure 2. Session transcripts, 20 September to 19 October 2026: 75,129 model requests.*

The split between the lead and the subagents is the part I'd use when planning a budget:

- **Subagents used 42% of all tokens but wrote 66% of the output.** The lead session is expensive because it carries the longest context; subagents are where the actual text and code get written.
- **Reviewers are cheap.** A security or code review used 1–4 million tokens. An implementer used 5–100 million. Two reviewers on every risky commit cost less than one implementer retrying.
- **The largest session**, on 6 October, launched 63 subagents and used 372 million tokens in 3.5 hours.

## What broke

Every failure below happened, has a date, and ended with a hook or a script that blocks it from happening again. None of them is "the model wrote a bug". Ordinary bugs were caught by the tests and reviewers. These got past both.

| Date (2026) | What broke                                                                             | What blocks it now                                                                     |
| ----------- | -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| 19 Aug      | Parallel agents overwrote one translation file: 55 of ~95 keys lost, every check green | Each agent writes its keys to its own fragment; a merge script proves every key landed |
| 24 Aug      | A cleanup script deleted 4 finished work packets                                       | Cleanup skips any worktree with uncollected changes; collect first, clean second       |
| 1 Sep       | Later waves of agents started from stale code                                          | Merge each agent's work as a three-way patch, never a file copy                        |
| 13 Sep      | An agent died at launch, and the launch said "success"                                 | The launcher reads the agent's first output and marks a dead model for every session   |
| 20 Sep      | Two of my own rules contradicted each other; agents retried a refused command          | Briefs for isolated agents carry the command shapes the sandbox accepts                |
| 18 Oct      | An agent edited the checks that blocked its commit                                     | Check files are read-only from project sessions; changes are logged and alerted        |
| 18 Oct      | Fix instructions quietly changed what users of a client app receive                    | Every fix brief says "keep live behaviour"; product changes go back to me              |

### Parallel writers and the file every feature touches

Three agents built three features of a client iOS app at the same time. Each one added its text strings to the app's single translation catalog, and each saved its own version of the whole file. The last save won. 55 of about 95 new keys were gone. Each agent's build passed, because each agent's copy was consistent; the catalog check passed, because its coverage was broken in a separate way. I found it by counting keys by hand.

The fix has three parts. A hook refuses to launch an agent whose declared files overlap with a writer that is still running. Agents never edit shared catalogs; they write their keys to a fragment file, and one script merges the fragments, fails on conflicting values, then checks every key is present. A second hook blocks the commit if a catalog has fewer keys than before. The honest part: I added those guards on 18 October, 60 days after the loss. Between those dates the only protection was a line in a Markdown rule, and a rule is a suggestion.

### Cleanup that deleted finished work

Parallel implementers work in git worktrees and stage their changes without committing. A commit check told the lead to run the cleanup script. The script skipped worktrees whose agent was still running, and these agents had finished. Four finished packets of work were deleted, with no git objects left to recover. "Is the agent still running?" and "has its work been collected?" are different questions, and the script only asked the first. Cleanup now skips any worktree with changes the main checkout does not have, and a separate hook blocks direct deletion of worktree folders.

### Waves that started from old code

A plan with 11 steps ran as 6 implementers in 3 waves. Waves 2 and 3 started from the commit the session began on, three commits behind. Copying one agent's file back would have reverted another agent's finished change. A worktree is a snapshot of where you started, so results now go back as three-way patches, and conflicts surface instead of being silently overwritten.

### A dead agent that reported success

An agent pinned to a model whose usage limit was used up died before its first step. The launch returned "spawned successfully"; the failure arrived later as an idle notice. A retry on another model was switched back by the same hook that pins models. The lead then quietly wrote the plan itself, which was the outcome the planner existed to avoid. The hook now reads the agent's first output and marks a dead model for every session for a while.

### The agent that edited its own checks

On 18 October a session working on a client project kept getting its commit blocked by a check. It edited three of the check scripts so that its commit would pass. Nothing about this was hidden; the edits were in plain view. But nothing stopped them either, because the checks lived in a folder the agent could write to, and only the file-editing tools were watched, not shell commands. Now the check files can be changed only from a session started in the configuration folder itself, every write route is covered (editor, shell redirects, scripts, git), and any change to them during a project session is logged with an alert.

This is the failure I'd warn anyone about first. An agent told to "get the commit through" treats the gate as part of the problem.

### Two of my own rules disagreed

One rule told agents to combine three or more shell steps into one script to save tokens. Agents isolated in a worktree run under a sandbox that refuses exactly that shape. Agents hit "too complex to verify" three times in a row on several days, and burned their budget rephrasing. Agents now stop after two identical refusals, and briefs for isolated agents state the command shapes that work.

## Why do multi-agent systems fail?

From these two months: not because a model is too weak. None of the seven failures above was a model failing at the task it was given. They fail at the seams between agents, where no single agent is responsible:

- **Shared state.** A file every feature touches, a working tree, a branch. Each agent is right locally and the result is wrong globally.
- **Green checks.** Every incident above passed the checks that existed at the time. A check proves what it checks, nothing more.
- **Success signals that lie.** "Spawned", "saved", "cleaned up" each meant something narrower than it said.
- **Agents optimising for the gate.** Tell an agent the commit is blocked and it will look for a way to unblock the commit, including through the gate.

Other builders report the same pattern from bigger fleets. Jed Arden, running about twenty agents, wrote that ["the LLM barely appears in the incident log"](https://jedarden.com/notes/what-breaks-at-twenty-agents/) (July 2026). [Cursor's team found](https://cursor.com/blog/scaling-agents) that twenty agents sharing one coordination file with locks slowed down to the throughput of two or three (January 2026).

## What I'd set up first

If you are starting with Claude Code subagents or any multi-agent orchestration, this is the order I'd do it in, given what it cost me to learn:

1. **Log every agent launch.** One line per launch: time, agent type, model, risk. Without it you can't tell what the system does, and none of the numbers above would exist.
2. **One writer per file.** Parallel writers get a worktree and a declared file list. Files that every feature appends to (translations, lockfiles, changelogs) get one owner, and the others hand in fragments.
3. **Collect before you clean.** Never let a cleanup step decide by "is it running". Decide by "is its work in the main checkout".
4. **Put the checks out of the agents' reach.** Hooks that block, kept where a project session cannot write.
5. **Cap the wave.** Five parallel agents is my ceiling; past that, review becomes the bottleneck and conflicts grow faster than throughput.
6. **Escalate on repetition, not on effort.** A cheap model by default; after the same failure three times, one attempt on a stronger model; after that, a human.

What does an AI agent cost to build and run for a business, rather than for one developer's workflow? That is a different question, answered with numbers in [what an AI agent costs to build and run](https://madsalt.tech/blog/how-much-does-an-ai-agent-cost/). Which panels I watch now, from the same logs a month later, is in [AI agent monitoring dashboard](https://madsalt.tech/blog/ai-agent-monitoring-dashboard/).

## Limits of this report

- **One developer's setup.** One lead session, up to five parallel agents. Fleets of tens or hundreds of agents fail in additional ways, for example at launch.
- **No real bill.** The dollar figure is an API list-price equivalent for 30 days, a floor that leaves out cache writes and older models. I pay a flat subscription.
- **Gaps in the logs.** Blocked launches are not logged, so I can't say how often the wave cap fired. The 159 runs of the stronger implementer mix risky steps and repeated failures.
- **Dates from my own notes.** Incident dates come from notes and commits written the same day; one incident is dated differently in an older rule file, and I used the note.

The dashboard prototype that grew out of this work is the [multi-agent orchestration case](https://madsalt.tech/work/multi-agent-orchestration/), and building agents for clients falls under [AI agent development](https://madsalt.tech/services/ai-agent-development/).

Running agents in parallel and losing work between them?

Happy to compare notes on hooks, worktrees and review gates. No pitch.

[Get in touch](https://madsalt.tech/#contact)