Imagine a software development team talking among themselves: one acts as the architect, another writes code, and a third serves as the auditor. On the screen, the labor appears cleanly divided. However, a 2025 study led by a UC Berkeley team introduces a sobering statistic. Popular multi-agent frameworks like ChatDev achieve an accuracy of only 33.3 percent on the ProgramDev software task benchmark.1
Clearly, combining three agents does not automatically fix everything. Sometimes, the performance gains compared to single-agent systems or simple best-of-N sampling remain razor thin. The fundamental question is simple: why do multi-agent systems fail?
What is an AI agent?
In this context, an agent is a unit composed of a language model, initial instructions, conversational history, and tool-use capabilities that interacts with its environment. A multi-agent system orchestrates these units through task decomposition, parallel execution, and role separation. MetaGPT mimics the procedures of a software company; AppWorld connects service agents such as email and music with an auditor; Magentic-One tackles open-ended web and file tasks. While architectures vary, the shared premise remains the same: a team outperforms a single model.
The Berkeley team examined seven open-source frameworks, reading over two hundred conversation traces where each trace exceeded fifteen thousand lines on average, with the assistance of six expert human annotators. The resulting classification is MAST, which stands for Multi-Agent System Failure Taxonomy. Fourteen failure types are grouped under three main headings. With a Cohen's kappa score of 0.88 among human annotators, the labels are consistent rather than random.1
Why do specification errors ruin multi-agent AI setups?
The first category covers approximately 41.8 percent of the traces. When system architecture, role definitions, or task constraints are ambiguous from the start, agents misinterpret their assignments. The most frequent issues include repeated steps, failure to follow task requirements, and an inability to recognize when a task is finished. In the ChatDev example, the prompt requests a standard Wordle game featuring a new five-letter word every day, yet the system repeatedly generates a static word list. Even when the prompt is written more explicitly, the static list and new errors persist. Consequently, the problem stems not merely from the user's ambiguous phrasing, but from how the system reads the specification.1
This does not mean writing instructions is unimportant. An intervention that tightens role specifications increases success in ChatDev by roughly 9.4 percent. Yet even with identical user prompts and base models, the gains remain limited. Fixing the specification is necessary, but it is not sufficient on its own.
How do multi-agent systems suffer from communication breakdowns?
The second category accounts for approximately 36.9 percent of the traces. One agent discovers the correct username format but fails to communicate it to another, leading the second agent to repeat the mistake. Other examples involve proceeding with false assumptions, drifting from the task, and mismatches between thought and action. On the surface, these appear simply as broken conversations, but isolating their root causes requires granular classification. Has information been withheld, has input been ignored, or has context been lost? Identical symptoms stem from different root causes.1
Classic software intuition applies here. If the API contract between two services is ambiguous, integration tests fail. Messages exchanged between agents constitute a contract as well; if specifications fail to state what must be transmitted, what must be asked, and what cannot be assumed, the team generates internal noise.
Why do verification steps fail in multi-agent systems?
The third category represents roughly 21.3 percent of the failures. Tasks finish prematurely, auditing proves inadequate, or verification produces incorrect results. A chess program generated by ChatDev passes all review rounds yet still accepts illegal moves. This happens because the auditor often settles for superficial checks like compilation or comment syntax rather than genuinely testing the rules of the game.1
Systems featuring a validator agent often exhibit fewer total errors. Nevertheless, ChatDev can remain capped at 33 percent accuracy on ProgramDev, struggling even with seemingly simple tasks rich in examples such as tic-tac-toe, chess, and sudoku. Adding an extra high-level verification step focused on overarching goals yields an absolute performance increase of 15.6 percent on ProgramDev. While this improvement is real, it remains insufficient for permanent reliability. The researchers' thesis is clear: many errors do not originate from single-model hallucinations, but from organizational design and coordination failures.2
What should engineers look for in multi-agent AI tools?
When encountering a tool advertised with multi-agent automation, the crucial question is not how many agents it employs. Are the roles explicitly written? Who passes what to whom? Who verifies the output, and how—through compilation or actual acceptance testing? Without answers to these three questions, scaling up the team often just scales up the noise. The value of MAST lies precisely here: by categorizing failure types, developers can ask whether the issue stems from specification, coordination, or verification rather than simply assuming the model is flawed.
The limitations of this research should be acknowledged transparently, as the study examines selected open-source frameworks and specific benchmarks rather than every commercial product individually. Nevertheless, the engineering lesson translates clearly: poor organization among intelligent individuals breeds disaster, and the same principle applies to agent teams.
The Berkeley team's classification demonstrates that failures usually stem from team organization rather than the model itself, as shown by ChatDev lingering at 33 percent accuracy on the ProgramDev benchmark consisting of example-rich tasks like tic-tac-toe and sudoku.
Notes and Sources
- Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., Zaharia, M., Gonzalez, J. E., and Stoica, I. (2025). "Why Do Multi-Agent LLM Systems Fail?". arXiv:2503.13657. MAST: 14 failure types, 3 categories (specification ~41.8%; agent misalignment ~36.9%; verification ~21.3%); 7 frameworks, 200+ traces; ChatDev ProgramDev accuracy 33.33%. arxiv.org/abs/2503.13657 ↩
- Project website, dataset, and LLM annotator: sites.google.com/berkeley.edu/mast. Intervention studies: role specification (+9.4% for ChatDev), additional high-level verification (+15.6% for ProgramDev). github.com/multi-agent-systems-failure-taxonomy/MAST ↩
This article is for informational purposes only and does not constitute software architecture, security, or product selection advice.
