Organisations building multi-agent AI pipelines face a measurement problem that standard benchmarking cannot resolve: individual agents can pass every capability test in your suite while the system they compose fails in ways that no single-agent analysis would predict.
That gap is not theoretical. It is the documented failure mode of production deployments in 2025 and 2026.
The Gradient Institute's technical report on risk analysis for governed multi-agent systems (Reid et al., 2025) offers the most rigorous taxonomy of that failure currently available. Commissioned with support from the Australian Government and grounded in a systematic review of emergent multi-agent AI behaviour, the report makes a point that engineering teams need to incorporate into their governance frameworks today: a collection of safe agents does not guarantee a safe collection of agents.
The agentic inflection point
The pace at which single-agent deployments are evolving into multi-agent architectures has outrun the governance frameworks most organisations have in place. The practical pressure is understandable. Different business units automate their domains independently; an HR agent processing new hires must coordinate with an IT agent provisioning access, and a finance agent setting up payroll. The efficiency case for enabling direct agent-to-agent communication is compelling, and framework proliferation (AutoGen, CrewAI, LangGraph) has made it technically accessible.
What has not kept pace is the risk thinking.
The framing most organisations carry into multi-agent governance was built for single-agent evaluation: does the model hallucinate? How accurate is its code? Does it follow instructions under adversarial prompting? Those questions remain necessary. They do not address the interaction dynamics of the systems now entering production.
The Sonar State of Code survey (Sonar, 2026), covering 1,100 developers, reports that 42% of all committed code is now AI-generated or AI-assisted, while 96% of developers do not fully trust its functional accuracy, and 59% spend moderate to substantial effort reviewing and correcting AI output. A randomised controlled trial by Becker et al. (2025) of 16 experienced open-source developers found that AI tooling increased task completion time by 19%, even as developers believed they had become faster. When agents produce that code and then review each other's outputs within an automated pipeline, those individual reliability gaps do not average out. They compound.
Why individual agent safety is necessary but insufficient
The professional instinct in software engineering is to decompose complex systems into components, verify each component against a specification, and infer system-level correctness from component-level correctness. That instinct is well-founded in deterministic software. It fails in multi-agent LLM systems for reasons that are structural, not incidental.
Reid et al. (2025) identify five such reasons.
When multiple agents interact, the number of possible system states grows exponentially; behaviour that was stable in isolation becomes unpredictable under agent-to-agent dynamics.
Each agent operates on partial information and makes decisions without visibility into the broader task state.
Agents cannot reliably evaluate each other's competence or credibility, so erroneous outputs propagate uncritically downstream.
Stable interaction patterns can fracture as agents continuously adapt to each other's behaviour; a software update to a single agent, even a revised system prompt, effectively creates a different system that requires full re-evaluation.
And early errors lock the system into failure trajectories that compound downstream and become progressively harder to reverse.
Berkeley's MAST taxonomy (Cemri et al., 2025) documents 14 distinct system-level failure modes in multi-agent LLM pipelines, none of which are detectable at the individual agent level. Reid et al. (2025) show that even under unified governance, where your organisation controls the configuration and deployment of every agent, these failure modes remain structurally present. They are properties of interaction, not of individual components.
The practical consequence for engineering leadership is direct: your current test suite is likely measuring the wrong thing. It tells you whether each instrument in the orchestra can play in tune. It does not tell you whether the orchestra produces coherent music.
The six failure modes your current risk framework does not cover
"Multi-agent systems don't just accumulate single-agent risks; they transform them. Cognitive limitations become cascading hallucinations, communication errors become system-wide miscoordination, and individual biases become collective blind spots through reinforcement dynamics." (Reid et al., 2025, p. 2)
Reid et al. (2025) identify six failure modes specific to governed multi-agent environments. Each is absent from standard single-agent risk frameworks, and each maps to concrete production scenarios.
1. Cascading reliability failures. LLM agents have "spiky" capability profiles: superhuman performance on certain complex tasks coexisting with profound failures on seemingly simpler ones. A single agent misreading a chart, interpreting "10.5K units" as "105K units," passes that figure downstream to agents that accept it uncritically. Unlike a human collaborator who might flag a suspicious number, a receiving agent builds on the flawed input as a valid premise. The error compounds at each node. By the time human managers notice unusual activity in the manufacturing scenario Reid et al. (2025) document, the company has committed millions in unnecessary material purchases, disrupted production of other product lines, and locked in logistics contracts it cannot fulfil.
The error rate of each individual agent is not the relevant metric. What matters is how errors propagate through the coupling structure.
2. Inter-agent communication failures. Natural language is an ambiguous medium when precision is required. When a Grid Management agent tells a Public Communications agent that "Substation 7 is now stable," the engineering definition (no longer at risk of cascading failure, but still fragile) diverges from the public relations interpretation: resolved, safe for users to resume normal consumption. The Communications agent broadcasts a restoration message; users switch on appliances; the fragile substation overloads. The failure originated not from an error but from a semantic gap that no single-agent benchmark would surface (Reid et al., 2025).
Communication failures also include context truncation: an agent summarising a long exchange for a downstream agent may discard exactly the exception case that downstream agent needed to handle correctly. The problem intensifies with every handoff.
3. Monoculture collapse. When all agents in a system are fine-tuned variants of the same base model, they share correlated biases and blind spots. A vulnerability that triggers one agent triggers all agents simultaneously, producing convergent outputs that appear reliable due to consensus. Reid et al. (2025) report evidence that similar model capabilities lead to static debate dynamics, converging on the majority view even when that view is factually incorrect.
The financial fraud detection system that deploys five variants of the same base model does not have five independent detectors. It has one detector with five labels on it. Convergent outputs look like high-confidence signal. In structural terms, they are a single point of failure wearing the costume of consensus.
4. Conformity bias. LLM sycophancy, the tendency of language models to prioritise agreement over accuracy, becomes a system-level hazard in multi-agent settings (Reid et al., 2025). A confidently stated confabulation from one agent generates conformity pressure on others. As agreeing agents accumulate, dissenting agents face increasing pressure to align. The consensus grows stronger with each reinforcing step, even though no individual agent had high confidence in the original claim.
This is most dangerous in collaborative swarm architectures, systems specifically designed to synthesise diverse perspectives, because the diversity assumption collapses when agents reward linguistic confidence over factual accuracy. The swarm produces a single high-confidence answer. That answer can be wrong in ways that no individual agent, interrogated in isolation, would produce.
5. Deficient theory of mind. Effective coordination requires agents to model what other agents know, want, and can do. When an inventory agent and a pricing agent each receive the same demand forecast and optimise independently, one placing massive orders, the other tripling prices, neither agent models how the other's action changes the shared environment. The outcome is warehouses full of expensive unsold inventory, despite each agent having behaved rationally given the information it had (Reid et al., 2025).
The agents are working correctly. They are simply failing to account for the fact that the environment each is reasoning about includes each other. No malfunction required.
6. Mixed motive dynamics. When agents pursuing individually rational objectives interact under shared governance, emergent behaviour can systematically undermine collective goals, even when all agents are formally aligned with organisational objectives. The sequence Reid et al. (2025) document is instructive: an inventory agent marks all orders as "critical" to bypass delays introduced by a cash-flow agent; that agent responds by requiring CFO approval for any order above $10,000; the inventory agent then splits large orders into $9,999 tranches to circumvent the threshold.
Both agents technically achieve their individual performance metrics. The organisation loses bulk purchasing discounts, experiences unpredictable stockouts, and destroys overall profitability. A collectively suboptimal equilibrium, reached through locally rational behaviour at every step, and invisible to any single-agent evaluation.
Know your architecture, know your exposure
One of the more practically useful contributions of the Gradient Institute report is its mapping of failure modes to canonical multi-agent configurations. The failure modes you face are not uniform across architectures.
A centralised orchestrator with specialised delegates concentrates risk at the orchestrator layer. If the orchestrator's task decomposition is inconsistent or its specialist selection degrades under complex scenarios, those failures propagate to every downstream agent simultaneously. Cascading reliability failures and a deficient theory of mind are the dominant concerns here.
A collaborative swarm (no central control, high communication frequency, a shared high-level goal resolved through distributed synthesis) is inherently exposed to conformity bias and monoculture collapse. The architecture is designed for diversity of perspective; the failure modes specifically erode that diversity.
A distributed autonomous task force, with persistent, role-specialised agents managing separate domains and infrequent but critical cross-agent coordination, is where mixed motive dynamics and inter-agent communication failures concentrate. Agents that do not interact continuously are less likely to have developed stable shared semantic conventions for the handoffs that matter most.
Mapping your production system to the canonical configuration that most closely resembles it is the first decision your risk governance process should make. It determines which failure modes warrant the most immediate investment in analysis and which risk analysis techniques are most applicable (Reid et al., 2025).
A 30-minute multi-agent risk audit
Before extending your agentic pipeline or adding new agent roles, run this diagnostic. The goal is to identify which failure modes your current configuration is most exposed to.
Map your agent network topology against the three canonical settings above: centralised orchestrator, collaborative swarm, or distributed task force. Your highest-exposure failure modes differ by configuration.
For each agent, identify the three outputs most likely to be accepted by downstream agents without verification. These are your cascade entry points.
Test each cascade entry point for capability brittleness: run the same task with minor input variations (rephrasing, numeric format changes, image quality degradation) and measure output consistency. High sensitivity to superficially innocuous changes predicts production cascade failures.
Review inter-agent communication logs for ambiguous or domain-specific terms. Flag any term whose meaning would shift if the receiving agent were trained on a different knowledge corpus than the sending agent.
Count how many agents in your network are fine-tuned variants of the same base model. If all agents share a base model, document the input classes that the base model handles poorly. Those are your monoculture blind spots.
Verify that at least one agent in your network holds an explicit critic or verification role whose architecture structurally prevents it from receiving social pressure from the agents it evaluates. Systems with no designated independent challenger are vulnerable to conformity bias regardless of individual agent quality.
For each pair of agents sharing resource constraints or carrying partially conflicting objectives, work out what locally rational behaviour looks like when the other agent behaves optimally. If the resulting interaction is not globally optimal, you have a mixed-motive dynamic in production.
Review your last three agentic pipeline failures and classify each: single-agent reliability failure, cascade from a peer output, communication ambiguity, or emergent coordination failure. The classification determines the appropriate remediation path.
Verify that your pre-deployment testing includes multi-agent interaction scenarios run over extended interaction sequences, not only single-agent capability assessments. Static benchmarks cannot reveal emergent failure modes (Reid et al., 2025). Simulation environments that replay production interaction patterns at scale are the most informative analytical tool currently available.
Confirm that your monitoring stack captures inter-agent communication logs and tracks failure propagation depth (the number of downstream components affected by a single agent-induced fault), not only end-to-end task completion rates. Monitoring only final outputs is the observability equivalent of removing all intermediate sensors from an industrial pipeline and measuring only whether the end product is on spec.
Next moves
For the Builder
Instrument your agentic pipeline to log inter-agent message passing with timestamps, agent identifiers, and input provenance. Without this data, post-incident failure cascade analysis is guesswork. Treat inter-agent message logs as first-class production observability, not a debugging afterthought.
When reviewing any output produced within an agent chain, treat an output that cannot be traced to a verifiable upstream input as a potential cascade seed. Calibrate your verification effort to the consequence level of the task, not the fluency of the output.
Build a personal test library of brittleness probes: input variations that target the capability troughs most relevant to your domain. Run each new model version against this library before trusting it in a production agent chain. When a model update arrives, your first question should not be "what improved?" but "what changed at the failure boundary?"
For the Manager
Audit your team's agent network against all six failure modes. Assign an owner to each high-exposure failure mode. Unowned risk is unmanaged risk, regardless of how sophisticated the system is.
Create psychological safety around reporting agent-to-agent failures. Engineers who quietly correct cascade errors without documenting them deprive the organisation of the failure-pattern data required to improve system governance over time. If your retrospectives discuss model quality but not failure propagation, your learning loop is incomplete.
Schedule a quarterly system-level red team exercise: deliberately inject a known erroneous output at the cascade entry point identified in step 2 of the audit, and measure how far the error propagates before detection. Tracking that boundary over successive quarters is your most honest measure of governance maturity.
Reconsider the structure of your agentic review processes. A review agent that sees its own previous output, or that of a directly upstream agent in the same chain, is not an independent reviewer. Structural independence matters, not role designation.
For the Roadmap Owner
When evaluating multi-agent platforms or frameworks, require vendors to demonstrate failure mode coverage across all six categories identified by Reid et al. (2025). A vendor whose safety documentation addresses only single-agent reliability is selling you a single-agent risk framework for a multi-agent problem.
Factor the cost of silent cascade failures into your AI ROI projections. Errors that compound through agent chains before reaching a human reviewer are harder to detect and more expensive to remediate than single-agent hallucinations. Budget for verification overhead proportional to agent network depth, not proportional to agent count.
Treat agent model diversity as a portfolio risk decision with quantifiable cost and benefit, not a technical implementation detail. A system where all agents share a base model carries correlated failure exposure; the cost of model diversification is bounded, the cost of a monoculture collapse event is not.
Resist the organisational temptation to measure agentic pipeline performance exclusively through task completion rates. A system that completes 95% of tasks while silently producing catastrophic cascade errors in the remaining 5% is not a high-performing system. It is a system with an unmonitored tail risk. The measurement framework must match the failure structure (Reid et al., 2025).
Closing thought
Software engineering has learned this lesson before. Brooks documented it for teams in 1975: adding people to a late project makes it later, because communication overhead grows faster than productive capacity. Perrow formalised it for industrial systems in 1984: tightly coupled systems with interactive complexity produce accidents that no component-level inspection can prevent. The multi-agent paradigm is the latest instance of the same structural insight: interaction creates failure modes that component analysis cannot predict.
The question for engineering organisations is not whether to adopt multi-agent architectures. The market pressure will answer that. The question is whether governance will evolve at the pace of deployment. How many of the six failure modes described above does your current risk framework explicitly address?
This is the last edition of Software Insights before the Easter break. I am heading to ICSE in Rio next, so if you are there, please say hello. The next edition lands on April 28.
Daniel Russo, Ph.D., is a Professor of Software Engineering whose research examines the intersection of human cognition and artificial intelligence. Through "Software Insights," he translates empirical research into actionable guidance for software practitioners and organizations.
Partner with Daniel to transform your organization through evidence-based approaches that bridge academic rigor with practical implementation. His consulting work helps organizations to adopt scientifically validated practices that improve software development outcomes, team performance, and innovation capacity.
Learn more about his approach to evidence-based organizational change
https://www.danielrusso.org/how-i-can-help-you/ (Öffnet in neuem Fenster)
References
Becker, J., Jorke, M., Ahrabian, K., Shang, R., Gao, G., Altman, R., Liang, P., & Yang, D. (2025). Measuring the impact of early-2025 AI on experienced open-source developer productivity. arXiv:2507.09089.
Cemri, M., Lin, M., Stengel-Eskin, E., Reddy, S., & Subramanian, S. (2025). Why do multi-agent LLM systems fail? arXiv:2503.13657.
Reid, A., O'Callaghan, S., Carroll, L., & Caetano, T. (2025). Risk analysis techniques for governed LLM-based multi-agent systems. Gradient Institute Technical Report. arXiv:2508.05687.
Sonar. (2026). State of Code: Developer Survey Report 2026. SonarSource.
Datum
31.03.2026