---
id: TECH-MARL-001
title: "Multi-Agent Systems"
alternative_names:
  - "Multi-Agent Reinforcement Learning"
  - "MARL"
  - "LLM Multi-Agent Systems"
category: "Technical → Agents and Interaction"
volume: III
chapter: 20
entry_type: "Technical Article"
level: 3
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "TECH-AGENT-001"
related_entries:
  - "TECH-RL-001"
  - "TECH-LLM-001"
  - "TECH-ROBOT-001"
  - "CAP-AGENTEVAL-001"
  - "SAFE-ALIGN-001"
---

# Multi-Agent Systems

## 1. Concise Definition
A **multi-agent system** is a set of two or more autonomous decision-making entities that act in a shared environment and whose outcomes depend on one another's behaviour. In machine learning the term covers **multi-agent reinforcement learning (MARL)**, where several learning agents optimise their own or a shared objective, and, more recently, **LLM-based multi-agent systems**, where several instances of language models with assigned roles exchange messages to complete tasks.

## 2. Overview & Significance
The single-agent framework in [[TECH-RL-001] Reinforcement Learning](./TECH-RL-001_reinforcement_learning.md) treats the environment as fixed. With several learners the environment from each agent's viewpoint changes as the others learn, which breaks the assumptions behind many convergence guarantees (Zhang et al., 2021). Multi-agent settings cover cooperation (teams of robots, coordinated software agents), competition (games) and mixed motives (negotiation), and they are relevant to safety because behaviours can emerge from interaction that no individual agent was trained to produce (Baker et al., 2020).

This article distinguishes two bodies of work with different maturity: the MARL literature, with formal models and benchmarked algorithms, and LLM-based multi-agent frameworks, which are recent and evaluated mainly on task benchmarks and case studies. Single-agent architecture is in [[TECH-AGENT-001] Autonomous AI Agents](./TECH-AGENT-001_autonomous_agents.md), which mentions multi-agent frameworks briefly; this article covers the interaction problem itself.

## 3. Historical Development
Multi-agent systems originated in distributed AI and game theory before deep learning. Zhang, Yang and Başar (2021) review MARL theory built on Markov (stochastic) games and extensive-form games, grouped into fully cooperative, fully competitive and mixed settings.

Deep MARL grew from 2016–2018. Lowe et al. (2017) proposed MADDPG for mixed cooperative-competitive environments. Foerster et al. (2018) introduced COMA, using a centralised critic and a counterfactual baseline to assign credit to individual agents. Rashid et al. (2018) introduced QMIX, which factorises a joint value function monotonically into per-agent values. Yu et al. (2022) reported that a carefully implemented multi-agent PPO performs strongly across four cooperative benchmarks, challenging the assumption that off-policy methods are necessarily better.

Large-scale game systems demonstrated multi-agent learning at scale: Jaderberg et al. (2019) reported human-level play in a 3-D first-person team game with population-based reinforcement learning; Vinyals et al. (2019) reported Grandmaster-level play in StarCraft II; Baker et al. (2020) showed hide-and-seek agents developing a sequence of strategies through an automatically generated curriculum; and Meta Fundamental AI Research Diplomacy Team (FAIR) et al. (2022) reported Cicero, which combined a language model with planning and reinforcement learning in the negotiation game Diplomacy.

From 2023, language models were composed into multi-agent systems: Park et al. (2023) simulated generative agents; Li et al. (2023) proposed CAMEL role-playing; Du et al. (2023) proposed multiagent debate; Hong et al. (2023) MetaGPT; and Wu et al. (2023) the AutoGen framework. Cemri et al. (2025) published a large annotated-trace study and taxonomy of failures in such systems.

## 4. Technical / Conceptual Explanation

### 4.1 Formal models
A **Markov game** for $N$ agents is a tuple $(\mathcal{S}, \{\mathcal{A}_i\}_{i=1}^N, P, \{r_i\}_{i=1}^N, \gamma)$ with joint transition $P(s' \mid s, a_1,\dots,a_N)$ and per-agent rewards $r_i(s,a_1,\dots,a_N)$. When agents observe only local information, the cooperative case is a **decentralised partially observable MDP (Dec-POMDP)**, with a shared reward and each agent conditioning on its own observation history. Agent $i$ seeks a policy $\pi_i$ maximising $\mathbb{E}\big[\sum_t \gamma^t r_i\big]$, but the value of $\pi_i$ depends on the other agents' policies $\pi_{-i}$. For general-sum games the solution concept is typically a Nash equilibrium, a profile in which no agent gains by unilaterally changing its policy; for two-player zero-sum games, equilibrium play is also minimax-optimal (Zhang et al., 2021).

### 4.2 Core difficulties
- **Non-stationarity**: from agent $i$'s perspective the transition kernel changes as $\pi_{-i}$ changes during learning.
- **Credit assignment**: with a shared team reward, determining each agent's contribution is hard. COMA computes a counterfactual baseline that marginalises one agent's action while holding the others' fixed (Foerster et al., 2018).
- **Scalability**: the joint action space grows exponentially with the number of agents.
- **Partial observability and communication**: what agents may observe or say to each other is a design choice.
- **Equilibrium selection and generalisation**: agents may fail against partners or opponents not seen in training.

### 4.3 Centralised training, decentralised execution
A common design is to give a critic access to global state or joint actions during training, while each actor uses only local observations at deployment. COMA uses a centralised critic with decentralised actors (Foerster et al., 2018); MADDPG likewise (Lowe et al., 2017). QMIX learns a mixing network constrained to be monotonic so that maximising the joint value is consistent with each agent maximising its own utility (Rashid et al., 2018). Yu et al. (2022) report that PPO with a centralised value function, little tuning and no domain-specific modifications is competitive on particle-world, StarCraft Multi-Agent Challenge, Google Research Football and Hanabi benchmarks.

### 4.4 Competition, self-play and populations
In competitive settings agents train against copies or populations of themselves. Jaderberg et al. (2019) used population-based training with learned internal rewards. Vinyals et al. (2019) combined supervised learning from human games with multi-agent reinforcement learning against a league of agents to reach Grandmaster level on the public Battle.net ladder. Baker et al. (2020) report six distinct emergent strategy phases in hide-and-seek, each creating pressure for the other team to adapt, and conclude that competition scaled better than intrinsic-motivation baselines on their transfer tests.

### 4.5 Language-model multi-agent systems
Here an agent is typically a language model instance with a role prompt, memory and tools; coordination happens via text messages. Patterns reported in the literature include: debate, in which several instances propose and critique answers over rounds, which Du et al. (2023) report improves mathematical and strategic reasoning and factual accuracy; role-playing pairs (Li et al., 2023); assembly-line pipelines that encode standard operating procedures so that agents check each other's intermediate outputs (Hong et al., 2023); a general conversation-programming framework (Wu et al., 2023); and agent societies with memory, reflection and planning that produce believable social behaviour (Park et al., 2023, with 25 agents in a simulated town). Cicero shows a hybrid: a language model for dialogue plus planning and reinforcement learning for strategy (FAIR et al., 2022).

## 5. Types, Components & Classifications
- **By objective structure**: fully cooperative (shared reward), fully competitive (zero-sum), mixed-motive (general-sum).
- **By information**: fully observable, partially observable, with or without communication channels.
- **By training paradigm**: independent learners, centralised training with decentralised execution, fully centralised, population-based self-play.
- **By agent type**: reinforcement-learning agents versus LLM-based agents with natural-language communication.
- **By topology (LLM systems)**: debate/peer groups, hierarchical (manager and workers), sequential pipelines, and free-form conversation.
- **By human involvement**: fully automated versus human-in-the-loop (supported in AutoGen according to Wu et al., 2023).

## 6. Applications & Concrete Examples
**Games (research milestones).** Team first-person play (Jaderberg et al., 2019), StarCraft II (Vinyals et al., 2019), and Diplomacy, where across 40 games of an anonymous online league Cicero achieved more than double the average score of human players and ranked in the top 10% of participants who played more than one game (FAIR et al., 2022). These are results in games with defined rules; they are not evidence of general negotiation competence outside them.

**Simulation and social modelling.** Generative-agent sandboxes (Park et al., 2023) are used to study believable behaviour; their evaluation is by human judgement of believability and by ablation, not by validated prediction of real populations.

**Software and task automation.** MetaGPT and AutoGen target software-engineering and other task benchmarks (Hong et al., 2023; Wu et al., 2023); see [[TECH-CODE-001] AI Code Generation](./TECH-CODE-001_ai_code_generation.md). These results are developer-reported on their chosen benchmarks.

**Robotics.** Coordinated teams of physical robots are an application area for cooperative MARL; this article cites no specific deployment and treats the area as research (see [[TECH-ROBOT-001] Robotics and Embodied AI](./TECH-ROBOT-001_robotics_and_embodied_ai.md)).

## 7. Advantages, Limitations & Unresolved Questions
**Advantages.** Division of labour and specialisation; parallelism; decentralised execution; in LLM systems, the possibility that cross-checking by multiple instances reduces some errors (Du et al., 2023; Hong et al., 2023).

**Limitations.** (1) Non-stationarity and weak theory: Zhang et al. (2021) describe the theory of MARL as relatively lacking compared with empirical results. (2) Training cost: the game-playing systems above required very large compute and engineered training leagues. (3) In LLM systems, the benefit of multiple agents over a single well-prompted model is not established in general. Cemri et al. (2025) analysed more than 1,600 annotated traces from seven multi-agent frameworks and proposed a taxonomy of 14 failure modes in three categories (system design issues, inter-agent misalignment, task verification), indicating that failures are frequent and structured. (4) Compounding errors, cost multiplication from many model calls, and difficult attribution of responsibility. (5) Safety: emergent behaviours (Baker et al., 2020) and interactions among deployed agents are hard to evaluate; systematic evidence on large-scale agent ecosystems is sparse.

**Unresolved.** Whether LLM multi-agent organisation yields gains that scale beyond what better single-agent prompting, tools or training provide; how to evaluate agent teams (see [[CAP-AGENTEVAL-001] Evaluating Autonomous Agents](../vol04_capabilities/CAP-AGENTEVAL-001_evaluating_autonomous_agents.md)); how to ensure cooperation is robust to partners not seen in training.

## 8. Current State of Knowledge
As of 2026-10-15: centralised-training methods (COMA, MADDPG, QMIX) and the effectiveness of tuned PPO baselines are documented on standard cooperative benchmarks (Yu et al., 2022; Foerster et al., 2018; Rashid et al., 2018) and are Well Supported for the tested benchmarks. Game-playing milestones are peer-reviewed. LLM multi-agent evidence is mostly Preliminary: framework papers (preprints, developer-reported benchmarks) and one failure-taxonomy study (Cemri et al., 2025). Systems and benchmark rankings change quickly and are not tabulated; see `MODEL-001`.

## 9. Future Trajectory & Strategic Outlook
**Observed trends.** Interest has shifted from game-playing leagues to LLM-based teams for coding and research tasks, alongside growing attention to failure analysis.

**Medium and long term.** Whether multi-agent organisation becomes the dominant way to build capable systems, or is absorbed into stronger single models with tool use, is an open question; this article asserts no forecast and attaches no `FCST-` record. Risks of interacting agents are discussed in [[SAFE-ALIGN-001] AI Alignment](../vol08_safety_governance/SAFE-ALIGN-001_ai_alignment.md) in general terms only.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-AGENT-001] Autonomous AI Agents](./TECH-AGENT-001_autonomous_agents.md): The single-agent loop that multi-agent systems compose.
  - [[TECH-RL-001] Reinforcement Learning](./TECH-RL-001_reinforcement_learning.md): MDP and policy-gradient foundations.
- **Related Methods & Technologies**:
  - [[TECH-LLM-001] Large Language Models](./TECH-LLM-001_large_language_models.md): Language backbone of LLM agent teams.
  - [[TECH-CODE-001] AI Code Generation](./TECH-CODE-001_ai_code_generation.md): A leading application of agent teams.
  - [[TECH-ROBOT-001] Robotics and Embodied AI](./TECH-ROBOT-001_robotics_and_embodied_ai.md): Physical multi-robot settings.
- **Evaluation and Safety**:
  - [[CAP-AGENTEVAL-001] Evaluating Autonomous Agents](../vol04_capabilities/CAP-AGENTEVAL-001_evaluating_autonomous_agents.md): Agent evaluation methodology.
  - [[SAFE-ALIGN-001] AI Alignment](../vol08_safety_governance/SAFE-ALIGN-001_ai_alignment.md): Alignment in multi-agent contexts.

## 11. References & Foundational Literature
1. Zhang, K., Yang, Z., & Başar, T. (2021). *Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms*. In K. G. Vamvoudakis et al. (Eds.), Handbook of Reinforcement Learning and Control (Springer); arXiv:1911.10635. [Markov and extensive-form game formulations and the state of MARL theory.]
2. Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., & Mordatch, I. (2017). *Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments*. NeurIPS 2017. https://arxiv.org/abs/1706.02275. [MADDPG.]
3. Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., & Whiteson, S. (2018). *Counterfactual Multi-Agent Policy Gradients*. AAAI 2018. https://arxiv.org/abs/1705.08926. [COMA: centralised critic and counterfactual baseline for credit assignment.]
4. Rashid, T., Samvelyan, M., Schroeder de Witt, C., Farquhar, G., Foerster, J., & Whiteson, S. (2018). *QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning*. ICML 2018. https://arxiv.org/abs/1803.11485. [Monotonic value factorisation.]
5. Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., & Wu, Y. (2022). *The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games*. NeurIPS 2022 Datasets and Benchmarks. https://arxiv.org/abs/2103.01955. [Tuned multi-agent PPO competitive on four cooperative benchmarks.]
6. Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L., Lever, G., Castañeda, A. G., et al. (2019). *Human-level performance in 3D multiplayer games with population-based reinforcement learning*. Science, 364(6443), 859–865. https://doi.org/10.1126/science.aau6249. [Population-based team play in a first-person game.]
7. Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., et al. (2019). *Grandmaster level in StarCraft II using multi-agent reinforcement learning*. Nature, 575(7782), 350–354. https://doi.org/10.1038/s41586-019-1724-z. [League-based multi-agent training to Grandmaster level.]
8. Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2020). *Emergent Tool Use From Multi-Agent Autocurricula*. ICLR 2020. https://arxiv.org/abs/1909.07528. [Emergent strategy phases in hide-and-seek.]
9. Meta Fundamental AI Research Diplomacy Team (FAIR), Bakhtin, A., Brown, N., Dinan, E., et al. (2022). *Human-level play in the game of Diplomacy by combining language models with strategic reasoning*. Science, 378(6624), 1067–1074. https://doi.org/10.1126/science.ade9097. [Cicero and its results over 40 games.]
10. Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023). *Generative Agents: Interactive Simulacra of Human Behavior*. UIST 2023. https://arxiv.org/abs/2304.03442. [25 LLM agents in a sandbox town; ablation of observation, planning and reflection.]
11. Li, G., Hammoud, H. A. A. K., Itani, H., Khizbullin, D., & Ghanem, B. (2023). *CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society*. NeurIPS 2023. https://arxiv.org/abs/2303.17760. [Role-playing framework for cooperative LLM agents.]
12. Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I. (2023). *Improving Factuality and Reasoning in Language Models through Multiagent Debate*. arXiv:2305.14325 (ICML 2024). [Multiagent debate and its reported gains.]
13. Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Zhang, C., Wang, J., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., Ran, C., Xiao, L., Wu, C., & Schmidhuber, J. (2023). *MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework*. arXiv:2308.00352 (ICLR 2024). [SOP-based role pipeline for software tasks.]
14. Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A. H., White, R. W., Burger, D., & Wang, C. (2023). *AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation*. arXiv:2308.08155. [Open-source multi-agent conversation framework.]
15. Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., Zaharia, M., Gonzalez, J. E., & Stoica, I. (2025). *Why Do Multi-Agent LLM Systems Fail?* arXiv:2503.13657. [Taxonomy of 14 failure modes from 1,600+ traces across seven frameworks.]

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Markov game / Dec-POMDP formalisms; Nash equilibrium concept | Established | Zhang et al. (2021) | High |
| Non-stationarity and credit assignment as core MARL difficulties | Well Supported | Zhang et al. (2021); Foerster et al. (2018) | High |
| CTDE methods (COMA, MADDPG, QMIX) work on tested benchmarks | Well Supported | Foerster et al. (2018); Lowe et al. (2017); Rashid et al. (2018) | Moderate to High |
| Tuned PPO is a strong cooperative baseline | Well Supported (single study, four benchmarks) | Yu et al. (2022) | Moderate |
| Game-playing milestones | Well Supported (peer-reviewed, game-specific) | Jaderberg et al. (2019); Vinyals et al. (2019); FAIR et al. (2022) | Moderate to High |
| Emergent multi-phase strategies in hide-and-seek | Well Supported (single lab) | Baker et al. (2020) | Moderate |
| Debate improves LLM reasoning/factuality | Preliminary | Du et al. (2023) | Moderate |
| LLM multi-agent frameworks outperform single agents in general | Contested / Not established | Hong et al. (2023); Wu et al. (2023); Cemri et al. (2025) | Low |
| Taxonomy of multi-agent LLM failures | Preliminary (preprint) | Cemri et al. (2025) | Moderate |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.