ProjekinAja. Reinforcement Learning Stage 4 · Advanced & Applied Lesson 4.3
● Stage 4 · Lesson 4.3

Multi-Agent RL

The real world is full of other agents. When many RL agents learn at once — cooperating, competing, or both — new dynamics emerge that single-agent RL never has to face.

⏱ ~16 min read 🤝 Cooperation & competition 🧩 Stage 4 · Advanced

🎯 By the end you'll understand

Why adding other learning agents breaks the comfortable assumption that the environment is stationary, the three settings of multi-agent RL (cooperative, competitive, mixed), the hard problems they create (non-stationarity, credit assignment, the joint-action explosion), and two big ideas that make MARL work — centralized training with decentralized execution and self-play.

KEY INSIGHTThe one idea to take away

The moment a second learning agent enters the room, the world stops standing still — every other agent is now part of your environment, and they're all changing at once.

Single-agent RL gets to assume the world plays fair: the same action in the same state tends to give the same outcome. Multi-agent RL throws that away. Your opponent learns, your teammate learns, the rules of the game you're learning shift under your feet while you study them. That single fact — a moving target — is the source of almost every difficulty in the field.

🔑 Hold onto this

In single-agent RL the environment is stationary — its dynamics don't change as you learn. In multi-agent RL, each agent's "environment" includes the other agents, and they're learning too. So the environment becomes non-stationary. Everything hard about MARL flows from this one shift.

CORE CONCEPTWhat changes when you add other agents

Think back to the cleaning robot from earlier in the course. It lived in three rooms, took actions, and the rooms behaved the same way every time — dirt was where the dirt was, doors opened the way they always did. That predictability is exactly what made learning a value function V(s) work: the robot could try things, see the result, and trust that the lesson would still hold tomorrow.

Now imagine a second cleaning robot in the same building, also learning. Suddenly the first robot's world is no longer fixed. The room it was about to clean might already be clean — because the other robot got there first. The path it learned to take might now be blocked. And here's the twist: the other robot is changing its own behaviour as it learns, so even the "rules" of how the building responds keep shifting. The first robot is trying to learn a target that is itself moving.

Formally, recall from Lesson 1.4 that single-agent RL assumes a Markov Decision Process with fixed transition dynamics P(s′|s,a) and a fixed reward R(s,a). From any one agent's point of view in a multi-agent system, those dynamics now depend on what everyone else does — and since the others are updating their policies over time, the effective transition and reward functions change as training proceeds. The MDP assumption quietly breaks.

📖 Multi-Agent Reinforcement Learning (MARL)

A setting where two or more agents learn simultaneously by interacting in a shared environment. Each agent chooses actions and receives rewards, but the outcomes — and often the rewards — depend on what the other agents are doing too.

📖 Non-stationarity

When the dynamics an agent is trying to learn change over time. In MARL it arises because the other agents are learning: from any single agent's perspective, the environment keeps shifting as its co-learners update their policies — so a fixed-target learner is always chasing.

This isn't a minor footnote. Most of the convergence guarantees you met in Stage 2 — Q-learning settling onto the optimal value function, for example — quietly assume a stationary environment. Take that away and the guarantees go with it. Two agents can chase each other in circles, each adapting to a version of the other that no longer exists by the time they finish updating.

FRAMEWORKCooperate, compete, or both

Not all multi-agent problems are the same — and the single thing that decides how they behave is the reward structure: do the agents win together, win at each other's expense, or a messy mix of both? This gives us three settings.

Cooperative. The agents share a goal. Often they literally share one team reward, so what's good for one is good for all. Think a fleet of warehouse robots fulfilling orders, or a set of traffic lights across a city all trying to minimise total congestion. The challenge here isn't "who wins" — it's coordination: getting agents to pull in the same direction without tripping over each other.

Competitive. One agent's gain is another's loss. The cleanest case is zero-sum: the rewards add up to zero, so +1 for me means −1 for you. Two-player board games — chess, Go — are the canonical example. Here the agents are adversaries, and getting better means anticipating and beating an opponent who is also getting better.

Mixed. The real world, mostly. Agents both cooperate and compete, often at the same time. Autonomous cars share the road safely (cooperation) while each still wants to reach its own destination quickly (competition). Players on a soccer team cooperate to win against the other team, but might compete for the ball or the glory. Markets, negotiations, and most real economies live here too.

SettingReward structureWhat the agents are doingExample
Cooperativeshared / common reward — good for one is good for allcoordinating toward one goalwarehouse robot fleet · city traffic control
Competitivezero-sum — my +1 is your −1out-thinking an adversarychess · Go · two-player games
Mixedpartly shared, partly opposedcooperating and competing at onceself-driving cars · soccer · markets
💡 The tell

Want to classify a multi-agent problem fast? Just ask what happens to everyone else's reward when one agent does well. If it goes up → cooperative. Down → competitive. "It depends" → mixed.

SEE ITWatch the setting change the incentives

Same two agents, same little world, same possible behaviours. The only thing we change is the reward structure — cooperative, competitive, or mixed. Watch how that one switch completely reshapes what "good behaviour" means and what each agent's score does.

🤝 Interactive · Cooperate, compete, or both

Two agents (Ada & Bo) split a shared resource. Pick the reward structure and see the behaviour that the incentives pull them toward.

🟦 Ada0
🟩 Bo0
Team total0

—

Notice: nothing about the agents or the world changed — only the reward structure. Yet cooperative agents converge on sharing to maximise the team total, competitive agents fight to starve each other, and mixed agents settle into an uneasy bargain. The reward structure is the strategy.

WEAKNESSThe hard problems MARL creates

Multi-agent RL inherits everything tricky about single-agent RL and then adds its own layer. Four problems dominate.

1 · Non-stationarity (the big one). As we've seen, every other learning agent makes your environment a moving target. Naïvely, if each agent just runs its own single-agent algorithm and pretends the others are part of a fixed world, they can fail to converge — each is optimising against an opponent that no longer exists by the next update.

2 · Credit assignment across agents. In a cooperative team that shares one reward, suppose the team scores +10. Which agent earned it? Maybe one did all the work and another loafed — but they get the same reward. This multi-agent credit assignment problem is on top of the across-time credit assignment you already know from Lesson 1.1. Without solving it, lazy agents get rewarded for their teammates' effort.

3 · The curse of dimensionality (joint action explosion). If you try to treat the whole team as one big "super-agent" choosing a joint action (one move for every agent at once), the number of combinations explodes. With n agents each having k actions, the joint action space has kn entries.

|joint actions| = knjoint action space grows exponentially in the number of agents

Just 10 agents with 5 actions each gives 510 ≈ 9.8 million joint actions — far too many to enumerate, let alone learn a value for. This is why we can't just bolt agents together and learn one giant policy.

4 · Communication, coordination & emergent behaviour. Agents often need to coordinate — sometimes by learning to communicate, sometimes by silently developing conventions (a "you go left, I'll go right" understanding). And because behaviours co-evolve, MARL frequently produces emergent behaviour: strategies no one programmed in, arising purely from agents adapting to each other. That's both the magic and the unpredictability of the field.

⚠️ Honest note: MARL is genuinely hard

Compared with single-agent RL, multi-agent methods are less stable and harder to train. Non-stationarity, credit assignment, and scaling to many agents are still active research problems with no clean universal solution. Treat the techniques below as the best current tools, not closed cases.

CORE CONCEPTCentralized training, decentralized execution

Here's the clever idea that tames a lot of the difficulty. At run time, each agent usually only sees its own local view — a robot can't read its teammates' minds, a car only sees its own sensors. So each agent must ultimately act on its own. But during training, we're in the lab, and we can cheat a little: we can give the learning process access to everyone's observations and actions all at once.

This is centralized training, decentralized execution (CTDE). Train with a god's-eye view; execute with blinkers on.

📖 Centralized Training, Decentralized Execution (CTDE)

A practical MARL pattern where, during training, a learner has access to global information (all agents' observations and actions); but at execution time, each agent acts using only its own local observation. You get the stability of central coordination while keeping agents independent in deployment.

Why does this help? The non-stationarity problem comes from each agent seeing the others as a confusing, shifting part of the environment. But a centralized critic during training can see what the others actually did, so the moving target stops moving from its perspective — it's conditioning on the very actions that were causing the confusion. Each agent still ends up with its own policy that runs on local information alone.

🧪 In your knowledge base: MADDPG

MADDPG (Multi-Agent DDPG) is the textbook example of CTDE. It extends the single-agent DDPG actor-critic to many agents: each agent keeps its own actor (a policy on local observations, used at execution), but during training each has a centralized critic that sees all agents' observations and actions. The central critic makes learning stable; the local actors keep deployment decentralized — CTDE in one algorithm.

INTUITIONSelf-play: the automatic curriculum

Now the competitive side — and one of the most beautiful ideas in all of RL. How do you train a world-class game-playing agent when there's no world-class opponent to learn from? You make the agent play against itself.

Imagine learning chess but your only opponent is a mirror — a copy of you. At first you're both terrible, so the games teach beginner lessons. But every time you improve, your mirror improves too, because it's a copy. So tomorrow's opponent is exactly as good as today's improved you — a little harder than yesterday. You're never crushed by something hopelessly strong, and never bored by something hopelessly weak. The difficulty rises in perfect lockstep with your skill. That's an automatic curriculum: a ladder of ever-stronger opponents that builds itself, for free.

📖 Self-play

A training method where an agent improves by playing against copies (or past versions) of itself. Because the opponent's strength tracks the agent's own, self-play generates an automatic curriculum of progressively harder challenges — no hand-designed opponents required.

The virtuous cycle looks like this: the agent plays itself → it discovers a slightly better strategy → that better strategy becomes the new opponent → which forces the discovery of an even better strategy → and around it goes. Each turn of the loop ratchets skill upward, often blowing past anything humans could have hand-crafted as a training opponent. This is precisely how the most famous game-playing systems reached superhuman play.

Press play to follow three rounds of the loop and watch the two strength bars, then swap the opponent for a fixed player and play it again.

Three rounds of self-play
The opponent is
Play a copy of itself Find a slightly better strategy It becomes the new opponent round 3
Playing strength
Agentlevel 4
Opponentlevel 4
The self-play loop: opponent strength rises in lockstep with the agent's own skill, building its own curriculum. Press play to follow three rounds.

EXAMPLEFamous results that made MARL famous

These ideas aren't just theory — they produced some of the landmark results in AI.

♟️

AlphaGo → AlphaZero (Go & chess)

DeepMind's systems reached superhuman play in Go — long thought decades away — largely through self-play. AlphaZero went further: starting from only the rules, with no human games at all, it taught itself Go, chess, and shogi to superhuman level purely by playing millions of games against itself. A textbook competitive, zero-sum self-play success.

🎮

OpenAI Five (Dota 2)

A team of five RL agents learned to play the complex video game Dota 2 and defeated world-champion human teams. It blends cooperation (five agents coordinating as a team) with competition (beating the enemy team) — a mixed setting — and was trained at enormous scale largely through self-play.

🙈

Hide-and-seek emergent tool use

In OpenAI's hide-and-seek experiments, two teams of agents competed. Through self-play, they invented a startling escalation of strategies nobody programmed: hiders learned to build forts with boxes, seekers learned to use ramps to jump in, hiders learned to steal the ramps — emergent behaviour arising purely from competitive co-adaptation.

💡 The common thread

Competition + self-play kept producing strategies that surprised even the researchers. When opponents push each other to improve indefinitely, you get an open-ended ladder of innovation — that's the engine behind most superhuman game-playing AI.

WEAKNESSCommon mistakes to avoid

✗ "Single-agent algorithms just work unchanged with many agents."

No. Dropping independent single-agent learners into a shared world (so-called independent learning) often fails to converge, because each treats the others as a fixed environment when they're actually learning too. That broken assumption is exactly why CTDE methods like MADDPG exist.

✗ "The environment stays stationary, like in single-agent RL."

It does not. The defining feature of MARL is non-stationarity: the other learning agents are part of your environment and they keep changing. Most single-agent convergence guarantees assume a stationary environment and so don't carry over directly.

✗ "Self-play needs a hand-designed opponent to learn against."

The opposite is the point. Self-play's whole appeal is that the opponent is a copy of the agent itself, so the curriculum is automatic — opponent strength rises with the agent's own. AlphaZero learned from zero human games, opponents included.

KEY INSIGHTWhere this connects in your course

  • Markov Decision Processes (1.4) — MARL is what happens when the MDP's stationary dynamics assumption breaks because other learners are in the loop.
  • Policy gradients / actor-critic (3.4) — the actor-critic structure is exactly what CTDE methods extend: local actors, centralized critics.
  • DDPG (3.6) — MADDPG, our CTDE example, is the multi-agent generalisation of DDPG.
  • Real-world & frontier RL (4.6) — multi-agent systems (markets, traffic, robot fleets, games) are a major frontier where these methods get deployed.
  • Agentic AI course — multi-agent systems — that course's multi-agent orchestration ideas (several LLM agents collaborating) rhyme conceptually with cooperative MARL: coordination, communication, and division of labour, just with language models instead of learned policies.

Hold the one sentence — other learning agents make your world a moving target — and CTDE and self-play are simply the two best answers we have to it.

GLOSSARYKey terms introduced

Multi-agent RL (MARL)
RL where two or more agents learn simultaneously in a shared environment, with interdependent outcomes.
Non-stationarity
When the dynamics an agent is learning change over time — in MARL, because the other agents are also learning.
Cooperative setting
Agents share a goal / reward; what's good for one tends to be good for all. Challenge: coordination.
Competitive setting
One agent's gain is another's loss, often zero-sum (+1 for me = −1 for you).
Mixed setting
Agents both cooperate and compete (e.g. self-driving cars, soccer, markets).
Self-play
Improving by playing against copies/past versions of yourself, creating an automatic curriculum of stronger opponents.
CTDE
Centralized Training, Decentralized Execution: train with global information, act on local observations.
Joint action space
The set of all combinations of every agent's action at once; size kn for n agents with k actions each.
Emergent behaviour
Strategies no one programmed, arising from agents adapting to each other (e.g. hide-and-seek tool use).

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1What's the single core difficulty that distinguishes multi-agent RL from single-agent RL, and why does it arise?›

Non-stationarity — the environment is no longer a fixed target. Because the other agents are also learning, they're effectively part of your environment and they keep changing their behaviour as training proceeds. The stationary-MDP assumption (fixed transition and reward dynamics) that single-agent guarantees rely on no longer holds.

2Classify these by setting and say why: (a) a fleet of warehouse robots, (b) two players in Go, (c) self-driving cars sharing a road.›

(a) Cooperative — they share a goal (fulfilling orders); one robot doing well helps the team. (b) Competitive / zero-sum — one player winning means the other loses. (c) Mixed — cars cooperate to avoid crashes but each competes to reach its own destination quickly. The test: what happens to everyone else's reward when one agent does well?

3What does "centralized training, decentralized execution" mean, and why does it help?›

During training the learner can use global information — all agents' observations and actions (e.g. a centralized critic). At execution each agent acts using only its own local observation. It helps because a critic that can see the other agents' actions removes much of the non-stationarity (the moving target stops moving from its perspective), giving stable training while keeping agents independent in deployment. MADDPG is the classic example.

4Why is self-play described as an "automatic curriculum," and which famous results used it?›

Because the agent plays against copies of itself, the opponent's strength rises in lockstep with the agent's own — never too easy, never too hard. Each improvement becomes the next opponent, ratcheting skill upward without any hand-designed ladder of opponents. AlphaGo/AlphaZero (Go, chess, shogi) and OpenAI Five (Dota 2) reached superhuman play this way; the hide-and-seek experiments produced emergent tool use through it.

Many agents, one moving world. Next: learning without live interaction →