Project 05AgentsMulti-agent reinforcement learning

Cooperative Multi-Agent Reinforcement Learning in Games

Coordination, shared objectives and credit assignment between learning agents

In a cooperative multi-agent game, each policy must act in the presence of another policy that may also be learning. Coordination can fail despite a shared goal, and a successful team reward may hide a contribution by one agent rather than both. A small cooperative puzzle or resource task provides a manageable setting for investigating these issues.

What is multi-agent reinforcement learning?

There are at least two different ways to train two agents that need to cooperate. Independent learners each run their own single-agent algorithm, updating only from their own experience, with the other agent treated as just part of the environment; from either agent's point of view, that environment keeps shifting as the partner also learns, which is a genuine theoretical problem called non-stationarity. A jointly-trained approach instead learns from both agents' experience together, sharing a value function or critic across them, which can assign credit properly but costs more to set up. Whether that difference actually matters for the resulting behaviour, or whether independent learners get there anyway in a simple enough task, is what this project investigates.

Project concept

Design a scenario, drawn from whatever kind of game or simulation interests you, where two agents genuinely need each other to succeed. One question sits at the heart of it: do two agents learning independently, each updating from its own experience with no shared or group-wise update, converge on real cooperation, or does that only reliably happen with a jointly-trained approach that learns from both agents together? Reward allocation, communication, role assignment and partner robustness are all viable angles worth considering alongside that.

Questions to consider

These are possible directions for your investigation, not a ready-made research question.

  • Does the non-stationarity facing independent learners actually prevent them from converging on cooperation, or does it just make training noisier or slower?
  • How does a shared outcome become useful learning feedback for individual agents, when no shared value function connects their updates?
  • Which observed behaviours indicate real coordination rather than accidental joint success, and does that distinction look different for the two training approaches?

What might you produce?

Some two-agent game or simulation, agents trained under both an independent-learners and a jointly-trained setup, and diagnostic tools such as replays or event traces that make cooperative and failed behaviour visible. The scenario is yours to design.

Possible approach

Two agents with discrete actions and a compact cooperative task is a practical starting point. Train the same task under independent learners (each agent runs its own algorithm, unaware it is training alongside another learner) and a jointly-trained approach (a shared critic or value function sees both agents' actions), keeping the task, algorithm family and training budget as comparable as the two setups allow.

Investigating the result

Compare joint task completion, coordination errors and learning variability between the independent and jointly-trained conditions, across repeated runs; non-stationarity is a source of instability, so a single training run from either condition is not enough to tell a real difference from noise. An authored cooperation policy, where feasible, gives a third point of reference.

Core concepts

  • Cooperative games
  • Multi-agent reinforcement learning
  • Shared and individual rewards
  • Credit assignment
  • Non-stationarity

Indicative technologies

Choose tools that suit the research. An engine is not required if a simulation can answer the question.

  • Python with Gymnasium-style environments or PettingZoo
  • NumPy or a lightweight 2D framework for a custom cooperative scenario, if PettingZoo's built-in environments do not fit
  • An RL implementation with multi-agent support, such as RLlib or a MADDPG/QMIX-style implementation

Keeping the scope manageable

Two agents and one cooperative task provide ample material to work with; the interesting questions about coordination and credit assignment show up just as clearly with two agents as with twenty. Save the swarms and competitive leagues for a project with more time.

Explore the literature

Search the library using relevant terms such as:

  • cooperative multi-agent reinforcement learning
  • multi-agent credit assignment
  • parameter sharing MARL
  • independent learners
  • PettingZoo environments

Papers and reading

  • Rashid, T., Samvelyan, M., Schroeder de Witt, C., Farquhar, G., Foerster, J. and Whiteson, S. (2018) QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. Proceedings of the 35th International Conference on Machine Learning (ICML), PMLR 80. PMLR (also arXiv).
  • Lou, X., Guo, J., Zhang, J., Wang, J., Huang, K. and Du, Y. (2023) PECAN: Leveraging Policy Ensemble for Context-Aware Zero-Shot Human-AI Coordination. Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems (AAMAS). ACM Digital Library (also arXiv).

Recommended reading:

  • Albrecht, S. V., Christianos, F. and Schäfer, L. (2024) Multi-Agent Reinforcement Learning: Foundations and Modern Approaches. MIT Press.