Imitation Learning from Gameplay Demonstrations
From recorded player actions to autonomous game-playing behaviour
Imitation learning offers an alternative to specifying a reward function: the policy learns from examples of decisions made by a demonstrator. A model may reproduce held-out actions accurately yet fail in closed-loop play because its own small mistakes lead to unfamiliar states. That discrepancy makes games a useful testbed for investigating learning from demonstrations.
What is imitation learning?
Instead of a reward signal, the agent is given recordings of someone else playing; pairs of "this was the situation" and "this was the action taken". Behavioural cloning trains a model to predict the recorded action from the situation, much like any supervised-learning problem. The catch is that the agent then has to act on its own, and small prediction errors can push it into situations the demonstrations never covered.
Project concept
Choose a game where actions and relevant state can be captured, whatever genre draws you in, then collect or obtain demonstrations. From there, two directions are both worth pursuing: training an agent to play from those demonstrations, and investigating whether it generalises beyond them, or using the demonstrations themselves as an analytical resource, examining what they reveal about strategy, skill, or mistakes rather than training an agent to reproduce them at all. The interesting part is what the demonstrations actually support, not just how closely something reproduces them.
Questions to consider
These are possible directions for your investigation, not a ready-made research question.
- What does it mean for a demonstration dataset to cover the game well enough?
- If training an agent: why might accurate action prediction fail to produce successful autonomous gameplay?
- If analysing demonstrations directly: what can they tell you about strategy or skill that raw performance numbers cannot, and what would you need to be cautious about inferring?
What might you produce?
Some demonstration capture or ingestion workflow and a documented dataset, together with either a trained imitation-learning agent and evidence of its behaviour during autonomous gameplay, or an analytical tool that extracts and presents what the demonstrations reveal. The game, demonstration source and direction are yours to choose.
Possible approach
Behavioural cloning is a feasible initial technique; record state-action pairs, train a supervised policy and then let it play. Decide how to separate demonstrations from evaluation scenarios, and whether collecting corrective examples is useful for your question.
Investigating the result
Measure task success on unseen scenarios and inspect failure trajectories; an agent can match the training demonstrations closely and still fail once play diverges from them, so test that directly rather than relying on training accuracy as a proxy. Separate data from different levels or runs appropriately.
Core concepts
- Behavioural cloning
- Distribution shift
- Demonstration datasets
- Supervised policy learning
- Train-test separation
Indicative technologies
Choose tools that suit the research. An engine is not required if a simulation can answer the question.
- Python, PyTorch and Gymnasium
- Recorded input/state traces from a simple game
- Godot, Unity or Unreal as a demonstration capture environment
Keeping the scope manageable
A small action space and one game task are sufficient to build on; the interesting question of whether imitation generalises shows up just as sharply in a small task as a large one. Learning directly from unlabelled gameplay video or training a general game-playing model belongs to a much bigger project than this one.
Explore the literature
Search the library using relevant terms such as:
- imitation learning game agents
- behavioural cloning
- covariate shift imitation learning
- learning from demonstrations
- DAgger imitation learning
Papers and reading
- Ross, S., Gordon, G. and Bagnell, D. (2011) A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 15. PMLR (also arXiv).
- Pearce, T. and Zhu, J. (2022) Counter-Strike Deathmatch with Large-Scale Behavioural Cloning. IEEE Conference on Games (CoG). IEEE Xplore (also arXiv).