Project 03AgentsMachine learning · Game AI

Learning NPC Behaviour: Adaptive Game AI Beyond Authored Rules

Learned decision-making compared with conventional authored game AI

Game AI is generally judged in the context of designed play rather than abstract optimality. An NPC's behaviour can come from several different learning mechanisms, and each produces a different relationship between what a designer intended and what the NPC actually ends up doing; some are difficult to interpret, some are fragile outside their training conditions, in different ways. This project explores that relationship in a manageable game encounter.

What does it mean for an NPC to "learn" its behaviour?

A finite-state machine or behaviour tree has every rule written down by a designer; a learned NPC's behaviour emerges instead from some training process. Three quite different mechanisms can produce that. Reinforcement learning has an agent try actions and adjust towards whatever earns a reward signal, with no example of "correct" play ever shown. Imitation or observation-based learning trains a policy directly on recorded examples of play. Evolutionary or search-based methods maintain and improve a population of candidate behaviours through variation and selection, rather than gradient-based training. Each gives a different relationship between designer intent and resulting behaviour, and that relationship is what this project investigates.

Project concept

Choose an NPC role that raises an interesting question for you, patrol, pursuit, evasion, companionship or something else entirely. Investigate a behavioural property that matters for your chosen role. A large game world is unnecessary; the encounter itself, and what a learned policy does differently from an authored alternative, is where the investigation happens.

Pathways into this project

Three starting directions, among others, to consider; a project can also compare two of them if that suits the question you want to investigate.

Reinforcement learning. The NPC learns through trial and error against a reward signal. The interesting failure mode is a gap between what got rewarded and what the designer actually wanted, an agent optimising detection rate might become predictable or unfun even while winning.

Imitation or observation-based learning. The NPC's policy is trained on recorded human or scripted play, without a reward signal at all. The interesting failure mode is brittleness outside recorded situations; behaviour that looks natural within demonstrated scenarios can fall apart the moment play diverges from them.

Evolutionary or search-based methods. A population of candidate behaviours is varied and selected over generations rather than trained by gradient descent. The interesting failure mode is convergence on a narrow, exploitable strategy that scores well against whatever it was evaluated on without generalising to a real player.

Questions to consider

These are possible directions for your investigation, not a ready-made research question.

  • What kinds of adaptation are possible with a learned controller, and what remains easier to author explicitly?
  • How should an NPC be assessed when task success and an enjoyable encounter are not the same thing?
  • What happens when the behaviour of the player or the environment changes?

What might you produce?

Some playable or simulated NPC encounter with at least one learned controller, whichever pathway you choose, a way to compare it against an authored alternative, and a way to inspect or replay behaviour alongside experimental evidence. The exact form is yours to decide.

Possible approach

A small 2D encounter is a practical way to expose the same state and action choices to an authored controller and a learned one, regardless of pathway. Consider how to make any comparison fair, and how you will observe behavioural differences rather than relying on a single demonstration.

Investigating the result

Meaningful measures might include detection or interception rate, time to respond, failure cases and performance on unseen scenarios. A stronger agent is not automatically a better game opponent; distinguish task effectiveness from design intent.

Core concepts

  • Finite-state machines and behaviour trees
  • Policy learning, imitation, or evolutionary search, depending on pathway
  • Agent evaluation
  • Generalisation
  • Observations and action spaces

Indicative technologies

Choose tools that suit the research and pathway. An engine is not required if a simulation can answer the question.

  • Gymnasium and Python with an RL library such as Stable-Baselines3, for reinforcement learning
  • PyTorch and recorded play traces, for imitation or observation-based learning
  • DEAP or a similar evolutionary-computation library, for evolutionary or search-based methods
  • Godot, Unity or Unreal, or a small custom state machine, for the authored comparison

Keeping the scope manageable

One encounter and a manageable behaviour problem is enough to work with. A full stealth game, crowd simulation or open-world AI system is more than the project needs; discuss with your supervisor how many learning mechanisms to bring into scope.

Explore the literature

Search the library using relevant terms such as:

  • reinforcement learning non-player characters
  • game AI behaviour trees versus reinforcement learning
  • imitation learning game agents
  • evolutionary computation game AI
  • NPC policy generalisation

Papers and reading

  • Mnih, V., Kavukcuoglu, K., Silver, D. et al. (2015) Human-Level Control Through Deep Reinforcement Learning. Nature, 518, 529-533. Nature (also arXiv preprint).
  • Risi, S. and Togelius, J. (2017) Neuroevolution in Games: State of the Art and Open Challenges. IEEE Transactions on Computational Intelligence and AI in Games, 9(1), 25-41. IEEE Xplore (also arXiv).

Recommended reading:

  • Sutton, R. S. and Barto, A. G. (2018) Reinforcement Learning: An Introduction. 2nd edn. MIT Press.
  • Yannakakis, G. N. and Togelius, J. (2018) Artificial Intelligence and Games. Springer.