Project 04AgentsReinforcement learning · Reward design

Reward Shaping in Reinforcement Learning

How the reward you design shapes what a game-playing agent actually learns to do

A reinforcement-learning agent only ever optimises the reward it is given, and how that reward is shaped, what it rewards, how often, how densely, changes what the agent ends up doing. This project investigates how reward shaping affects training in a game of your choosing, comparing what a given reward formulation produces against what training on it was meant to achieve.

What is reward shaping?

A reinforcement-learning agent only knows what it is told through its reward signal, a number given after each action. Reward shaping means designing that signal: whether it is sparse (feedback only occasionally) or dense (feedback often), and what specifically it rewards. The agent will find the easiest way to maximise that number, which is not always the way you imagined when you wrote it.

Project concept

Choose a game and a training objective within it, then investigate how varying the reward affects what the trained agent does. Some games make this especially interesting: a long, puzzle-heavy game like Pokémon Red gives an agent very little to learn from between rewards, which is one reason reward shaping matters there, but any game where the reward could plausibly be designed several different ways is a fair starting point.

Questions to consider

These are possible directions for your investigation, not a ready-made research question.

  • How does a sparse reward compare to a denser, shaped version of it, in what the agent learns and how quickly?
  • Does a reward formulation that trains well produce behaviour you would actually call successful, or does it find an easier way to maximise the number?
  • What happens to training when the shaping signal and the true objective start to disagree?

What might you produce?

A game or environment with a clear training objective, agents trained under two or more reward formulations, and behavioural evidence, trajectories, replays, learning curves, comparing what each one produced. The game and the reward formulations are yours to choose.

Possible approach

Pick one fixed game and vary the reward while holding everything else constant: same environment, same algorithm, same training budget. Define what counts as genuine success independently of the reward itself, so that a reward formulation which trains well can still be checked against it.

Investigating the result

Compare each reward formulation on that independent success measure, not only on the reward it produced during training; a smoothly climbing reward curve confirms the agent is optimising the signal it was given, not that the resulting behaviour is what you actually wanted. Repeated runs help distinguish a real effect of the reward design from ordinary training variance.

Core concepts

  • Reward shaping
  • Sparse and dense rewards
  • Potential-based reward shaping
  • Reward hacking
  • Evaluation metrics versus training objectives

Indicative technologies

Choose tools that suit the research. An engine is not required if a simulation can answer the question.

  • Python and Gymnasium
  • Stable-Baselines3 or another RL implementation
  • An optional simple visual replay or game-engine front end

Keeping the scope manageable

One fixed game and a small set of thoughtfully chosen reward formulations is enough to make the comparison; two or three well-justified variants beat many arbitrary ones. Multiple algorithms, complex physics and sophisticated visual perception are not the primary objective.

Explore the literature

Search the library using relevant terms such as:

  • reward shaping reinforcement learning
  • potential-based reward shaping
  • reward hacking
  • sparse reward exploration
  • specification gaming

Papers and reading

  • Ng, A., Harada, D. and Russell, S. (1999) Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. Proceedings of the 16th International Conference on Machine Learning (ICML). ACM Digital Library.
  • Skalse, J., Howe, N., Krasheninnikov, D. and Krueger, D. (2022) Defining and Characterizing Reward Gaming. Advances in Neural Information Processing Systems (NeurIPS) 35. NeurIPS Proceedings (also arXiv).

Recommended reading:

  • Sutton, R. S. and Barto, A. G. (2018) Reinforcement Learning: An Introduction. 2nd edn. MIT Press.