Project 07EvaluationSupervised learning · PCG

Predicting Difficulty in Procedurally Generated Game Levels

Modelling challenge using level features, observed performance or simulated play

Difficulty is often approximated using level length, hazards or shortest paths, but these features may be poor proxies for a particular player or task. A machine-learning model can combine evidence from level structure and play, provided the outcome being predicted is meaningful. This project invites investigation of how difficulty can be defined, measured and predicted for a chosen game.

What is a supervised prediction model, here?

Supervised learning trains a model on paired examples, a level's features going in, some observed outcome (deaths, completion time, a rating) coming out, so it can estimate that outcome for a level it has never seen. The model is only as good as the outcome it was trained on, and level length or hazard count might be poor stand-ins for what actually made a level hard for a given player.

Project concept

For a game or level type that interests you, gather a dataset of levels alongside some evidence of their challenge, then explore whether computational models can estimate difficulty for levels they have not seen. The reference could come from simulated agents, observations of play or another defensible source you choose; each supports a different interpretation of what "difficulty" means.

Questions to consider

These are possible directions for your investigation, not a ready-made research question.

  • What does difficulty mean for the chosen mechanics, and for whom is it being measured?
  • How much can level structure tell us before anyone plays the level?
  • Do predictions remain useful when the generator or the playing agent changes?

What might you produce?

Some level-and-observation dataset, a difficulty estimation prototype or model, a visual or analytical presentation of predictions, and evidence about their reliability. How you assemble this is up to you.

Possible approach

A small level generator or an existing dataset can supply examples. Explore candidate definitions and how to obtain reference labels before selecting predictive models. Comparing against a simple, easy-to-explain rule (level length or hazard count, say) shows whether the trained model is actually doing better than that, or just matching it.

Investigating the result

Assess predictions on levels not used to build the model, and examine systematic errors. Agent-based measurements should not be presented as human-perceived difficulty without additional evidence.

Core concepts

  • Difficulty operationalisation
  • Supervised regression or classification
  • Feature engineering
  • Dataset bias
  • Validation and generalisation

Indicative technologies

Choose tools that suit the research. An engine is not required if a simulation can answer the question.

  • Python, pandas and scikit-learn
  • NetworkX or simple pathfinding agents
  • Optional game-engine level visualisation

Keeping the scope manageable

One game or level format and one justified interpretation of difficulty is enough to build a solid investigation on. Difficulty means different things in different games, so a rating that claims to work everywhere would end up meaning nothing in particular.

Explore the literature

Search the library using relevant terms such as:

  • game level difficulty prediction
  • procedural content generation difficulty metrics
  • player modelling difficulty estimation
  • level feature engineering
  • generalisation across generators

Papers and reading

  • Jennings-Teats, M., Smith, G. and Wardrip-Fruin, N. (2010) Polymorph: A Model for Dynamic Level Generation. Proceedings of the 6th AAAI Conference on AI and Interactive Digital Entertainment (AIIDE). AAAI.
  • Roohi, S., Guckelsberger, C., Relas, A., Heiskanen, H., Takatalo, J. and Hämäläinen, P. (2021) Predicting Game Difficulty and Engagement Using AI Players. Proceedings of the ACM on Human-Computer Interaction, 5(CHI PLAY), Article 231. ACM Digital Library (also arXiv).

Recommended reading:

  • Yannakakis, G. N. and Togelius, J. (2018) Artificial Intelligence and Games. Springer.