Procedural Level Critic: Automated Evaluation of Generated Levels
Investigating what makes a generated level assessable by a computational system
Automatic level evaluation is often discussed as though a single quality score were possible. In practice, the qualities a designer cares about may be structural, functional, aesthetic or experiential, and they need not correlate. This project invites you to investigate which aspects of a generated level can be assessed computationally, what evidence those assessments require and what an automated critic could contribute to a design workflow.
What is a computational critic?
Rather than a person reading a level, a program reads it; analysing its geometry, simulating play through it, or comparing it against reference examples. The output might be a single diagnostic ("this room is unreachable") or a machine-learning model trained to flag patterns a human evaluator would notice.
Project concept
Develop a prototype capable of examining generated levels and communicating an assessment to a designer or generator. You will need to investigate what is worth assessing in your chosen genre and which qualities are observable from geometry, play traces, reference examples or human judgements. A critic need not begin with a learned model; determining where learning is useful is itself part of the investigation.
Questions to consider
These are possible directions for your investigation, not a ready-made research question.
- What different meanings might “a good level” have, and which are measurable rather than matters of interpretation?
- Where might structural tests, simulated play, learned models or human assessments disagree? How might that be resolved?
- How could an automated assessment be communicated so it supports rather than replaces designer judgement?
What might you produce?
A level-evaluation prototype, such as an analysis tool, reusable library or editor integration, that accepts levels and produces interpretable assessments or diagnostic visualisations, supported by an investigation of the limits of its judgements.
Possible approach
Choose a manageable level format and explore the literature on level evaluation before selecting assessment criteria. Prototype one or two assessment techniques, establish a credible source of reference evidence and examine what the critic gets wrong as well as what it gets right.
Investigating the result
Compare the critic's judgements with relevant independent observations or references, examine disagreement and assess whether its outputs would inform a real design decision; a convincing diagnostic visualisation is not itself proof the underlying assessment is accurate. If recruiting people for ratings or interviews, secure the necessary ethics approval.
Core concepts
- Procedural content generation
- Structural and experiential properties
- Computational evaluation
- Simulation-based assessment
- Construct validity and operationalisation
- Predictive modelling where justified
Indicative technologies
Choose tools that suit the research. An engine is not required if a simulation can answer the question.
- Python with an appropriate level representation
- Pathfinding or graph-analysis libraries, if relevant
- scikit-learn or PyTorch where a learned critic is justified
- A game engine or editor integration where interaction matters
Keeping the scope manageable
Investigate a single game or level type and a defensible subset of its qualities; even one property, examined rigorously, is a real contribution. Trying to define game quality in general, or build an autonomous designer, would dilute the investigation rather than strengthen it.
Explore the literature
Search the library using relevant terms such as:
- procedural content generation evaluation
- computational game level evaluation
- level quality metrics
- playability and level design
- expressive range and controllability
- player experience and level assessment
Papers and reading
- Smith, G. and Whitehead, J. (2010) Analyzing the Expressive Range of a Level Generator. Proceedings of the 2010 Workshop on Procedural Content Generation in Games (PCGames '10), FDG. ACM Digital Library.
- Withington, O. and Tokarchuk, L. (2023) The Right Variety: Improving Expressive Range Analysis with Metric Selection Methods. Proceedings of the 18th International Conference on the Foundations of Digital Games (FDG). ACM Digital Library (also arXiv).
Recommended reading:
- Shaker, N., Togelius, J. and Nelson, M. J. (2016) Procedural Content Generation in Games. Springer.