A Conceptual Framework for Assessing Performance in Games and Simulations

The military’s need for high-fidelity games and simulations is substantial, as these environments can be valuable for demonstration of essential knowledge, skills, and abilities required in complex tasks. However assessing performance in these settings can be difficult—particularly in non-linear simulations where more than one pathway to success or failure may exist. The challenge lies not in capturing the raw data arising from game-play, but in interpreting what a player’s actions and decisions mean in the broader context of cognitive readiness for a particular job function or task.

The aim of our current research is to develop a conceptual framework for assessing complex behaviors in non-linear, 3-D computer-based simulation environments. Central to this framework is the incorporation of both a domain ontology (which depicts the key constructs and relationships that comprise the domain being simulated), and one or more Bayesian networks (which catalog the probabilities of various sequences of actions related to the constructs in the ontology). For the current research, the domain is damage control related to fire-fighting onboard naval ships, and the two key constructs being assessed are situation awareness and decision-making.

A 3-D, computer-based simulation depicting the interior of a naval ship has been developed. Assuming the role of a damage control investigator, the player is tasked with identifying, addressing, and reporting on a variety of potential, imminent, and existing fires and fire hazards. Using a dynamic Bayesian network, all actions and decisions related to situation awareness, communications, and decision-making are evaluated and recorded in real time, and are used for both formative and summative assessments of performance. Using this conceptual framework, our goal is to provide a generic model of assessment that can be incorporated into both new and pre-existing computer-based simulations that depict cognitively complex scenarios.