Specifying and Refining a Measurement Model for a Simulation-Based Assessment

The challenges of modeling students’ performance in simulation-based assessments include accounting for multiple aspects of knowledge and skill that arise in different situations and the conditional dependencies among multiple aspects of performance in a complex assessment. This paper describes a Bayesian approach to modeling and estimating cognitive models in such situations, in terms of both statistical machinery and actual instrument development. The method taps the knowledge of experts to provide initial estimates for the probabilistic relationships among the variables in a multivariate latent variable model and refines these estimates using Markov chain Monte Carlo (MCMC) procedures. This process is illustrated in the context of NetPASS, a complex simulation-based assessment in the domain of computer networking. We describe a parameterization of the relationships in NetPASS via an ordered polytomous item response model and detail the updating of the model with observed data via Bayesian statistical procedures ultimately being provided by Markov chain Monte Carlo estimation.