Publications & Resources

Modeling Conditional Probabilities in Complex Educational Assessments

Nov 2002

Robert J. Mislevy, Russell Almond, Lou Dibello, Frank Jenkins, Linda Steinberg, Duanli Yan, and Deniz Senturk

An active area in psychometric research is coordinated task design and statistical analysis built around cognitive models. Compared with classical test theory and item response theory, there is often less information from observed data about the measurement-model parameters. On the other hand, there is more information from the grounding psychological theory, and the task designer’s insights into which patterns of skills lead to which patterns of performance. We describe a Bayesian approach to modeling these situations, which uses experts’ judgments to produce prior distributions for the conditional probabilities in a multivariate latent-variable model, and MCMC estimation to refine the estimates. Task-design schemas and expert judgments are used in the first phase to structure the conditional probability table—that is, conjunctive, compensatory, or disjunctive models, or combinations thereof. Machinery from gradedresponse IRT is used to translate experts’ judgments about task requirements into prior distributions for model parameters, which in turn imply values for all the conditional probabilities. Bayesian estimation methods are then used to update the distributions for the model parameters in response to observed data. The approach is illustrated with examples from the Biomass biology assessment prototype.

Mislevy, R. J., Almond, R., Dibello, L., Jenkins, F., Steinberg, L., Yan, D., & Senturk, D. (2002). Modeling conditional probabilities in complex educational assessments (CSE Report 580). Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).