Publications & Resources

Assessing Student Representations of Inferential Statistics Problems

Dec 2001

Nancy C. Lavigne and Robert Glaser

An active area in psychometric research is coordinated task design and statistical analysis built around cognitive models. Compared with classical test theory and item response theory, there is often less information from observed data about the measurement-model parameters. On the other hand, there is more information from the grounding psychological theory, and the task designer’s insights into which patterns of skills lead to which patterns of performance. We describe a Bayesian approach to modeling these situations, which uses experts’ judgments to produce prior distributions for the conditional probabilities in a multivariate latent-variable model, and MCMC estimation to refine the estimates. Task-design schemas and expert judgments are used in the first phase to structure the conditional probability table—that is, conjunctive, compensatory, or disjunctive models, or combinations thereof. Machinery from graded response IRT is used to translate experts’ judgments about task requirements into prior distributions for model parameters, which in turn imply values for all the conditional probabilities. Bayesian estimation methods are then used to update the distributions for the model parameters in response to observed data. The approach is illustrated with examples from the Biomass biology assessment prototype.

Lavigne, N. C., & Glaser, R. (2001). Assessing student representations of inferential statistics problems (CSE Report 553). Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).