Publications & Resources

Intuitive Test Theory

May 2004

Henry I. Braun and Robert J. Mislevy

Psychologist Andrea diSessa coined the term “phenomenological primitives”, or p-prims, to talk about nonexperts’ reasoning about physical situations. P-prims are primitive in the sense that they stand without significant explanatory substructure or explanation. Examples are “Heavy objects fall faster than light objects” and “Continuing force is needed for continuing motion.” P-prims are based on experience. They are not a coherent system; they may even contradict one another. People assemble from them a model of sorts to reason about a given situation. Intuitive physics is wrong from a physicist’s point of view, but it works just fine to play fetch with your dog or push a couch across the room. It fails when you want to build a skyscraper or send a rocket to the moon. This paper considers p-prims that underlie reasoning about assessment, the basis of what one might call intuitive test theory. Examples are “A test measures what it says at the top of the page,” and “Scores from any two tests can be made interchangeable, with a little equating magic.” Testing p-prims underlie discussions of test theory in the classroom, in the news, and in policy-making. Again, intuitive test theory works reasonably well for everyday uses like Friday’s math quiz. It fails when you want to design an adaptive test, or measure the change in the proportion of students reading Above Basic from a matrix-sampled assessment such as NAEP.

Braun, H. I., & Mislevy, R. J. (2004). Intuitive test theory (CSE Report 631). Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).