Intuitive Test Theory

Psychologist Andrea diSessa coined the term “phenomenological primitives”, or p-prims, to talk about nonexperts’ reasoning about physical situations. P-prims are primitive in the sense that they stand without significant explanatory substructure or explanation. Examples are “Heavy objects fall faster than light objects” and “Continuing force is needed for continuing motion.” P-prims are based on experience. They are not a coherent system; they may even contradict one another. People assemble from them a model of sorts to reason about a given situation. Intuitive physics is wrong from a physicist’s point of view, but it works just fine to play fetch with your dog or push a couch across the room. It fails when you want to build a skyscraper or send a rocket to the moon. This paper considers p-prims that underlie reasoning about assessment, the basis of what one might call intuitive test theory. Examples are “A test measures what it says at the top of the page,” and “Scores from any two tests can be made interchangeable, with a little equating magic.” Testing p-prims underlie discussions of test theory in the classroom, in the news, and in policy-making. Again, intuitive test theory works reasonably well for everyday uses like Friday’s math quiz. It fails when you want to design an adaptive test, or measure the change in the proportion of students reading Above Basic from a matrix-sampled assessment such as NAEP.