On the Cognitive Validity of Interpretations of Scores From Alternative Concept Mapping Techniques

The emergence of alternative forms of achievement assessments and the corresponding claims that they measure “higher order thinking” have significantly increased the need to examine their cognitive validity (Glaser & Baxter, 1997; Linn, Baker, & Dunbar, 1991). This study evaluates the validity of connected understanding interpretation of three mapping techniques. The study focused on the correspondence between mapping-intended task demands, inferred cognitive activities, and scores obtained. We analyzed subjects’ concurrent and retrospective verbalizations at different levels of competency performing the mapping tasks and compared the directedness of the mapping tasks, the characteristics of verbalization, and the scores obtained across techniques. Our results led to the following general conclusions: (a) Consistent with a previous study, we found that the three mapping techniques provided different pictures of students’ knowledge. (b) Inferred cognitive activities across assessment tasks were different and corresponded to the directedness of the assessment task. The low-directed technique seemed to provide students with more opportunities to reflect their actual conceptual understanding than the high-directed techniques.