Publications & Resources

A Cognitive Task Analysis, With Implications for Designing a Simulation-Based Performance Assessment

Aug 1998

Robert J. Mislevy, Linda S. Steinberg, F. Jay Breyer, Russell G. Almond, and Lynn Johnson

To function effectively as a learning environment, a simulation system must present learners with situations in which they use relevant knowledge, skills, and abilities. To function effectively as an assessment, such a system must additionally be able to evoke and interpret observable evidence about targeted knowledge in a manner that is principled, defensible, and fitting to the purpose at hand (e.g., licensure, achievement testing, coached practice). This article concerns an evidence-centered approach to designing a computer-based performance assessment of problem-solving. The application is a prototype licensure test, with supplementary feedback, for prospective use in the field of dental hygiene. We describe a cognitive task analysis designed to (a) tap the knowledge hygienists use when they assess patients, plan treatments, and monitor progress, and (b) elicit behaviors that manifest this knowledge. After summarizing the results of the analysis, we discuss implications for designing student models, evidentiary structures, task frameworks, and simulation capabilities required for the proposed assessment. Don Melnick (1996), who for many years directed the National Board of Medical Examiners’ computer-based patient management assessment project, recently remarked, “It is amazing to me how many complex “testing” simulation systems have been developed in the last decade, each without a scoring system” (p. 117). A foundation for sound assessment must be laid long before a simulator is complete and tasks are written. This article summarizes research designed to lay the foundation for a computer-based performance assessment of problem-solving. A quote from Messick (1992) captures the essence of the evidence-centered approach to assessment design that guides our work: [We] would begin by asking what complex of knowledge, skills, or other attributes should be assessed, presumably because they are tied to explicit or implicit objectives of instruction or are otherwise valued by society. Next, what behaviors or performances should reveal those constructs, and what tasks or situations should elicit those behaviors? Thus, the nature of the construct guides the selection or construction of relevant tasks as well as the rational development of construct-based scoring criteria and rubrics. (p. 17) We focus on the methods, results, and implications of a cognitive task analysis that addresses these evidentiary issues, to ground the design of a simulation-based assessment of problem-solving and decision-making in dental hygiene.

Mislevy, R. J., Steinberg, L. S., Breyer, F. J., Almond, R. G., & Johnson, L. (1998). A cognitive task analysis, with implications for designing a simulation-based performance assessment (CSE Report 487). Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).