Publications & Resources

Estimating the Consistency and Accuracy of Classifications in a Standards-Referenced Assessment

Apr 1998

Michael James Young and Bokhee Yoon

An important feature of recent large-scale performance assessments has been the reporting of pupil and school performance in terms of performance or proficiency categories. When an assessment uses such ordered categories as the primary means of reporting results, the natural way of reporting on the quality of the assessment is through the probabilities of consistent and correct classification of students. This paper applied a method introduced by Livingston and Lewis (1995) for calculating those probabilities. The use of this procedure to extend Lord’s strong true score theory to tests containing other than multiple-choice items has created an important tool for test developers. The data used in the paper are from the New Standards Reference Examinations in Mathematics and English-Language Arts that were administered to students in Grades 4, 8, and 10 in spring 1996. The results of these analyses showed that for total score composites, the range of students accurately classified as having “met the standard” is from 85% to 98%, and the range of students consistently classified as having “met the standard” is from 77% to 96% across grades and content areas.

Young, M. J., & Yoon, B. (1998). Estimating the consistency and accuracy of classifications in a standardsreferenced assessment (CSE Report 475). Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).