Alternative Approaches to Scoring: The Effects of Using Different Scoring Methods on the Validity of Scores From a Performance Asessment

The purpose of this study was to investigate the degree to which modifications to the format of the scoring rubric and the associated training procedures affect the technical quality of the resulting scores and the perceived utility of each scoring method for positively influencing a teacher’s instructional decisions. Two different methods were used to score responses to six middle-school science performance tasks. Although both types of scoring utilized in this study could be characterized as focused holistic, the format and training associated with how the scoring criteria are presented and utilized by raters were modified to create two different methods&emdash;Focused Holistic (F-H) and Analytic Impression (A-1). Evidence from the study suggests that the F-H and A-1 methods are not equally preferable for making decisions about individual students or groups of students. However, the raters were overwhelmingly in favor of the A-1 method for obtaining useful information to improve instruction.