Toward the Instructional Utility of Large-Scale Writing Assessment: Validation of a New Narrative Rubric

If performance assessments are to be used for making important decisions about children, then the rubrics used to score the assessments must be technically sound. In “Toward the Instructional Utility of Large-Scale Writing Assessment: Validation of a New Narrative Rubric,” CRESST researchers examined the technical qualities of a new scoring rubric for student writing. “The purpose of our study,” wrote the authors, “was to gather evidence of validity for the Writing What You Read (WWYR) rubric, through technical comparisons with an established narrative rubric that has consistently demonstrated sound technical capabilities in large-scale assessments of elementary level writing.” Designed for classroom use, the WWYR rubric contains scales for theme, character, setting, plot, and communication, and a sixth holistic scale for overall effectiveness. Teachers use of the new rubric has led to improved teacher understandings of student writing. The results of the study point to the difficult nature of creating valid and reliable scoring rubrics for performance assessments. “Our findings,” wrote the authors, “regarding the reliability and validity… yielded promising but mixed evidence of the utility of the Writing What You Read rubric for large-scale assessment.” “The results of a generalizability study,” said researcher John Novak, “suggests that three of the six scales could be used reliably if each narrative is scored by two raters.” He added that achieving technically acceptable results for the remaining three scales would require as many as four raters. The authors recommend further research into the structure and content of writing rubrics, the types of materials rated, and the methods of rater training.