Cross-State Comparability of Judgments of Student Writing: Results From the New Standards Project

Ten states participated in a cross-state scoring workshop during the summer of 1991. Writing products from three separate grade levels, elementary, middle school, and high school, were used in the workshop and states participated at one or more of these grade levels. Participating states provided a sample set of student writing produced in response to a single prompt or writing assignment as part of the state’s regular operational writing assessment program. Three cross-state scoring sessions were held during the course of the 4-day workshop. At each scoring session, readers from a given state received a set of papers from another state that were then read and scored using the standard procedures of the scoring state. At the completion of the workshop the papers from each providing state had been scored independently by readers from three other states. Correlations of the scores assigned by readers from one state with those assigned by readers from another state were generally quite high. There were some large differences, however, in the level of scores that were assigned to papers even in instances in which the different states’ relative ordering of papers was almost as similar as the within-state rating reliabilities would allow. Implications of these results for defining common standards across states are discussed.