Publications & Resources

Alignment of Mathematics State-level Standards and Assessments: The Role of Reviewer Agreement

Jun 2006

Noreen Webb, Joan Herman and Norman Webb

In this report we explore the role of reviewer agreement in judgments about alignment between tests and standards. Specifically, we consider approaches to describing alignment that incorporate reviewer agreement information in different ways. The essential questions were whether and how taking into account reviewer agreement changes the picture of alignment between tests and standards. This study showed a wide range of reviewer agreement during the process of aligning standards and assessments, with substantial reviewer disagreement about important elements such as correspondence between objectives and items on the assessments. Taking this reviewer disagreement into account changed conclusions about alignment, not only showing weaker alignment than previously demonstrated, but also changing the profiles of alignment about, for example, relative coverage of specific standards. The results of this study point to the need for greater clarity in objectives and standards, more extensive reviewer training during the alignment process, and possibly also inspection of items to uncover characteristics that may lead to uncertainty among reviewers.

Webb, N., Herman, J., & Webb, N. (2006). Alignment of mathematics state-level standards and assessments: The role of reviewer agreement (CSE Report 685). Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).