Measurement Issues in the Alignment of Standards and Assessments: A Case Study

This study examined the impact of rater agreement on decisions concerning the alignment between the Golden State Examination (GSE) in High School Mathematics and the University of California (UC) Statement on Competencies in Mathematics. UC faculty and high school mathematics teachers (n = 20) rated the mathematics items of the GSE relative to the expectations identified in the UC competency statement, identifying item features related to content and dimensionality. Raters assigned values for a primary topic, a secondary topic, item/topic centrality, and depth of knowledge. Agreement within these criteria was the basis of the assessment of alignment. Results showed considerable variability in judgments across raters and different pictures of alignment depending on the particular subset of raters providing the ratings. Results also varied by rater type and the method of determining rater agreement.