Problems in Stabilizing the Judgment Process

This article analyzes a series of measurement problems that jeopardize the reliability and validity of competency-based writing assessments. The paper distinguishes between two indicators of rating variability: (1) rater drift—rater’s progressive deviation within a scoring session from previously shared criteria; and (2) scale instability—differential application of criteria by raters in different scoring sessions. Examples from research illustrate the nature and magnitude of rating fluctuations. Promising techniques are described for stabilizing raters’ judgments and documenting scale stability.