Publications & Resources

Problems in Stabilizing the Judgment Process

Dec 1980

Edys Quellmalz

This article analyzes a series of measurement problems that jeopardize the reliability and validity of competency-based writing assessments. The paper distinguishes between two indicators of rating variability: (1) rater drift—rater’s progressive deviation within a scoring session from previously shared criteria; and (2) scale instability—differential application of criteria by raters in different scoring sessions. Examples from research illustrate the nature and magnitude of rating fluctuations. Promising techniques are described for stabilizing raters’ judgments and documenting scale stability.

Quellmalz, E. (1980). Problems in stabilizing the judgment process (CSE Report 136). Los Angeles: University of California, Los Angeles, Center for the Study of Evaluation.