Drawing Sound Inferences Concerning the Effects of Treatment on Dispersion in Outcomes: Bringing to Light Individual Differences in Response to Treatment

Individual differences in response to a given treatment have been a longstanding interest in education. While many evaluation studies focus on average treatment effects (i.e., the effects of treatments on the levels of outcomes of interest), this paper additionally considers estimating the effects of treatments on the dispersion in outcomes. Differences in dispersion can, under certain circumstances, signal individual differences in response to a given treatment, thereby helping us identify factors that magnify or dampen the effects of treatments that might otherwise go unnoticed. Much of this paper focuses on quasi-experiments in nested settings, which are commonly encountered in multi-site evaluation studies. In such settings, studying differences in dispersion as well as in means (e.g., differences in levels of outcomes for treatment and control group students) entails jointly modeling mean and dispersion structures in a hierarchical modeling (HM) framework. This paper shows how a well-elaborated dispersion structure based on substantive theories mitigate the problem of confounding by cluster characteristics, while a well-elaborated mean structure helps avoid confounding by individual characteristics, with regard to inferences concerning dispersion. We illustrate these ideas with analyses of the data from a study of the effectiveness of two innovative instructional programs relative to traditional instruction in elementary mathematics classrooms. We employ a fully Bayesian approach and discuss its advantages in modeling dispersion. We further discuss possible extensions of the methodology to other evaluation settings, including longitudinal evaluation settings.