Publications & Resources

On the Role of Multilevel Item Response Models in Multisite Evaluation Studies for Serious Games

Apr 2016

Li Cai, Kilchan Choi, and Megan Kuhfeld

In this book chapter, we propose a generalizable solution to measurement error and multilevel modeling issues in multisite randomized controlled trials (RCTs). We argue that traditional use of summed scores or conventional IRT-based scaled scores are not optimal as outcomes. We propose a multilevel two-tier (MTT) item factor model for scaling item responses in multisite RCTs with pre- and post-tests. The MTT model is demonstrated with data from a multisite RCT assessing the instructional effects of video games on pre-algebra math topics. The new outcome measure derived from MTT analysis shows more sensitivity to intervention than standard measurement approaches.

Cai, L., Choi, K., & Kuhfeld, M. (2016). On the role of multilevel item response models in multisite evaluation studies for serious games. In H. F. O’Neil, E. L. Baker, & R. S. Perez (Eds.), Using games and simulations for teaching and assessment: Key issues (pp. 280-301). New York, NY: Routledge.