Publications & Resources

A Distance Learning Testbed

Jul 2006

William L. Bewley, Gregory K. W. K. Chung, Jin-Ok Kim, John J. Lee, and Farzad Saadat

Because of the great promise of distance learning for delivering cost-effective instruction, there is great interest in determining whether or not it actually is effective, and—more interesting—determining what variables of design and implementation make it more or less effective. Unfortunately, much of the research has been based on simple comparisons of distance learning to the “traditional” method of instruction rather than examining the variables influencing the effectiveness of distance learning. In addition to not manipulating or controlling important independent variables, the dependent measures used in such studies are often inappropriate, ranging from the obviously inadequate (e.g., the “smile test”) to standardized tests that have known psychometric properties but are not aligned with course objectives, to homegrown measures that appear to be aligned with instructional objectives but are of unknown reliability and validity. We have addressed the problem of limitations in dependent measures with research on measures of student achievement based on families of cognitive demands, and have developed assessment models for these families that can be used to design assessments across a variety of subject matters. We have also developed computer-based assessment tools implementing the models, including tools for data collection, scoring, analysis and reporting, assessment authoring, and knowledge acquisition and representation. With support from the Office of Naval Research we have developed a distance learning testbed to apply these models and tools to distance learning research and evaluation. This paper describes our summer of 2004 testbed implementation and presents three examples of the research conducted in the testbed on methods for assessing human performance via distance learning technologies.

Bewley, W. L., Chung, G. K. W. K., Kim, J.-O., Lee, J. J., & Saadat, F. (2006). A distance learning testbed (CSE Report 695). Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).