The Interchangeability of Assessment Methods in Science

This article describes a study that investigated the interchangeability of four different assessment methods for measuring middle-school students’ understanding of science concepts. The four methods compared were hands-on tasks with associated multiple-choice and written justification items, written analogues of the hands-on tasks, and two types of multiple-choice items that were not related to hands-on or written analogue tasks. Some students took the hands-on test before the written test, and some students took the written test before the hands-on test. Observed and disattenuated correlations were examined. Multivariate generalizability analysis was used to obtain disattenuated correlations. The results indicate that hands-on and written analogue tests are not interchangeable, but multiple-choice and written justification items linked to hands-on and written analogues could be considered interchangeable if correlations between .76 and .96 are an acceptable criterion for interchangeability. In addition, a number of interesting order effects were found.