Determining How Well a Test Measures Your Objectives

This paper describes a procedure for in depth analysis of a limited number of tests being considered for selection by a school, district, project, or state personnel. This procedure involves listing the objectives that it would be desirable to measure, determining the relative importance of each of these objectives, having “judges” match test items to these objectives, and then correlating the relative importance of each objective with the extent to which it is covered by a test. Variations in the procedures are presented as well as adjustments for differences in average item difficulties across clusters of items and for differences in test length. There also is a discussion of how well a cluster of items assigned to an objective actually covers that objective. Finally conditions under which the procedures described are and are not applicable are considered and appendices illustrating specific directions and procedures are provided.