Publications & Resources

Using a Matched Sampling Methodology to Evaluate Program Effects: An Illustration From the University of California Outreach Programs

May 2003

Denise D. Quigley, Jorge Muñoz, and Alison Jacknowitz

Policymakers and educators are committed to increasing the competitive eligibility of high school students applying to the University of California and to increasing the representation of poor and underrepresented minority students on UC campuses. The University of California’s primary strategy to accomplish these goals is through School University Partnerships and academic development programs. Policymakers in conjunction with the University of California have invested substantial resources toward these common goals and programs. As a result, policymakers and educators want to know whether and to what extent the investment in School University Partnerships with supporting academic development programs is making a difference in the choices and lives of students. The University of California Office of the President (UCOP) has funded a number of research and evaluation efforts to investigate these questions.

This report is the first in a series that will evaluate the systemwide effects of School University Partnerships with supporting academic development programs on individual student-level achievement and student-level eligibility. The purpose of this report is to establish the feasibility of using a matched sampling methodology to investigate program effects and the methodology’s ability to establish the matched samples of elementary, middle, and high schools whose progress on key indicators can be tracked over time. The purpose of this report is not to evaluate the effects of the program. At this time, evaluating the effects of the program is premature. Moreover, the evaluation of program effects should be coupled with information about the implementation of the program and its individual components. This report establishes the methodology and the matched samples. It has three main sections. The first (Background) summarizes the history of the UC programs. The next section (Methodology) reviews the literature on matched sampling methods and outlines the strategy used for establishing the comparison groups. The following section (Illustrative Analyses) illustrates how differences in demographic characteristics and achievement outcomes can be assessed using the established comparison groups for the UC elementary, middle and high Partner schools.

Quigley, D. D., Muñoz, J., & Jacknowitz, A. (2003). Using a matched sampling methodology to evaluate program effects: An illustration from the University of California outreach programs (CSE Report 598). Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).