December 21, 2018

Using Feature Analysis to Examine Career Readiness in High School Assessments

Authors:
Jenny C. Kao, Kilchan Choi, Nichole M. Rivera, Ayesha Madni, and Li Cai
This report is the second in a series considering career-readiness features within high school assessments. Experts in English language arts and math were trained to rate a selection of active Grade 8 and Grade 11 Smarter Balanced items using feature set lists that were refined within the items’ respective content areas (30 features for ELA items; 22 features for math items). A total of 264 ELA items and 186 math items were rated. ELA items contained between three and 13 career-readiness features, with an average of 6.0. The most frequent features were importance of being exact or accurate, written comprehension, time sharing, deductive reasoning, and reading comprehension. Math items contained between two and 15 career-readiness features, with an average of 7.8. The most frequent features were deductive reasoning, analyzing data or information, reading comprehension, number facility, and processing information. Feature ratings of the target items were analyzed with item metadata difficulty parameters in order to explore relationships between features and item difficulty. A number of career-readiness features showed associations with item difficulty, notably, reading comprehension for Grade 8 math and deductive reasoning for Grade 11 ELA. Because career-readiness features can be used to explain item difficulty, results suggest that such features are prevalent in content-based assessments, and inferences for career readiness can thus be drawn from test performance.
Kao, J. C., Choi, K., Rivera, N. M., Madni, A. & Cai, L. (2018). Using feature analysis to examine career readiness in high school assessments (CRESST Report 858). Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).