Smarter Balanced Assessment Consortium: Mathematical Reasoning Project Quantitative Analyses Results: Grades 4, 8, and 11

In the context of the current report, feature analysis is defined as the qualitative rating of tasks against a set of attributes, followed by a subsequent quantitative analysis to determine how these attributes determine task performance. The four main components of the feature analysis process include feature rating, step-by-step analysis, cognitive labs, and quantitative analysis. For the feature analysis for the Mathematical Reasoning Project in this current report, we focus on the five following questions, with particular emphasis on Questions 3, 4, and 5:(1) What particular attributes/features does each item contain? (2) What are the dominant attributes/features across items? (3) What particular attributes/features of items/tasks contribute to increased or reduced item difficulty across items/tests/tasks and why? (4) What feature combinations contribute to increased and reduced difficulty and why? (5) What features are most valid to elicit and assess key learning outcomes?
The current report combines item feature analysis across Grades 4, 8, and 11 with item response analysis across the same grades using a logistic linear test model (Fischer, 1973, 2005). The primary purpose of this analysis is to examine the relationship between item difficulty and identified item features on fourth, eighth, and eleventh grade items. This report will summarize and provide an analysis of these findings related to each respective grade. This report contains select descriptive statistics across feature categories and items per grade and item-based examples highlighting specific feature and item difficulty components. The purpose of the descriptive statistics across feature categories is to determine whether any patterns appear that can inform the item difficulty analysis.