Final Report for Validation of Problem-Solving Measures

In this report we summarize several studies regarding the measurement of problem solving and investigation of the impact of providing content information (Yes/No) and scoring criteria (Yes/No) for two knowledge mapping tasks proposed for use in an international assessment context (the International Life Skills Survey). Reliability and validity issues related to (a) the use of knowledge maps for measuring content understanding, (b) domain-specific problem-solving prompts for measuring participants’ problem-solving strategies, and (c) a self-regulation questionnaire for measuring levels of participants’ metacognition and motivation are presented within the larger context of measuring problem solving. Analyses showed that providing scoring instructions to participants did not significantly affect their performance on knowledge mapping tasks, including measures of content understanding, number of terms of used, and number of links constructed. For one of the two tasks, no gender effects were found for knowledge mapping measures. However, for the other task, there were significant effects found for two knowledge mapping measures, where males constructed significantly greater numbers of links and achieved significantly higher content understanding scores. Finally, providing content information resulted in higher performance scores. In general, there was sufficient reliability and validity to use these new measures at the cross-national level but not at the individual level.