Benchmarking Text Understanding Systems to Human Performance: An Exploration

Benchmarking in the context of this report means comparing the performance of intelligent computer systems to the performance of humans on the same task. Computer responses to questions based on specific reading texts are referenced back to human responses to the same questions about the same texts. The results of this report support the belief that we can compare system performance to human performance in a meaningful way using performance-based measures. This study provides direction for researchers who are interested in a methodology for assessing intelligent computer systems.