November 27, 2024
Integration of metrics into MyNavy Learning training effectiveness and adaptive learning: Considerations for implementation and validity
Authors:
Alan D. Koenig, John J. Lee, Eva L. Baker, Joanne K. Michiuye, and Harold F. O’Neil
This report examines strategies to enhance the effectiveness of the U.S. Navy’s MyNavy Learning (MNL) adaptive learning training approaches. It analyzes how to identify performance gaps through proper assessment design and data interpretation, emphasizing the relationship between cognitive demands and task performance. This report outlines a framework that begins with establishing clear assessment objectives and target audiences, followed by developing measures that map to specific competency constructs. This mapping enables precise identification of areas requiring remediation while eliminating unnecessary training topics, thereby optimizing the MNL adaptive learning system’s recommender engine. The report addresses validity as a property of assessments that varies based on intended use and outcome measures. It emphasizes that assessment validity depends on comprehensive measurement of relevant competency constructs, informed by subject matter experts and authoritative sources. These considerations form recommended best practices for enhancing MNL’s adaptive learning system and implementing performance assessments in operational settings.
Koenig, A. D., Lee, J. J., Baker, E. L., Michiuye, J. K., & O’Neil, H. F. (2024). Integration of metrics into MyNavy Learning training effectiveness and adaptive learning: Considerations for implementation and validity (CRESST Report 874). UCLA/CRESST.