November 20, 2024

Approaches for combining competencies and metrics as a single weighted metric

rainbow divider
Authors:
John J. Lee, Alan D. Koenig, Kilchan Choi, Michael Senteney, Eva L. Baker, Joanne K. Michiuye, and Harold F. O'Neil
This report discusses developing a single composite metric or score from various U.S. Navy training programs to allow for comparisons of training effectiveness across programs. It describes incorporating three levels of the Kirkpatrick model currently used to evaluate training: reaction, learning, and behavior. The goal is to create a composite score combining Levels 1, 2, and 3 that can serve as a key performance indicator for evaluation purposes. Several weighting methodologies are proposed. The document outlines various Navy data sources that could be included in the composite score, such as (Personnel Qualification Standard (PQS) time to qualify, Surface Training Readiness Management System (STRMS) assessment scores, advancement exams, and baseline questionnaires. Validity and reliability of the composite score are important considerations that expert weighting of the measures may help establish. An example approach is described using CRESST data to illustrate the qualitative and quantitative weighting methodologies.
Lee, J. J., Koenig, A. D., Choi, K., Senteney, M., Baker, E. L., Michiuye, J. K., & O’Neil, H. F. (2024). Approaches for combining competencies and metrics as a single weighted metric (CRESST Report 873). UCLA/CRESST.
rainbow divider