Major Blueprint Changes: How One Organization and Vendor Gathered and Used Psychometric Evidence to Support and Validate the Change

I.C.E. Exchange Program Description: In 2024, the National Registry of EMTs launched a new examination blueprint for the certification programs for Paramedics and Advanced Emergency Medical Technicians. This examination represented a significant departure from the current blueprint, which consisted of five domains measured through an objective multiple-choice examination combined with rater-scored skills-based examination. The new examination replaced the in-person skills-based component with an objectively scored scenario-based component measuring a new domain of clinical judgment.

In support of this change, staff of the National Registry and their psychometric partner, Pearson VUE, developed and implemented a comprehensive sequence of analyses to support the validity of the new examination and assist with psychometric and programmatic decisions for the new examination. Significant thought and collaboration resulted in a carefully planned approach for analyses conducted within a short timeline. This presentation will describe the planning, data collection, and analyses conducted. These analyses included evaluation of dimensionality, item dependency, scoring models, as well as traditional item and timing analyses.

This session will also describe how the organization used the results to communicate to stakeholders about the value of the new blueprint as well as ongoing considerations for the collection of additional validity evidence and other analyses to support the examination.

Thank you Excelsoft for Sponsoring our Test Development and Administration Track.