Real Data from the field: Examining Return on Investment Using AI-assisted TD Tools

I.C.E. Exchange Program Description: The potential for AI to transform assessment and certification is perhaps the single most debated topic in test development right now. So far, however, limited evidence has been shared about demonstrated efficacy and return on investment (ROI) from the field. As experts in assessment, we must get beyond the hype of AI and answer a central question: Can assessment and certification move forward with AI to optimize validity and reliability while simultaneously saving time and resources, and if so, how?

This interactive session will provide the opportunity for attendees to analyze actual operational data coming back from the field using AI-assisted test development tools in multiple disciplines with different certification programs. Of significant importance is that true ROI analytics must go well beyond claims of “saving SME time coming up with an initial idea”. Specifically, a weak draft item requires significant editing time, sometimes even more time than coming up with a high-quality draft initially. So, what is the reality of the performance and output of AI-enabled tools in assessment?

This session will take a deep dive into AI-enabled tools at different stages in test development (beyond initial idea generation) such as full item development times, use of the AI tool in editing, item review times, item acceptance rates, Job Task Analysis (JTA), mapping items to outcome and cognitive complexity frameworks, and analysis of features such as AI-enabled rationales.

The presenters will feature actual practitioners and experts from: Ascend Learning (representing different programs), BOC (“Board of Certification (for the Athletic Trainer)”, assessment design expert and psychometrician/AI tool-developer. The focus will be an up-close analysis of the technology, how it is implemented, and quantitative and qualitative metrics of performance from across the test development process along with lots of opportunity for discussion.

Thank you PSI for Sponsoring our Credentialing Innovations Track.