Test Development and Psychometrics

Does It Make a Difference? A psychometric comparison between an in-person oral exam and a remote oral exam

Monday, November 18, 2024

11:30 AM – 11:40 AM East Coast USA Time

Location: Innovation Hall, Sparkle Ballroom - E-Poster Gallery

Main Contact/Presenter(s)

LM

Lidia Martinez

Senior Psychometrician
American Osteopathic Association, United States

Presenter(s)

TM

Tara McNaughton, PhD

AVP, Psychometrics & Assessments
American Osteopathic Association, Illinois, United States
Nicholas Williams, MEd

Psychometrician
American Osteopathic Association
Chicago, Illinois, United States
SP

Sarah Pirani

Psychometrician I
American Osteopathic Association, United States

I.C.E. Exchange Program Description: The COVID-19 pandemic forced many test constructors to adapt their in-person oral exams to an online remote platform. Is there a difference in measurement when examiners score test takers face-to-face versus online? We explore measurement error and rater effects with generalizability theory and differential rater functioning analysis on a remotely administered oral exam and an in-person oral exam for a surgery medical certification.

*PTC Members may find value in this session

Thank you Excelsoft for Sponsoring our Test Development and Administration Track.

Learning Objectives:

Understand each source of variance in the two modes of administration via generalizability theory
Understand systematic interactions between rater scoring styles (leniency/severity) and assessment conditions (in-person vs remote)
Understand the psychometric advantages and disadvantages of both modes of administration