Level Up Your Assessments: Practical Applications of Differential Item Functioning Analysis

I.C.E. Exchange Program Description: As part of the Diversity, Equity, and Inclusion initiatives, many credentialing organizations started collecting demographic information about their candidates. This creates the opportunity to conduct differential item functioning (DIF) analysis to identify items where one group has a different chance of answering correctly compared to another group, despite having similar underlying skill levels. This panel will present two organizations’ experiences of utilizing DIF to improve their assessments and two studies that investigate the impact of focal group selection and item lexical characteristics.

A medical certifying board incorporated DIF analysis into its regular key validation process for each exam administration while a national psychological licensing examination board did an ad hoc analysis and review of its active item bank. The details of DIF methodologies utilized (e.g., statistical methods and selection of reference and focal groups), percentage of items flagged, composition of the reviewing committee, reviewing process and results from both organizations will be shared.

Asian is commonly treated as a single category when conducting DIF. However, substantial cultural differences within the Asian group could potentially impact DIF results. A comparative analysis from a 3rd certification board showed a notable difference, with approximately 7 times more items being flagged when Asian subgroups were analyzed independently compared to Asian as one group. The subgroup DIF patterns (Chinese, Indian) will be compared with the overall Asian DIF pattern.

Thank you Excelsoft for Sponsoring our Test Development and Administration Track.

DIF typically serves as a screening tool to flag items that could potentially be biased, and subject matter experts have been heavily relied on to make their judgment of whether bias exists and items should be removed from scoring or the item bank. The 4th presentation leverages Machine Learning and Natural Language Understanding models to understand how item lexical characteristics could be associated with the probability of showing DIF. Implications for item writing will be discussed.