Generative AI, the Augmented SME and the Augmented Translator: large language model assessment development in multiple languages

I.C.E. Exchange Program Description: Can SME test developers worth their salt have the best of all worlds by being valued for their discernment while being spared tedious, repetitive tasks in the assessment development process?
Large language models have existed well before ChatGPT. The sudden global publicity of AI engines has created disproportionate expectations as well as distrust, misuse, and ethical concerns. This applies to item generation in monolingual settings as well as in multilingual settings. The presenters will discuss both.
Is the way forward a combination of a highly specialized model designed to generate assessment content with the SME as an indispensable resource to review, assemble, edit, and finalize items?
If this "augmented SME" model is workable, can it be expanded to function in multilingual contexts?
This panel session will take a broad view of the risks involved in using generative AI to produce assessments (hallucination, recursion, lack of validity and reliability, bias replication and exacerbation).
It will examine the feasibility of treating language models as a productivity tool that should remain inseparable from the psychometric expertise of subject matter experts. Cross-pollination between mature translation technology, large language models and the gold standard for item translation and adaptation will also be discussed.

A linguist and a psychometrician will discuss the implications, the potential, and the limitations of
- building a language model designed in English but trained with corpora in additional languages to generate output in those languages;
- selecting human experts to review such output; and
- translating the output of generative AI, or not.

Insights will be in the form of perspectives, roadmaps, and caveats.

Thank you Excelsoft for Sponsoring our Test Development and Administration Track.