Who Does It Better? A Study of AI Versus Human Categorization of Open-Ended Survey Responses

I.C.E. Exchange Program Description : While there is a flurry of activity in finding applications of AI models for different tasks, consideration should be given when applying these groundbreaking programs. As AI art generators continually demonstrate, using AI to translate a user's description into an image can lead to artifacts, oddities, and weirdly drawn hands. When compared to human generated art, it can be lacking in substance and nuance. In a similar vein, is AI able to translate survey comments in such a way that is categorizable and actionable when compared to human efforts? Join as we investigate this question by utilizing AI for the parsing of open-ended survey responses into different categories for processing and review. Using AI to streamline this process would save time and make follow-up on survey feedback easier for users. Before committing to such an action, a pilot study of parsing survey comment data using AI into distinct categories was conducted. We share our results of the accuracy of the AI categorization project compared to humans completing the same task across two separate post-examination surveys. The study results will help inform others into any potential trade-offs with using AI for similar categorization tasks.

Thank you PSI for Sponsoring our Credentialing Innovations Track.