Making Semi-Structured Interviews Scalable: An Experimental Evaluation of AI Conversational Interviewing

Public opinion research has long faced a trade-off between depth and scale: while structured surveys facilitate large-scale data collection, they restrict respondents to predefined options, limiting the researcher's ability to uncover the diverse considerations underlying public sentiment. Conversely, conversational interviews yield rich, open-ended insights but are resource-intensive and difficult to scale. This study examines the potential of using large language models (LLMs) as interviewers to conduct conversational interviews at scale. AI Conversational Interviewing aims to illuminate complex networks of ideas, memories, and arguments that remain inaccessible to standardized surveys, yet at a sample size infeasible for human-led interviewing. We present evidence from 962 respondents (recruited via Prolific and Payback) who completed both an AI-moderated interview and a standardized survey on migration policies. The pre-registered study pursues three primary objectives: (1) assessing the practical viability of AI interviewing by analyzing participant experiences and evaluations; (2) experimentally comparing data quality and respondent preference across voice-based, chat-based, and mode-choice conditions; and (3) demonstrating the unique analytical value of conversational text data for addressing research questions beyond the reach of closed-ended items. Our findings indicate that AI Conversational Interviewing is a viable and valuable addition to the social science toolkit for capturing people’s opinions and is liked by respondents roughly as much as standardized surveys, with negligible differences in data quality between the experimental voice / text / choice conditions. By providing open data and open-source materials, this study contributes to the burgeoning literature on leveraging artificial intelligence to expand the horizons of public opinion measurement.