1,000 Speeches vs. GPT-4o: Analyzing Position Shifting in the European Parliament

The main goal of this paper is to make a methodological contribution to the rich and growing text-as-data literature in comparative legislative research by developing and comparing a large language model (LLM) approach to prior work that relied on manual coding, automated analyses of heavily preprocessed chunks of words, as well as research using roll call vote data. To demonstrate and validate our approach, we employ OpenAI’s Generative Pre-trained Transformer 4 Omni (GPT-4o)* and a transparent, repeatable multi-step process to code entire speeches of over 200 Members of the European Parliament (MEPs) in European parliamentary debates across two election cycles and highlight the advantages, similarities, and differences between our own approach, human coding, and Wordfish scores. We also address issues of inter-coder reliability, present alternative measures of uncertainty in the coding process, and explore the implications of prompt engineering for research into political speech that turns from content to sentiment analysis and the analysis of audience perception, reception, and engagement. Substantively, our MEP position data allows us to not only study variation between MEPs or their European political groups, but within MEP variation across speeches and over time. We use it to initially replicate work that treated MEPs’ position as fixed and focused on the traditional left-right dimension before investigating the timing of when exactly MEPs portray themselves as either concerned about the security implications of migration or asylum seekers’ human rights. Specifically, we test two competing hypotheses on why and when MEPs adjust their statements on refugees in the context of their national, intra-party nominations and the elections for the European Parliament. (* Disclosure: This project has been supported by OpenAI’s Researcher Access Program.)