New and Improved Methods for Identifying Election Pledges

A growing body of research analyzes the pledges made during election campaigns, and under what conditions parties keep those promises. Previously, election pledges have been extracted manually by expert researchers who interpret qualitative raw sources. In this paper, I provide a novel approach to coding campaign promises from political text in a transparent and reproducible way. Building on recent advancements in crowd-sourced coding and semi-supervised classification (Benoit et al. 2016; Manning et al. 2008), the method introduces quantitative text analysis and machine learning to the research area of pledge fulfillment. A Naïve Bayes classifier, trained with expert codings of party manifestos, extracts potential pledge content from various text sources. The pledge selection is validated by comparing the automated classification to a reliability test among pledge scholars and the aggregated coding by instructed non-experts. After the validation I show that both experts and crowd-workers often disagree on the coding of manifesto statements. The computational preselection of potential pledges can tackle this problem by simplifying the collection of more than one coding per sentence of the filtered text, either through manual reading or online crowdsourcing. With multiple codings researchers can derive estimates of uncertainty at the level of sentences. As a result, the semi-supervised screening could improve the transparency and reduce the costs and time associated with coding election pledges. Moreover, I show how the automated text classification can extract election promises from previously neglected sources, such as leaders’ debates, media reports, and large text corpora of parliamentary speeches. Finally, I utilize the new method to assess whether political parties across established democracies have generally increased the number of election promises they make over time.