An Agenda for Comparative Research on Electoral Pledges: Prospects and Limitations of Human and Machine-Based Methods in Pledge Research

This paper sets out a research agenda for comparative research on election pledges, which identifies the lines of inquiry that we believe should be prioritized given both their relevance to democratic theory and the knowledge that has been generated to date. We also consider the relevant methods for pursuing these lines of inquiry, paying particular attention to the prospects and limitations of human and machine-based methods, and ways of integrating human and automated methods. Our review of the literature on election pledges summarizes the insights that this expanding field has generated into the making and breaking of pledges. The field has provided unique insights into the effects and non-effects of a range of political institutions and power-sharing arrangements on pledge fulfillment, including the distinctions between coalitions and single-party executives, and majority and minority governments. The field has provided fewer comparative insights into the wider contexts in which pledges are made and fulfilled. Notwithstanding forays into these areas, we have only limited understanding about how pledges are created: What actors are influential in pledge making? Why do parties make pledges in some areas but not in others? These are questions that can be addressed by tracking formulations of pledges in communication about politics preceding the writing of the manifesto. In addition, there is only limited knowledge on the conditions under which and ways in which election pledges reach voters in modern political communication. The research agenda provides a balanced and realistic assessment of the prospects and limitations of human and machine-based methods for opening these lines of inquiry. To date, research on election pledges has mainly used established qualitative methods based on human interpretation and coding of manifestos and relevant sources for assessing fulfillment. The research agenda acknowledges that some of these new lines of inquiry can be opened with recently developed methods for automated text analysis. Automated methods can analyze vast amounts of text, are cheap to use and more reliable than human coding. At the same time, we identify the reasons why machines will not be replacing human judgment in this field any time soon. In support of our argument, we assess recent attempts to test whether supervised and unsupervised machine learning methods can detect pledges in manifestos, to the same extent as human coders. We conclude that these attempts have largely failed, because machines cannot replace human judgment and contextual understanding, which is much needed in pledge coding. We identify other more realistic prospects for automated text analysis in a large-scale qualitative research. In general, these realistic prospects involve human judgment being used as inputs into machine processing and research questions directed to large amounts of text. For instance, once humans have identified specific election pledges, automated methods can be used to study how traditional and new media portray the pledges. The paper concludes by drawing the broader implications of this research agenda for mixed-methods for studying political communications in a range of different fields.