New off-the-shelf methods: Assessing pre-trained transformers for negative campaigning detection

Negative campaigning has received considerable academic interest (Haselmayer 2019). With candidates and parties increasingly adopting social media platforms to disseminate their messages, there is a wealth of digital communication data available for researchers. Since manual content analyses are time-consuming and labour-intensive, automated methods become increasingly important. Off-the-shelf dictionaries, which have been available for some time, are easy to use, but their quality has been widely criticized (e.g., Boukes et al. 2020). Today, a new class of off-the-shelf methods promises fast and easy annotations: Language models that have been adapted for specific tasks (most often based on BERT (Devlin et al. 2019)) and easy-to-use software implementations (e.g., the “text” package for R (Kjell and Schwartz n.d.)) make the benefits of modern machine-learning techniques more accessible. Custom fine-tuning of pre-trained language models is a suitable method to measure negative campaigning (e.g., Stromer-Galley et al. 2021). However, fine-tuning requires a fair amount of manual labour to generate training data. Therefore, we focus on modern off-the-shelf approaches that avoid the cumbersome manual annotation process. Specifically, we test the performance of a BERT model fine-tuned for sentiment classification (Guhr et al. 2020) and a BERT-based natural language inference (NLI) model (Laurer et al. 2022) for detecting negative campaigning in social media messages. The latter is able to predict classes that were not present during training (zero-shot classification). In a first step, we create a simple baseline model that only considers whether a political competitor is mentioned by matching strings (M0). Afterwards, we expand our baseline model by adding the results from a dictionary-based sentiment analysis (M2) and comparing them to an approach that features the BERT-based sentiment analysis instead (M3). These models are then compared to the performance of the zero-shot classifier (M4). Using 2,459 hand-coded German political Facebook posts published during the 2019 EP election campaign (Baranowski et al. 2022), our preliminary results show that these methods do not unconditionally fulfil their promise. The simple baseline model performs similarly well overall (F1-Score). Combining the baseline with sentiment analyses achieved higher precision at the cost of a lower recall. Therefore, this approach can be useful to reduce false positives and detect only messages which truly contain negative campaigning. In this respect, the BERT-based sentiment analysis slightly outperformed the dictionary. The BERT-NLI model showed promising results overall (F1-Score) but still could not beat out the simple baseline model. We conclude that there is still no free lunch in the automated detection of negative campaigning.