Media discourse plays a key role in shaping public opinion about migration, migrants and migration policies (Boomgaarden & Vliegenthart, 2009). Previous research shows that media-mediated information regarding migration and migrants and its negative or positive tone has at least as much influence on attitudes towards migration, migrants and migration policies as the actual numbers of migrants in each country (Eberl et al., 2018). Nevertheless, to date, there is no research focusing on the influence of media discourse on attitudes towards migration, migrants and migration policies in Slovakia. There are also as of yet no analytical tools in the Slovak language that would enable the exploration of media discourse based on a robust analysis of large amounts of media content. One of the most commonly used analytical techniques is supervised machine learning (Barberá et al., 2021). A key aspect of content analysis of text using supervised machine learning is the conceptually and theoretically sensitive validation of the automated content analysis of large-scale text data, which complements the computational part of text analysis and is provided by qualified experts who understand the topic under investigation (Song et al., 2020). Such validation by experts enables the interpretation and contextualisation of complex, socially constructed and conceptually volatile media content and sentiment/evaluation/stance that is characteristic of political and social science research (Nelson et al., 2021). In this process, a statistical model is trained on a certain amount of annotated data (e.g., paragraphs of text with certain content categories/topics). The learning algorithm adjusts the parameters of the statistical model so that it is able to generalize, i.e., consistently assign content categories to a large number of unannotated text units. In this paper we discuss the challenges and opportunities related to expert annotation validation of automated stance detection in a large corpus (960.000) of Slovak media outputs published between 2003 and 2024 by using a recently developed language model for Slovak - SlovakBERT (Pikuliak et al., 2021) to be trained on tasks specific to our project, e.g. stance analysis towards migration. Specifically, we focus on A) resolving the discrepancies between various conceptualisations and operationalizations of sentiment, evaluation, and stance and their impact on performance of classification tasks by human and computational annotators (Bestvater & Monroe, 2022; Overbeck et al., 2023); B) dealing with multiplicity of stance/evaluation targets that introduce additional variation to stance classifications (Baden & Tenenboim-Weinblatt, 2018); C) selecting the appropriate unit of analysis (sentence, paragraph, media output), e.g. separately analyzing headlines and article bodies (van Atteveldt et al., 2021); D) recognising annotator/coder bias as noise or systemic variation worthy of further examination (Sap et al., 2022). Our paper contributes to the methodological knowledge on conducting automated analysis of textual content focused on migration and related topics, especially in lower resourced languages.
