Learning from Party Splits - Validating Methods of Automated Text Analysis in Intra-Party Settings

Automated text analysis has become a popular tool for assigning policy positions to political actors. As a growing literature indicates, these techniques offer particularly promising insights for the study of intra-party preferences and party cohesion. This paper makes a first attempt to systematically evaluate the performance of the automated Wordscores and Wordfish techniques in purely intra-party settings. Following a case study approach, the Paper validates Wordscores and Wordfish preference estimates derived from party congress speeches (n=342) by applying a true triangulation of methods. The paper studies five party congresses (SPÖ 1946, 1947, 1948; FPÖ 1990, 1992) preceding two cases of party splits in Austria: the SPÖ/SAP split in 1948 and the formation of the Liberales Forum (LIF), which splintered away from the FPÖ in 1993. Using information on the later party splits (candidate lists of the splinter parties and other materials), the individual adherents of the splinter-groups are a priori identified for the party congresses studied. The analysis then evaluates whether Wordscores and/or Wordfish are capable of measuring the preference-splits between party wings prior to the actual party split. In a second step the automated preference estimates are compared to estimates obtained by manual coding of the same speeches. By combining these two sources of external validation – one derived from the subsequent behavior of intra-party actors, one based on the same textual data as the automated estimates – we analyze the validity of Wordscores and Wordfish in great detail and hence draw more robust conclusions about the scope of the techniques’ applicability to intra-party settings.