Introduction to Quantitative Text Analysis

Most social science concepts are not directly observable, so text analysis can provide a useful method for measuring quantities of interest that would otherwise be difficult to estimate. By analysing the speeches of legislators, for example, we can classify them as charismatic, populist, authoritarian, liberal, and so on. Similarly, by analysing the content of newspaper editorials, we can infer whether the media in question were biased in favour of a particular candidate during an election campaign.

Text analysis is a specific type of content analysis, typically defined as a method whose goal is to summarise a body of information in the form of text, in order to make inferences about the actor behind this body of information. This implies that text analysis can be seen as a data reduction method since its goal is to reduce the text material to more manageable bits of information.

Text analysis can be also seen as a method for descriptive inference. Weber (1990, p. 9) for instance, defines content analysis as ‘a method that uses a set of procedures to make valid inferences from text’. The idea is that, by analysing the textual output of an actor, we can infer something about this actor. This conceptualisation of content and text analysis implies that we can use it as a tool for measurement in the social sciences. In this view of content analysis we are concerned with replicability and objectivity, (Neuendorf 2002, pp. 10-15), and therefore we should distinguish text analysis from other approaches/methods such as discourse analysis, rhetorical analysis, constructivism, ethnography and so on.

This course will familiarise you with manual and computer-assisted text analysis. Following Krippenforff (2004) and Neuendorf (2002), it will introduce you to the basic concepts and building blocks in content analysis designs. We will address and discuss the following questions:

• Coding scheme What are the theoretical underpinnings of the coding scheme? How are the categories selected and operationalised? What are the coding units? How is coding performed? Is our coding scheme valid?
• Selection of documents What guides the selection of texts? Are texts sufficiently comparable? Are our documents valid and reliable indicators of the quantities of interest? How can we acquire and process text for computer-assisted text analysis?
• Aggregation Are texts coded by different coders? If so, how are their results aggregated? If not, how can we ensure inter-coder reliability? What statistical measures can be used to estimate inter-coder reliability?
• Scaling Are we estimating the quantities of interest directly? If not, how do we scale data in order to estimate the quantities of interest? Is our scaling valid and reliable?

For manual text analysis, the course will also look at the often-overlooked distinction between the analysis of manifest content and judgemental coding. For computer-assisted text analysis, it will introduce you to a variety of popular methods, such as the use of content analysis dictionaries (including sentiment analysis), scaling methods (wordscores, wordfish), and supervised and unsupervised learning approaches (including topic models). We will discuss the relationship between reliability and validity, illustrate methods for estimating inter-coder reliability, and explore the links between manual and computer-assisted text analysis in terms of validation and training of supervised classification methods.

The course will be taught via a mix of lectures, seminars and hands-on exercises. The examples used to illustrate the promises as well as the pitfalls of content analysis will be concerned with various applications across the social sciences (e.g. sentiment analysis of the press, frames analysis of social movements, estimating the positions of political actors, agenda-setting in the EU), while the majority of the exercises will involve following, step-by-step, code provided in the R statistical software, so previous knowledge of R is not necessary. In most of the seminars we will use R Studio.

Download R Studio

You will also have the opportunity to present your own project in class and receive constructive feedback.

You should be familiar with basic statistical concepts such as measures of central tendency (mean, median), dispersion (standard deviation), tests of association (Pearson’s r) and inference (χ², t-test).

These materials are covered in the first few chapters of introductory statistics or data analysis textbooks. A useful example is Pollock P.H. III, The Essentials of Political Analysis, fourth edition (Washington, DC: CQ Press, 2012), Chapters 2, 3, 6, and 7.

Some familiarity with R statistical software is also desirable but not necessary. In most of the classes we will use R Studio.