Measurement: Approaches Bridging Concepts and Data

If broadly conceived as a link between concepts and data, measurement is indispensable for all empirical social science. Sometimes, measurement is easy but most of the time it is not.

A study of local public health may count the number of health centers or the number of staff working in such centers in a city. But what is this a measure of? Is it a good measure?

A study of employee engagement among firefighters may roll out a standard nine-item engagement battery in a survey. But why so many items? Why those nine? And to what extent can we believe what the firefighters reply?

A study of constructive deviance among nurses may approach how they construct meaning of their interaction with clients and how they work around or counteract rules. But how do we elucidate such meaning from their everyday language? After all, few nurses will ever use the term constructive deviance. 

This course introduces measurement issues from variance-based, case-based, and interpretivist perspectives as well as common measurement problems for social science research.

The social sciences tend to deal with concepts – like public health capacity, employee engagement, and constructive deviance – that are not immediately visible. Moreover, the sources of information we have to measure these concepts are rarely direct and are often statements or stories from people, who may not be responding to what they are asked, may hide what they believe from researchers or themselves, and who do not speak the language of the social science academy.

We will examine in detail topics surrounding the often problematic business of social science measurement, including:

1.    Can everything that matters be measured?
2.    Content validity
3.    Measurement validity
4.    Social desirability bias
5.    Context effects on responses in surveys and interviews
6.    Latent variable modeling (confirmatory factor analysis and structural equation modeling)
7.    Measurement issues in process tracing and case studies
8.    Judgement and theory in historical measurement
9.    Interpretative mediation between everyday language and scientific concepts
10.    The transferability of qualitative and quantitative measurement standards to interpretative research.

The course is structured around three approaches: Variance-based, case-based, and interpretative.

Day 1

We introduce the main differences between the three approaches in terms of concepts and measurement. We talk about the extent to which each approach leans on – or can lean on – the others, and the extent to which they conflict with each other either fundamentally or in common practice. 

Day 2

We tackle human beings as sources of information. We talk about a large literature, mainly from survey methodology, about sources of bias when asking questions of people. These biases include social desirability bias, where people report distorted views of their attitudes or behaviours either strategically or to present themselves better to interviewers or themselves, and context effects, where prior questions influence answers to subsequent ones. We discuss the applicability of these and other biases to research, relying on a variety of tools including interviews and participant observation. 

Day 3

We dive deep into variance-based approaches to measurement. In particular, we examine in depth the latent variables perspective on measurement developed in psychology. In this approach, the notion that we cannot directly observe many of our concepts is tackled head on and modelled statistically using confirmatory factor analysis (CFA) and structural equation modelling (SEM). While the session itself will not teach you how to apply these tools, an additional, voluntary workshop during the week will provide an introduction to CFA and SEM in the lavaan package for the R statistical environment.

Day 4

We dive equally deep into case-based approaches to measurement. In particular, we look at measurement in case study research, whether this research is based on historical sources or interviews. We talk about common measurement pitfalls in process tracing research, and apply process tracing tests to attempt to get around them. Moreover, we talk about the roles that theory, histography, and expert judgement plays in historical case study measurement. 

Day 5

We dive into interpretative approaches to measurement. We look at the extent to which the advice from Days 3 and 4 apply to this approach and talk about what standards do apply if the interpretative approach is different. We talk about taking meaning building seriously and what this means to concepts and measures of those concepts. Finally, we debate the feasibility and problems of quantification in interpretative work.

We strongly encourage you to bring your own research questions, concepts, hypotheses, and proposed measures – even if you are only in the early stages of research – to the course on Day 1.

This course will help you measure core concepts in your research. There will be plenty of opportunity to reflect on measurement issues in your own research, and develop ways to overcome them. 

Basic understanding of statistical methods.

Some knowledge of concept formation would be useful.