How to enhance the external validity of survey experiments? A discussion on the basis of a research design on political gender stereotypes in elections

(Quasi-) experimental methods are rather scarce in political science, but its use tend to increase in recent years (e.g. Blais, Lachat, Hino, & Doray-Demers, 2011). An experiment is a deliberate test of a causal proposition, typically with random assignment to conditions (Druckman, Green, Kuklinski, & Lupia, 2011b). Researchers employ experimental designs in order to combat problems of causal inference: it is thought that experiments are the best possible way to address the problem of third variables and potentially spurious relationships (Mutz, 2011, p. 9). As the conventional wisdom suggests, experiments are widely valued for their internal validity, but they lack external validity (Mutz, 2011). External validity is commonly used to refer to the extent to which the causal relationship holds over variations in persons, settings, treatments and outcomes (Shadish & Cook, 2005, p. 83). The concept refers more generally to whether conceptually equivalent relationships can be detected across people, places, times and operationalizations (Anderson & Bushman, 1997, p. 21). There is, however, no deductive or empirical basis for claiming that experimental findings are less generalizable than those of other methods (Mutz, 2011). In this paper, we will focus on the ways in which we can overcome problems of external validity by discussing the research design of our PhD project on political gender stereotypes in elections. In our experimental design, the central research question is whether political gender stereotypes also prevail in a Proportional Representation system. We will conduct a survey experiment in which respondents will be confronted with hypothetical candidates, who present themselves in an audio message in which they mention their position on the list and their policy position on a particular issue. The experiment will be set up as ‘between subjects’ design, in which different groups (differing on the gender of the candidates and on their list position) are compared with each other (Druckman, Green, Kuklinski, & Lupia, 2011). We will discuss how we will deal in our experimental design with four different elements to increase external validity: whether the participants resemble the actors who are in real life confronted with these stimuli, whether the context within which actors operate resemble the context of interest, whether the stimulus used in the study resembles the stimulus of interest in the real world, and whether the outcome measures resemble the actual outcomes of theoretical or practical interest (Druckman, Green, Kuklinski, & Lupia, 2011a). By linking methodological insights on experiments with a specific experimental design, we come to the heart of what this section on experimental research aims to do.