Advanced Multi-Method Research

Mixed-method research is an enduring topic in the social sciences (e.g., Creswell and Piano 2011), but multi-method research (MMR) more narrowly is a relatively new topic in the “US methods debate”. After long standing antagonistic discussions about the pros and cons of small-n and large-n methods, we now find a growing consensus that each method has its distinct advantages and that they work best in combination with each other. This course builds on the debate about MMR and focuses on its unique advantages and challenges for empirical researchers seeking to combine two (or more) methods.

On day 1, we lay the foundation and discuss different varieties of MMR. As regards the large-n method, the focus rests on regression analysis and Qualitative Comparative Analysis (QCA) as, arguably, the most widely applied cross-case methods in MMR (emphasis on both cross-case methods depends on the method the course participants are implementing). First, the process of making an informed choice between regression analysis and QCA is explained. In addition, we discuss the conditions under which it is better to begin with case studies and utilize the large-n method afterwards, and when it is better to apply the large-n technique first and process tracing second. Furthermore, we have a brief discussion of fundamental terms such as causal effects and causal mechanisms and levels of analysis.

On day 2, we begin with a reflection on concepts and concept formation in MMR as the cornerstone of all empirical research. The session is based on two interrelated claims one finds in the literature. First, it is argued that concepts are thin in large-n and thick in small-n research. Second, it is claimed that this discrepancy creates problems of conceptual stretching undermining causal inference in MMR. We elaborate on whether these assertions are warranted and, to the extent that they are accurate, how concept formation can be improved in MMR.

The topic of day 3 is case selection on the basis of results derived from the large-n analysis. First, it is shown that case selection strategies differ depending on whether one is running a regression analysis or QCA. Building on this insight, we expand on the designation and choice of different types of cases – e.g., typical and deviant cases – on the basis of large-n results. Following Fearon and Laitin’s (2008) plea for random case selection for process tracing, we further consider the pros and cons of random case selection versus different types of intentional case selection.

Because of the importance of case selection and the variety of arguments in the literature, the beginning of day 4 finishes the discussion of this topic. Afterwards, we turn to the question of generalization. Generalization is rarely considered for the large-n method – regression analysis and QCA alike – but we also spend some time considering the generalization of large-n results. The main focus lies on the generalization of the inferences generated via process tracing. According to a common line of reasoning, generalization of case study inferences is only possible under conditions that are difficult, if not impossible, to reconcile with MMR. Running counter to this, we discuss a statistical procedure for generalizing process tracing inferences and detail the conditions under which it is applicable.

On the last day, day 5, we have a wrap-up session. We take a closer look at published empirical studies with regard to the topics of day 1 to 4 and with a specific focus on what can be called causal consistency or causal coherence. This means that one’s theoretical expectations as regards the large-n results and process tracing insights should fit with each other. Similarly, the inferences that one derives from large-n and small-n analyses should be coherent. For example, a lack of fit occurs when process tracing leads to the conclusion that multiple factors work in conjunction, while the regression analysis models the effect of covariates as independent from each other. We discuss several sources and manifestations of inconsistency, strategies for achieving coherence in the specification of observable implications related to causal effects and causal mechanisms, and raise the awareness for making consistent causal inferences on effects and mechanisms.

On day 1, 2, 3, and 5, participants receive small assignments due the next day that will be discussed at the beginning of each section of the following day. The assignments deal with MMR studies published in journals or books.

Participants at a more advanced stage of their MMR are invited to bring their large-n data with them in order to discuss specific issues and to immediately attempt to implement the lessons learned in the course.

The course does not discuss basics of regression analysis, QCA, case studies, process tracing, or any other method one might use in multi-method research. Participants are expected to have acquired skills on these methods when taking this course because it specifically focuses on how to combine them. Having sufficient knowledge of the methods you aim to use is important because it is important for understanding some of the principles of multi-method research.