Applied Mixed Methods Research

Political science is an intrinsically interdisciplinary approach to the study of politics. Not only does this imply a variety of epistemological and ontological approaches, but also a range of methodological approaches. Contemporary political science research increasingly features blends of two or more different social science methodologies, widely known as multi-method researrch (MMR). The value of drawing on more than one approach is that many ”big” research questions can only be answered by examining processes taking place across multiple temporal and spatial domains at different levels of analysis. To assess the nature and scope of these processes, today’s researchers increasingly draw on more than one methodology to shed light on multiple facets of such political problems in hopes of achieving greater explanatory leverage as well as descriptively rich empirical results.

This course covers the logic, advantages and unique challenges associated with multi-method research in political science. In the first two days, we explain the uses and attractiveness of a MMR approach and identify questions that are paticularly well-suited for an MMR approach. We also explore the limitations and challenges students are likely to face in applying MMR to their research projects, as well as the range of options available to scholars desiring to undertake multi-method research. The first two days cover these issues in detail. We explore the unique challenges related to concept formation and theory development in multi-method research. We also tackle the unique challenges of hypothesis testing and data requirements associated with different types of MMR. The remaining three days explore the range of MMR types, using concrete examples.

We begin with the more traditional positivist MMR blends found in mainstream positivist research projects. These ”quantitative-qualitative” approaches typically combine regression analysis with a small-N case analysis. Another variety combines different qualitative, positivist research traditions—typically historical process-tracing in one or two cases—with comparative or medium-N (QCA) analysis of multiple cases across space or time. A third technique combines micro-level experimental or other survey research undertaken at the individual level with macro-level analysis undertaken at higher levels of analysis, such as cross-temporal or –sectional analysis of states, institutions, groups, or institutions. The next day focuses on MMR projects that cross epistemological lines, combining positivist and non-positivist techniques in an appoach known as ”analytical eclecticism.” While less mainstream than positivist hybrids, the combination of interpretivist research techniques with positivist research design principles or positivist methods has gained momentum across a number of social science fields. The final day focuses on the ”write-up,” namely how to derive empirical generalizations across different analyses and how to integrate these findings in a single research report.

The course is divided into three main parts: (1) explaining why researchers might want to combine multiple methods in a single research project, when they should avoid doing so, and how to tell the difference; (2) showing how concept development and measurement can be used to bridge divergent ontological and epistemological divides in an integrated research design, and (3), exploring different combinations of social science methods using real examples of social science research. Throughout, we will spend equal time on seminars and workshops--a format intended to assist students in developing mixed-method designs tailored to their specific research aims.

In the first part of the course, we explore the advantages and disadvantages of MMR as well as the type of research questions most appropriate for multi-method research designs. On Day 1, we review the both the promises and strengths of MMR as well as its potential pitfalls. These mostly relate to the not-inconsiderable problem of conceptual stretching when formulating and operationalizing concepts for use across different domains and/or different sets of data—Day 2 extends this discussion. Most of the seminar on Day 1 is devoted to mapping the different epistemological, ontological and methodological differences that must be reconciled in developing a MMR project, as well as the arguments commonly raised against combining methods across disparate research traditions. We discuss the ”paradigm wars” as they relate to MMR as well as how to mix research methods in a way that leverages their joint payoffs without violating any important methodological assumptions.

Day 2 begins by discussing in greater detail the type of research questions that lend themselves to multi-method research. We then review of the most well-known MMR techniques, starting with nested analysis—a hybrid qualitative-quantitative research approach that uses the results of quantitative analysis to inform the scholar’s choice of cases for qualitative analysis—the results of which are combined in a single academic article or book. We also review other qualtitative-quantitative hybrids as well as purely quantitative and purely qualitative hybrids (particularly those designed for cross-temporal or -spatial analysis). Using real examples of social science scholarship, we explore the most effective uses of muti-method research designs—what makes them work and how they help researchers make persuasive claims across a variety of empirical domains.

We also emphasize the importance of maintaining conceptual consistency across different methodological approaches to ensure the validity of inferences from multiple analyses. The most common issue is how to ensure consistency between ’thick’ concepts, most often used in qualitative, ethnographic, and/or interpretivist research, and ’thin’ concepts, mainly used numerical measurement in quantitative analyses.

The second part of the course (Days 3 and 4) is aimed at executing different types of MMR designs, namely positivist hybrids and postivist-interpretivist hybrids. Although non-positivist research traditions do not have the same standards of evaluation of positivist social science, we nevertheless discuss different threats to causal inference in each hybrid. These include threats to concept validity and measurement commensurability that come from translating concepts across different domains and empirical analyses. We also explore how to derive and evaluate empirical claims using different types of empirical analysis, levels of analysis or spatial or temporal units. Using concrete examples of MMR social science research, we dissect how each author handled problems of endogeneity, reverse causation and equifinality when making causal inferences in positivist empirical analyses.

Day 3 explores more traditional multi-method research designs in the positivist tradition. The most formalized MMR technique is nested analysis, which offers guidelines for how researchers can conduct large-N analyses to test for hypothesized effects of the independent or treatment variable across many cases, and then use the results of these tests to select ”pathway” or average cases that can shed light on the precise causal mechanism at work through small-N analysis. In the seminar, we discuss how MMR was designed to achieve these goals through a concrete example of nested analysis (Ross). We also discuss how to ensure concept validity and consistency in other hybrids, including QCA-comparative analysis (Howard), and field and/or survey experiments (readings TBA). Students will be asked to think about principles of achieving causal homogeneity across cases through principles of case selection, how to generalize from small-N and large-N analyses, and how to combine inferences from two or more methods to evaluate empirical claims in a single academic project.

Day 4 focuses on MMR that combines positivist and non-positivist (interpretivist) methods in single research project. Such projects have their origin in the ”constructivist turn” in social science that privileges on the role of ideas and communication (meaning making) in the production of political behavior. Scholars in this tradition routinely utilize principles of positivist research design when selecting cases, measuring concepts, and identifying empirical domains (including texts or respondents), while interpretivist techniques such as discourse analysis or interpretivist ethnography are used in the actual empirical analysis (Cramer 2012; Tannenwald 1999). Researchers have also combined interpretivism with positivist methods of process-tracing, comparative and counterfactual analysis to assess claims about the function of speech acts or ideas in foreign policy or other political behavior across time and space (Guzzini 2012; Krebs 2015). This class will focus on the problems that come from combining approaches from research traditions that differ on whether one can arrive at generalizable theories or make causal inferences about the relationship between ideas or discourse, on the one hand, and political or social behavior, on the other. 

The third and final part of the course (day 5) draws on the material from the previous four days, focusing in particular on how to make empirical inferences based on the data and then how to ”write up” MMR results. We cover techniques for developing a convincing narrative or ”story” that will relate back to the original research question that is guiding the project. Particular attention will be given to how to grab and maintain the readers’ interest while building a case for the researcher’s central argument. Researchers using disparate empirial analyses and types of data need to pay careful attention here as to how to weave these data together without doing violence to the assumptions undergirding the different methods of analysis.

This course does not have any special prerequisites, but students should have a basic familiarity with the comparative method and qualitative research design.