Publication Bias in Qualitative Comparative Analysis

Science advances by finding and not finding evidence for causal relationships. However, studies of quantitative research show that it is characterized by publication bias because most published findings are positive. Publication bias can occur because of research practices such as p-hacking and fishing; the exclusive submission of positive findings to journals; reviewers and journal editors favoring positive results over negative results and making publication decisions accordingly. In this paper, I discuss the incidence, possible causes and remedies of publication bias in Configurational Comparative Research (CCR) with a particular focus on Qualitative Comparative Analysis (QCA). A review of QCA studies from 2015 to 2017 shows that publication bias exists. All solutions that are reported and their constitutive terms reach the conventional consistency threshold of 0.75 and therefore represent positive results. I further show that most published work is confirmatory as opposed to exploratory and publication bias therefore a point of concern deserving further consideration. I continue with discussing sources and remedies against publication bias in CCR. As in quantitative research, the file drawer problem of not submitting negative results could be one reason that is difficult to study empirically. The same holds for selective reviewer and editor preferences. The well-known solution to both problems is results-blind review. I discuss what results-blind review means for CCR. For the studies that I review, I demonstrate that the dialogue between ideas and evidence, which is often argued to be characteristic in QCA, is rarely practiced. Instead, CCR is implemented as a plain data analysis technique anchored in set theory. I conclude that results-blind review is possible in CCR and explain how it could be implemented in practice by formulating a protocol that could be used by journals and reviewers of blinded QCA studies. For the third source of publication bias that is linked to research practices, I first explain what the equivalents of concepts such as fishing and p-hacking are in CCR and what additional options for the deliberate production of positive findings are available. I use simple hypothetical examples and simulations to demonstrate that it is always possible to produce a positive result. Based on these demonstrations I develop a comprehensive preregistration protocol for CCR that empirical researchers can use if they want to preregister their study. I conclude by detailing research questions that would allow one to preregister a QCA study even if it relies on observational data.