Comparing Biases of Probability and Non-Probability Samples in Germany

The use of non-probability samples is on the rise not only in market research, but also in electoral research and social science applications. Organizations like AAPOR still hold, that there is no reliable way to establish representativity of a sample, other than probability sampling (Baker et al. 2013). For the US-context there already are a number of studies who established the strong bias of non-probability samples before, but even after the application of weighting procedures in comparison to proper probability samples (e.g. Yeager et al. 2011, Dutwin/Buskirk 2017). Other authors, however, claim, that the sampling biases are of moderate strength only and that their effects depend very much on the usage of the collected data. While they admit, that descriptive information about the target population does have a high chance of being biased, they claim on the other hand, the biases become less important if not negligible, if analysis focusses on correlation and causation instead of description (Ansolabehere/Schaffner, 2014). To our knowledge, all studies comparing the quality of probability and non-probability samples that have been done so far use data from surveys in the United States. While the general results obtained for the United States might therefore apply to other contexts as well, we believe it still is important to establish the specific effects of different sampling modes in those other contexts. We therefore apply the analytic framework laid out by Dutwin and Buskirk (2017) to different kinds of survey data collected in Germany. We use the ALLBUS as a “gold standard” survey, which works with a stratified address based random sample, an RDD-sample, and the German Internet Panel (both probability samples as well) and two studies by Yougov and Dalia-Research who work with different non-probability mechanisms for sampling from the online population. These data sets will be used for an analysis of their respective raw as well as weighted biases. With this comparison of surveys we contribute to the knowledge about the concrete effects of sampling biases, adding information about other then US survey contexts to the extant knowledge. Ansolabehere, Stephen, und Brian F. Schaffner. 2014. Does Survey Mode Still Matter? Findings from a 2010 Multi-Mode Comparison. Political Analysis 22: 285-303. doi: 10.1093/pan/mpt025. Baker, R.; Brick, J. M.; Bates, N. A.; Battaglia, M.; Couper, M. P.; Dever, J. A. et al. (2013): Summary Report of the AAPOR Task Force on Non-probability Sampling. In: Journal of Survey Statistics and Methodology 1 (2), S. 90–143. Dutwin, David; Buskirk, Trent D. (2017): Apples to Oranges or Gala versus Golden Delicious? In: Public Opinion Quarterly 81 (S1), S. 213–239. Yeager, David S.; Krosnick, Jon A.; Chang, LinChiat; Javitz, Harold S.; Levendusky, Matthew S.; Simpser, Alberto; Wang, Rui (2011): Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples. In: Public Opinion Quarterly 75 (4), S. 709–747.