Estimating Heterogeneous Treatment Effects in Survey experiments

Abstract

In this one-hour seminar, we discuss innovative ways to test for heterogeneous treatment effects in survey experiments. The discussion will follow two presentations (see below for more information). We are also pleased to have Dr Filipa Sá (King’s College, London) joining us as a discussant. Presentation 1: “Justifying EU immigration policy implementation: Explaining citizens’ diverging responses to contentious domestic justifications” by Thijs Lindner (Erasmus Universiteit) Abstract: National governments increasingly justify the domestic implementation of contested EU policies, yet little is known about how different groups of citizens respond to these justification strategies. We examine how governmental communication strategies influence policy legitimacy in the context of EU immigration policy implementation. Using an original preregistered survey experiment in the Netherlands and Germany, we compare blame-shifting to the EU with two defending strategies: responsibility-based and responsiveness-based justifications. We find that citizens’ responses to these strategies vary and are contingent on their socioeconomic positions and sociocultural orientations. As hypothesized, responsibility-taking significantly increases legitimacy among high-income and highly educated citizens and those holding positive orientations towards immigration. In contrast, none of the justification strategies produced systematic responses among low-income groups and lower-educated citizens. We also find that EU-sceptical citizens respond positively to all types of governmental justification when they hold more positive orientations towards immigration. These findings demonstrate that governmental justification strategies surrounding the domestic implementation of EU immigration policy generate markedly divergent reactions across the public, particularly between the perceived “losers” and “winners” of globalization. Presentation 2: “Using Flexible Machine Learning Methods to test for Competing Treatment Effect Modification in Social Science Survey Experiments” by Guido Priem (KU Leuven) Abstract: This presentation will show how to use flexible Machine Learning (ML) methods to test for heterogeneous treatment effects in a high-dimensional survey experiment. As a test case, we use a single-profile conjoint experiment embedded in the LISS-panel infrastructure. We present respondents to a series of instances of religious speakers speaking to an audience and ask respondents to judge if they should be allowed to speak or not. We randomly vary who is speaking, what is said, and where it takes place, allowing us to separately estimate the impact of each attribute on decisions to tolerate religious expressions. To detect individual differences, we use a novel machine learning method based on Bayesian Additive Regression Trees (BART) to systematically explore how the dominant attribute in the decision varies according to a range of socio-political characteristics included as pre-existing data in the LISS panel. Leveraging the breadth of pre-existing data in the LISS panel, we rely on an extensive list of background information on the respondents to assess this heterogeneity. In doing this, this study is the first to model how the interplay between individual characteristics and contextual attributes shapes tolerance decisions about religious expressions in a concrete intergroup setting.