Short is better: Examining the properties of the Instructional Manipulation Checks through an experimental approach

The rapid diffusion of the web surveys has raised the issue of the scarce attention individuals pay when answering to self-administered questionnaires. In order to measure the attention of the respondents, a new tool, Instructional Manipulation Check (IMC) also known as screener, has been introduced in survey research. IMCs basically consist in trap questions, which are correctly answered only if respondents read carefully the instructions concerning the tasks that are asked them. Several studies argued that IMCs might also be useful in awakening “lazy” respondents. Although the tool is becoming popular in the fields of social psychology and political science, low attention has been devoted to its characteristics. Our paper focuses on the nature and the formulation of IMCs, aiming to answer two main research questions: Are IMCs simply tools to detect inattentive respondents or do they also awake their attention? Does the length of the text of an IMC (and therefore the cognitive strain employed to correctly answer it) affect the performance of the instrument to detect inattentive respondents? We test these questions by means of a factorial survey experiment, that randomizes both the length (short, medium, long) and the position (before or after a test task) of an IMC. Differently from most IMC experiments carried out in ad hoc studies, our experiment is carried out in a real web electoral survey (Italian National Election Panel 2013-2015, N=3,000). Properties of the tools are analyzed by considering the quality of the answers to a battery of items on attitudes toward democracy. Preliminary results show that on the one side respondents’ attention is better assessed by shorter rather than longer (and more difficult) IMCs; on the other side, there is no evidence of an awakening effect of IMCs on the quality of answers to political questions.