Something Old, Something New…: Assessing Public Opinion by Validating Social Media Data

Studies correlating Twitter data – whether tweet counts or more refined measures of sentiment – with opinion poll trends have become increasingly common in the election forecasting literature (e.g. Williams and Gulati, 2008; Tumasjan et al., 2010; Ceron et al., 2015). Yet for every predictive success there seems to be a failure, and a cautionary view seems increasingly to be dominant in the literature (e.g. Gayo-Avello et al., 2011; Jungherr, 2013). The reasons for caution – notably the unrepresentativeness of the user community, the dominance on Twitter of certain voices that then have only one vote in the election, and the difficulties of sentiment analysis – are all problems that can be mitigated by combining new media data with ‘old’ survey data drawn from these users themselves. In this paper, based on Twitter data from the period January-April 2016 in the run-up to the Scottish Parliament election in May, we report on a pilot project attempting just such a combination. By surveying users identified by the use of relevant hashtags, and gauging the relationship between Twitter measures of their sentiment and survey measures of their attitudes, we can go some way towards two related goals: i) validating the individual-level social media data; ii) understanding the aggregate-level disjuncture between Twitter sentiment and polls. Both help us to respond to the plea from Gayo-Avello et al. for a “model explaining the predictive power [or lack thereof] of social media” (2011, p. 490).