Assessing the Quality of VAA Questionnaires

While the questionnaire of the VAA is the main source of information regarding the positions of the user, thus far it has received only limited attention from scholars. Here, I use various techniques from survey methodology to better establish: a) the “quality” of the data the VAA generates and b) to identify “difficult” questions in the VAA questionnaire. Difficult questions are those questions that were confusing or hard to understand for the user and which thus may lead them to provide an answer that is different then their true position. First, I test whether certain questions lead to an unexpected response on the Likert-scale as often used in VAAs. As I expect that greater expected agreement on an item would also lead the user to choose an option that would signify this agreement, I use the “dirty data index” to test whether this is the case. A high score on this index, which is calculated by comparing the PCA solution to the questions with their CatPCA solution, signals questions that were possibly difficult for users to understand, generating data of a lower quality as a result. Second, I will test whether the assumed ordinality of the responses is violated by using MCA to see in which order the questions load on the main dimension. Third, I look at the pattern of usage of the “Neutral” option, which can provide an easy escape option for difficult questions. If used as a substantial option, an analysis using MCA will show it on the same dimension as the other options. If not, and users use it to hide their non-opinion, it will load together with the “Skip” option on a second non-substantial dimension. Throughout, I will use the data generated by EUVox, a VAA that was developed for the 2014 European Parliament elections. Given that the VAA had a nearly identical questionnaire in all countries, EUVox allows us to discover differences in question difficulty and usage of the different response options between these countries. I will conclude with several observations on how VAA designers can use the above techniques to identify and possibly remedy difficult questions, and how users of the data the VAA generates can check its quality.