Estimating the Recommendation Accuracy in Candidate-based Voting Advice Applications

Voting Advice Applications (VAAs) typically require users to answer questionnaires before receiving party or candidate recommendations. Naturally, the recommendations become more accurate as users answer more questions. When users finish the questionnaire early, however, the certainty of these recommendations is unknown. In this work, we provide a measure to quantify this certainty. Specifically, we develop an algorithm that estimates the Candidate Recommendation Accuracy (CRA) – the overlap between early and final recommendations – after each question during the questionnaire. Through simulations based on existing user data, we find that this algorithm is more accurate than heuristic estimates. Additionally, we show that this algorithm can be used to identify stable recommendations – candidates who will likely be among the final recommendations – based on confidence thresholds. Finally, we conduct a user experiment on the AQVAA platform, testing different ways of communicating the recommendation certainty to users. Our results show that users quit the questionnaire later when the estimated CRA is lower. We find that users appreciate the displayed certainty, especially when it is framed through stable recommendations. We conclude that the personalized display of the CRA can enhance user experience and spark curiosity in VAAs while providing a robust estimate of the recommendation quality for users who finish the questionnaire early.