A Gold Standard of Institutional Assessment? Analyzing Political Biases in International Organization Evaluation Reports

Evaluation has become a significant part of international organizations’ (IOs) policy cycles and decision-making procedures. Every year, hundreds of costly evaluation reports are produced by major international organizations (IOs) and the number is growing. These reports inform IO decision-making procedures and are generally perceived as a ‘gold standard of institutional assessment’. Yet there is virtually no comparative empirical research on the substance of IO evaluations and how the findings are presented. In this paper, we conceptualize, operationalize and measure political biases in over 12,000 evaluation reports across 19 major IOs. We argue that powerful IO stakeholders – member states and IO bureaucracies – can use evaluations to utilize their pre-defined strategic interests. For instance, while IO administrations are interested in overly positive evaluation results, member states seek resource optimization and control. Depending on who has more power over evaluation resources, evaluation units anticipate their interests both in the content and form of evaluation reports. Drawing on qualitative text analysis and text-as-data approaches such as automated sentiment analysis, we show both the existence and variation of such political biases in IO evaluation reports. Our findings challenge the traditional understanding of value-free evaluations and have broader implications on our understanding of evidence-based policymaking in IOs.