Understanding the information quality of public comments on bureaucratic policymaking in the European Union: a text-as-data approach

A key assumption underpinning stakeholders’ participation in public policymaking is that stakeholders possess relevant information that might otherwise be unavailable to decision-makers and allow them to make better informed decisions that lead to better policy outputs and generate policy outcomes that enjoy higher levels of input and output legitimacy. A key mechanism for facilitating this information transmission is the introduction of direct public comments on bureaucratic acts and initiatives. While these comments are widely perceived as a democratising tool of bureaucratic policymaking, important questions were raised about the extent to which they manage to provide useful policy inputs that meet the (high) information needs for expertise and technical information that bureaucratic policymakers have in light of their frequent engagement in evidence-based decision-making. While these questions received important theoretical answers, they remained largely unexamined empirically. Our study addresses this gap and asks: Are there systematic differences in the information quality of public comments provided by different categories of stakeholders? And, relatedly, under what conditions are stakeholders more likely to provide comments that are of high(er) information quality? We answer by building a conceptual framework that recognises the multi-dimensional nature of information quality of stakeholder inputs in bureaucratic policymaking. We propose a conceptualisation of information quality along three dimensions: (1) textual sophistication, (2) the extent to which they provide constructive and actionable feedback, and (3) the extent to which they engage explicitly with the issues raised by policymakers in consultation documents. Theoretically, we developed a set of expectations about varying levels of information quality of comments across stakeholder categories and policy acts. Based on a text-as-data approach we develop several measures that tap into these three dimensions and apply them to 20,769 public comments generated as part of 1,037 public commenting events in the EU supranational policymaking. Our preliminary descriptive analyses reveal three distinctive empirical patterns. First, citizens’ comments are systematically of a lower information quality across all policy acts and measures of information quality. Second, there are no, or very few, and rather small distinctive differences amongst organisational stakeholders in terms of information quality. Third, legislative proposals generated public comments that are on average of higher information quality than those generated by roadmaps and IIAs or delegated and implementing acts.