Scholars and policy advocates frequently call for better transparency of algorithms (Andrews, 2019; Fink, 2018). One of the main challenges of such transparency is that it is difficult for developers of algorithms to make them work effectively while also revealing all the coding and development steps that go into creating an algorithm (Kitchin, 2014).
As a potential solution to this problem, the author tests an unsupervised machine learning (ML) approach to a public comments regulatory development process. Similar approaches have been applied to citizen (e.g., Clark, Morris, and Lomax, 2018) and politician (e.g. Imai, Lo, and Olmsted, 2016) preferences mined in social media text, but public comments in formal processes of regulatory development may also yield relevant insights especially as the texts are explicitly addressed to specific regulatory proposals. The research question thus posed is: can a machine learning approach to public comments in a public policymaking process deliver better policy insights and transparency?
The approach involves using 5,547 public comments scraped from the United States federal website regulations.gov using the site’s application programming interface (API). The comments relate to a 2013 proposed rule by the United States Transport Security Administration (TSA) on the use of new full body imaging scanners in airport security terminals. A manual coding process first produces a codebook. Keywords from the codebook are used to ‘read’ the public comments using the regulations.gov API, and key topics in the texts are modelled using latent Dirichlet allocation (LDA) with a second layer sentiment analysis.
The ML approach results in salient topic clusters with positive and negative sentiment scores that could be used as a tool by citizens and policymakers to understand large amounts of text such as in an open public comments process. To evaluate these results further, the author compares the results of the ML model with the actual final proposed TSA rule, noting that there were many areas where the final rule was at odds with the public comments. Finally, the author discusses the challenge of embedding ML approaches in recommender systems in public regulatory development processes. Proposals are put forward for how this can be done in a way that advances transparency and accountability.
References
Andrews, L. (2019). Public administration, public leadership and the construction of public value in the age of the algorithm and ‘big data’. Public Administration, 97(2), 296-310.
Clark, S. D., Morris, M. A., & Lomax, N. (2018). Estimating the outcome of UKs referendum on EU membership using e-petition data and machine learning algorithms. Journal of Information Technology & Politics, 15(4), 344-357.
Fink, K. (2018). Opening the government’s black boxes: freedom of information and algorithmic accountability. Information, Communication & Society, 21(10), 1453-1471.
Imai, K., Lo, J., & Olmsted, J. (2016). Fast estimation of ideal points with massive data. American Political Science Review, 110(4), 631-656.
Kitchin, R. (2014). The data revolution: Big data, open data, data infrastructures and their consequences. London: Sage.
