WA105 - Automated Web Data Collection with R

Automated web data collection with R is a course designed to provide a primer on web scraping with R, i.e. the automated gathering of data and text from the web. Web scraping can enable new perspectives on social science research problems, but it can even help to develop new subjects altogether by making data available that just recently came into being – e.g., social media and blog posts or webpage linkage networks. Similarly, web scraping pushes the scales on the amount of data available to researchers.

Although web scraping is not new, it is only starting to be recognized by a broad social science audience. In the past, statistical software used by social scientists like SPSS, Stata or R did not have web scraping capabilities, while software that had the means for web scraping like PHP, Pearl or Python was alien to social scientists. Furthermore, web scraping tends to be presented either purely technical in the form of ‘from programmers for programmers’ manuals or as specialized single-issue case studies that make it hard to get the general picture. Being well aware that social scientists usually want to do substantive research instead of learning yet another software, the course builds on a thorough but gentle, hands on web scraping textbook written by social scientists for social scientists: Automated Data Collection with R - A Practical Guide to Web Scraping and Text Mining. The book rests on three pillars: (1) providing an introduction to web technologies and associated tools, (2) presenting handy R packages and guidance for web scraping and (3) illustrating real life applications with case studies.

The main goal of the course is to provide a solid overview on the most important web technologies, how they are connected and which tools are available in R to handle them. After completing the course, you should be able to handle simple scraping scenarios: downloading various file formats, extracting lists and tables from HTML documents or retrieving easily accessible information.

R and RStudio

Participants to bring own laptops.

Munzert, Simon, Rubba, Christian, Meißner, Peter, and Dominic Nyhuis (2014): Automated Data Collection with R. A Practical Guide to Web Scraping and Text Mining. Wiley

Barberá, Pablo, 2015: Birds of the Same Feather Tweet Together. Bayesian Ideal Point Estimation Using Twitter Data. Political Analysis 23:76–91.

Barberá, Pablo, and Gonzalo Rivero, 2014: Understanding the Political Representativeness of Twitter users. Social Science Computer Review 1–18.

Chadefaux, Tomas, 2014: Early warning signals for war in the news. Journal of Peace Research 51:5–18.

Gayo-Avello, Daniel, 2013: A Meta-Analysis of State-of-the-Art Electoral Prediction From Twitter Data. Social Science Computer Review 31:649–679.

Gill, Michael, and Arthur Spirling, 2015: Estimating the Severity of WikiLeaks U.S. Diplomatic Cables Disclosure. Political Analysis 23:299–305.

Gohdes, Anita R., 2015: Pulling the plug: Network disruptions and violence in civil confict. Journal of

Peace Research 52:1–16.

Hassanpour, Navid, 2013: Tracking the Semantics of Politics: A Case for Online Data Research in Political Science. PS: Political Science & Politics 46:299–306.

King, Gary, Jennifer Pan, and Margaret E. Roberts, 2013: How Censorship in China Allows Government Criticism but Silences Collective Expression. American Political Science Review 107:326–343.

Rød, Espen Geelmuyden, and Nils B. Weidmann, 2015: Empowering activists or autocrats? The Internet in authoritarian regimes. Journal of Peace Research 52:1–14.

Sagi, Eyal, and Morteza Dehghani, 2014: Measuring Moral Rhetoric in Text. Social Science Computer Review 32:132–144.

Shaw, Aaron, and Bejamin Mako Hill, 2014: Laboratories of Oligarchy? How The Iron Law Extends to Peer Production. Journal of Communication 64:215–238.

Street, Alex, Tomas A. Murray, John Blitzer, and Rajan S. Patel, 2015: Estimating Voter Registration Deadline Effects with Web Search Data. Political Analysis 1–2.

Zeitzof, Tomas, 2011: Using Social Media to Measure Confict Dynamics: An Application to the 2008 - 2009 Gaza Confict. Journal of Confict Resolution 55:938–969.