Effective Data Management with R

It's impossible to put a precise number on it, but it's a safe bet to say that somewhere between 40% and 60% of the work in most data analysis projects needs to be allocated to mundane tasks of data management. In some cases involving custom data sets obtained through Internet scraping, this share can easily grow to 70–80%. The goal of this course is to present a set of commonly-used and effective R packages that minimize the hassles and errors that invariably appear in the course of processing data. At the end of the class participants should be comfortable with reading in data in R that comes in a variety of formats, with cleaning and recoding such data, summarizing and aggregating it, and with merging it with other types of data. Although these might seem as simple goals for a 3-day class, the multitude of data types that can be encountered in the course of an analysis means that they are anything but simple. String variables (e.g. electoral district names), timestamps (e.g. from Twitter posts) and geographic coordinates (e.g. placement of polling stations) frequently have to be merged with administrative data that can come in a variety of formats, such as Excel, SQL or JSON. This course offers you a set of tools that can help you along in these tasks, and others like them.

In the course, we follow the typical structure of most data preparation projects: a) reading in data in R, b) cleaning, checking and recoding the variables needed for analysis, c) summarizing and aggregating the data, d) reshaping data based on our specific needs and, e) merging/matching the data with other data sources. For each task, we introduce and test a series of packages that can assist the user. To begin with, we go over some of the most common types of data sources that users have to input into R. The focus is less on common types of statistical data, e.g. .SAV or .POR (for SPSS) or .DTA (for Stata). Rather, we concentrate on typical formats encountered on the Internet and produced by government agencies or organizations: .CSV and .XLSX (Excel), .JSON, .HTML, .MDF (SQL database), and even .PDF. In the course of this section, I demonstrate the capabilities of the rio package, along with foreign, readstata13, readxl, XML, rjson, RMySQL, and googlesheets.

In the second part of the class, which is also frequently the most time consuming in an applied project, we deal with tasks of cleaning, checking and recoding variables. Data which has already been partially cleaned and validated for us, as encountered in SPSS or Stata data sets, will pose few problems. In most other situations, we are dealing with freshly-collected information, which requires heavy cleaning and checking. I cover here the most common problems that appear in text data (tweets, electoral district names, or city names), along with handling time stamps that appear on web pages, Facebook posts or tweets. In the course of this work, I rely on the stringr, lubridate, and zoo packages. Following this, I focus on how to clean and validate variables that will be used in the course of one’s analyses. The section concludes with how to restructure data based on the principles of “tidy data”, so as to render it in a format suitable to most R analyses and graphical functions. In the course of these tasks I demo the functions available in the janitor, validate and tidyr packages for R.

In the third part of the class we focus on a slightly smaller set of topics. We start with how to use R for the analysis of very large data sets, for which the most common types of R functions are very slow. I introduce in this section the data.table package, which speeds up considerably some data management tasks. We continue with a few common ways of summarizing and aggregating data, with the help of the dplyr and plyr packages. These are frequently needed in the process of data checking and recoding for time-series cross-section, or pooled cross-sectional, data sets. data.table and plyr will also prove to be useful in the course of merging different data sets, for the purpose of adding new variables to our analyses. Finally, at the end of the course, we look at a few ways in which the reshape2 package can help us with data reshaping, such as converting between “long” and “wide” data formats.

This course is certainly not a statistics course, in the sense that we do not cover any statistical concepts or methods. Rather, the point of the course is to make R a friendlier work environment for the part of the research project that comes before statistics enters the scene: preparing the data for analysis. Due to this focus, participants who are looking for an R-based introduction to statistics should opt for an alternative class. Neither can the class be considered an introduction to R course. I do not cover the object-based logic of R, the different types of objects R works with, or how R can be used for basic statistical analyses and graphs. The focus will constantly be on data cleaning, recoding, reshaping, and merging.

Participants should possess introductory-level knowledge of the R statistical environment: what types of objects it uses, how to manipulate these objects, and the basic rules of R syntax (e.g. how to use R functions, the difference between “=” and “==”).