Advanced Topics in Applied Regression

As an estimation framework, OLS (Ordinary Least Squares) has clear advantages over alternatives when it comes to ease of understanding, speed, elegance and robustness in the face of distributional “irregularities” in data. At the same time, the instances when OLS can safely be employed in a standard way with social science data are rather few and far between. This class aims to expand the statistical toolbox of participants by presenting a set of regression tools that can be applied in situations when standard OLS would lead to suboptimal results.

We start with an in-depth coverage of OLS assumptions, emphasizing in particular the need for continuous (or dichotomous) measures in our regressions, a normal distribution of residuals, homoskedasticity, and linear relationships. We discuss, in turn, how OLS estimates of effect and uncertainty are impacted by violations of these assumptions, and what tools we have available in R to diagnose these problems. I make the point that these assumptions are frequently not met in the course of many analyses, leading to biased estimates and, therefore, shaky conclusions. The topics that follow in the course represent a few of the modeling strategies available to researchers when such assumptions are clearly invalid. While more complex than OLS, they are nevertheless still part of the “least squares” framework, and somewhat easier to apply than more complex procedures, like multilevel modeling.

We first discuss heteroskedasticity: what its implications are for estimates, how it can be detected in the course of a standard analysis, and how commonly it appears as a problem. To address this issue, I present two potential solutions. The first is cluster-corrected standard errors, which might remedy estimates of uncertainty if the underlying problem is generated by clustering in the data. Cluster-corrected SEs continue to be a very popular approach in a variety of disciplines, which is why they are covered in depth here. The second, more general, solution is the use of Weighted Least Squares (WLS). Both subtopics are discussed from a theoretical perspective, as well as in a practical setting, in the laboratory. The third day of the course is taken up by the issue of effect heterogeneity across different subpopulations in the sample. In practice, this will involve an in-depth discussion of interactions in linear models. We cover two-way and three-way interactions, both for continuous and dichotomous predictors, as well as how to present marginal effects in a graphical way. As we will see, interactions are frequently a source of confusion in published work, and continue to frequently be misinterpreted. In the final part of the day, we discuss how the interaction framework might be applied when the subpopulations are actually samples from different countries, through the use of fixed effects. As in the previous days, the theoretical coverage is followed by applied lab work, using R and empirical data.

We continue, in the fourth session, with how to model non-linear relationships between variables. Rather than use statistical transformations of the predictors, we rely in this section on polynomials and regression splines. I show how these tools allow the researcher to model increasingly complex relationships between predictors and outcomes. We conclude the section with a presentation of nonlinear least squares, as an estimation method. The final section is taken up by the topic of robust regression. Unlike the second day, though, we refer to regression estimates robust to data outliers, rather than robust to violations of homoskedasticity. Data outliers can severely impact both the magnitude of the regression coefficients and the standard errors produced. We cover three types of procedures in this session: M estimation, bounded-influence regression, and quantile regression. We take up this discussion again in the laboratory, with practical examples of the differences between standard OLS estimation and robust regression in the presence of outliers.

By the end of the course participants should be able to recognize the situations where OLS does not produce adequate estimates, and to identify the specific cause(s) for this breakdown. After this diagnosis, they should be able to either re-estimate using a more appropriate model specification, or to apply the needed corrections to the initial estimates. Finally, they should feel capable of interpreting the estimates from the revised models, and of summarizing the procedures they have implemented in a concise way.

The topic of how to go beyond the standard assumptions of OLS is much broader than the topics covered here. Due to lack of time, I cannot also offer even an overview of a variety of resampling methods (bootstrapping, jackknifing), or of how to correct for estimation problems arising from the absence of full information on some variables (missing data). Those who are interested in these particular topics for their research should aim for one of the other courses offered at either the Winter or the Summer School, dealing with these specific topics. Despite its close connection with the issues of heteroskedasticity, and of varying effects across different subpopulations, I also cannot cover the topic of multilevel modeling. Suitable courses for MLM are offered both in the Winter and Summer editions of the School, and I encourage those interested in the topic to take either of them.

Due to the advanced nature of the course I expect participants to have good knowledge of linear regression. They ought to be familiar with running such a regression, interpreting coefficients and standard errors, assessing model fit, and diagnosing problems with the estimation procedure. In addition to this foundation of statistical information, participants are also required to have good knowledge of the R statistical environment. This refers to common procedures such as reading in data, cleaning and recoding variables, as well as running a regression in R and manipulating the output object (e.g. extracting coefficients or standard errors).