Advanced Topics in Applied Regression

As an estimation framework, OLS has clear advantages over alternatives when it comes to ease of understanding, speed, elegance and robustness in the face of distributional 'irregularities' in data. At the same time, the instances when OLS can safely be employed in a standard way with social science data are few and far between. This course aims to expand your statistical toolbox by presenting a set of regression tools that can be applied in situations when standard OLS would lead to suboptimal results.

Day 1
We begin with an overview of regression assumptions, like measurement scales, collinearity, homoscedasticity, autocorrelation and linearity. If you are familiar with regression, you probably know what most of these mean. But what happens if they are violated to some extent? What happens when they are violated extensively? Where to draw the line? We use R to simulate examples of these assumption violations and through them test and see how they impact on the results of our regressions.

Day 2
The effects on one variable often depend on the level of another variable or perhaps there is a certain group structure in the data. We focus on specifying and interpreting interactions, including three-way interactions. We'll see how, most often, plotting is the only way to make such interactions interpretable and meaningful and we explore some of the possibilities that R can offer us here. We also focus on how one can use interactions between dummy variables to model groups in data and we conclude with the fixed-effects model, which is the simplest, but often inadequate, approach to modelling the multilevel structure in your data. 

Day 3
Linear regression, as its name implies, assumes that relationships are linear, which is a very strong and often an erroneous assumption. We will be looking into how one can resolve this problem by transforming variables so that the relationships become linear through polynomial regression, which in most cases involves applying quadratic or cubic functions to regressors. We also briefly look at the usefulness of regression splines in modelling non-linear relationships.

Day 4
We look at resampling as a tool for hypothesis testing and the evaluation of bias/uncertainty. While applicable beyond regression, resampling can be used also in this context for estimating the uncertainty of our results when some of the assumptions of regression are not met. We look at bootstrapping – sampling with replacement – and the jackknife – leaving one case out of the analysis at a time – and their implementations in R. The first is especially useful for estimating confidence intervals in a non-parametric manner; the latter can be used to estimate bias in the model and in some cases also provides a simple way to evaluate how much the results of an analysis depend on single cases. For bootstrapping, we consider case resampling and residual resampling.

Day 5
We examine model selection and model averaging, starting from the question of how to assess the overall quality of the model, not just the statistical significance (which is often meaningless) of the coefficients of the variables it includes. Much of the class will focus on the Bayesian Information Criterion (BIC), a common measure of model fit for generalised linear models, among other things, and thus also applicable to linear regression. Through its connection to the Bayes factor, it allows us to compare, given certain assumptions, models to each other in terms of their relative probability of being 'true'. By extension, this also allows us to fit several models, all of which are likely to be true, and combine the results. This can be useful when we do not have a well-defined theory to test (i.e. the assumption of full model specification is violated) and want to give a good, empirically grounded summary of the possible associations in the data.

These various regression assumption violations and handling them are tied together through working with R and using R to exemplify and simulate many of the problems that we go through. We will use materials prepared in R specifically for this course, that we will go through and discuss together. Whenever possible, we look at how one can use R to present problems and results visually (using ggplot), i.e. make visualisations of our models and results. Therefore, the course has a much more hands-on focus than some of the readings might lead you to believe. The equations are only for the connoisseurs, not for the practitioners.

The classes will not have a strict 'theory versus application' divide. We begin each day with a brief discussion of the theoretical/methodological nature of the problem, but for most of the days we will look at how these problems manifest themselves in actual and simulated data, and at how we can use R to overcome such problems and find adequate solutions.

This course builds on the many common problems encountered in linear OLS (Ordinary Least Squares) regression that are often ignored or overlooked.

You should have knowledge of linear regression. If you are reasonably familiar with running OLS, interpreting coefficients and standard errors, assessing model fit, and diagnosing problems with the estimation procedure, then you are more than ready for this course.  

You also need basic knowledge of the R statistical environment (R as a language and the use of an IDE like RStudio). This refers to common procedures such as reading in data, cleaning and recoding variables, as well as running a regression in R.