Introduction to Machine Learning for the Social Sciences

A general introduction into the logic of machine learning, explaining how ML is related to more familiar statistical methods.

We will discuss the ML emphasis on prediction (and how this is only another side of the explanation coin) and substantive effect over statistically significant ones; the problems and potential solutions to overfitting, and the bias/variance trade-off.

We talk about the building blocks of machine learning models and algorithms: the role of hyperparameters, tuning, cross-validation, and supervised vs. unsupervised learning.

Throughout our discussions we will refer to (and apply in R) the K-nearest-neighbour classifier to illustrate those conceptual points.

Lecture + Lab

Conceptual issues (and how ML relates to conventional statistical methods in social sciences):

• Prediction and explanation;
• Supervised and unsupervised learning;
• Cross-validation and overfitting;
• Bias/variance trade-off;
• Hyperparameters and tuning;
• K-nearest neighbor classifier to illustrate these issues

Mandatory readings

James, G., Witten, D., Hastie, T., Tibshirani, R. 2013
An Introduction to Statistical Learning with Applications in R
New York: Springer, Ch. 2 and 5

Cranmer, S. J., and Desmarais, B. 2017
What Can We Learn from Predictive Modeling
Political Analysis 25(2): 145–166

We discuss classification problems, some of the most common problems for which ML can be applied in social sciences.

It is especially useful, for example, as substitute to human coding when there is some raw data available – think of classifying political parties as left or right, or countries as democratic or not, etc.

We cover some of the most common algorithms in use, such as support vector machines (SVM), and decision trees.

Lecture + Lab

• How to evaluate model performance and predictive accuracy with categorical outcomes
• Support vector machine
• Logistic regression
• Decision trees

Mandatory readings

James, G., Witten, D., Hastie, T., Tibshirani, R. 2013
An Introduction to Statistical Learning with Applications in R
New York: Springer, Ch. 8

We move to regression problems – i.e., continuous outcomes.

We look again at decision trees and proceed to tree-based models, such as random forests and boosting. These methods tend to have excellent predictive performance, and are also able to generate interpretable models. They allow users to identify what are the most important variables for predicting the outcome (substantive importance, not statistical significance), what means they can be applied to any problem in which linear regression could be used.

Lecture + Lab

• How to evaluate model performance and predictive accuracy with continuous outcomes
• Tree-based methods
  • Boosting
  • Random forest

Mandatory readings

James, G., Witten, D., Hastie, T., Tibshirani, R. 2013
An Introduction to Statistical Learning with Applications in R
New York: Springer, Ch. 9

Other flexible methods for supervised learning.

We introduce variable selection models, which are especially useful for cases of many more variables than observations (p > N), common in text analysis.

We also explain and demonstrate deep learning, the state of the art tool in artificial intelligence today.

Lecture + Lab

• Further supervised methods
• Variable selection methods
  • Lasso, ridge, and elastic nets
• Deep Learning
• Ensembles

Mandatory readings

James, G., Witten, D., Hastie, T., Tibshirani, R. 2013
An Introduction to Statistical Learning with Applications in R
New York: Springer, p. 214–228

Chollet, F., and Allaire, J. J. 2017
Deep Learning with R
Manning, Ch 1: What Is Deep Learning?

These are cases where we know the characteristics of our observations in independent variables, but not on the dependent, so it is not possible to train the model on known outcome values.

We discuss and apply clustering methods, which identify groups of observations in the data given the independent variables, outlier detection, and latent variable models for dimensionality reduction – here, familiarity with Principal Component Analysis and Structural Equation Modelling is an advantage, but not a necessity.

We also cover topic models, a class of unsupervised learning for text data.

Lecture + Lab

• Clustering
  • K-means
  • Hierarchical clustering
  • Model-based and partial clustering
• Dimensionality Reduction
  • Principal Components Analysis
• Topic Models

Mandatory readings

James, G., Witten, D., Hastie, T., Tibshirani, R. 2013
An Introduction to Statistical Learning with Applications in R
New York: Springer, Ch.10, p.373–400

Conway, D., and White, J. M. 2012
Machine Learning for Hackers
Sebastopol, CA
O’Reilly, Ch. 8. p.205–214

Cranmer, S. J., and Desmarais, B. 2017. “What Can We Learn from Predictive Modeling” Political Analysis 25(2): 145-166

James, G., Witten, D., Hastie, T., Tibshirani, R. 2013. An Introduction to Statistical Learning with Applications in R. New York: Springer, Ch. 2 and 5.

James, G., Witten, D., Hastie, T., Tibshirani, R. 2013. An Introduction to Statistical Learning with Applications in R. New York: Springer, Ch. 8.

James, G., Witten, D., Hastie, T., Tibshirani, R. 2013. An Introduction to Statistical Learning with Applications in R. New York: Springer, Ch. 9.

James, G., Witten, D., Hastie, T., Tibshirani, R. 2013. An Introduction to Statistical Learning with Applications in R. New York: Springer, p. 214-228.

Hastie, T., Tibshirani, R., Friedman, J. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2 ed. New York: Springer, Ch. 11, 16.

Chollet, F., and Allaire, J. J. 2017. Deep Learning with R. Manning, Ch 1 (What Is Deep Learning?).

James, G., Witten, D., Hastie, T., Tibshirani, R. 2013. An Introduction to Statistical Learning with Applications in R. New York: Springer, Ch. 10 (373-400).

Conway, D., and White, J. M. 2012. Machine Learning for Hackers. Sebastopol, CA: O’Reilly, Ch. 8 (p. 205-214).