Python Programming for Social Sciences: Collecting, Analysing and Presenting Social Media Data

Nowadays, very large and diverse kinds of data are becoming available to researchers. Online social media data in particular has great potential to provide a new approach to social science questions.

However, these kinds of data have diverse data structures, which often differ from traditional social science data. Some are provided by a structured way through application program interfaces (API): e.g. Twitter and Facebook; others could be semi-structured data such as web pages. Moreover, those data could be in different formats and the available data size much larger than before.

Although widely used software packages such as R, STATA, and Matlab are practical for statistical analysis, they are of only limited use for gathering, transforming, managing, and analysing new, massive and diversely structured types of data.

As an alternative to these packages, many scholars have begun to use Python, a popular, versatile script language with a large user community. Python has become popular because it is open source, easy to learn (even for beginners), and allows researchers to handle massive datasets quickly.

A large, rapidly expanding set of libraries helps users solve complex problems with ease. These include Tensorflow and Keras: newly developed libraries for deep learning.

Familiarity with Python language opens up new possibilities for conducting your research in a more efficient way.

The course covers:

• fundamental principles of programming and their implementation in Python language
• collecting social media data
• handling large amounts of data in diverse formats
• using Python libraries for data analysis.

The course will include concrete examples of how to collect, manage and analyse social media data, especially Twitter.

Course structure

First, I introduce the basic concepts of programming. You will learn types of data, operators, conditions, loops, functions, data structure, and objected-oriented programming.

Then you will learn how to implement the programming in Python language. I will set programming tasks for you to solve, to teach you how to program in an efficient way. 

After basic programming, I will introduce a couple of methods for obtaining social media data, such as scraping web pages, and using API – in particular Twitter’s. I will also introduce useful Python libraries for data collection, such as urllib and Beautiful Soup.

I will demonstrate several analytical methods for text data:

• how to obtain basic statistics of the collected data
• how to conduct a simple text analysis
• text classification using supervised machine learning algorithms (Naive Bayes and Support Vector Machine).

You will learn basic regular expressions to handle text data and Python libraries for the analysis such as Numpy, Pandas, NLTK, scikit-learn etc.

I will ask you to submit a small programming assignment every day. The task will be directly related to the content of the corresponding day and should take one or two hours maximum.

Required literature

Lubanovic, Bill. 2014
Introducing Python: Modern Computing in Simple Packages
O’Reilly Media

Mitchell, Ryan. 2015
Web Scraping with Python: Collecting Data from the Modern Web
O’Reilly Media

Installation and setup
Please install and configure Python and PyCharm on your laptop before the course starts, using these step-by-step instructions.

Experience in other languages
You should have some experience with basic programming/data analysis in other languages, e.g. R, Matlab, STATA. In other words, you should be able to write basic codes in the corresponding languages: e.g. assigning a value to a variable, writing for loop, if condition. 

If you do not fulfil the above requirements
If you have problems following the installation instructions, or do not have experience in other languages, take the course WA108, Basics of Programming in Python.