ECPR

Install the app

Install this application on your home screen for quick and easy access when you’re on the go.

Just tap Share then “Add to Home Screen”

ECPR

Install the app

Install this application on your home screen for quick and easy access when you’re on the go.

Just tap Share then “Add to Home Screen”

Your subscription could not be saved. Please try again.
Your subscription to the ECPR Methods School offers and updates newsletter has been successful.

Discover ECPR's Latest Methods Course Offerings

We use Brevo as our email marketing platform. By clicking below to submit this form, you acknowledge that the information you provided will be transferred to Brevo for processing in accordance with their terms of use.

Survey Design and the Online Survey Lifecycle

Course Dates and Times

Monday 6 to Friday 10 March 2017
Generally classes are either 09:00-12:30 or 14:00-17:30
15 hours over 5 days

Kathrin Thomas

kathrin.thomas@abdn.ac.uk

University of Aberdeen

In this course the complete online survey lifecycle is covered. We discuss design and implementation of surveys from the initial planning phase to the data preparation as a final step. The course is taught from a Total Survey Error perspective weighing up data quality at each step of the process against associated costs. First, we look at the survey mode choices that should be made, how different ways to sample respondents follow from that choice, and think about survey nonresponse. On the second day we cover measurement of survey constructs focusing on the actual survey content. On the third and fourth day we focus on the design of online surveys using the obtained knowledge of the previous two days. Finally, we cover mixed-mode survey designs; what happens if different survey modes such as telephone and web surveys are combined in one design and how can we maintain high quality data. The course will be applicable to surveys of individuals, households, and organisations.


Instructor Bio

Kathrin Thomas teaches at the University of Aberdeen. 

Previously, she was Senior Research Specialist in the Department of Politics at Princeton University, where she worked on the Arab Barometer.

Her main research interests are survey research methodology as well as social and political behaviour. Kathrin's current research includes measuring sensitive behaviour and attitudes using Randomised Response Techniques and other experimental designs in surveys, interviewer effects, and other methodological issues in survey practice.

Kathrin has extensive experience with survey data design, management, and analysis from her work on the Arab Barometer; the European Social Survey; the Austrian National Election Study; the Comparative Study of Electoral Systems; and applied survey practice at Kantarpublic (former TNS BMRB).

Why take a course on (online) survey design?

Surveys are everywhere and especially web surveys are a popular method for data collection. Within the social sciences, the majority of empirical studies rely on surveys to collect data on demographics, attitudes and behaviour. Setting up a survey may seem to be a relatively simple process. Everyone can ask questions! In practice however, conducting a survey often turns out be hugely complicated, for various reasons. First of all, the number of choices for the basic design of your survey can seem overwhelming. You have to think about the choice of survey mode, obtaining a good sample, limiting nonresponse, asking good questions and analysing data, all within time and costs constraints. To make things even more complicated, each individual design choice affects other aspects of the survey design. For example, choosing to do a survey online is generally cheap and quick, but it will be hard to obtain a representative sample, and some questions are hard to ask online. Moreover, the right choice of a survey design depends on your study population and your research question.

This course introduces you to (online) survey design. We will discuss the various stages that you encounter in doing a survey, and will evaluate the trade-offs between different design choices you may face. We do this from the perspective of Total Survey Error. The overall goal is to limit the overall error of your survey, in order to enable you to give the best answer possible to your research question.

Focus of the course

The course aims to give an overview of the survey design and survey process from a Total Survey Error perspective especially focusing on the online survey lifecycle. Web surveys are increasingly used due to time and cost efficiencies, the ever-improving technological possibilities, and the increase in Internet use.

What will not be covered

In this course, we will not cover the analysis of survey data. That is, how to analyse your survey data once you have collected them. For this, you will need to take (or have taken) a general statistics course for social scientists. The course will also not cover qualitative interviews. Our course focuses on doing a survey with structured, closed–ended questions. Finally, mixed-mode surveys are discussed (internet, face-to-face, telephone and mail), but only at a basic level.

How will the course work?

The instructor will give interactive lectures introducing the topics of the day. There will be ample room for discussion, and I encourage students to contribute their own experiences and questions. There will be room for discussing specific topics you may wish to know more about. The lectures include some practical group exercises and demonstrations. In the afternoon, students are expected to finish exercises related to a problem that is linked to the materials discussed in the morning, and optionally read some more in-depth literature. On day 1, you will design your own sample and choose a survey mode. On day 2 you work on designing and evaluating your own questionnaire. On day 3 you pre-test your questionnaire with fellow course participants, and on day 4 you will use an online software tool to implement your survey. On day 5 we take a closer look at a hypothetical dataset in SPSS and focus on data screening and cleaning. I encourage students to work on their own projects during the exercises, but will provide example questionnaires and datasets for those students that do not have their own survey project (yet) to work on.

  • No previous experience in survey research is needed, however, some practical experience in conducting surveys and analysing data will be beneficial.
  • It is not necessary to have any skills on using statistical software. The course focuses on the design of surveys, not on the analysis of survey data.
Day Topic Details
1 Survey Representation

2 hours of lectures

1 hour of practical exercises

The first day sets the scene of the course. First, we discuss the strengths and weaknesses of surveys as a research design in comparison to other research methods. We introduce the Total Survey Error (TSE) framework and discuss how the survey mode affects the potential for different survey errors. Dimensions of survey mode – computer- vs. paper-based, interviewer-assisted vs. self-completion and aural vs. visual – are contrasted. Then, we dive into the availability of sampling frames and their coverage of the population. Finally, we discuss nonresponse issues.

Exercises on:

- develop a survey design (mode and sampling strategy)

- nonresponse error

4 The Online Survey Process (2)

2 hours of lectures

1 hour of practical exercises

 The Online Survey Process (2)

On the fourth day we continue with the second part of the online survey lifecycle. We discuss new online developments such as the use of mobile phones and tablets. Also differences across several online tools for developing web surveys, free and paid ones, are discussed.

Exercise on:

- Implement your survey by using an online survey tool

3 The Online Survey Process (1)

2 hours of lectures

1 hour of practical exercises

Day three starts with an outline of the online survey lifecycle from the initial planning phase to the data preparation stage as a final step. Especially we look into methods of pre-testing online survey questions, including the qualitative technique of cognitive interviewing. Finally, we discuss the advantages and disadvantages of online panel surveys.

Exercise on:

- Survey pre-testing

2 Survey Measurement

2 hours of lectures

1 hour of practical exercises

The second day looks into the psychology of survey response. We assess why operationalizing our research questions into survey questions can be intricate, which survey factors may affect responses and how different respondents differentially understand our questions. Interviewer effects on measurement are discussed here, as well as the effects of other (self-administered) survey modes on measurement error. In addition, ways of assessing measurement error in survey data are presented.  

Exercises on:

- Develop a short questionnaire taking into account the design principles covered during class or improve your own questionnaire.

- Evaluating and improving existing surveys

5 Mixed Mode Surveys

2 hours of lectures

1 hour of practical exercises

 

On this last day we cover mixed-mode surveys; designs in which multiple modes are used for contacting and /or data collection. We apply the principles of the Total Survey Error framework to mixed-mode surveys and discuss the advantages and disadvantages.

To conclude the course we look back at the TSE framework and evaluate how different costs associated with survey design decisions might affect data quality. We discuss the methods that can be used to assess the different components of the TSE framework and discuss how we may trade-off survey quality against survey costs.

Exercise on:

- Data screening and cleaning

Day Readings
1

- Groves, R.M. et al. (2009). Survey Methodology (2nd edition). New York: Wiley. Chapters: 2, 3, 4, 5 &

- Groves, R.M., & Lyberg, L. (2010). Total survey error: Past, present, and future. Public Opinion Quarterly, 74, 849-879.

2

- Dillman, D.A, et al. (2014). The Tailored Design Method (4th edition). New York: Wiley. Chapters: 4 & 6.

- Groves, R.M. et al. (2009). Survey Methodology (2nd edition). New York: Wiley. Chapter 7.

- Tourangeau, R. & Yan, T. (2007). Sensitive questions in surveys. Psychological Bulletin, 133, 859-883.

3

- Groves, R.M. et al. (2009). Survey Methodology (2nd edition). New York: Wiley. Chapter 8.

- Presser, S., Couper, M.P., Lessler, J.T., Martin, E., Martin, J., Rothgeb, J.M., and Singer, E. (2004). Methods for Testing and Evaluating Survey Questions, Public Opinion Quarterly, 68, 109-130.

- Toepoel, V. (2015). Doing Surveys Online. London: Sage Publications. Chapters 3 & 5.

4

- Lugtig, P. & Toepoel, V. (2016). The use of pcs, smartphones, and tablets in a probability-based panel survey. Effects on survey measurement error. Social Science Computer Review, 34, 78-94.

- Toepoel, V. (2015). Doing Surveys Online. London: Sage Publications. Chapters 8 & 9.

5

- Dillman, D.A, et al. (2014). The Tailored Design Method (4th edition). New York: Wiley. Chapter: 11

- Groves, R.M. et al. (2009). Survey Methodology (2nd edition). New York: Wiley. Chapter 10.

- de Leeuw, E.D. (2005) To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21, 233-255.

Software Requirements

SPSS (version 18 of higher).

Hardware Requirements

None

Literature

None, see ‘Readings’ above.