Survey Experiments and Pre-Analysis Planning

Survey experiments have become one of the most widely used tools in social science research. They are flexible, cost‑effective, and well‑suited to testing causal claims across a wide range of substantive questions. At the same time, the credibility of experimental findings depends heavily on transparent research practices—in particular, pre-registering hypotheses and analysis plans before data collection begins. This course addresses both the design and the documentation side of experimental research.

The course is structured around the OSF Pre‑Analysis Plan (PAP) framework. Each session introduces new methodological content and immediately applies it to your own research questions. You will arrive on Day 1 with a rough idea and complete the final online session with a near‑complete PAP that is, in principle, ready to register.

Key topics covered

Day 1 (in person) Open science and research design foundations

We discuss the replication crisis, questionable research practices, and the logic of preregistration. You will formulate research questions, develop hypotheses, and complete the initial sections of their PAP on OSF.

Day 2 (in person) Causal inference and experimental design

We cover counterfactuals, randomisation, and the full landscape of design options— between- and within-subjects, factorial, vignette, conjoint, field, and survey experiments — along with validity, ethics, and sampling. You will complete the design and sampling sections of their PAP. Between Days 2 and 3, you need to prepare a draft list of outcome measures drawing on established surveys (e.g., ESS, WVS).

Day 3 (in person) Measurement and survey design

We go deep on item wording, scale construction, attention and manipulation checks, and survey flow. You will complete the variables and analysis‑plan sections of their PAP. By the end of Day 3, the PAP is in principle ready to register, though you are encouraged to refine it based on later feedback.

Days 4–5 (online) PAP presentations and peer feedback

Each participant presents their design (approximately 10 minutes), followed by structured feedback from peers and the instructor. The course concludes with a discussion of next steps: how to move from a completed PAP to fielding a study, and emerging opportunities in experimental research including platforms for digital experimentation.

Throughout, the course pays particular attention to participants’ own substantive interests. Depending on the group, special attention can be paid to survey experiments in digital environments.

Assessment

Active participation in all sessions and completion of the two deliverables below are required to pass the course and obtain 3 ECTS credits.

1. Written Pre‑Analysis Plan (PAP)

A Pre-Analysis Plan submitted between Day 3 and the online sessions. The PAP covers the research question, hypotheses, experimental design, sampling strategy, outcome operationalisation, and planned analyses. The bulk of this document is developed during class sessions; the submission represents a finalisation and write-up of that in-class work.

2. PAP Presentation

A short structured presentation on Day 4 or Day 5, followed by active participation in peer feedback.

Optional post‑course assignment (+1 ECTS)

Students who wish to receive a fourth ECTS credit may submit a revised and finalised PAP two to three weeks after the course ends. The revision should incorporate feedback received during the online sessions and result in a polished, near-registration-ready document. This assignment equates to approximately 25 hours of additional work and will be graded (A–D-, with D- as the passing threshold).

How the course will work in-person and online

The course is structured into five live sessions, each lasting three hours. The first three sessions will take place from Tuesday 8 – Thursday 10 September at Jagiellonian University. The remaining two sessions will take place on Monday 14 and Tuesday 15 September, online. You must attend all sessions to complete the course.

The instructor will also conduct Q&A sessions and offer designated office hours for one-to-one consultations.

Prerequisite Knowledge

The course is designed at an introductory to intermediate level. It does not require prior experience with experimental methods, but you should have a working knowledge of quantitative research methods and basic statistical analysis.

The course does not include data analysis (you will not run models or work with data); however, you need sufficient statistical background to specify your planned analyses in your Pre-Analysis Plan (e.g., knowing which models you intend to run and why). Familiarity with basic research design is also expected. Familiarity with basic research design is also expected.

You should arrive with a research question you are genuinely interested in pursuing. You will be asked to submit a short paragraph (around 150 words) describing your question and interests before the course begins, so the instructor can tailor examples and in-class feedback to the group’s needs.

Course engagement includes:

• 15 hours of live teaching
• 6 hours of preparatory reading before the course begins
• Daily in-class work on the Pre-Analysis Plan
• Approximately 3 hours for finalisation and write-up of the PAP between Day 3 and the online sessions

An optional post-course assignment is available for those who wish to obtain 1 additional ECTS credit (4 ECTS total). For full details, see the Assessment section under the In‑Depth tab.

Required Readings

You are expected to have read the following before the course begins. These texts provide the theoretical and methodological background that the sessions will build on and will not be taught from scratch in class.

• Green, D. P. (2022). Social Science Experiments. Cambridge University Press. (Chapters 2, 4, 5 + 6)
• Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
• Gaines, B. J., Kuklinski, J. H., and Quirk, P. J. (2007). The logic of the survey experiment reexamined. Political Analysis, 15, 1–20.
• Schuldt, J. P., Konrath, S. H., and Schwarz, N. (2011). ‘Global warming’ or ‘climate change’?: Whether the planet is warming depends on question wording. Public Opinion Quarterly, 75, 115–124.

Learning commitment

You will engage in a variety of activities designed to deepen your understanding of the subject. Live teaching sessions form the core of your learning experience, but the learning commitment will extend beyond these. This ensures that you engage deeply with the course material, participate actively, and complete assessments to solidify your learning.

If you have registered and paid for the course, you will be given access to our Learning Management System (LMS) approximately two weeks before the course start date. Here, you can access course materials such as pre-course readings. 

Disclaimer

This course description may be subject to subsequent adaptations (e.g. taking into account new developments in the field, participant demands, group size, etc.). Registered participants will be informed at the time of change.

By registering for this course, you confirm that you possess the knowledge required to follow it. The instructor will not teach these prerequisite items. If in doubt, please contact us before registering.