Before and after: how to tell from a survey whether something actually changed
You ran a survey before the training and one after, and the score went up. That rise is not yet an effect. Here is what else moves a before-and-after number, and how to design a pre-post survey that can tell them apart.
By SurveyLane · The team building SurveyLane
You send a survey before a leadership training and the same survey six weeks later. Confidence in giving feedback goes from 3.1 to 3.7 on a five-point scale, and the slide for the steering group writes itself. Careful, though. That 0.6 is the gap between two measurements. Several other things push the same number up or down, and a pre-post design only tells you what the training did if you build it to pull those apart.
A difference is not an effect
Donald Campbell and Julian Stanley listed the usual suspects in 1963, in Experimental and Quasi-Experimental Designs for Research, and methods textbooks still teach their threats to internal validity. Four of them turn up in almost every survey you run twice.
History is whatever else happened between the waves. A reorganisation, say, or a new manager. Maturation is change that would have come anyway, because people get better at a job they started three months ago. Testing is the effect of the first questionnaire: twenty questions about feedback make people think about feedback. Instrumentation is any change to the measuring tool, which for a survey means wording, scale, order or channel.
None of these gets its own column in your export. They all end up inside the same 0.6.
Regression to the mean picks your worst scores for you
Say you give a programme to the five teams with the lowest engagement scores. Next quarter four of them have improved. Part of that was guaranteed.
Every score mixes a stable part with noise: a bad week, a conflict that has since blown over. Pick teams for an extreme score and you pick teams whose noise was unusually bad that time. Measure again and the noise is as likely to go the other way, so the average drifts back to the middle. Francis Galton described this in the 1880s. HR and CX teams still fall for it every quarter.
If you have to select on a low score, compare against similar low scorers who got nothing. They will improve too, and their improvement is your baseline.
People move their own ruler too
Before a course on feedback, plenty of people rate their own skill as fairly good, because they don't yet know what good feedback looks like. Afterwards they do, and they judge themselves against a stricter standard. Their skill went up, but so did the bar, so the self-rating moves less than the skill. It can even drop.
George Howard and Patrick Dailey called this response-shift bias. In a 1979 paper in the Journal of Applied Psychology (volume 64, pages 144 to 150) they evaluated a training programme for interviewing skills and found that the shift in standards affected how trainees estimated the effect of the training. So for a programme that teaches people what competence looks like, a plain self-report comparison tends to underestimate the change. A flat result doesn't prove the programme did nothing.
Questions about behaviour ("How many one-to-ones did you hold last month?") have no bar that moves this way, so include a few. The post on questions about behaviour and recall explains how to write them.
The retrospective pretest, and what it costs you
One fix is to ask about "before" afterwards. The after survey asks each item twice: "How would you rate your ability to give critical feedback now?" and "Looking back, how would you have rated it before the training?" Both answers use the same updated bar.
Howard and five colleagues tested this retrospective pretest in five studies, published in Applied Psychological Measurement in 1979 (volume 3, pages 1 to 23), and proposed it as a more accurate way to measure change in self-reports.
It has its own costs. Putting "before" and "now" side by side signals that a difference is expected, and some people will supply one. Someone who spent two days on a course wants to believe it helped. Laura Griner Hill and Drew Betz revisited the method in the American Journal of Evaluation in 2005 (volume 26, issue 4, pages 501 to 517). Their advice: use a traditional pretest to judge a programme's effect, and a retrospective pretest to learn how participants experienced their own change. If you can, ask both. Where they disagree, people's understanding of the topic shifted.
Keep the instrument identical
Instrumentation is the threat you cause yourself. Someone tidies a question between waves, swaps a five-point scale for a seven-point one, or moves a block forward. Each change moves the answers, and you can't tell that movement from the effect.
Freeze the pre-survey once the first response arrives: same wording, same options in the same order, same question order, same channel. Email on a laptop before and a QR code on a phone after is a change of mode, and mode alone moves answers. See repeating a survey without breaking your trend and question order effects. Fix the wording before wave one, with the methods in testing a survey before you send it.
Matching people without breaking anonymity
You can compare everyone who answered before with everyone who answered after. Or you can link each person's two answers and look at change per person. The second is much stronger: it takes differences between people out of the comparison, so a smaller real change shows up with the same number of respondents.
Linking needs a key, and that collides with any anonymity you promised. The post on anonymity and confidentiality explains why breaking that promise costs you honest answers.
A middle path is a self-generated code. At the start of both surveys, ask for things only the respondent knows, such as the first two letters of their mother's first name and the day of the month they were born. "MA14" links the waves without a name. Some codes won't match because someone typed them differently, so report how many pairs you linked. And in a team of six, a code combined with department and role can still point at one person. Decide in advance which breakdowns you won't report.
Who answered both times
The second wave is rarely the same people as the first. People leave or drop out halfway. Others just never get round to it. The people who found the course useless are the least likely to open a survey about it, so the after group often looks more positive even when nobody changed their mind. Campbell and Stanley called this experimental mortality; the post on sample quality covers non-response more generally.
Two checks catch most of it. Compare the before scores of people who answered both waves with those who answered only the first. If the drop-outs started lower, your after average flatters the programme. And report change for matched pairs next to the raw averages.
A comparison group is worth more than a perfect question
The biggest improvement has nothing to do with wording. Measure a group that didn't get the intervention at the same moments. History and maturation then hit both groups, and so does testing, and the difference between their changes is a far better estimate of the effect.
Inside a company this is often easy. Roll the training out in two cohorts and survey both at the same times; the second cohort is your comparison until its turn comes.
Say the trained group goes from 3.1 to 3.7 and the waiting group from 3.2 to 3.5 over the same six weeks. The training added about 0.3. The other 0.3 came from something else, the season or the new performance cycle. The groups needn't start out equal, as long as you compare changes. But a pilot team that volunteered is a poor match for the teams that didn't.
When to send the second wave
A questionnaire at the end of the last session captures how people feel about the day. Energised, usually, and a little glad it's over. How they work hasn't had a chance to change yet.
For a skill used weekly, six to eight weeks is a fair first guess. For something done once a quarter, wait a quarter. Ask about concrete recent behaviour. "In the past two weeks, how many times did you give a colleague critical feedback face to face?" is harder to inflate than "I feel confident giving feedback."
How big a change you can see
Forty people will catch a large change and miss a modest one. If you hope for a quarter of a point on a five-point scale, check first whether your group can show it. Otherwise "no significant difference" mostly means "not enough people".
Matching helps, because change per person varies far less than the scores do. The post on how many respondents you need explains margin of error and subgroups.
And don't test thirty items separately and celebrate the two that came out significant. A couple will cross the line by chance. Pick the two or three outcomes the programme is meant to move before you start.
Reporting change honestly
Give responses per wave and the number of matched pairs. Show distributions, because a programme that lifts the bottom quarter looks nothing like one that nudges everybody a little. Put a comparison group's change beside the treated group's.
Then say what the design can carry. "Scores of people who answered both times rose by 0.4; a comparison group rose by 0.1" is a claim you can defend. "The training raised confidence by 15 percent" usually isn't. More in reporting survey results.
Setting up a pre-post survey in SurveyLane
Build and test the pre-survey, then copy it for the after wave so wording and order stay identical. Put the self-generated code at the start of both. Add the retrospective items to the after survey only, as their own block.
Use conditional logic to hide programme-specific items from the comparison group. Once results are in, the read-only MCP server lets an AI assistant link the waves on the code and compare change per group, as in analysing surveys with AI. Check its pair count against your own.
Frequently asked questions
Is a retrospective pretest better than a normal pretest?
Neither wins in general. A normal pretest is the safer basis for judging a programme's effect, because people answer it before they know what the programme teaches. A retrospective pretest avoids the moving bar but invites people to report the improvement they expect. Use both if you can.
Can I compare before and after if the survey was anonymous?
Yes, at group level, or per person if you add a self-generated code. A group-level comparison can't separate a change in people from a change in who responded, so compare the before scores of those who answered once with those who answered twice.
How long should I wait before sending the after survey?
Long enough for the behaviour you care about to happen a few times. For weekly skills, six to eight weeks is a reasonable start. A survey on the last day measures how people felt about the session.
What if I cannot have a comparison group?
Then make a smaller claim. Measure more than once before the intervention to see the existing trend, add behavioural questions, and don't pick participants because they scored low. Report it as a change over time. It says little about the effect of the programme.