Weighting survey data: correcting for who actually answered
A high response count still leaves you with the wrong mix of people. Weighting corrects the imbalance after collection. Here is how post-stratification and raking work, what they cost you in precision, and the bias they cannot fix.
By SurveyLane · The team building SurveyLane
You collected a thousand responses and the mix is wrong. Two-thirds are women, the under-30s barely showed up, and one region answered at twice the rate of the rest. The data is real, but it does not look like the population you meant to describe. Weighting is where you correct that imbalance after the fact, by making some responses count for more than others. Done well, it pulls a skewed sample toward the truth. Done carelessly, it hands you false confidence in numbers shakier than they look.
Weighting is what you do after sampling goes sideways
Weighting is not a first move. It is a repair. The earlier you get the right people into your sample, the less you have to lean on it, which is the whole argument in sampling quality: who fills in your survey and who doesn't. But you will almost never get a perfectly balanced sample, because some groups always answer more readily than others. Weighting exists for that gap between the sample you wanted and the one you got.
If a group is underrepresented among your respondents, you let each of their answers stand in for more people. If a group is overrepresented, you dial each answer down. The weighted totals then reflect how the population is built rather than how your respondents happened to break down.
What a weight actually is
A weight is a multiplier on each response that says how many people in the population that one respondent stands for. In an unweighted survey every response counts the same, and weighting breaks that on purpose. A respondent from an underrepresented group might get a weight of 1.8, and one from an overrepresented group a weight of 0.6.
Say 18-to-29-year-olds are 20% of your population but only 10% of your sample. To fix it you give each young respondent a weight of roughly two, so that as a block they carry the 20% share they should have. Every mean, percentage and cross-tab you compute afterward uses those weights, so the young respondents pull twice their headcount.
Design weights versus adjustment weights
Two different reasons can make a response deserve a different weight, and it helps to keep them apart. Design weights come from the sampling itself. If you deliberately oversampled a small group so you would have enough of them to analyze, you knew their selection probability was higher, and the design weight is just the inverse of that probability.
Adjustment weights are the after-the-fact kind, and they are the ones most teams mean when they say weighting. You did not plan the imbalance. Nonresponse handed it to you. So you nudge the sample to match known population figures on age, gender, region, education, whatever you have good targets for. Design weights correct for how you sampled. Adjustment weights correct for who bothered to answer. A serious analysis often uses both.
Post-stratification: the simplest correction
Post-stratification is the most intuitive adjustment. You split the population into strata, cells like "women, 30-44, north," work out what share each cell should be, and reweight your respondents so each cell hits its target share. If your data matches the population on the variables that drive your outcome, this alone can remove a lot of bias.
The weakness is the cell grid. Post-stratification needs the full joint distribution, so you have to know the population share of every combination of every variable at once. Cross three or four variables and you get dozens of cells, some with a handful of respondents or none at all. An empty cell cannot be weighted, and a cell with two people gives those two enormous, jumpy weights. So it stays clean on one or two variables and falls apart when you want many.
Raking when you can't fill every cell
Raking is the workaround, and it is the method most large surveys actually run. Instead of the full joint distribution it needs only the margins: the overall age split, the overall gender split, the overall regional split, each on its own. It adjusts the weights to match one margin, then the next, then the next, looping back until every margin lines up at once. The technical name is iterative proportional fitting.
Pew Research Center's methods team, in a January 2018 study by Andrew Mercer, Arnold Lau and Courtney Kennedy, calls raking the most common approach precisely because it asks so little of you. You bring marginal proportions for each variable, which you can usually get, rather than a full cross-tabulation, which you usually cannot. The trade is that raking matches each margin without guaranteeing any particular joint cell is right.
Raking, in one loop:
start every weight at 1
repeat until stable:
scale weights so the AGE totals match the population
scale weights so the GENDER totals match the population
scale weights so the REGION totals match the population
(each pass nudges the others; a few passes and they all settle)
Where your population targets come from
Weighting is only ever as good as the targets you weight to, and every method above assumes you know the true population shares. Get those from an authoritative source, a national census or official statistics or a well-run reference survey, and never from a guess or a competitor's marketing deck. If your targets are stale or wrong, weighting does not fix your data. It drags it confidently toward the wrong answer.
This also decides which variables you can weight on at all. You can only correct for a variable if you both measured it in your survey and have a trustworthy population figure for it. That is why age, gender, region and education are the usual suspects. If the thing skewing your sample is something you have no population target for, you cannot weight on it.
The bias weighting cannot touch
Weighting corrects the variables you weight on and only those. If the people who ignored your survey differ from the people who answered in a way your weighting variables do not capture, weighting leaves that bias fully intact. Matching the age and gender split does nothing about the fact that, say, satisfied customers answered and angry ones deleted the email.
The Pew study makes this concrete. Demographic weighting alone reduced bias only slightly, and the real gains came from adding variables tied more closely to what the survey measured. So weighting is no substitute for getting responses from the right people, a point that runs back through getting your survey invitation into the inbox to sampling itself. If nonresponse is tangled up with your actual topic, no reweighting on demographics will save you.
The price of weighting: variance and effective sample size
Weighting has a cost: precision. When responses carry unequal weights, your estimates bounce around more than the raw sample size suggests, because a few heavily weighted respondents sway the result. The design effect is the factor by which weighting inflates the variance of your estimates. A design effect of 1.5 means your weighted sample is as precise as a simple random sample one and a half times smaller.
That shrinkage has a name: the effective sample size, which is always smaller than your real count.
effective sample size n_eff = (sum of weights)^2 / (sum of squared weights)
Collect 1,000 responses and weight them hard, and your effective sample size might be 650. Your margins of error should come from that 650, not the 1,000 on the box. Report the 1,000 as if it were clean and you overstate your own precision.
Trimming extreme weights
The cure for runaway variance is trimming. A single respondent from a tiny, underrepresented cell can end up with a weight of 15, so one person speaks for fifteen. If they answered oddly, they yank the whole estimate along. Trimming caps weights at a ceiling. Pull everything above, say, 5 back down to 5, and no single response can dominate.
Trimming is a deliberate trade. Capping a weight means that cell no longer perfectly matches its population target, so you accept a sliver of bias to buy back a chunk of stability. That is often the right call: a slightly biased but steady estimate beats one that swings wildly with one respondent. Decide the ceiling before you look at the results, so the choice is a rule rather than a nudge toward the answer you were hoping for.
Weighting opt-in and online samples
Opt-in and online panels are harder than a probability sample that merely came back unbalanced, because the people who joined were never randomly selected, so the imbalance can run deeper than demographics. The Pew study tested two heavier tools for this. Propensity weighting models each respondent's probability of being in the sample versus the population and weights on that, keeping every case but risking wild, highly variable weights. Matching pairs each respondent with a similar member of a reference sample and throws away the ones with no match, which shrinks your usable data.
Neither is magic, and the honest reading of that research is that method matters less than variables. Adding adjustment variables genuinely related to the outcome helped far more than any clever algorithm run on demographics alone. If you work with opt-in samples, spend your effort finding and measuring variables that predict what you care about, then weight on those.
Weighting without fooling yourself
Treat weighting as a documented decision, not a switch you flip at the end. Write down the variables, the population targets and their source, the method, and any trimming rule, all before you see how the weights move your headline number. Then report both the weighted and unweighted results. If they agree, weighting was a light touch. If they diverge sharply, that gap is a finding in itself: your sample was badly skewed on something that matters, and readers deserve to know how much of your conclusion rests on the weights.
Weighting sits next to screening out bad responses, covered in spotting low-quality survey answers, and it pairs with the judgment you already apply to analyzing open-ended answers. If you explore your data conversationally through analyzing surveys with AI the safe way, make sure the weights travel with it, so an AI summary reflects the population rather than your raw respondent pool.
Frequently asked questions
When should I weight my survey data?
Weight when your respondents differ from the population on variables that matter to your conclusions and you have trustworthy population targets for those variables. If your sample already matches the population closely, weighting will barely move your numbers and mostly adds variance, so it is not worth it. The stronger the skew on something that drives your outcome, the more weighting earns its place, and a large gap between the weighted and unweighted result tells you how much the imbalance mattered.
What is the difference between post-stratification and raking?
Post-stratification needs the full joint distribution of your weighting variables, meaning the population share of every combination of categories, and it reweights each of those cells to its target. Raking needs only the separate margins, the overall split for each variable on its own, and it loops through them until they all match at once. Use post-stratification when you weight on just one or two variables and have complete cell data. Use raking when you have several variables and can only get their individual distributions.
Can weighting fix a biased sample?
Only partly, and only on the variables you weight on. Weighting rebalances the traits you measured and have targets for, such as age or region, but it does nothing about differences it cannot see. If the people who skipped your survey differ from the people who answered in a way unrelated to your weighting variables, that bias survives weighting completely. It is a correction for imbalance, not a replacement for reaching the right people.
Why does weighting reduce my effective sample size?
Unequal weights make your estimates lean more heavily on a few respondents, which raises their variance. The effective sample size is the size a simple random sample would need to be to match that precision, and it is always smaller than your raw count when weights vary. A thousand responses with heavy weights might behave like six or seven hundred, so compute margins of error from the effective size rather than the headline number.
