How many responses do you actually need? Sample size and margin of error

"How many people do we need?" Every survey opens with that question, and most teams answer it worst of all. The honest answer is a short calculation, not a round number someone half-remembers from a meeting. Get it right and you stop collecting the moment the data is precise enough to act on. Get it wrong and you either quit early and read noise as signal, or you chase thousands of extra responses that shift the answer by a fraction of a point.

More responses buys precision, not the truth

Sample size controls exactly one thing: how much your estimate would wobble if you drew a different random sample of the same size. Statisticians call that random sampling error. It shrinks as the sample grows, and it is all that more responses will fix.

Bias sits there untouched. If the wrong people answer, a bigger sample just measures the wrong people more precisely. Take a satisfaction survey that only the delighted and the furious bother to finish: it does not improve at 4,000 responses, it just reports the same skewed number with a tighter interval drawn around it. The guide to who answers your survey and who doesn't covers whether you have the right people at all. Read it first, because this piece only covers how many of them you need.

What the margin of error actually measures

The margin of error is the half-width of the confidence interval around a percentage. A poll says 52%, plus or minus 3 points, at 95% confidence. Here is what that sentence actually claims: repeat the same probability sample many times and about 19 in 20 of the intervals it produces, 49% to 55% in this case, would contain the true population value.

Two warnings come with it. It says nothing about whether your one interval happens to catch the truth; it describes how the method behaves over many samples. And it only covers sampling error. A leading question, a broken frame, the people who never replied, none of that lands anywhere in the number. Pew Research Center spells this out in its work on the variability of survey estimates: the figure captures noise from sampling and leaves the bias from everything else alone. So quote a margin of error only when you genuinely drew a probability sample.

The three dials: confidence, margin, and the split

Three inputs decide the number, and you fix all three before a single invitation goes out.

Confidence level is how often you want the method to land inside its interval over the long run. The default is 95%, which sets the multiplier, z, to 1.96. Drop to 90% and z falls to 1.645, which costs you fewer responses. Push to 99% and it climbs to 2.576, which costs a great many more.

Margin of error is the precision you can act on. Plus or minus 5 points is fine for a rough read; plus or minus 2 is expensive and, for most internal work, overkill. Halve the margin and you roughly quadruple the sample, so this dial drives the cost harder than any other.

The split is the proportion you expect to find, written p. Precision is worst when opinion divides evenly, at p = 0.5, because that is where the variance p(1-p) peaks. You rarely know the answer before you ask, so plan at 0.5. It hands you the largest sample any single proportion could demand, which means you are covered whatever comes back.

The formula, worked once

For a proportion drawn from a large population, here is the sample size you need:

n0 = z^2 * p * (1 - p) / e^2

z = 1.96   (95% confidence)
p = 0.5    (worst-case split)
e = 0.03   (target margin, 3 points)

n0 = 1.96^2 * 0.5 * 0.5 / 0.03^2
   = 3.8416 * 0.25 / 0.0009
   = 1067

Always round up, never down, because half a person does not exist and rounding down quietly breaks the guarantee. Run the same formula at a few other margins:

e = 0.05  ->  n0 = 385
e = 0.03  ->  n0 = 1,068
e = 0.02  ->  n0 = 2,401
e = 0.01  ->  n0 = 9,604

The diminishing returns jump straight out. Tightening from plus or minus 5 to plus or minus 3 costs about 680 responses. Tightening from plus or minus 3 to plus or minus 1 costs another 8,500. Past a point, precision gets very expensive very fast.

That 1,068 at plus or minus 3 is why so many national polls land near 1,000 respondents; Pew's own table puts it at 1,067 cases. It is where precision becomes good enough for most decisions and the next increment costs far more than it returns. So 1,000 makes a fine default for one headline number about one population, and nobody's universal answer: start comparing groups or tighten the margin and the right number moves.

The confidence level costs more than you think

Turning the confidence dial up has a price. Hold the margin at plus or minus 3 points and the split at 0.5, then watch the sample climb:

90% confidence (z = 1.645)  ->  n0 = 752
95% confidence (z = 1.960)  ->  n0 = 1,068
99% confidence (z = 2.576)  ->  n0 = 1,844

Going from 95% to 99% almost doubles what you pay, for a guarantee a product or CX call rarely needs. For most internal work 95% is the right setting, and 90% is defensible when the stakes are low and you only want direction. Reserve 99% for decisions where being wrong is genuinely costly.

Small populations: the correction that saves you responses

Every number so far assumes the population is effectively infinite. When it is small, known, and every member is reachable, you need fewer responses than the basic formula suggests, because each response tells you proportionally more about a small group than a large one. The finite population correction handles that.

n = n0 / (1 + (n0 - 1) / N)

Target: plus or minus 5 points, so n0 = 385

Population N = 2,000  ->  n = 385 / (1 + 384/2000)  = 324
Population N = 500    ->  n = 385 / (1 + 384/500)   = 218

This comes up constantly in real surveys. An employee survey at a 500-person company does not need 385 responses for a plus-or-minus-5 read. It needs about 218. Apply the correction and a target that looked out of reach comes back down to something you can hit. Skip it and you set a goal you cannot meet, then read the shortfall as failure when the design was fine.

The subgroup trap

This is where most sample plans quietly fall apart. The margin of error you calculated belongs to the group whose responses you count. Split the total into segments and every segment carries its own margin, wider, driven by its own smaller count.

Picture a survey with 1,000 responses and a tidy plus-or-minus-3 headline. The region you most want to compare holds 40 people, and 40 responses carry a margin near plus or minus 15 points, so any gap you spot between two segments like that is mostly noise. Work out which comparisons matter before you send, size the smallest cell to a margin you can act on, and let the total fall out of that. Sizing the total first and slicing afterwards is the most common way a well-run survey ends up unable to answer its own question.

Weighting and clustering: the design effect

The clean formula assumes a simple random sample where every response counts the same. Real samples seldom oblige. The gap has a name, the design effect, written deff: the factor by which your variance beats that of a simple random sample the same size, and it inflates how many responses you actually have to collect.

Weighting is the usual reason. Weight responses to fix an unbalanced sample and you raise the variance of your estimates, because more of the adjustment rests on fewer people. Pew notes that piling on weighting variables tends to widen the margin of error. A design effect of 1.5 means 50% more completed responses to reach the precision the simple formula promised. If you already know you will weight hard, plan for a deff between 1.3 and 2.0 and multiply your target by it. The guide to weighting survey data covers when that correction earns its keep.

From completes to invitations: work backwards

The number you calculated is completed responses, not invitations sent. To reach it, divide by the rate at which invitations turn into completes, and that rate is often brutally low. This is where the plan meets reality.

completes needed = 1,068
expected completion rate = 8%

invitations to send = 1,068 / 0.08 = 13,350

List smaller than that? You have three levers. Accept a wider margin. Apply the finite population correction, if the population is genuinely closed. Or lift the completion rate, which is the honest one. A shorter survey gets finished more often, as the guide to survey length and drop-off lays out, and the invitation has to reach the inbox first, which the guide to delivering survey invitations works through. An invitation sitting in spam is a non-response you caused, and it counts against this arithmetic in full.

Estimating a number is not the same as detecting a difference

Sizing a sample to estimate one proportion is the easy version. Sizing one to detect a difference between two groups is a different animal, and mixing the two up burns fielding budgets all the time.

Say you want to be confident that 55% really differs from 50%, a five-point gap. That takes roughly 1,500 completed responses in each group, not 1,000 across the whole survey. Smaller gaps cost far more; a two-point difference runs into the thousands per group. So settle up front on the smallest difference that would change what you do, the minimum detectable effect. If a three-point move would not change a single decision, do not pay to measure it that finely. It is why underpowered tracker surveys report no significant change quarter after quarter: they were never sized to catch a change worth acting on.

Put the number last, not first

One caution to close on. Sample size buys precision against random sampling error and nothing else. A tight margin under a self-selected web poll is theatre, a narrow interval sitting dead centre on the wrong number. So settle the population, confirm the frame reaches it, write clean questions, and only then size the sample to the decision. Work a real case backwards from the hardest requirement: if you run CX for 40,000 customers and want to compare three plan tiers at plus or minus 4 points, the smallest tier sets the floor, a light weighting adds a design effect, and a 10% completion rate turns your target into the pile of invitations you actually send. Then write the limitations note before any data arrives, so a decent number never gets oversold later.