Reporting survey results without misleading the people who act on them

A careful survey can still end in a wrong decision, and the last step is usually where it happens. A manager reads one slide, sees "62% want feature X", and nobody asks 62% of whom. Treat the report like a questionnaire for the reader: every chart asks them something, and an ambiguous chart gets answered with whatever they already believed. The rules from writing better survey questions carry over, and this guide covers the rest.

Lead with the decision, not with question 1

Most reports follow the questionnaire, one chart per question. That order suited the respondent. The reader wants to know what to do.

Start from the decision the survey was meant to inform and put the two or three findings that bear on it on page one. "Should we move onboarding to self-service?" is a decision. What matters for it is how new customers rate the current onboarding, where they got stuck, and whether the ones who struggled are the ones you would lose. The rest goes into an appendix.

Write each finding's headline as a sentence with a number in it. "Onboarding" tells the reader nothing. "Four in ten new customers needed help to finish setup" gives them something to check and argue with. It also makes you decide what the chart shows before you show it.

Every percentage needs its base

A percentage without a base is half a number. "62% prefer the new pricing page" reads very differently when it is 62% of 1,400 customers than when it is 62% of the 29 people who saw the new page and then got a follow-up question.

Put the base under every chart, in the same spot each time, in words a reader understands: "Base: customers who used the export feature in the last 30 days, n = 312". A bare "n = 312" is not enough.

With conditional logic this gets more important. Routing means different people answered different questions, so the base moves from chart to chart. If question 8 only went to people who rated support below 5, then "71% said response time was the main problem" describes unhappy support users. It says nothing about your customer base as a whole. Readers will not work that out on their own. Write the filter into the base line every time.

Decide what to do with "don't know" before you count

People who answer "don't know" or "not applicable", or who skip the question, are still in your data. You have to decide where they go, and that decision moves the headline.

Say 500 people are asked whether the new dashboard is easier to use. 240 say yes, 110 say no, 150 have not used it enough to judge. On all 500, 48% find it easier. Drop the "don't knows" and it becomes 69%. Both numbers are correct. The first is about your users. The second is about users who had formed an opinion.

You can use either. What you cannot do is hide which one you used. Pick the base that fits the decision, say so, and show the excluded group as a number. If 30% of people could not judge the new dashboard yet, that is a finding too.

Rounding, totals and multi-select

Three answers of 33.3% each round to 33, 33 and 33, which makes 99. Someone will notice. A note fixes it: "Percentages may not total 100 because of rounding." Do not nudge a number so the column adds up.

Multi-select is the reverse: if people could tick every reason that applied, the bars total well over 100. Say so on the chart, because 180% looks like a bug. And match decimals to your sample. 47.3% from 180 responses suggests a precision you do not have.

Show the distribution, not just the average

A mean hides the shape of the answers, and the shape is often the finding. A 3.0 on a five-point scale can mean everyone picked the middle. It can also mean half picked 1 and half picked 5. One is mild indifference. The other is a split customer base that needs a very different response.

For rating scales, show the whole distribution as a stacked bar, points in their natural order, and give the top-two-box share next to the bottom-two-box share if you want one summary figure. Put the scale labels from your rating scale design on the chart too. "1" to "5" means nothing to a reader without the questionnaire.

Drop one habit: sorting the points of a scale by size. The chart looks tidy and becomes unreadable, because "somewhat agree" lands next to "strongly disagree". Sort unordered categories, like "main reason for cancelling", by size. Leave ordered scales in order.

A difference is less certain than either number

This is where reports overstate most often. Satisfaction is 64% on the annual plan and 57% on the monthly plan, and the room concludes annual customers are happier. Maybe. Each figure has its own uncertainty, and the gap between them has more.

Pew Research Center's Andrew Mercer wrote in a 2016 explainer on the margin of error: the margin for the difference between two candidates is generally about twice the margin for each candidate's share. His subgroup example is worse. A poll with a margin of about 3 points overall had about 8 points for a subgroup making up 15% of the population, and about 16 points for a difference inside that subgroup.

Now look at your own report. A seven-point gap between two segments of 200 respondents each is very likely noise. If you want to claim the gap, test the gap, and if the test does not back it up, write "no clear difference". Our guide on how many respondents you need shows how to work out precision before fieldwork. The same goes for change over time: 41 to 44 between two waves is not yet a trend.

Crosstabs and the forty-comparisons problem

Crosstabs are where reports go fishing. Split ten questions by four segments and you have forty comparisons. At the usual 95% threshold you should expect about two of them to look significant by chance, even if nothing real is going on. And those two will be the most eye-catching cells in the report, because chance is good at producing surprises.

Decide before you look which comparisons the survey was built to answer, and report those as findings. Everything else gets labelled as exploratory: worth checking in the next wave, nothing more. If a subgroup result would change a decision, confirm it with a follow-up first. And if your tool marks significant cells automatically, remember it ran every test it could.

Charts that people read correctly

In a 1984 paper in the Journal of the American Statistical Association, William Cleveland and Robert McGill tested how accurately people read quantities back from different visual encodings. Position along a common scale did best. Angles and areas did much worse. In practice, people compare bar lengths far more accurately than pie slices.

For survey results that gives a short list:

• Use horizontal bars for answer categories, so long labels stay readable.
• Start bar axes at zero. An axis that starts at 50% turns a three-point gap into what looks like a doubling.
• Use a pie or donut only for a two-way split, if at all. Five slices of similar size are guesswork.
• Give the same answer the same colour in every chart. If "very satisfied" is dark green on page 2, it is dark green on page 9.
• Print the number on the bar so nobody has to estimate from gridlines.

The aim is a chart that is hard to misread. Whether it looks good is secondary.

Quotes from open answers illustrate a theme

A good quote does more in a meeting than any chart. Pick the three sharpest complaints out of 600 open answers and the room will assume they speak for everyone.

Use a quote to show what a theme sounds like, and put a count on the theme: "About one in five open answers mentioned export formats, for example: ...". Choose quotes that are typical of the theme over ones that are extreme. If AI did or helped with the coding, check every quote you use against the raw response, because a summary can paraphrase a sentence nobody wrote. Our guide to analysing open-ended answers covers coding themes so the counts hold up.

The method note AAPOR expects

The American Association for Public Opinion Research has a Code of Professional Ethics and Practices, last revised in April 2021. Section III lists what has to be disclosed immediately in any report of publicly released survey results. It was written for public polls, but it makes a decent checklist for internal reports:

• how the data were collected, and who sponsored and conducted the research
• the exact wording and presentation of the questions
• the population under study and how the sample was drawn and recruited
• the modes and dates of data collection
• sample sizes and, where applicable, the precision of the results
• how the data were weighted, if at all
• how the data were processed and checked for quality
• a general statement of the design's limitations

For an internal report that fits in a small box on the last page:

Method: online survey sent by email to all 4,820 customers
active in Q3. Fieldwork 2-16 October. 1,212 complete responses
(25%). No weighting applied. 37 responses removed for speeding
or straightlining. Full questionnaire in appendix A.
Limitation: customers who churned before October were not invited.

One line in the code deserves attention: reports of non-probability samples should only give measures of precision when they explain how those were defined and calculated. Most customer and employee surveys are non-probability, because you invited everyone and some chose to answer. A "margin of error ±3%" under such a survey claims a precision the design cannot deliver, and nonresponse is the bigger risk anyway. Our guide on weighting survey data goes into what you can and cannot correct afterwards.

Put the limitations next to the findings

Limitations tend to end up in grey text on the last slide. Move each one next to the finding it affects: if only 12% of your enterprise customers answered, say so under the enterprise chart. For each headline finding, write one sentence starting with "This would be wrong if", and keep the ones you cannot rule out.

Then hand the report to someone who did not work on the survey and ask for the three main findings. If their answer differs from yours, rewrite the report.