Making sense of open-ended survey answers, and where AI fits

Most survey teams pour their design effort into the rating questions and treat the comment box as an afterthought. That is backwards. The closed questions tell you what happened. The one open-ended question at the end tells you why. Open text is also where analysis quietly falls apart, because reading a few thousand free-form answers is slow work that no dashboard does for you. So the answers get skimmed, or pasted into a chat model that hands back a tidy summary nobody checks. This is a guide to doing it properly, and to knowing the line where an AI model stops helping.

Why open-ended answers are worth the trouble

A rating tells you the size of a feeling, not the cause. A customer who scores you 3 out of 5 has given you a number you can average, but the number is mute about whether they were annoyed by the price, the onboarding, one bad support ticket, or a competitor's ad that morning. The open-text answer beside that score is the only place the reason shows up in their own words, before you filter it through your hunch about what the problem probably is.

Open text is also the part of a survey you did not pre-decide. Every closed question is a hypothesis: you guessed the answers and printed them as options. If your guess was incomplete, the question cannot tell you, because the missing option is not there to pick. The open box is where the answer you never thought to offer turns up. Throw it away and you keep only the findings you already expected.

Coding is the skill everything else depends on

The catch is that open text does not arrive as data. It arrives as prose, in wildly varying lengths, spelling, languages and levels of effort, and none of it slots into a chart until someone imposes structure on it. That structuring step is called coding, and it is the whole game.

Coding means reading each answer and tagging it with one or more categories, so that "the checkout kept timing out" and "payment page froze twice" both land under a code like checkout reliability. Once every answer carries codes, you can count them, cross them against the rating, and report that 31% of detractors mentioned checkout reliability, instead of quoting three vivid comments and hoping they are typical. Coding is what turns anecdote into evidence.

The set of codes and their definitions is your codebook. A good one has categories that are distinct enough not to overlap, specific enough to act on, and defined in a sentence each so that two different people tagging the same answer reach the same code. Vague codes like "user experience" are useless because everything fits under them. Analysing open text is really the work of building and applying a solid codebook. And that is exactly the work people hope AI will do for them.

Deductive and inductive coding are not the same job

This split decides where AI is safe, so it is worth being precise. Deductive coding means you bring the codebook to the data: you have your categories already, and the task is to apply them consistently across thousands of answers. Inductive coding means you build the codebook from the data, reading answers with no fixed categories and letting the themes come up out of what people wrote. Most real projects do both: draft a codebook inductively from a first sample, then apply it deductively to the full set.

The two jobs ask for different things. Applying a known code is a matching task with a clear right answer. Discovering a theme is interpretive. It depends on tone, on context, and on the judgement to notice that three differently worded complaints are the same underlying problem. Hold on to this distinction, because it maps almost exactly onto what a language model can and cannot be trusted to do.

Where AI is genuinely reliable: applying a codebook you wrote

Hand a model a clear codebook and ask it to tag answers against it, and it performs about as well as a person. A blinded comparison published in PLOS Digital Health in April 2026 tested several models against human analysts on healthcare interview data. On deductive coding, applying pre-defined codes, the models came out non-inferior to humans: mean agreement of 93.5% for the models against 92.7% for the humans. The strict hallucination rate, where a model invented evidence that was not in the text, was 1.2%. For the boring, high-volume half of coding, that is real value. A model will tag ten thousand answers against your codebook overnight, consistently, without flagging on answer number four thousand. You keep the interpretive control and delegate the mechanical scaling. The guide to analysing surveys with AI over MCP covers how to wire a model to your response data in a read-only way, so it can do this tagging against data it cannot alter.

Where AI gets themes wrong: discovering them for you

Now flip the task. Ask a model to read raw answers and tell you what the themes are, with no codebook, and it gets far less dependable. In the same study, only one of the tested models cleared the non-inferiority bar on inductive analysis. The models were weakest exactly where it counts: sensitivity to nuance and latent meaning, the kind of reading that catches an implied complaint or a theme nobody stated outright. The comprehensive error rate, which counts partial mismatches and misattributions rather than only outright fabrication, was 12.4%.

The failure is rarely a wild invention. It is quieter, and that makes it worse. A model produces a plausible, well-organised set of themes that reads as authoritative, and along the way it flattens the outliers, merges two distinct problems under one heading, or promotes a phrasing it saw once into a "major theme." You cannot catch this from the summary, because the summary always looks confident. You catch it by having read the raw answers yourself first. A model handed a thin batch of answers will still return five neat themes without warning you the sample was too small to carry them.

A workflow that uses AI for what it is good at

Put the two findings together and the division of labour writes itself. Read a sample by hand first, a few hundred answers, enough to see the shapes that keep repeating. This is the inductive step, and it stays human because it is the step AI is worst at. Draft your codebook from that reading, one sentence per category. Then hand the codebook and the full dataset to the model for the deductive step, tagging every answer at scale. Spot-check its work by re-coding a random slice yourself and comparing. If you disagree on more than a small fraction, your codebook is ambiguous, not the model.

What the human never hands over is deciding what the categories are and whether the results hold up. The model gets the repetitive matching in the middle, where the volume lives and where a person's consistency slips but a model's does not.

Quantifying open text without lying with the numbers

Once answers are coded you will want percentages, and percentages are where open-text analysis oversells itself most. A code that shows up in 8% of comments is not the same as an issue affecting 8% of customers, because only some people wrote a comment at all, and the ones who did sit at the more motivated, more annoyed or more delighted end of your sample. Report coded open text as a share of the people who commented, never as a share of everyone.

Watch small numbers too. A theme carried by nine comments out of four thousand is a signal worth chasing, not a trend to size on a slide. The guide to designing scale questions explains why a number needs enough responses behind it before it means anything, and coded open text is no exception.

Failure modes to watch for

A few traps recur. Answer attribution: feed a model many answers at once and it can blur which comment said what, which is part of why the comprehensive error rate above ran higher than the fabrication rate. Agreeable themes: models lean toward the tidy summary, so a contradictory or awkward finding can get smoothed out of existence. The vanishing rare case: the single comment describing a serious accessibility failure or a legal risk matters far more than its one-in-three-thousand frequency suggests, and a summary driven by frequency will drop it. And low-effort or machine-written answers can pollute the input before coding even starts. The guide to spotting low-quality responses covers how to filter those out first.

None of this means avoid AI. It means read a real sample of the raw text yourself, keep the rare-but-serious comments in view, and treat any theme the model surfaces on its own as a hypothesis to verify, not a finding to publish.

The analysis starts when you write the question

The cleanest open-text analysis is the one you set up months earlier by asking a good question. "Any other comments?" produces a graveyard of "no", "n/a" and venting that is hard to code because it is about nothing in particular. A pointed prompt, like "What one thing would have made this easier?", produces answers that already cluster, because you narrowed what people would talk about before they started typing. The guide to writing better survey questions applies directly. A focused open question is kinder to the respondent, and it is the difference between a codebook that almost builds itself and one you have to force onto scattered replies.

So ask one good open question rather than three vague ones, and decide before you send what decision the answers are meant to inform.