Synthetic respondents: can an AI answer your survey for you?

Fieldwork is slow and respondents are expensive, so the pitch is easy to like: describe your target group to a language model, ask it to answer as five hundred of those people, and have results by lunch. Several research tools now sell exactly that under names like synthetic respondents, silicon samples or AI personas. The published research gives a more specific answer than the marketing. Synthetic answers can get an average roughly right and still be wrong in every way that matters for a decision.

What a synthetic respondent is

A synthetic respondent is a language model prompted to answer a questionnaire as a particular kind of person. You give it a profile, for example "a 52-year-old nurse in Rotterdam who votes centre-left and has used our app for two years", then your questions. It produces an answer. Repeat with a few hundred profiles drawn to match a population and you have something that looks like a dataset.

Nobody has asked a person anything. The model is predicting what text would plausibly follow a description of that person, based on what it absorbed during training.

The study that started it: silicon samples

The idea got serious attention from a 2023 paper in Political Analysis by Lisa Argyle, Ethan Busby, Nancy Fulda, Joshua Gubler, Christopher Rytting and David Wingate, titled "Out of One, Many". They took GPT-3 and conditioned it on thousands of sociodemographic backstories built from real participants in large American surveys, including the American National Election Studies. Then they asked it political questions as each of those people.

The generated answers tracked the real ones closely enough that the authors coined a term for the property: algorithmic fidelity. A model has it, they argued, when its output for a group reflects the patterns of ideas, attitudes and demographics you would find in real people from that group. They proposed four tests for it. The strictest: the relationships between variables should match what real surveys show.

That paper is careful. Its claim is that the model holds a lot of information about how attitudes cluster in the American electorate, and that social scientists can use this. Nowhere does it say you can stop surveying people.

The averages look right, the spread does not

A year later James Bisbee, Joshua Clinton, Cassy Dorff, Brenton Kenkel and Jennifer Larson put the idea under pressure, also in Political Analysis, in a paper called "Synthetic Replacements for Human Survey Data? The Perils of Large Language Models". They had ChatGPT adopt personas and rate eleven social and political groups on a feeling thermometer, then compared the results with the 2016 to 2020 waves of the American National Election Studies.

At first glance it worked. The average scores ChatGPT produced were close to the real averages. Underneath, the answers had much less variation than real people give. And when the authors ran the same regressions on synthetic and real data, the coefficients often differed significantly.

This is the failure that catches people out. A topline average is the number you check first, so it is the number that reassures you. Most survey decisions depend on something else: how spread out opinions are, which group differs from which, what predicts what. A sample with too little variance makes every group look more uniform and every estimate look more precise than it is. The post on sample size and margin of error explains why variance drives your confidence intervals. With synthetic data that confidence is unearned.

Change one word, get a different answer

Bisbee and colleagues found two more problems that matter to anyone who wants to repeat a study. Minor changes in prompt wording shifted the distribution of synthetic answers. And the same prompt gave significantly different results when they ran it again over a period of three months.

Real surveys are sensitive to wording too, and the post on question order effects shows how much a neighbouring question can move an answer. The difference is that you can test and document wording effects in a human survey, and the people do not change underneath you between Monday and Friday. A hosted model can be updated by its vendor without notice. If your quarterly tracker runs on synthetic answers, a change between quarters may come from your customers or from a model update, and you cannot tell which.

That makes synthetic data a poor fit for anything you plan to repeat. The post on repeating a survey without breaking your trend lists everything that can quietly break comparability. An unannounced model change belongs at the top of that list.

Whose opinions does the model hold?

Shibani Santurkar and colleagues at Stanford asked a broader question at ICML 2023: whose opinions do language models reflect by default? They built a dataset called OpinionQA from Pew Research Center's American Trends Panel and compared model answers with those of 60 demographic groups in the United States.

The gap between the models and these groups was substantial. The authors put its size on a par with the gap between Democrats and Republicans on climate change. It did not go away when they explicitly told the model which group to answer as. Some groups were represented particularly poorly, including people aged 65 and over and people who are widowed.

For a survey designer that is a direct warning. The groups a model handles worst tend to be the ones that are already hard to reach, and hard-to-reach people are the main reason anyone wants a shortcut. If you hoped to fill the gap left by older respondents with synthetic ones, you are filling it with the model's least accurate guesses. The post on sample quality covers nonresponse among exactly these groups.

Flattened groups and stereotypes

Angelina Wang, Jamie Morgenstern and John Dickerson took this further in a paper published in Nature Machine Intelligence in 2025. They ran human studies with 3,200 participants across 16 demographic identities and compared their answers with identity-prompted output from four language models.

They describe two failures. Misportrayal: when asked to answer as a member of a group, the models' answers sat closer to what outsiders imagine that group would say. Members of the group answered differently. Flattening: the answers lost the diversity that exists within a group, so "women" or "older people" came out as one voice. They also point out that prompting by identity treats a single label as if it explains a person, which essentialises the group.

Their techniques for reducing these problems at inference time helped, but by their own measurement they did not remove them. Their advice is caution wherever the identity of the respondent matters to the task. In a survey it almost always does.

Why the model cannot know your respondents

A language model learned from text that existed before it was trained. It knows a great deal about how people in general talk about politics, brands and work. It knows nothing about the specific people you want to hear from.

Your customers used a version of your product that shipped last month. Your employees work under a manager the model has never heard of. The question you care about is usually the one nobody has written about yet, otherwise you would not need a survey. On those questions the model falls back on the most typical answer for someone who fits the profile. That is where flattening comes from: averages without the people who disagree.

Where synthetic answers do earn a place

None of this makes language models useless around a survey. It means using them where a plausible answer is enough and a true answer is not required.

The most useful one is stress-testing the questionnaire. Ask a model to answer as a confused, rushed or sceptical respondent and see where it stumbles. An ambiguous term or a missing answer option often shows up within minutes. It is an extra pair of eyes. You still run the human pretest afterwards.

Generated answers are also a cheap way to push hundreds of paths through your conditional logic. You catch the branch that leads nowhere, or the question that never appears for anyone.

Dummy data lets you build your reports, dashboards and coding frames before fieldwork closes. When the real responses arrive, the analysis is ready to run.

And a model can suggest answer options you forgot, or a reason for churn you had not thought of. Real respondents then tell you whether it holds.

In every one of these uses the synthetic data is thrown away before any number is reported.

Questions to ask before you buy a synthetic panel

If a vendor offers synthetic respondents as a replacement for fieldwork, ask how they know it works. A credible answer includes all of these:

• Validation against a real human sample on questions like yours. Well-known public polls do not count, because the model has probably seen them during training.
• Results for variance and for differences between subgroups. Averages on their own prove little.
• Which model version produced the answers, and what happens to comparability when it changes.
• How sensitive the results are to small changes in the prompt and the persona description.
• What they recommend for groups that the published research shows models portray poorly.

If you use them anyway, keep them apart

Label synthetic records clearly and keep them in a separate dataset. Do not mix them with human responses. Do not weight them up to fill a quota either, and a synthetic number never goes in a report under the same base as a real sample. Record the model, version, date and full prompt with the output, so anyone can see what produced a number. And compare at least one question against real responses before anyone draws a conclusion.

When you analyse your real data with AI, the same principle applies the other way round. The model has to work from what people actually wrote. The guide to analysing open answers shows how to check AI-generated themes against the text, and the SurveyLane MCP server gives an assistant read-only access to real responses, so there is something real to summarise.

Keep real people for real decisions

A survey is worth running because it can surprise you. The answer you did not expect, from the group you underestimated, is usually the one that changes a decision. Synthetic respondents are built to produce the expected answer. That surprise is what you were paying for, and they take it out. So let a model tighten your wording and speed up your analysis. Then send the survey to people.