How to spot low-quality survey responses
A clean export can still be full of junk. Here is how to spot satisficing, straightlining, speeders and AI-written answers before they skew your survey results.
By SurveyLane · The team building SurveyLane
A clean-looking dataset can still be wrong. Someone clicks through your survey without reading, ticks "agree" the whole way down, and their answers land in the export and move the averages anyway. The wording was fine. The responses were not. This is the second half of survey work, and it decides whether the charts you build mean anything.
What satisficing actually is
Respondents want to be done. When a question takes effort, some give the answer that is good enough to move on instead of the accurate one. Psychologists call it satisficing. It shows up as blank fields, a straight column of identical ratings, or a survey finished faster than anyone could have read it. The Pew Research Center runs these exact checks. In the methodology for its 2026 political typology, fielded in November 2025, researchers flagged people who left questions blank at very high rates or always picked the first or last option, and dropped the worst cases before weighting. If a probability panel bothers, so should you.
Straightlining and the first-answer reflex
Straightlining is picking the same point on every scale, a vertical line of 3s down a grid of rating questions. Sometimes it is honest. More often it means the person stopped reading. You cannot tell from one answer, so look across the block: ten matrix items answered identically is the signal, not any single row. The first-answer reflex is the close cousin, where someone always takes the top option because it sits nearest the cursor. Both get worse the longer a survey runs. That is one more reason to cut questions you cannot act on. When you write each question to map to one decision, a shorter survey gives satisficing less room to build.
Speeders, blanks and filler
Timing is the cheapest quality signal you have. If a careful reader needs four minutes and someone finishes in fifty seconds, those answers are guesses. Record completion time and read the bottom of the distribution. Missing data is the next flag. A respondent who skips every optional field is telling you they checked out. In open-text, watch for answers that repeat, contradict the closed questions, or read like padding: "good", "n/a", "asdf". One of these alone convicts nobody. Two or three together usually do.
The AI-written answer problem
Open-text used to police itself, because gibberish was obvious. That has changed. A respondent, or a bot farming an incentive, can paste a fluent, on-topic paragraph from a language model in seconds, and it survives a skim. The tell is that it stays generic: it answers in the abstract but names nothing specific to your product, your wording, or what the person was doing. Ask for a concrete detail, like which screen or what they were trying to finish, and the machine filler falls apart while a real complaint gets sharper. Do not ask an AI to grade AI here. Use it to cluster and surface, then read the flagged answers yourself.
Design so bad answers stand out
The best cleaning happens before launch. A well-placed attention check, something like "select somewhat agree for this item", catches inattentive respondents without insulting careful ones, as long as you use it once and not five times. Reverse-worded items inside a block expose straightliners, because an honest answer flips where a straight line does not. And a short survey with balanced scales just produces less noise to sift later. These are design decisions, not repairs you bolt on afterward.
Clean before you analyse, not after you present
Decide your exclusion rules before you look at the results, and write them down. Too fast. Too many blanks. Failed the attention check. Straightlined the grid. Inventing rules after you have seen the numbers is how you talk yourself into deleting answers you did not like. Run the pass once, keep the removed rows in a separate file, and note how many you dropped. Because SurveyLane holds responses in a real database, you can ask the AI analysis layer over MCP to list completion times or flag identical answer patterns. The call on what counts as junk stays with you.
A quick screening pass
- Sort by completion time and read the fastest five percent.
- Scan rating grids for vertical lines of identical answers.
- Count blanks per respondent and flag the high end.
- Read open-text for padding, duplicates and generic AI-style paragraphs.
- Apply the rules you declared in advance, and record how many rows you removed.
Frequently asked questions
How many responses is it normal to remove?
There is no fixed rate. It depends on your channel and your incentive. An unincentivised survey to a known audience might lose almost nothing, while an open link with a prize draw can shed a double-digit percentage to speeders and bots. What matters is that you set the rules before you looked, and that you report the number you dropped.
Are attention checks worth an extra question?
In a longer survey, yes. One well-worded instructional check somewhere in the middle catches inattentive respondents for the price of a few seconds. Overdo it and you irritate careful people and train everyone to hunt for the trick, so one is usually plenty.
Can I let the AI analysis remove low-quality responses for me?
Use it to surface, not to decide. An AI layer is good at clustering open-text and pointing at odd patterns, but it also invents themes that are not in the data, so a person should confirm each exclusion. Keep the judgement and the audit trail on your side.
Does a low completion rate mean my data is bad?
Not on its own. People abandon long or irrelevant surveys, and the ones who finish can still answer carefully. Completion rate is a design signal. Response quality is about how the people who did finish actually answered. Treat them as two separate problems.
