Translating a survey without changing what it measures

When you send one survey in four languages, you are hoping to get one dataset. What you often get is four surveys that look alike. Each translation can be accurate word for word and still ask a slightly different question, and you only find out later, when a gap between countries turns out to be a gap between translations.

Accurate words, different question

A survey question is a measuring instrument, and a translation has to measure the same thing. Matching dictionary entries is the easy part. Pew Research Center says in its methods notes that it aims for questions that are "comparable at the level of meaning, not simply literal translations of the original English versions."

"Satisfied", "tevreden", "zufrieden" and "satisfait" overlap a lot, though not completely, and respondents answer what the word means to them.

For every translated item, ask: if a respondent in this language held the same opinion, would they understand the question the same way and pick the same answer?

One word can move your results

The best documented case comes from American health surveys. The standard self-rated health question offers excellent, very good, good, fair and poor. Spanish versions usually render "fair" as "regular". Spanish-speaking respondents picked that option far more often than English speakers picked "fair", so Hispanic respondents looked less healthy than other measures suggested.

Gabriel Sanchez and Edward Vargas tested the wording directly. In a survey of Latino adults, they randomly gave half of the Spanish-speaking respondents "regular" for "fair" and the other half "más o menos". The "regular" group was more likely to report poor health, compared with English speakers and with Spanish speakers who saw "más o menos". "Regular" reads more like "normal" or "okay". It is a reasonable dictionary translation that sits somewhere else on the scale.

Translation can change the context, too

Sunghee Lee and Norbert Schwarz compared two large American surveys of adults aged 50 and over, in a 2014 paper in the American Journal of Public Health. The Health and Retirement Study asked the self-rated health question before any other health questions. The National Health Interview Survey asked it after questions about health limitations.

Among English speakers, 75.4 percent reported positive health in one survey and 80.5 percent in the other. Among Spanish speakers it was 41.3 percent without the health context and 68.3 percent with it. That is a 27-point swing. The earlier questions told Spanish speakers what kind of "health" the survey meant, and only when the question came after them did their answer predict mortality.

A question can lean on its context more in one language than in another. Reorder, cut or add questions in one version and you may change the meaning there and nowhere else. The post on question order effects covers how earlier questions colour later ones. Across languages, keep the order identical.

Why back-translation gives false comfort

The classic check is back-translation. Someone translates your English survey into German, a second person translates it back into English, and if the two English versions match, the translation passes.

It feels rigorous. Yet a stiff, literal version reverses beautifully, while a natural one that adapts an idiom may come back looking wrong. Philipp Walde and Birgit Völlm, writing in Frontiers in Psychiatry in 2023, list the weak spots: errors in the backward step go unnoticed, the method pulls the translation too close to the source, and a translator who does not know the context can choose the wrong term in both directions, so the round trip matches while the meaning is off.

Their own example is "Pflegedienste" for care services. It translates and back-translates cleanly. In German, though, the word is used only for care of the ill or the elderly, which did not fit their questionnaire.

The team approach: TRAPD

Most large cross-national surveys, including the European Social Survey, use TRAPD, a method developed by the survey methodologist Janet Harkness. The letters stand for Translation, Review, Adjudication, Pretesting and Documentation.

In practice, at least two people translate the questionnaire independently. They then sit down with a reviewer and go through the versions item by item, picking or combining the best wording. One person, the adjudicator, makes the final call where the team cannot agree. The result is tested with real respondents in that language. And every decision gets written down with its reason.

Two independent translations are useful because they disagree, and where they disagree the source question is usually ambiguous.

You do not need the European Social Survey's budget. For a customer or employee survey, ask two colleagues who are native speakers to translate on their own, then spend an hour comparing with someone who knows what each question is meant to measure. Note why you chose each wording.

Write the source survey so it can be translated

Many translation problems start in the source. Pew sends new questions to professional linguists before translation, to judge how easily they translate. You can run the same filter while you write.

Idioms are the first thing to go. "How likely are you to go the extra mile for this team?" has no clean equivalent in most languages. Vague quantifiers drift too: "pretty often" means different things in different languages, while "in the past month, how many times" does not. Double negatives are hard enough in one language, so "Do you disagree that the onboarding was not clear?" has no chance in four. Name things concretely. "Your manager" travels better than "leadership", which can mean the board in one language and your team lead in another. And keep one idea per question, because each half of a double-barrelled question can drift on its own.

These are the rules from the guide to writing better survey questions, applied with a translator in mind.

Scales do not translate one to one

Rating scales carry most of the risk, because the labels are short and every word counts. "Strongly agree" might become "helemaal eens", "stimme voll und ganz zu" or "tout à fait d'accord". Each is a fair rendering. The distance between the top label and the one below it will still differ, and if one language's top label sounds more extreme, fewer people pick it.

Then there are response styles. Anne-Wil Harzing's 2006 study of respondents in 26 countries found large differences in how people use scales. Respondents from Spanish-speaking countries gave more extreme answers and agreed more readily. Japanese and Chinese respondents were far less extreme. Her work also suggests that making people answer in English instead of their own language squeezes the differences between countries together.

Translate scale labels as a set and check that the steps feel evenly spaced in each language. Label every point, so respondents do not have to guess what the gaps mean; the post on designing rating scales explains why fully labelled scales are more reliable. Item-specific scales ("very easy" to "very difficult") hold up better than agree-disagree formats, which invite acquiescence everywhere.

Tone of address is a design decision

English has one "you". Dutch has "je" and "u", German "du" and "Sie", French "tu" and "vous". The choice sets the tone of the whole survey. A formal "Sie" in a survey for young app users feels stiff. An informal "du" in an employee survey at a conservative firm feels presumptuous.

Use the register your brand already uses with these people in that language, and use it everywhere, from the invitation email to the reminder. Switching halfway is jarring.

Some things need adapting

Some content has to be rebuilt per country.

Education levels are the obvious one. A Dutch list with MBO, HBO and WO means nothing to a French respondent, so map each country's system to categories you can compare, and settle that mapping before launch. Income brackets are another: converting €40,000 gives an odd number, and the bracket means something else where wages differ. Dates and numbers trip people up as well. 03/04 is March in the US and April in Europe, and 1.500 is fifteen hundred in German but one and a half in English. Use date pickers and spell out formats.

Examples like "a call with the Belastingdienst" only work where people know them, and regions and job titles need local lists.

Write down how each local category maps to the one you will report. That mapping is part of the questionnaire.

Test every language version

Pretesting has to happen in each language. Walde and Völlm pretested their German translation with forensic patients in two rounds of cognitive interviews. In the first round, 6 of 9 patients struggled with "Individuum" for "individual". After the team adapted the wording, 7 of 8 patients in the second round understood the revised versions well. Back-translation would never have flagged it, because "Individuum" is a correct translation.

Ask three or four native speakers from your real audience to fill in the survey while thinking aloud, and ask what they understood by the key questions. Then soft-launch per language and compare skip rates and answer spreads on the same questions. A question skipped far more often in one language usually has a translation problem. The guide to pretesting a survey explains how to set up both.

Analysing a multilingual dataset

Store the language each respondent answered in as its own variable. Without it you cannot tell a real difference between groups from a translation effect.

Before pooling, look at the key questions per language. If one version behaves oddly, say a spike on one option or a skip rate well above the rest, investigate before you report a combined figure. Language and group are tangled, though, so treat a language gap as a reason to dig further before you blame the translation.

For open answers, code in the original language where you can, or keep the original next to any machine translation so a coder can check it. The post on analysing open-ended responses covers the coding itself.

Repeating a translated survey

Once a translation is in the field, freeze it. Pew reuses earlier translations for questions it has asked before, so results stay comparable over time. If you polish the German wording of a tracking question between waves, you cannot tell whether the German trend moved because opinions changed or because the question did. If a translation is truly wrong, fix it and mark the break. The post on repeating surveys shows how.

In SurveyLane, translations are stored with the survey itself, for each question, answer option and scale label. Every language version shares one structure and one set of routing rules, so they cannot drift apart.