Ranking, rating or MaxDiff: how to measure what people really prioritise

You want to know which of ten roadmap items to build first, so you ask customers to rate each one from "not important" to "very important". Eight of the ten come back above four out of five. You have learned that people like getting things. You still don't know what to build first. To measure priorities you need a question that makes people choose, and each of the usual formats breaks somewhere.

Why rating scales flatten priorities

A rating question asks about one item at a time. Nothing stops anyone from giving every item the top score, and for anything that sounds like a benefit, plenty of people do. Faster load times, better exports, more integrations, lower prices: who would call those unimportant? A rating tells you whether something is welcome. Almost everything is.

It gets worse. Half-attentive respondents pick the same point for item after item, which methodologists call nondifferentiation. And the top box is a ceiling: once someone has said "very important" to the third item, there is nowhere higher to go for the fifth one that matters more.

So you get means bunched between 4.1 and 4.6, with gaps smaller than the noise. Teams rank them anyway and present 4.4 against 4.3 as a finding. It isn't one, however well you built the rating scale.

What the research says about ratings and rankings

Duane Alwin and Jon Krosnick compared the two formats using the 1980 General Social Survey, where a list of valued qualities was asked once as ratings and once as a ranking. Their paper appeared in Public Opinion Quarterly in December 1985. For the sample as a whole, both methods put the qualities in much the same order. Underneath, they disagreed: the patterns of who valued what, person by person, looked different depending on the format.

In a 1988 follow-up, Krosnick and Alwin pointed to one cause: respondents who rated many items alike distorted the correlations. For a rough group order, ratings often get close. To learn what one person would give up, or to segment people by priorities, you need a forced choice.

So ask what decision the data feeds. "Is this feature wanted at all?" is a rating question. "Which of these do we build first?" is about trade-offs, and you need a format that forces them.

When a plain ranking question is the right tool

The simplest forced choice is a ranking: here are five items, put them in order. It works when a few conditions hold.

The list is short. Five items is comfortable and seven is about the limit. Past that, people rank the first and last few with care and shuffle the middle.

Every respondent knows every item. If a third of your sample has never used the export feature, their rank for it is a guess. Use conditional logic to show only the items someone can judge.

You only need the order. A ranking says A came before B, and nothing about whether A was far ahead or barely ahead.

When that holds, a ranking is quick to answer and easy to explain. A top three is often better still: less work, and the top is where people put their care anyway.

Where ranking questions break

Long lists are the obvious failure. Ask someone to order fifteen items and you get a careful top three, a careful bottom two, and ten positions filled in so the form will accept it. Forcing a complete ranking turns honest indifference into numbers that look precise, and the effort spills over into later questions.

Items near the top of the displayed list also tend to be ranked higher, because people start reading there. Randomise the starting order per respondent.

Then there's the interface. Drag-and-drop is fiddly on a phone, where your finger covers the item and scrolling and dragging fight over the same gesture. It is also hard with a keyboard or a screen reader. Give the widget a number field or a dropdown per item as well, or those respondents quietly drop out. The post on accessible surveys goes into detail.

Ranks are not scores

Every respondent's ranks add up to the same total. Put item A first and everything else moves down a place. The numbers depend on each other by design. Statisticians call this ipsative data, and it breaks the assumptions behind ordinary correlation, factor analysis and regression. A factor analysis on raw ranks finds structure that the question format put there.

There are safer ways to report a ranking. The share of respondents who put each item first is easy to read and matches the decision most people care about. The share in the top three is steadier, especially with small samples. Mean rank works as a rough summary, but it assumes the step from first to second equals the step from sixth to seventh, and it hides splits.

That last problem is the big one. An item half your sample ranks first and the other half ranks last gets the same mean as an item everybody puts in the middle. Show how the ranks are spread for each item; the post on reporting results has more on that.

Constant sum: when the size of the gap matters

Sometimes the order isn't enough. If you're dividing a budget, you want to know whether A is twice as important as B or only slightly ahead. A constant-sum question gives people a fixed number of points, usually 100, to split across the items.

That captures intensity. One respondent puts 70 points on reliability; another splits evenly. A ranking gives both a neat order. Constant sum shows that only the first has a clear priority.

The cost is effort. People have to do arithmetic, and without a running total on the form, lots of answers add up to 90 or 115. Keep it to about six items and test it first. The points have the same fixed total as ranks, so the same analysis warnings apply.

How MaxDiff works

MaxDiff, short for maximum difference scaling and also called best-worst scaling, gets around the long-list problem by never showing the whole list at once. It grew out of Jordan Louviere's work in the late 1980s. One of the first published applications, by Finn and Louviere in the Journal of Public Policy & Marketing in 1992, measured public concern about food safety.

The respondent sees a small set of items, picks the most important and the least important, and moves on to the next set:

Which of these matters MOST to you, and which matters LEAST?

  [ ] most   Exports to Excel and CSV          [ ] least
  [ ] most   Single sign-on for your team      [ ] least
  [ ] most   Response alerts by email          [ ] least
  [ ] most   Survey templates                  [ ] least

Over many sets, each item shows up several times in different company. If A was best and D was worst, A beats B, C and D, and D loses to all three. That's five comparisons from two clicks.

People pick extremes reliably. Spotting the best and worst of four is easy; separating number six from number seven out of fifteen is hard. MaxDiff asks only the easy question, many times, and the analysis rebuilds the full order.

Designing a MaxDiff exercise

Sawtooth Software, whose tools are widely used for MaxDiff, publishes design guidance. Show three to five items per set, since more than five causes fatigue and errors. If you need precise scores per respondent, show each item at least three times. Never show more than half the list in one set, so eight items means at most four per set. The method needs at least six items, and Sawtooth recommends staying around 30 or fewer.

Do the sums before launch. With 15 items, 5 per set and 3 showings each, you need 15 × 3 ÷ 5 = 9 sets. Nine quick screens is fine. With 30 items, 4 per set and 3 showings, it's about 23 sets, and that's a lot of clicking. Cut the list first.

Most of the work is in the items. Every item should answer the same question; a list mixing features, prices and brand traits gives a comparison nobody can use. Keep items similar in detail. A vague item like "better quality" tends to beat "export to Google Sheets", because people read their own wish into it. Remove overlap, since two near-identical items split the vote. And fix the framing: "most important when choosing a tool" and "most important to improve next" can order the same list differently.

Let the software build the sets. Hand-built sets almost always over-show some pairs and never show others.

Reading MaxDiff scores

The simplest analysis is counting. For each item, take the times it was picked as best, subtract the times it was picked as worst, and divide by the times it was shown. The result runs from −1 to +1 and already gives a usable order. Dedicated tools fit a choice model and produce scores per respondent, often rescaled to add up to 100.

Keep two things in mind. MaxDiff scores are relative: they compare the items with each other and say nothing about whether any of them matters in absolute terms. If nobody wants anything on your list, MaxDiff still crowns a winner. Anchored MaxDiff adds a question asking whether an item is important at all, so you can draw a line between what matters and what doesn't.

The scores also belong to this list. Add or remove items and the shares move, so don't compare studies with different lists. To see which priorities actually move satisfaction, as opposed to which ones people say matter, pair MaxDiff with a key driver analysis.

Segments are where MaxDiff earns its keep. Split by role, plan or region and see where groups disagree. Each segment needs enough people, though.

Choosing between rating, ranking, constant sum and MaxDiff

Start from the decision.

Use ratings when you need to know whether each item is wanted on its own terms, or when you're tracking one item over time.

Use a short ranking, or a top three, when the list has seven items or fewer and the order is all you need.

Use constant sum when the size of the gap drives the decision, like a budget split.

Use MaxDiff when the list is long, the items are close in appeal, or you want to compare segments. Add an anchor if you also need an absolute bar.

Whatever you choose, follow it with one open question: "Is there anything important to you that wasn't on this list?" A forced choice only measures the options you thought of. Those answers show what you missed, and the post on open-ended answers shows how to work through them.