Spotlight B: the Net Promoter Score and correlation

In Spotlight B in my book on surveys, I write about:

• Net Promoter Score® (NPS®), also described on the SatMetrix Net Promoter Score website
• Correlation where two things vary in similar ways, a topic discussed on many other sites such as Math Is Fun Correlation
• The fact that ‘correlation does not imply causation’. Somethings are coincidentally correlated but having nothing to to with each other. For example, I quoted one from Tyler Vigen’s extensive selection at Spurious correlations

NPS® appeals to large corporations

Net Promoter Score® is derived from the now-ubiquitous “How likely is it that you would recommend [company X] to a friend or colleague?” question that has an 11-point answer range from zero to 10.

is much loved by large organisations, ever since Fred Reichheld wrote about it in the Harvard Business Review in 2003, where he said that “One question was best for most industries” (original emphasis). More recently, the book co-authored by Reichheld and Markey has been a best-seller. Obviously the idea of only asking one question and only basing corporate growth on a single number is highly attractive to busy executives.

• The one number you need to grow, Harvard Business Review
• The Ultimate Question book (several editions widely available, any will do)

NPS® can be jarring in other circumstances

Unfortunately, a question that might indeed have some value for a corporate brand that is specifically aiming to grow the brand has become heavily used in many other circumstances, some of which seem at best heartless to the people who have to answer the question. The phrase “best for most industries” (note both ‘most’ and ‘industries’) has often been forgotten.

For example, in the UK for several years our National Health Services mandated the use of a similar question for assessing health service delivery. I personally did not feel comfortable answering a question along the lines of  “Would you recommend us to a friend or family member?” when I was leaving hospital after being seriously ill. I was definitely delighted that I had recovered, but I also definitely did not want to think about any friend or family member possibly needing similar treatment. In 2019, they changed the question to “Overall, how was your experience of our service” – a question that I also consider lacks nuance, but at least I personally no longer find it downright upsetting.

• NHS England guidance on using the Friends and Family test, 2029 .pdf

The statistical discussion has a few challenges

Reichheld’s original research isn’t quite as clear-cut, in my view, as the fanfare about it might suggest. There’s a lot about how important loyalty is for a brand, but there’s also some confusion between announced loyalty and actual continued purchase behaviour: they equate ‘would recommend and has purchased in the past” with “would recommend and then actually did purchase in the future”.

This is particularly muddy because of different usages of the word “predicted”. In everyday life, we say “the weather forecast correctly predicted rain” to mean that the forecast told us rain would come some time in advance of the rain, and the rain duly arrived when it was supposed to.

In statistics, “predicted” means “correlated within the context of this data”. It doesn’t mean “a forecast of future events”. So you can say that a high level of “recommend to a friend or colleague” predicted loyalty, even when the loyalty behaviour was a purchase that happened BEFORE answering the ‘recommend to a friend or colleague’ question. Statisticians often use the word “explained” in the same way, which can be a surprise. I’ve learned to think carefully about the actual sequence of events when reading this sort of material.

The prediction of future behaviour is never exact – but the customer may reflect on past behaviour and use that to gauge their answer.

NPS® may not measure what it claims to measure

In a short Spotlight, I could not go into all the problems of Net Promoter Score®. If you would like a more detailed critique of the measure, Jared Spool has long campaigned against it and did a detailed take-down in 2017:

• Net Promoter Score Considered Harmful (and What UX Professionals Can Do About It)

If you must use NPS®, read the book and do testing

Nevertheless, many of us will encounter colleagues, clients, or stakeholders who are convinced that using a ‘recommend to a friend’ question (or similar) is important for reasons of their own. If this happens to you, I recommend that you:

• Get hold of a copy of Reichheld’s book (easily available from your local library or the second-hand book vendor of your choice) and read it so that you are fully aware of the method described in it. This will provide you with plentiful ‘yes, and’ questions to ask such as ‘yes, and can you tell me about the plans for the investigation into reasons for score changes that we will do alongside?”
• Make sure that you test the planned question in interviews. You’ll have a “defined group of people”, the ones who you want to answer your survey. So find some folks in that defined group and ask them about the topics you plan to include in your survey, and also ask them what the planned question means to them in their own words. Be prepared to be surprised: if you’re working with a large brand where brand loyalty is relevant, the question might work better than we expect.

If you must use NPS®, get it right

And: get the actual question right.

The screenshot below is one of the many in my library, sent after online shopping from a brand that also has physical stores. The  questionnaire ‘how likely to recommend’ question  . , so failed on the ‘one question’ idea from Net Promoter Score®.

Even more annoyingly, it comes right after two very similar ‘how likely are you to’ questions that use the confusing term ‘likelihood’. It’s followed by a tortuously-worded question that is trying to encourage us to leave both positive and negative feedback.

And most of all, the Net Promoter Score® question is the wrong way around, leading off with ‘Extremely Likely’ and then going in descending order through the options down to ‘Not at All Likely”. Do not put an ubiquitous question into a different format – it’s a recipe for confused data.