Think Customers: The 1to1 Blog

Don,

Let me address your questions about why you would hire Satmetrix to "deploy a one questions survey".

If you want to integrate customer feedback into your enterprise, design information work flows to get information to your front line employees and design action planning to improve your loyalty based on that feedback, that's where you will benefit from Satmetrix.

Satmetrix was deeply involved in the research behind Net Promoter and we have over 10 years of experience helping enterprise companies measure and act on customer feedback using Net Promoter and other measures. One question is not always the right approach; it depends. We help clients design enterprise programs that get them on the journey of building a customer-centric enterprise. Measuring NPS is only one component of the program approach needed to realize the benefits.

The important point is building a customer-centric enterprise. The beauty of Net Promoter is it is simple and actionable. Simple allows business leaders to quickly evaluate their customer loyalty and drives adoption. Actionable comes from nurturing promoters, addressing detractors and moving the passives. How can anyone argue with these benefits vs. a complex index that requires a statistician to explain and has no identification of what you can do about it?

We consult with clients to design the right program for their needs and take action to change the score. After all the score isn’t where the game is won, the game is won when you change it.

You can learn more about clients seeing results at www.satmetrix.com or www.netpromoter.com.

Thank you,
Deborah Eastman
CMO
Satmetrix

Dear Don –

I appreciate your viewpoint, but would like to make a few additional points.

First, I know that you did not mean to infer this, but while you mention that I work for IPSOS Loyalty, my work never interferes with my scientific research. If I ever felt pressure at all in this regard, which I do not, I would quit my work immediately. [I am confident that as a scientist, your partner Martha Rogers would say the same thing.] As a result, if Net Promoter had worked as promised, that is exactly what we would have reported. [And clearly, my academic co-authors feel the same.]

I respect your inherent belief that increasing customers’ willingness to say that they would recommend the firm benefits the firm economically. Unfortunately, knowing it in your gut (i.e., truthiness) doesn’t mean it is the truth. As you showed in Return on Customer, there are a myriad of issues involved in making a real economic impact. And our work did not just find that Net Promoter was "not superior," we found it was not a good predictor at all.

And, unfortunately, I am aware of another scientific paper currently under review at a leading journal by one of the top minds in the field that shows what you believe to be true regarding recommend intention just isn’t so. I will not steal his thunder, but in the end the truth will win.

Sincerely,

-- Tim Keiningham

Speaking of NPS, have you got a question for Fred Reichheld or Dr. Laura Brooks at Satmetrix? Now is the time to pick their brains.

We're collecting questions for the two experts to answer as part of a new Q&A series with industry experts. Please send your questions to me at eglagowski@1to1.com, and stay tuned for the upcoming article.

Note: Not all questions will be answered, depending on volume and space constraints.

What if NPS does not accurately direct companies in the direction of superior growth and instead leads the company in the wrong direction? If it is not the accurate assessment that it purports to be, companies can invest great sums of money in the wrong area.

I am reminded of the cartoon of the football running back with his helmet over his eyes and you see a chaotic path of his running around the field. The caption is "its speed that counts, not the direction." The direction that use of NPS may lead the company may be similar to the outcome of the football running back.

Measurements are important! While I am not willing at this stage to pronounce NPS correct or not, I think we (those of us in academia and those who live and work in the marketplace) must be very clear that the measures we use are the correct ones, whatever they are and can be proven to be, not the popular ones.

Thanks for your long and extremely well documented comment, Tim, and thank you also for providing the download links to your two papers and the other documents, as well. I hope everyone takes the time to read these documents because they are very illuminating.

On the face of it I have no basis to disagree with your conclusions that:
1. NPS is not the best predictor of a company’s growth, and
2. NPS is not the best indicator of increased customer loyalty.

However, I’m not overly troubled by either conclusion, because I generally don’t believe the more far-reaching claims of consultants and business authors to begin with (remember, I’m a consultant and business author myself!). Frankly, when Reichheld’s NPS work first came out I didn’t put much stock in the patently unbelievable claim that it was superior to every other metric on the face of the Earth.

But what I liked very much about the NPS idea was that it was extremely simple to explain, intuitively straightforward, and practical to implement. In my opinion, whether it is the BEST metric of growth or loyalty is just not as important as the fact that (a) NPS is generally indicative of increased customer loyalty, better customer service, a better company reputation, and a longer-term value proposition for a company, and (b) NPS can be communicated to management and implemented easily.

Now remember, just like your own consulting firm, Ipsos Loyalty, we here at the Peppers & Rogers Group and at Carlson Marketing are technically in competition with Bain, Satmetrix, and Reichheld himself. We have no business reason whatsoever to endorse Reichheld’s ideas and every business reason to find fault with them.

On the other hand, it’s my belief that when it comes to tracking their long-term success with customers, our clients are better served by good-but-imperfect metrics than by no metrics at all, which is the situation most of them are in at present. Most companies find it extremely difficult to resist the siren call of short-term results. Adopting a more customer-centric approach meets great resistance in this climate, because a company must incur measurable costs in the short term, while creating not-so-measurable value in the long term. The shift away from purely short-term priorities often starts with a few proponents of customer centricity, or of developing longer-term customer relationships, or of simply trying to improve service. These early proponents need data to support their case, and getting their unbelieving colleagues to invest heavily in more complicated market research – even though that might be the wiser course – is often just not possible, partly because no one really sees the benefit of long-term metrics to begin with.

So yes, I certainly agree that it’s disappointing if biased data sets or flawed research methods were used to establish the primacy of NPS over other measures, because it does undermine the credibility of marketing research, in general. If that occurred, then it really is deplorable. But in my mind that’s a separate issue, going to the question of whether NPS is or is not competitively superior to other metrics, in terms of its predictive or prescriptive power.

But whether NPS makes sense for a firm to use or not – especially in the absence of any corporate commitment to other, more sophisticated research tools – boils down to a very simple question:

Does a company’s economic value improve if it undertakes policies designed to increase the proportion of people who say they would recommend it to a friend, and to reduce the proportion saying they would not?

Sorry, but in spite of all the other controversy surrounding the NPS issue, I just can’t see how the answer to this question could be anything but “yes.”

If any of your readers are interested in learning about the successes companies - such as HSBC, Philips, LEGO, Experian, T-Mobile, Schwab, and more - are having with Net Promoter, I'd like to turn their attention to the Net Promoter Conference (in London) blog. See companies listed in the "categories" section at right for quick scanning of company presentation write-ups.

Indeed, knowing a number can not impact a business. It's the necessary follow-on to improve that number. Your readers may be interested in the Net Promoter blogs by Jeanne Bliss, author of "Chief Customer Officer."

"We Know Our Net Promoter Score. Now What?"
"Mining the Gold: Listening Hard to Detractors"

Here's the link to Jeanne's blogs: http://netpromoter.typepad.com/jeanne_bliss/.

Her next blog, due out soon, focuses on promoters.

Dear Don –

While I understand your desire for a “quick way to get a handle on whether your company is building enough customer equity to sustain your growth and profit in the future,” and a “leading indicator of lifetime value change,” I am unaware of any scientific research that shows Net Promoter to do this. In fact, there are several scientific papers (two of which I co-authored) that conclusively demonstrate that the claims made by Reichheld regarding Net Promoter’s linkage to growth and customers’ future loyalty behaviors are false. Furthermore, the results uncover VERY strong evidence of bias in the research reported by Reichheld.

JOURNAL OF MARKETING

Professors Bruce Cooil (Vanderbilt University), Tor Wallin Andreassen (Norwegian School of Management), Lerzan Aksoy (Koç University), and I have a paper that appears in the current issue of the Journal of Marketing that investigates Reichheld’s claims regarding the linkage between Net Promoter and growth.

There are two key findings from the research:

1) We did not find Net Promoter to be a good predictor of growth at all.

2) We found very strong evidence of research bias in the research reported by Reichheld in support of Net Promoter. In particular, we were able to replicate a subset of Reichheld's reported data for his best case scenarios and compare it to a metric he claimed was examined and found to have a 0.00 correlation to growth, the ACSI. Our findings clearly show that when using Reichheld’s own data, Net Promoter wasn’t superior to the ACSI. It is difficult to imagine a scenario other than research bias as the cause of this finding. This is a serious problem. We expect published research to be free of bias in management science, just as we do in all other fields of study. Managers have adopted Net Promoter based upon the belief that solid science underpinned the claims attributed to the metric. In fact, there would have been no HBR paper introducing Net Promoter without the research.

The Journal of Marketing is allowing the article to be downloaded for free at the following URL:
http://www.atypon-link.com/AMA/toc/jmkg/71/3

MANAGING SERVICE QUALITY

The current issue of Managing Service Quality released another paper we co-authored that investigates other aspects of the Net Promoter research reported by Reichheld. The research examines different customer satisfaction and loyalty metrics and tests their relationship to customer loyalty behaviors. The goal was to test the robustness of the customer-level analysis reported by Reichheld, which served as the foundation of the Net Promoter research. Contrary to Reichheld's assertions, the results indicate that recommend intention alone will not suffice as a single predictor of customers' future loyalty behaviors. Use of multiple indicators instead of a single predictor model performs better in predicting customer recommendations and retention.

Managing Service Quality is making the paper available for free for download at the following URL:
http://tinyurl.com/2aq3aj

These findings, taken in conjunction with the findings of our research reported in the Journal of Marketing regarding the relationship between Net Promoter and growth, call into question the robustness of the entire analysis reported by Reichheld.

OTHER PAPERS

Two other papers of note that investigate and challenge the claims of Net Promoter are:

1) “The Single-Question Trap,” by Gina Pingitore, Neil A. Morgan, Lopo L. Rego, Adriana Gigliotti, and Jay Meyers that appears in the current issue of Marketing Research.

2) “Would You Recommend This Paper to Your Friends?” by Mark Molenaar. This paper won two awards at this year's South African Market Research Association conference. It is available at the following URL:
http://www.researchsurveys.co.za/papers/samra/2007/SAMRA%20paper%20-%20Mark%20Molenaar%20-%20May%202007.pdf

COLLOQUY MAGAZINE ARTICLE

An article in Colloquy magazine does a very good job of discussing the current debate regarding Net Promoter. It paints the debate as one of "religion versus science." Specifically, "For NPS promoters, faith trumps science. Like proponents of Intelligent Design, they respond to facts with platitudes." For Reichheld's part, he decided "not to invest time responding." Given that the science is not on the side of Net Promoter, there is little wonder to this decision. This article can be downloaded at the following URL:
http://www.colloquy.com/article_view.asp?uid=4030

CONCLUSION

Scientific researchers have consistently shown that the claims attributed to Net Promoter are false. Furthermore, given that our findings show that Net Promoter was not superior to the ACSI when using Reichheld’s best-case scenarios, it is virtually impossible to imagine a scenario other than research bias as the cause. This is a VERY SERIOUS problem. We expect research published in our most prestigious journals to be free of bias in management science, just as we do in all other fields of study. We would not consider this kind of problem acceptable had the research been conducted in medical, psychological, or physics research; the same standards apply in management science.

Managers have adopted Net Promoter based upon the belief that solid science underpinned the claims attributed to the metric. In fact, there would have been no Harvard Business Review paper introducing Net Promoter without the research. This also has serious implications regarding the credibility of Reichheld’s book, The Ultimate Question. Additionally, biased “research” that is published in a prestigious management journal contaminates not only management practice, but also management science, as it will be used by scientists as a basis for future research.

Net Promoter “Believers” typically focus on “testimonials” from users or “qualifications” regarding the usefulness of the metric in response to evidence contradicting their earlier assertions and beliefs. Unfortunately, as “researchers” this is the equivalent of ignoring the 900-pound gorilla in the room; the research upon which Net Promoter is based is the sole reason for its existence. If it is biased, then it is disqualified—period.

It is vital that we not be apologists or revisionists when it comes to issues of research bias. Our credibility must never be in question regarding the research we publish in prestigious journals; the truth matters. Therefore, discussions about Net Promoter by researchers (practitioner and academic) must first adequately address this issue. If not, then why do any research at all, as we can simply present the answers we want to believe as supporting evidence and be done with it? Given the evidence we uncovered, however, we seriously doubt that there will be an acceptable answer to the issue of bias in Reichheld's reported research. [Ironically, in The Ultimate Question, Reichheld emphasizes the importance of eliminating bias from research (pp. 106-111).]

While we can all agree for the need to have measures that are easily understood and used by managers, that is completely irrelevant to the issue at hand. Regardless of whether or not one chooses to believe in Net Promoter, we all must insist that the evidence used to support the metric be unbiased. This issue is so overridingly important to very core of what we as researchers (academic and practitioner) do and stand for that it must be addressed. Our credibility in science and in practice is based upon honest and fair evaluations of data...if this is contaminated and unchallenged, then there is no reason to believe anything we say.

Sincerely,

-- Tim Keiningham

Sorry for the misstatements, John. But why would I need to hire SatMetrix or any other professional research firm to deploy a one-question survey to my own customers?

Don,

I like Fred's concept of the NPS and our company has adopted it recently.
It's certainly an evolution from measurable customer satisfaction, that can evaporate before the survey's analyzed, or conceptual customer loyalty, that can easily be misinterpreted for "putting up with". Unlike those measures that give hints about past and present behaviour, NPS gives a clue to likely future behaviour.

Here's the rub however, unless you ask the next 2 questions as well, the "ultimate question" may not help you understand how to keep your preciously won advocate.

-Why? On what basis are you willing to make that recommendation?

-In the future what do we have to do to continue earning your trust beyond what we do for/with you now?

Any variation of those 2 questions add actionable insight to the "yes" or "no" responses you get to from your NPS effort. And since I don't believe in "ultimates" , I suspect we'll find greater wisdom and insight as we eplore even greater "customer intimacy" relationships by finding additional questions that we can ask, and even more important in a customer experience driven economy, be asked by, our customers.

We're heading in the direction of ongoing and meaningful dialogue with customers rather than having to make value assuptions about one another
and NPS is helping to get us there. Large surveys don't do that either.

Dear Don,

Before the debate gets going, may I please clarify a number of things in your intro?

1. InfoQuest is not a British research company. Headquarters are in Connecticut, where the business was founded back in 1989. Yes, I am based in the UK.

2. Time Consuming? The InfoQuest survey, which can pose up to 60 questions and statements and has an average worldwide response rate (based on more than 80,000 b2b surveys), typically takes less than 10 minutes.

3. Expensive? I have only one comparison I can make, which is with SatMetrics where InfoQuest was less that half the price.

4. Premiums? I’m not sure what you mean by “premiums” but I guess that you are talking rewards for those who complete the survey. InfoQuest does not get involved with that sort of business.

Thank you very much.

John Coldwell
UK Managing Director
InfoQuest

Don, There are so many reasons why NPS is a flawed concept it's hard to know where to begin. Just a few reasons why the alleged advantages do NOT outweigh the disadvantages:

-- NPS doesn’t help explain WHY a customer would recommend the firm. Let’s say a bank finds out that 10% of its branches score much higher than the average on the NPS and that 10% score much lower. What has it learned? Nothing. It isn’t actionable. You might argue that it provides clues as to where to dig in… but wouldn’t it be more useful to find out the root causes in the first place?

-- NPS measures intention, not behavior. Consumers don’t always do what they say they’ll do. Far more important is who actually refers the company to friends and family — not who might do so.

-- NPS doesn’t capture inherent consumer differences. Forrester Research has found that Gen X and Gen Yers are more likely to recommend a firm to their family and friends than Baby Boomers and Seniors. So if your NPS increases from one year to the next, is it because you improved the products and services you deliver, or does it simply reflect an underlying change in the demographics of your customer base?

-- NPS can incent undesirable behavior. One marketing exec told me about an interaction he had at his car dealer’s repair shop. When he arrived to pick up his car, the shop manager said “if there’s any reason you wouldn’t check off the ‘likely to recommend’ box on the customer satisfaction survey, please let me know before filling out the survey.” Do you want your firm’s personnel running around asking customers to say they’d refer the firm to friends and family — or doing the things that earn a referral?

-- NPS uses funds better deployed elsewhere. Measurement doesn’t come for free. Firms that have built an infrastructure to measure customer satisfaction are now being encouraged to build an infrastructure around measuring NPS. It’s not worth it. There are better things to measure — like what causes a customer to refer a firm in the first place.

-- The "simplicity" argument holds no water. Common practice is to only consider customers as promoters if they give a 9 or 10 on the 10-point scale. But in comparing NPS between time periods, it’s quite possible that the net score could increase while a significant number of customers shift from 7s and 8s to 1s and 2s. Not so simple.

Bottom line: There are better measures to hang your hat on.

Why not look at a different NPS -- Net Purchaser Score (the net difference between the number of people who bought your product and the number of people who returned it)? This metric:

1. Measures behavior (not intention)
2. Encompasses all customers (not just a sample)
3. Directly impacts the bottom line (not indirectly)
4. Can be measured in real-time (or at least more often than surveys)
5. Is simple!

I could write a book about why NPS isn't worth adopting, but I think Tim Keiningham will beat me to it.