InSights - What you didn't know you didn't know

Beware Of The Evil Case Study

Business is “supposedly” all about ROI…

So to attract new business most marketers believe we have to be able to prove it. A three year P & L accounting of a customer’s return on their investment in this wonderful product or service could do the trick…

Except they’re expensive and difficult if not impossible to track, and too boring, and too complex for most to read.  Let’s face it, even if we could get them, P & Ls are not “sexy” enough to attract attention in the market clutter of claims, clamor and clever hype!We need pictures, graphics (charts and bar graphs are great) and human interest. The Ideal solution: the Case Study. If the headline is compelling, and the story plausible it will be accepted as real. There are so many (everybody has them) that they are accepted at face value. The fact that the GREAT majority is trivial, irrelevant or actually misrepresentations gets forgotten.The problem:
Most don’t pass the legitimate “smell test” of a truly analytic or statistical analysis.

The most obvious flaws:

1. Even terrible products or services are liable to work sometimes and with some customers, so let’s pick the best of these and present them as typical!

2. Most case studies typically fail to exceed the now well recognized “placebo effect” the short term benefit that can produce 20-60% improvements just from the customer believing this solution is logical and bound to work. This is especially true when the “sample or example” is small scale (less than several thousand), and the tracking lasts for only a short period of time (less than 2-3 years). By the way, how many case studies have you seen reporting results across multiple customers and over long periods of time?

Seth Godinmay have put it best: “We don’t like to admit that we tell stories, that we’re in the placebo business… Of course, we need to persuade ourselves that it’s morally and ethically and financially okay to participate in something as immeasurable as the placebo effect. The effect is controversial and it goes largely unspoken. “Sadly this manipulation and misrepresentation, has gotten so acceptable and pervasive there is even a company: Mega Placebos Plus ™, that promises to bring the “awesome power of placebos to you in dozens of new ways.” They claim “Absolutely nothing works as well as the multi-purpose placebos and placebo services we offer”. They guarantee it. Because “every product from Mega Placebos Plus ™ is so easy to use–there’s really nothing to it! And there’s nothing to worry about!”

3. Statistical word games are also big. For example, in one of our core businesses, assessing individuals to identify the best candidates for hire and the best areas for development for incumbents, the proof of effectiveness is accuracy in predicting the level of future performance of the candidate or incumbent. The statistical measure is a “validity study” or predictive analytics that confirm that higher assessment scores predict better performance. However, many competitors substitute reliability of their assessment (which simply says that an individual who takes the test twice will get the same score. i.e. if he gets a good score but fails to perform…the next time he will likely get a high score again (and probably fail to perform again.)

The second trick, typical with personality tests, type indicators, and style tests is to show the validity scores for the assessment measuring what its scales are labeled instead of future performance. In selection we really shouldn’t care if a candidate is extroverted we should only care if he/she will succeed in the job regardless if he/she is extroverted or not. Why? Because the real research has demonstrated that even in sales many of the best salespeople are extroverted…and many are not!

Bottom line: “Lies…damn lies…and statistics” (if you don’t understand the statistical tricks) is True and we haven’t even talked about the marketing tools of “operant conditioning”or the more controversial persuasion techniques.