Business is “supposedly” all about ROI. So, to attract new business most marketers believe we have to be able to prove it. A three-year P & L accounting of a customer’s return on their investment in this wonderful product or service could do the trick. Except they’re expensive and difficult if not impossible to track, and too boring, and too complex for most to read. Let’s face it, even if we could get them, P & Ls are not exciting enough to attract attention in the market clutter of claims, clamor, and clever hype! We need pictures, graphics, and human interest. The ideal solution: a Case Study. If the headline is compelling, and the story plausible, it will be accepted at face value. The fact that the GREAT majority are trivial, irrelevant or actually misrepresentations gets forgotten. The problem:
many don’t pass the legitimate “smell test” of a truly analytic or statistical analysis.

The most obvious flaws:

1. Even terrible products or services are liable to work sometimes, and with some customers, so let’s pick the best of these and present them as typical.
2. Most case studies typically fail to exceed the now well recognized “placebo effect” the short-term benefit that can produce 20-60% improvements just from the customer believing this solution is logical and bound to work. This is especially true when the “sample or example” is small scale (fewer than several thousand), and the tracking lasts for only a short period of time (less than 2-3 years). By the way, how many case studies have you seen reporting results across multiple customers and over long periods of time?

Seth Godinmay may have put it best: “We don’t like to admit that we tell stories, that we’re in the placebo business. Of course, we need to persuade ourselves that it’s morally and ethically and financially okay to participate in something as immeasurable as the placebo effect. The effect is controversial and it goes largely unspoken. “Sadly, this manipulation and misrepresentation, has gotten so acceptable and pervasive there is even a company: Mega Placebos Plus™ that promises to bring the “awesome power of placebos to you in dozens of new ways.” They claim, “Absolutely nothing works as well as the multi-purpose placebos and placebo services we offer.” They guarantee it because “every product from Mega Placebos Plus™ is so easy to use–there’s really nothing to it! And there’s nothing to worry about!”

3. Statistical word games are also big. For example, in one of our core businesses, assessing individuals to identify the best candidates for hire and the best areas for development for incumbents, the proof of effectiveness is accuracy in predicting the level of future performance of the candidate or incumbent. The statistical measure is a “validity study” or predictive analytics that confirm that higher assessment scores predict better performance. However, many competitors substitute reliability of their assessment (which simply says that an individual who takes the test twice will get the same score. i.e., if they get a good score but fail to perform, the next time, they will likely get a high score again (and probably fail to perform again.)

The second trick, typical with personality tests, type indicators, and style tests is to show the validity scores for the assessment measuring what its scales are labeled instead of future performance. In selection, we really shouldn’t care if a candidate is extroverted, we should only care if they will succeed in the job regardless if they are extroverted or not. Why? Because the real research has demonstrated that even in sales, many of the best salespeople are extroverted…but many are not!

Bottom line: “Lies…damn lies…and statistics” (if you don’t understand the statistical tricks) is true and we haven’t even talked about the marketing tools of “operant conditioning” or the more controversial persuasion techniques.