Dan Waldschmidt

by Dan Waldschmidt

September 30, 2014

Why A/B Testing Is For Idiots.

One of the biggest lies ever invented for business leaders is that A/B testing is a viable strategy for business growth. Find a company anywhere who achieved massive, mind-blowing success by A/B testing their way to greatness. I dare you to find one. I’m begging you to find one.

It’s like the pink unicorn of business mythology.

The concept sounds wonderful — inspiring and hopeful — but it’s all just a big fairy tale. A distraction from reality. A friendly lie grown more credible every time it’s told. A/B testing is a flawed strategy. It doesn’t work.

Don’t get me wrong — testing any two variables will show you which of those variables performs better. That’s called experimenting. It works in science, so why shouldn’t it work incredibly well in business?

But is experimenting really the point of business growth?

The problem with A/B testing isn’t that you’re experimenting with the nouns and verbs that you use in your marketing language or the design that you create for your web presence. All of that is good and reasonable and likely to get you closer to where you want to be.

The real problem with A/B testing is that it just creates more information. Information that might not be credible, believable — or, more importantly, actionable.

Just because you get results doesn’t mean that your conclusions are accurate.

Maybe your testing is flawed in the first place.

Can you really be assured that two datasets deliver different results simply because of the wording changes you made? Can you rule out timing, demographic nuance, personal life experience, and a list of other dynamic variables that lead to fickle outcomes?

And even if the results you get back seem credible to you, is the data set you’re working with large enough to really make a difference? A lot of the discussion around A/B testing assumes that you’re making changes that thousands upon thousands of viewers will be a part of deciding the outcome.

That’s usually not the case. The data is flawed.

Data sets are so small and unqualified that it’s hard to be sure that the results you’re looking at would remain consistent if you were to do that testing all over again — which leads to the biggest issue of all when it comes to the problems of A/B testing.

Information has never been the secret to business growth. More information, better information, faster information — they are all great assets for business leaders. But information isn’t the secret to being successful. Transformation is.

And that’s the biggest lie sold by those selling A/B testing to you as a strategy.

That you will transform your business because you have more information. You won’t. It’s a lie. Knowing more doesn’t automatically lead to doing more. It’s helpful and important and something to which you should aspire, but it’s not the secret to massive, mind-blowing success.

Transformation is. And that’s something that a A/B test simply can’t do. You can’t A/B test your way to taking radical positions in the marketplace. You can’t poll test and qualify the potential of “doing it your way”.

You can’t measure that in numbers.

There is power in bucking conventional wisdom, defying the elders of your industry, and pursuing a better way with boldness. Any test you run on that will call you crazy. No matter how you word it, phrase it, design it, or polish it up, the A/B test results of radical behavior all come to the same conclusion — that you shouldn’t be doing it.

And that’s the reason why it’s such a big lie — because the one thing that can help you the most scores the lowest any way you test it.

Do it your way.

Don’t believe the lies of those who wish you to believe but success is nothing more than pushing the right knobs on the computer keyboard.

Stand for something. Lead boldly.

Watch your business transform before your eyes.

Go be awesome.


About the author

Dan Waldschmidt

Dan Waldschmidt doesn’t just talk about leveling up. He’s obsessed with it. He's set records as an ultra-runner and been the personal strategist for the leading business leaders of our time. He wrote a book, called EDGY Conversations that accidentally became a worldwide bestseller and continues to share his insights from the stage as a keynote speaker and on the blogs and podcasts you will find here. Most days, you'll find Dan heads-down, working on breakthrough strategies for his clients at EDGY Inc, a highly-focused, invite-only, business strategy execution company based out of Silicon Valley.