Why A/B Testing Is For Idiots.

One of the biggest lies ever invented for business leaders is that A/B testing is a viable strategy for business growth.
Find a company anywhere who achieved massive, mind-blowing success by A/B testing their way to greatness. I dare you to find one. I’m begging you to find one.

It’s like the pink unicorn of business mythology.

The concept sounds wonderful — inspiring and hopeful — but it’s all just a big fairy tale. A distraction from reality. A friendly lie grown more credible every time it’s told. A/B testing is a flawed strategy. It doesn’t work.

Don’t get me wrong — testing any two variables will show you which of those variables performs better. That’s called experimenting. It works in science, so why shouldn’t it work incredibly well in business?

But is experimenting really the point of business growth?

The problem with A/B testing isn’t that you’re experimenting with the nouns and verbs that you use in your marketing language or the design that you create for your web presence. All of that is good and reasonable and likely to get you closer to where you want to be.

The real problem with A/B testing is that it just creates more information. Information that might not be credible, believable — or, more importantly, actionable.

Just because you get results doesn’t mean that your conclusions are accurate.

Maybe your testing is flawed in the first place.

Can you really be assured that two datasets deliver different results simply because of the wording changes you made? Can you rule out timing, demographic nuance, personal life experience, and a list of other dynamic variables that lead to fickle outcomes?

And even if the results you get back seem credible to you, is the data set you’re working with large enough to really make a difference? A lot of the discussion around A/B testing assumes that you’re making changes that thousands upon thousands of viewers will be a part of deciding the outcome.

That’s usually not the case. The data is flawed.

Data sets are so small and unqualified that it’s hard to be sure that the results you’re looking at would remain consistent if you were to do that testing all over again — which leads to the biggest issue of all when it comes to the problems of A/B testing.

Information has never been the secret to business growth. More information, better information, faster information — they are all great assets for business leaders. But information isn’t the secret to being successful. Transformation is.

And that’s the biggest lie sold by those selling A/B testing to you as a strategy.

That you will transform your business because you have more information. You won’t. It’s a lie. Knowing more doesn’t automatically lead to doing more. It’s helpful and important and something to which you should aspire, but it’s not the secret to massive, mind-blowing success.

Transformation is. And that’s something that a A/B test simply can’t do. You can’t A/B test your way to taking radical positions in the marketplace. You can’t poll test and qualify the potential of “doing it your way”.

You can’t measure that in numbers.

There is power in bucking conventional wisdom, defying the elders of your industry, and pursuing a better way with boldness. Any test you run on that will call you crazy. No matter how you word it, phrase it, design it, or polish it up, the A/B test results of radical behavior all come to the same conclusion — that you shouldn’t be doing it.

And that’s the reason why it’s such a big lie — because the one thing that can help you the most scores the lowest any way you test it.

Do it your way.

Don’t believe the lies of those who wish you to believe but success is nothing more than pushing the right knobs on the computer keyboard.

Stand for something. Lead boldly.

Watch your business transform before your eyes.

Go be awesome.


0 Replies to “Why A/B Testing Is For Idiots.”

  1. link bait? While I agree that you aren’t going to explode brand awareness doing A/B testing, you can really get beneficial results by doing A/B testing on landing pages. Our site gets anywhere between 500K to 1M visitors a month to specific landing pages. With that many visits you can really see the difference you are making. There is a reason why unbounce and optimizely are exploding.

    1. I don’t disagree that testing just about anything provides you enough information to make incremental progress. But is that really the point of our business efforts? To grind it out at a 3% to 5% growth?

      Find me one amazing business who has grown at 500% per year who has done that via A/B testing. Don’t tell me the names of people selling tools. Show me one Apple or Google or IBM or Tesla who grown by A/B testing…

      That’s my challenge — name one. It’s a cool tactic. It’s probably a piece of an overall strategy. But it’s worthless with a daring strategy, bold action, and killer foresight.


  2. Hey Dan!
    I handle business development and some other things over at http://instapage.com. Id love to extend a totally free version of our software for you to try on your website. It instantly plugs into your current site and you can create multiple A/B variants and drive traffic from any source without spending any extra time, the software does this for you automatically.

    If having a set of data in front of you that shows you one variant of a page converted 80% more than the other 3 wouldn’t you rather spend your $10,000 PPC budget on that page and not the other 3 that converted poorly?

    I’d love to show you and your readers more about what A/B testing can do for marketers, using our software or not. Thanks for the contribution and opinion!

    Kieran Daniels
    Director of Business Development

    1. NO. I don’t care about the data — it’s just a distraction. We subscribe to the theory of “have something awesome to say”. You don’t need to A/B test that. You just say it.

      I’m not a Neanderthal when it comes to technology. Frankly, we consider ourselves epic nerds in The EDGY Empire, but I’m not sold on quick answers to complex business challenges. There is no “automatic” when it comes to legendary business breakthrough.

      That all being said — landing pages and visitor intelligence are all cool ideas. They’re just not all they’re sold to be. That’s my “beef”…


  3. I can’t replace “A/B testing” with anything else in your sentence “Find a company anywhere who achieved massive, mind-blowing success by A/B testing their way to greatness.” and find such a company. My point is, no single action can be credited with delivering “massive, mind-blowing success”. (BTW, “transformation” is a great thing but it’s not something I can start doing tomorrow, so it doesn’t qualify.) Incremental improvements are fine and there are several ways of doing it, one of them being A/B. Proponents of A/B testing I know (e.g. Optimizely) never claimed that A/B testing is the only thing a business needs to do to grow. And, I know several businesses, including my own, that have achieved 5-15% incremental growth by using A/B testing. Amazon is also quoted as an example of a company that has achieved a lot of positive results with A/B. As for test sample quality, A/B testing is the only form of testing I know that conducts experiment on live traffic, so its quality is beyond compare. As for test sample size, tools like Optimizely allow the website owner to decide the confidence value with which they want the test to report a certain conclusion and work out the sample size accordingly. Someone who is happy with 67% CV can stop the test after 1 week. Someone else who wants 90% CV can keep running the test for a longer duration. The tool itself doesn’t restrict the sample size.

    1. I agree that no one single tactic has the power to achieve mind-blowing success. That’s what is most troubling to me about this discussion. A bad strategy will always yield bad results — regardless of the tactics.

      For instance, if you have lousy customer service you can A/B test your up-front engagement but it doesn’t matter much. You are losing most of your work on the back end.

      In the big picture of things I just don’t think you win by choosing what the crowd wants to hear. You have a belief and values and let then guide you to where you want to be. You don’t follow. You lead.


  4. Hi Dan! Wow, even from so much negative comments I’m seeing here, I’m buying this article. As a person who’ve practiced A/B tests for years to develop an effective CRO strategy for my e-commerce clients and also able to generate revenue of 7-figures in 6 months from zero via effective strategic partnerships and successful paid campaigns, I always felt like A/B testing is just another marketing gimmick. Yes, it does impact 5%-15% MoM growth (best case 30-50%) each time I take successful findings from experiments and make them live to all traffic. But still, not that ideal 500% growth that I wanted to achieve to impress our stakeholders and investors and eventually get us to the next round of fundraising.
    And yes, if you have a $10k budget on PPC to land them on the right landing page, I still feel that the decision making to spending it right is to ensure you have the b*lls to create a superb, one-of-a-kind looking landing page with revolutionary features that addresses actual customer problems that has a mass market demand instead of A/B testing simple copies and visuals. It’s like the ‘Do-or-Die’ kind of thing – it’s the same mindset when we all started our business; it’s whether we get to the unicorn, or we’ll crash – in between is just simply flowery (the icing) to make monthly reports look good; but the actual fact is that’s not the thing that gets us the money in our pocket every month.

    Even the Senior Director of Growth in Intercom thinks “Growth Hacking is Bullshit”- https://blog.intercom.com/growth-hacking-is-bullshit/ – it’s not that he’s entirely disagreeing to the whole industry of growth hacking – but he did highlight that minor tweaks don’t actually solve the actual problem of why we’re not gaining that revenue we want. It’s because simple copies and visuals won’t solve customer’s problem, but the real actual product feature is (solution). Growth hacks should be superbly mind-blowing and it has to be pioneering. Any other marketers who tend to think it’s something they can follow as well – has to realise that once it’s used, it’s totally outdated. Market adapts so quickly that we need to be 100x more creative in the way we deliver and market our business.

    Dude, I would love to connect with you.


  5. I wouldn’t go as far as “idiots” but definitely for kids. Not long ago read blog about A/B testing whether button on landing page should be orange and round or blue and square!? Seriously! So… first they designed something, then they payed around with it, then tested and finally… wrote a blog about it. Is this what marketers are being payed for these days? Kindergarten. Cool kids playing with crayons and getting badges for their scribbling’s.

Leave a Reply

Your email address will not be published. Required fields are marked *