20 Simple Ways to Fuck Up your AB tests

Companies waste a huge amount of time on A/B tests that tell them nothing or actively mislead them. Craig Sullivan, a conversion optimization consultant, says he has made over 60 different kinds of mistakes with split testing. In this talk he goes through the most common ones, technical and cultural, and argues that there is almost always a people problem behind a technology problem.

You'll learn why stopping a test the moment your tool shows 95 % confidence is a mistake, how long to run a test and why you should test whole business cycles. Craig explains why, in his experience, 40 % of tests are technically broken, why executives' guesses don't work, and what it means to test things you didn't know you could test. He sets out five pillars of a working optimization practice and ends with where he thinks testing is heading.