This is the forth post in a series, you may want to read the previous posts first: part 1, part 2 and part 3.
Now that we learned all about split testing you probably already want to test two designs of your web site to see if you can improve your sales (and if you don’t you should) but there’s one more small thing you should know before starting – what to test first.
The first thing you need to test is your A/B testing software and your site, you do this by running your current design against itself – an A/B test where both option A and option B are identical.
Why are we doing this? first we want to see how a test runs – if you don’t have a huge amount of traffic you will probably see one of the options gaining a sizable advantage over the next for a short while, if everything is working over time this advantage should get smaller until both options get the the same conversion rate.
The time it takes both options to converge is the minimum time you should run future tests.
You should also remember the behavior of this test in future tests – if one options performs much better then the other it doesn’t mean a thing until the result is statistically significant.
And last but not least - this is a good test of your tools, if the two options do not converge or your tool report statistically significant results you know your tool is not reliable and you should switch tools.
In the next posts of the series I’ll talk about my tools and share some results.
posted @ Wednesday, March 10, 2010 5:45 PM