Multivariate Testing: Mrs. Peacock In The Conservatory With The Candlestick

It’s true that A/B testing, or better yet, A/B..N testing, is the art of scientifically testing how one page performs against a second and a third (and a fourth.) However, web marketers now have at their disposal tools that are much powerful than A/B testing. A/B merely splits the traffic. But multivariate testing (MVT) enables you to play Clue with your web page (and it’s not just for the Big Guys anymore.)

So let’s say that you want to test a page of your site. You are considering two alternative headlines, a new picture, three different buttons, and a new block of text. That means that you have 3*2*4*2=48 variations (because, for each element you are testing, you have both the new version and the current version.) Can you imagine creating 47 different pages to test? Of course, you don’t have to create the 48th page, the one you already have that one – the current page — but that’s still a lot of work, even if it is cut and paste.

Now we will see the beauty of multivariate testing. All you need to do is create the three new buttons, the alternative picture, the one new block of text and the two alternative headlines. Then, work with a software package like Offermatica or Google Site Optimizer.

The software serves up variations of the tests each time and after you have enough data, you will win your game of Clue: you know which title works with which picture and which test and which button. “Works” will have been defined by you at the beginning of the test. Often, it is defined as “which combination increases my conversion rate the most?” (The software packages give you a place to define a “success URL,” which is often your “Thank you very much” page.)

It is perhaps not fair to put Offermatica and Site Optimizer in the same sentence. Offermatica uses a methodology that figures out the winning combination even without testing every combination. This means that you can have less traffic and still find out the answer in a reasonable amount of time. Site Optimizer, on the other hand, tests every one of those 48 combinations, so you need more traffic to get a statistically significant result. On the other hand, Site Optimizer is free.

How much traffic is enough traffic to use Site Optimizer? Well, you always want your test to run for at least a week, and probably don’t want to run it for much longer than a month. If you run the test for too long, the environment around you will change (politics, the weather), and the data from the beginning of the test no longer match your conditions. (Would you use test data from 1999 to lay out your website?  Didn’t think so…) If the page you are interested in testing gets 100 views/day, it currently has a conversion rate of 2%, and you are expecting to increase that to a 3% conversion rate (from that page) with your changes, your can get a statistically signifcant test in 36 days. Of course, if you don’t see those lifts in conversion coming from your new page, you’ll need to keep your test alive until you have more data to declare a significant winner.

Maybe it will be Colonel Mustard in the billiard room with the rope…