Website Optimizer: 5 Non‑conversions Required

Did you know that you needed five non-conversions for each of your combinations in Google’s Website Optimizer? Yeah, me neither.

When I was at Google last week, I was showing the results of a 72 way test to Eric, one of the engineers who is behind Google’s Website Optimizer product. (In case you are not familiar with WO, it is Google’s free multivariate software. It enables website owners to test multiple pieces of a page and even a funnel or series of pages to see which combinations of the variables convert the best. Read why multivariate testing is like a game of Clue.)

Well, anyway, I showed it to Eric, and he pointed out that even though we had a few combinations with greater than a 97% chance to beat the original, one variation was’t even showing up in the Combinations tab. “And that’s your best one so far,” he said. It converted 14 out of 17 times, the dashboard proclaimed. The reason that the combination data didn’t show, he explained, was that there needed to be enough data in general for that test, and there needed to be at least five conversions and (here is the part I didn’t expect) five non-conversions. In the case of this particular combination, there were only three non-conversions. (17-14, right?)

So why the need for non-conversions? After all, the more conversions, the merrier, right? Well in fact, no. “It does best when the conversion rate is 50%” Eric emailed to me.

In order to begin to wrap your head around this (assuming you aren’t a statistician), you have to stop thinking that conversions are good. Instead, there are two states here, a or b. Conversion or non-conversion. Heads or Tails.

So let’s say that we take a finite, maybe only 20, visitors and estimate the conversion rate based on the fraction of the 20 that converted. Is that a good estimate of the true conversion rate? The holy “Law of large numbers” in statistics says that the average conversion of a finite set of visitors becomes a good estimate of the true value as the number of trials becomes large. But, the fine print in this law states that the number must be really large when the true mean (the true conversion rate) is very small (very few convert) or large (nearly all convert). In fact, for the estimate based on finite visitors to be good you need to have enough counter examples. “Counter examples” are non-conversions when the conversion rate is high, or conversions when the conversion rate is low.

I know, you want to know where they got the number five from, and why it’s an absolute number and not a percent. Me too. I’m thinking that the issue is, it can’t be a percent, because if you have only 20 visits, you need a high percent, and if you have 100 visits to that combination, you need a low combination. By fixing a specific amount, you make sure you get something, for both conversions and non-conversions. For both heads and tails. But that last part is speculation. Now if you want to learn something really cool about WO, go over to ROI Revolutions’s blog and read Shawn Purtell’s magnificent piece on the marriage of GA and Website Optimizer.

Robbin Steif