Analytic Troubleshooting: Narrow The Data Scope

August 29, 2006

Twice in the last 24 hours, I had trouble with analytics — once with Omniture’s SiteCatalyst and once with Google Analytics. Both times, I “solved” the problem (more accurately, finally understood it) by making my time period shorter and shorter. But I’ll only show the Omniture example.

The Omniture/SiteCatalyst problem was that the customer was seeing more orders than items put into the shopping cart. So even if everyone only bought one item (somewhat unlikely), and everyone checked out (incredibly unlikely), there still weren’t enough items to justify all those purchases. We had already verified that the purchase numbers were pretty good, and it wasn’t that hard to see that items put into the checking cart, as a percent of visits, was incredibly low compared to recent history.

“So when did this start?” asked Omniture LiveSupport (a question I should have asked myself.)

I pulled a report for the last 60 days and found this:

You can see the crash on August 1, and when I “zoomed in” on August 1 and did a report by hour, I was able to find exactly when it happened. IT tied it to the change that they made. (By the way, the report is Commerce > Shopping Cart > Cart Additions.)

Now, this seems very straightforward. I had a similar program with Google Analytics (and that one is actually being reported as a change request, or a bug), but only by narrowing and narrowing my parameters was I able to delete enough variables to understand that the problem was truly a GA bug.

Time isn’t the only parameter to use when troubleshooting but it sure is a great place to start.

Robbin Steif
LunaMetrics