Compensating For The "Streetlight Effect" Observational Bias In Google Analytics

Observational biases often infect analysis, and can convince businesses to make incorrect assessments of their website data. In 2010, David H. Freedman coined the term “Streetlight Effect” for an observational bias represented in an old humorous science anecdote. This Streetlight Effect can very easily affect the insight gained from Google Analytics, should you not be aware of it, and not take measures to properly shed light on your data.

What is the Streetlight Effect?

The “Streetlight Effect” is named after an old late night parable told by scientists gathered ’round the flickering bunsen burner’. It goes as such: Late one night, a rookie police officer came across a stumbling visibly drunk man in a parking lot, intently examining the ground. “What are you doing, drunk man,” asked the police officer. “I dropped my wallet,” the drunk man slurred back at the police officer, reeking of gin and stale cigarette smoke. (blog author note: I’m trying to give the story a little more character than it normally gets) The police officer looks around and seeing no wallet asks, “Are you sure you lost your wallet here?” The drunk looks at the officer and belches out  “No, actually I dropped it over in that dark alley.” The rookie officer is stunned. “If you dropped your wallet in the alley, why are you looking for it here in the parking lot?” he asked incredulously. “The light here’s better,” hiccuped the drunk.

Freedman writes about why scientific studies are often wrong, and how the “Streetlight Effect” is to blame in his book Wrong.

Put more simply, the “Streetlight Effect” is when we look for something, where it’s easier to look, not necessarily in the best place to find the most accurate results. This is a frequent unintentional problem that people have when using Google Analytics, particularly when they work in default configurations.

A Common Streetlight Effect Victim: The Shopping Cart

A good example of a common victim to the Streetlight Effect within Google Analytics is the shopping cart, and the ecommerce flow visualization. Let’s take a generic ecommerce website. By default, Google Analytics won’t even track the transactions on the site. That’s a very dark alley. How can you gain any insight into how your website helps or hurts your ecommerce transactions without tracking the revenue!

Assuming we turn on and track our transactions, that’s not the end of the dark alley. We now have transactions, but we haven’t set up a transaction as a goal, nor have we set up the steps of the funnel.

Here’s an example of a funnel visualization I saw recently.

It steps the visitor from the shopping cart, to the checkout, to the goal completion of placing an order.  It shows 55% of the visitors leaving during the checkout process. That might look like a brightly lit parking lot, but it’s not. It’s actually a dark alley. it turns out that the checkout page contained ajax code with 5 different internal pages and steps asking the user for different information, shipping information, delivery information, etc. But we had no data for those steps. The recommendation here was to separate those steps with virtual pageviews* so we could shine light onto that checkout page by breaking it out into 5 separate pages and steps in the goal funnel.

*Quick note on virtual pageviews. Some people avoid using virtual pageviews because of fears of pageview inflation. This is certainly true if you’re using a virtual pageview rather than an event on pages for when a banner shows to a user, or something of that nature. However if you have someone proceeding through a form, and they have to click to proceed to the next stage of the form, you can often infer that as a perfect example of a time TO USE a virtual pageview. Even though the main page isn’t changing, the user is forcing a change of the “page” virtually by proceeding to the next stage of the form.

We start with a dark alley with no data on transactions at all, and go to one where we find out that mobile users leave in droves in step 2 of the checkout process and therefore convert at an absurdly low rate, helping us identify what actually turns out to be a technical problem with our shopping cart.

Other Common “Dark Alleys”

Another common Dark Alley is to have AdWords Cost Source Settings not applied. If you are running paid ads you should be sure to properly connect your AdWords account with your Google Analytics data. If you are running paid ads on another third party service, you should take advantage of the new Cost Data Import feature within Google Analytics to import your spending in these other services and connect them to your website data. The only way to truly know if your ad spending is working is to connect your ads with their ultimate goals on your website.

Social Engagement (ie clicking on a like button) is also generally not properly implemented by default, and therefore a Dark Alley. Initially, Google Analytics will only track Google+ clicks on your website, and not Facebook, Twitter, or any other social website. Who is clicking these buttons? How do they relate to your data and conversions? It’s fairly easy to set up your website to contain the code which allows Google Analytics to record these social events, and the only way to shed light on the Dark Alley of social sharing is to make sure you set it up correctly.

Downloads are a big source of data darkness. By default Google Analytics will not track someone who clicks a link to download a PDF on your website. If a big part of your website involves providing downloads to your customers, it’s important to track which documents are actually downloaded, and by whom. By setting up event tracking on those downloads you’re able to see which documents get the most downloads. Interestingly, you still have a minor dark alley here because you won’t see any documents which weren’t downloaded at all.

How to Counter the Streetlight Effect using Google Analytics

1. Goals & Funnels (w/ virtual pageviews)

As we mentioned above, be sure to know what your goals are on your website. Once you know what your goals are, be sure to set them up as goals within Google Analytics. If there are steps in a process a user has to follow to get to the goal, be sure to set them up as steps so you can view the goal in a Funnel Visualization. If you need to set up some virtual pageviews to compensate for an ajax form, you should do so.

2. Apply AdWords Cost Source Settings

Don’t forget to apply these cost source settings. It’s generally not hard to set up correctly unless you have a fairly complicated AdWords account setup or use a third party agency, but usually one way or another you should be able to get them connected.

3. Implement Social Engagement Tracking

For the most part this involves having code placed in the head and foot of all your pages. On dynamic sites this is very easy. On static sites of thousands of pages this could be a bit harder to implement. If you use a tag management system, it can be even easier. Here are a couple of links to older blog posts on how to implement social engagement tracking as well as a wordpress plugin to do it for you.

Tracking Your Social Engagement With Google Analytics

A Google Analytics Social Engagement Tracking WordPress Plugin

4. Event Tracking on Downloads, External Links, and More

Event tracking might be the biggest way outside of goals for shining light on your Dark Alleys. If something happens on your website that stands outside the normal pageview, you might want to consider using events to track them. downloads. clicks on external links to other websites. clicks on social sharing (in addition to the track social above). clicking on a banner. viewing a specific type of widget. clicking on a navigational menu. picking a category in an advanced search. You name it. If the user is doing or seeing something that you otherwise can’t track, you might want to consider putting an Event on it. There are tons of ways you can use events, but here are a couple of examples. One covers how to track YouTube video plays embedded on your site, and the other is used to track form abandonment.

Easily Track YouTube Videos with Google Analytics

How to Track Form Abandonment in Google Analytics

5. Custom Variables (and soon Custom Dimensions)

When is one user on his desktop the same user on his phone, in your analytics? Who are your members vs your non-members? Which users are male or female? What’s their favorite Sci-Fi franchise: Star Wars or Star Trek? There are so many pieces of information you can know about your users, that it’d be a shame to have those hide in a Dark Alley. Shed some light on this information by using Custom Variables. You’re limited to 5 with the free version of Google Analytics, but that will be expanding soon with the release of Universal Analytics to 30 dimensions. Do your members use your website differently than non-members? Do the Star Trek fans spend more money in your online store than your Star Wars fans? Need some ideas for Custom Variables? Check out my old post here.

20 Ways to Use Google Analytics Custom Variables

6. Utilize Custom Alerts

I’m a fan of Custom Alerts. I rave about them to any who listen. Sometimes it’s hard to identify the Dark Alleys. They’re dark after all, and you’re in a brightly lit parking lot. A full suite of custom alerts can help you identify when odd things are happening on your site, which can help you identify new areas for you to shine a spotlight on. Here’s my list of a bunch of Custom Alerts that you can use to get started.

55+ Google Analytics Custom Alerts – The Check Engine Light For your Data

One Last Note: Try to Limit Guesswork

The key to avoiding the streetlight bias, is to learn to shed light on the dark alley, not avoid it. If you want to improve your revenue, you’re not going to do that looking at your revenue by source/medium if you don’t have that revenue being tracked. Think of the dark alley also as the lines that connect the two brightly lit streetlamps. It’s your goal to place enough lights in enough places through the use of events, virtual pageviews, transaction tracking, and custom variables, that you can connect everything together.  Never settle for looking for your wallet in that parking lot. Sometimes there’s nothing you can do, like when that dark alley actually belongs to a third party that won’t let you put your tracking code on their site. But otherwise, do your best to destroy all your dark alleys.