The Art And Science Of Analysis, Part 2

January 7, 2013 | Dorcas Alexander

In part one I wrote that you should use both sides of your brain to do analysis. Use curiosity and intuition, and at the same time rely on structure and evidence. What exactly do I mean? Let’s take a look at a real-life example.

Suppose I work for a university and we’ve introduced a new section on our website. We hope the new section, a center for news and events, will boost views of content that’s previously been overlooked or underutilized. Three months after the section launch, I’ve been asked to find out how this new section is doing. What do the data show?

Define What You’re Looking For

First, I need to define what “success” means. I can’t know how the new section is doing if I don’t quantify what it would mean to “do well”. I’m at a university, so I have many audiences – current and prospective students, faculty, staff, researchers from other institutions (to name a few). What do I hope these audiences are viewing in the new section and what actions do I hope they will take?

  • Who is in my target audience(s)?
  • What content do I want them to view?
  • What action(s) do I hope they will take as a result?

My right brain considers all the possibilities, while my left brain figures out which ones are quantifiable (and which ones I’ve actually collected data for). In this example, let’s say I’m hoping that prospective students are viewing content related to student activities and events, and that they are motivated to browse degrees and courses and (the ultimate goal) to become registered students.

In the ideal scenario, I’m already tracking those pages and actions, because I already knew they were valuable. I may or may not have identified prospective students in any way other than the fact that they are browsing degrees and courses. If I’m using Google Analytics, I may have set a custom variable to identify prospective students’ subsequent visits based on some previous action or page view.

Define the Most Useful Data Sets

After I decide what I’m looking for, I want to collect several sets of data to analyze. The two most basic parameters I can use to define my data sets are the date range and data source. By varying these two parameters, I get data sets that produce meaningful comparisons.

In the case of the new section that launched three months ago, I am interested in how its performance has trended in that time. So I’ll look at daily or weekly totals over the three-month date range and examine the trend. Is the new section experiencing slow, gradual growth? Did it spike initially and then fall off? There may be many factors that affect the “curve”, which might resemble more of a mountain range with multiple peaks and valleys.

What internal factors such as semester start/end dates, events unique to my university, or specific marketing efforts could have had an effect? What external factors such as current events, weather, or even international holidays might be involved? I’ll need to use my right brain to consider as many as I can.

  • What date range is relevant?
  • What internal and external factors affect data points in that time frame?

Since it didn’t exist three months ago, I can’t compare the new section’s performance to its previous performance. But I can compare visits that include the new section to visits that don’t include it. How often do prospective students browse courses and degrees (or register) when they view the new section of the site?

Without comparing these two data sources, we’d have no idea whether the number of actions taken were significant. Just because people visit the new section and then browse courses doesn’t mean they wouldn’t have done so anyway. So we want to look at what percentage of people normally browse courses without viewing the new section.

Follow-up avenue: What percentage of people browsed courses last year during the same three-month time period? If it’s significantly different from this year’s percentage (who didn’t view the new section), what factors could account for that difference? Left brain: calculate the significance, right brain: probe for the answer to “why?”

  • What data set can I compare and contrast with this new data set?
  • What additional date range may be relevant?

Ask Why and How Can We Do Better

Finally, I want to see where there are opportunities to improve. What actions can I take that might bring more prospective students to the new section of the website, and get more of them to browse degrees and courses (a different section) and ultimately register?

I’ll look through my data for clues to how visitors found the new section of the site. What keywords did they search for? Were any of my marketing efforts particularly successful? How were visits that started on a page of the new section different from visits that started on the home page or some other section?

I’ll also want to examine more closely how much effect the new section had on visitors taking those valued actions. Did visitors go immediately from the new section to courses or degree information? Where exactly is my new section driving visitors – which content comes next (or do they leave the site altogether)?

  • What keywords or marketing efforts can I use to bring valuable visitors?
  • Which content in the new section is the most useful and which content should be dropped or rewritten?

Writing up this relatively simple example made me realize even more what a non-linear path I take to analysis. Every question branches into more questions. I still think it’s a good thing to stray “out of focus” for several branches, in order to cast a wide net and consider more possibilities.

Just remember to bring the analysis back “into focus” by returning to your definition of success. In my example, three months really isn’t very much data with which to gauge success, but it will provide a preliminary answer to “how’s the new section doing?” and definitely create a benchmark for future analysis.

Your turn: There are many branches this analysis could take, but I didn’t list here. If you’ve faced a similar scenario, what questions did you investigate? Please share in the comments.