Web analytics faux pas

This is a guest post by Ed Freyfogle, co-founder of property search engine Nestoria.

It’s often claimed that the advantage of the internet versus traditional media is that
everything can be measured. Rarely mentioned though is the pain experienced by industry
insiders working in the fog of confusion caused by such a hyper-analytical environment.
In this rapidly evolving new medium fundamental misunderstandings are surprisingly common.

A few of the most striking, and common, examples:

• Mistaking users for customers – Some people use the terms ‘users’ and ‘customers’ interchangeably. That’s nonsense. Customers are people who pay you. Users are people who use your service. There can certainly be overlap (ex: Amazon), but for most media sites customers means advertisers. Don’t misunderstand, I’m not saying users aren’t important. You should certainly focus on user experience, but mixing customers with users shows miscomprehension of the media business model. If the terms are interchanged, alarm bells should start ringing.
• Using the term ‘hits’ – Before last.fm there were things called record companies. They produced hits. I have no idea what is meant when I hear someone say a website has a lot of “hits”. And evidently nor do they.
• Belief in pageviews – Conceptually pageviews seem straightforward: a measure of the number of times users view your content. Sadly it’s not that simple. The interweb is swarming with bots querying servers for content that no human will ever see. Secondly, in the age of AJAX and widgets it’s hard to quantify what exactly a pageview is. Most importantly though, users are spending more time online, generating (or “consuming” as hip media exec likes to say) more and more pageviews. Facebook and Bebo are prime examples. As supply of pageview inventory rises, value (CPM) drops. None of this is news – see last year’s discussion re: Death of the Pageview – but it’s amazing how people still cling desperately to the pageview metric.
• Rampant use of averages – any number worth knowing the average of is worth turning into a histogram. You don’t have ‘average’ users. Most likely your site is visited by power users, recurring visitors, and occasional passer-by’s. Using averages is a fast path to shaky conclusions.
• Blind faith – There’s nothing worse than ‘true believers’ – those who never question the numbers put forth by their analytics system. Software has bugs, can be badly configured, garbage in still leads to garbage out, and different tools measure different things. As an example, Google Analytics often disagrees with Google Adwords. Tools just give you data, data needs to be analysed and understood to become information.

Things I haven’t covered: confusing correlation and causation, those who endlessly argue why comScore and Hitwise are all wrong, and (my personal favorite) flippantly tossing ‘it could be seasonality’ into any discussion of analytics. In short, the true potential of the mass of data the internet generates lies not in its quantity but in its thoughtful application.

BTW, in the three minutes it took to read this post, my RSS reader generated a few thousand pageviews.

(Photo from maisonbisson @ flickr)