Compete.com, a Kantar Media company, today released its ranking of the top 250 websites for August 2011 (click here for the top 50).

We can’t repeat this often enough, but take these stats with a grain of salt when it comes to accuracy.

We’ve looked at the top 250 all the same, because the list tends to depict trends also visible with other analytics service providers, such as comScore and Google Trends.

According to Compete.com data, which is pulled from its PRO Enterprise edition, notable traffic increases registered in August 2011 were to Facebook.com, Bing.com, YouTube.com, Amazon.com and eHow.com (all up roughly 4 percent month-over-month) and Twitter.com (up about 12.7 percent month-over-month and over 27.5 percent compared to August 2010).

Microsoft also appears to be on fire: traffic data for Bing.com, Live.com, MSN.com and Microsoft.com suggest steady traffic increases for all. Another winner: Weather.com, with traffic up 16.4 percent month-over-month and 34 percent compared to August 2010.

Some of the traffic losers include blogging platforms such as Blogspot.com (down 5.5 percent compared to July 2011) and WordPress.com (down 2.7 percent) as well as information websites such as YellowPages.com (down 7.8 percent) and iMDB.com (down 6.6 percent).

Other notable ‘losers’: Netflix.com (down over 4 percent compared to July 2011) and daily deal sites Groupon.com and LivingSocial.com, which are down 28.6 percent and 4.25 percent month-over-month, respectively.

It’s worth noting that all of the websites who’ve seen their traffic decrease in August 2011, according to Compete, are up (some significantly) year-over-year – with the exception of iMDB.com.

Honorable mention goes to Myspace.com, which is down over 55 percent compared to August 2010.

Earlier today, I was checking out some new questions in the TechCrunch topic area on Quora. One in particular caught my eye: How was TechCrunch traffic affected by their major redesign in July 2011?

This has been something I’ve seen asked here and there given the radical changes we implemented — and, I assume, given the audience issues Gawker faced after their recent redesign. Mostly, people seem to want to know: is TechCrunch tanking?

I was set to weigh in, when I noticed that someone else already had. This person (not affiliated with TechCrunch) painted a picture in which our site was essentially crashing and burning since the redesign (the answer has since been removed by Quora, presumably due to down-voting). Their source? Compete.

Dear Internet, I thought we’ve been over this? If you cite Compete or Alexa for anything other than making fun of them, you’re officially a moron. How bad is their data? Well, in the case of Compete, if you reversed their chart, then it would be much closer to being correct than it currently is. I seriously wonder if they’re tracking anti-visits or some new metaphysical stat I’m not aware of?

TechCrunch will set a new all-time record for traffic this month (both in uniques and pageviews), breaking the previous record — set last month. And that broke the previous record set the month before that. In other words, things are on the up and up. How do I know this? I have the luxury of seeing directly-measured results from both Google Analytics and WordPress.com’s own analytics area. Both confirm that Compete and Alexa are absolutely worthless when it comes to this type of measurement.

I mean seriously, the data from each service is so bad that I’m not clear how either is still operating. I have to cut the stat-searching public some slack because they are two of the only free public tools out there for gauging traffic data. But seriously, you’d be better off just guessing.

Yes, Quantcast is much better, but they have their own issues too, which is why we don’t expose data publicly that way.

The newer Google Trends for Websites seemed to be halfway decent for a while, but in the past year, they’ve also fallen off a cliff in terms of accuracy, it seems. Given that Google never talks about this product any more, I’m going to assume it’s one of many that has fallen into neglect — which is too bad.

As for Alexa, which is owned by Amazon, I have no idea what’s wrong with them. It seems to be more of a landing page for advertisements above all else now.

And yes, I know Compete focuses on U.S. Internet usage. But looking at our charts, that segment of the data is even more inaccurate when compared to reality. Compete has always been rather quiet about the way they actually gather data — and at times, it has sounded rather sketchy — but the fact of the matter remains constant: the data stinks.

But I will give Compete some credit: their data showcasing their own downfall does appear to be fairly accurate, as backed up by comScore.

As for the Quora question itself, I answered it there. Long story short: traffic is up pretty significantly since the redesign. Thanks all!

When it comes to public-facing web analytics, basically, they all suck. We in the press are sometimes forced to use tools like Alexa and Compete for comparison’s sake, but using either for absolute numbers is extremely flawed and basically worthless. Naturally, those companies always disagree with us when we say such things. But a new bit of information may put that disagreement to the test.

The Compete chart for Compete.com is perfect. If the chart is to be believed, Compete is in a total tailspin. According to their data, they’ve dropped from about 750K unique visitors in January 2010 to roughly 250K in December 2010. The numbers for total visits are even worse (dropping from 3.25 million to about 750K). Both have been in decline every month since March. And neither shows signs of stopping.

For all I know, these numbers are correct. But all I have to go on is what Compete is telling me. And something tells me that Complete themselves will dispute them, just like every single other site on the planet does when you use Compete numbers in a blog post.

For what it’s worth, Alexa doesn’t seem to have enough data to rank themselves. How convenient. Or sad.

Update: And here are the numbers from comScore (a much, much more reliable service) showing a similar plunge in U.S. numbers (below). But as you can see the absolute numbers are far different.

[thanks David]

A couple of weeks ago, comScore came out with a report that said Microsoft’s Bing had reached an all-time high market share of 11.8% in November 2010.

According to rival Compete, however, Bing’s market share is actually much larger than that.

Based on search data from its panel of more than two million US-based internet users, the Kantar Media company says Bing-powered search engines as a whole grew 4.3 percent month-over-month in query volume, driving Bing’s total market share up by 1 percentage point.

Bing (‘MSFT’) and Yahoo’s search products (which are powered by Bing these days) had 14.4% and 14.6% market share, respectively, which means the combined market share of the search engines rose to a healthy 29% in November 2010, according to Compete’s search data.

Bing.com also saw the highest growth in the number of unique visitors, with a month-over-month increase of 7.4 percent (Yahoo’s number of UVs actually declined .3 percent). In November 2009, Bing’s market share was just over 10%, according to Compete.

Both ASK and AOL’s share remained flat from October 2010 to November 2010, which means there’s only one search engine whose market share effectively declined last November: Google‘s.

According to Compete, Google has seen its query volume decline for the second month in a row now, with a recent 1.1 percent month-over-month drop. Compete registered 66.4% market share for the search engine, down a noteworthy 7 percentage points compared to November 2009.

Excuse the blurry screenshot, but this was the best I could do (see source image).

Black Friday, or the day after Thanksgiving, has always been known as one of the year’s biggest holiday shopping days. But in the past five years, Cyber Monday, the Monday following Thanksgiving, has become a a serious shopping day for online sales and promotions. Last year, Cyber Monday brought in $887 million in sales compared to $595 million in online spending on Black Friday. Today, Compete is releasing a survey that indicates that once again consumers could be choosing Cyber Monday for shopping as opposed to Black Friday.

According to Compete’s data, 45% of respondents indicated they will do their holiday shopping on Cyber Monday, versus 37% who plan to shop on Black Friday. Interestingly, Black Friday shoppers are planning to spend more money than Cyber Monday shoppers, with Black Friday shoppers averaging an expected $353 and Cyber Monday shoppers averaging an expected $233.

Compete says the increased spend is likely due to more people shopping for high-end items such as electronics, clothing, toys and games on Black Friday than on Cyber Monday. The report shows that 11%, 32% and 23% more people will shop for these items, respectively, on Black Friday than on Cyber Monday.

Last year Hitwise reported that Amazon took the top spot on Cyber Monday and Black Friday.

It’s not surprising that Cyber Monday is becoming more popular than Black Friday, especially if deals are equally as good on both days. Who wants to wait in long lines at 4 am the day after Thanksgiving if they can just click Buy It Now on their computer?

Online analytics company Compete has just published its ranking of the top 50 websites for September 2010, giving some insights into current visitor trends (and not absolute numbers, as the company tends to undercount traffic for most websites).

Compete’s data compilation shows increasing traffic to Microsoft’s search engine Bing (up 11.7 percent for the month and 108.5 percent for the year) as well as Ask.com (up 8.7 percent for the month and 75.3 percent for the year). On the other side of the spectrum we – unsurprisingly – find MySpace (unique visitors down 5.53 percent for the month and 19.1 percent for the year) and MapQuest (down 5.8 for the month and 22.1 percent for the year).

If MySpace’s redesign will help buck the trend remains to be seen.

Perhaps more surprisingly, Yahoo’s Flickr.com seems to have lost some of its shine lately, showing a 14 percent decline in unique user visits in September 2010.

And what about IAC’s Ask.com, which actually jumped over online juggernauts such as Amazon.com and MSN.com last month, according to Compete’s data. The search engine is now ranked in sixth place, trailing sites like Wikipedia, Yahoo, Google, Facebook and YouTube.

Other winners include Disney’s Go.com and Mozilla.com, which showed the largest monthly unique user visitor gains (15.8 percent and 30.5 percent, respectively).

We were recently pointed to this post from last month by Ben Edelman, a Harvard privacy advocate. In it, he details the data the Upromise toolbar collects and sends out. This toolbar is used by college students looking for savings on various items across the web, and can be quite useful. But until a few weeks ago, it appears they were also sending web browsing (and more personal) data to Compete without anyone’s knowledge. Writes Edelman:

As shown in the “host:” header of each of the preceding communications, transmissions flow to the consumerinput.com domain. Whois reports that this domain is registered to Boston, MA traffic-monitoring service Compete, Inc. Compete’s site promises clients access to “detailed behavioral data,” and Compete says more than 2 million U.S. Internet users “have given [Compete] permission to analyze the web pages they visit.”

He continues:

Upromise’s installation sequence does not obtain users’ permission for this detailed and intrusive tracking. Quite the contrary: Numerous Upromise screens discuss privacy, and they all fail to mention the detailed information Upromise actually transmits.

The Upromise toolbar installation page touts the toolbar’s purported benefits at length, but mentions no privacy implications whatsoever.

If a user clicks the prominent button to begin the toolbar installation, the next screen presents a 1,354-word license agreement that fills 22 on-screen pages and offers no mechanism to enlarge, maximize, print, save, or search the lengthy text. But even if a user did read the license, the user would receive no notice of detailed tracking. Meanwhile, the lower on-screen box describes a “Personalized Offers” feature, which is labeled as causing “information about [a user's] online activity [to be] collected and used to provide college savings opportunities” But that screen nowhere admits collecting users’ email addresses or credit card numbers. Nor would a user rightly expect that “information about … online activity” means a full log of every search and every page-view across the entire web.

Shortly after Edelman’s post (and a follow-up PCMag.com post), Upromise changed their privacy policy to alert their users that this data is being sent out. But the company declined to state how long the issue had been going on.

Privacy implications aside, it’s interesting that this is one of the ways Compete was gathering data. And it would be good to know where else they get it from. On their site, they only vaguely note that they have “developed a unique methodology created by experts in the fields of mathematics, statistics and the data sciences to aggregate, transform, enhance and normalize data in order to estimate U.S. Internet traffic.” They also claim to have over two million members — but apparently, at least some of them (such as the Upromise toolbar users), don’t know they’re members.

I’ve sent a message to Compete asking them what other means (other toolbars, etc) they use to gather their data. In light of this Upromise fiasco, it seems wise that they should disclose that kind of information. I’ll update if and when I hear back.

You can read the whole post over at Calacanis.com. He doesn’t pull any punches (in fact he goes on a tangent about punching bullies in the face as a kid) He suggests that companies refuse to pay Comscore for the service, and that investors short the stock.

Comscore investor Fred Wilson laid into Jason with a couple of comments on a copy of the post on Posterous. He also randomly dragged me into the argument (I think he’s still mad about the Zynga stuff):

jason, since you’ve slandered me, i’ll respond here.
you don’t know what you are talking about. comscore (SCOR) is a public company. you can go look at their financials. they aren’t exactly printing money. it’s hard to measure the internet and they spend well over $100mm per year doing just that. they aren’t “shaking down” anyone. their move to a hybrid model is a reaction to many of the criticisms that people have had of their panel model over the years. but it isn’t cheap to manage that data either. someone has to pay for this. or of course we could all just let google do it for free. we know how that will play out. eric schmidt has said “analytics are infinitely monetizable” well for google they are. if we want a third party keeping everyone honest, the market has to pay something for it. as i said, go look at comscore’s financials and you’ll see they aren’t exactly getting rich doing that.

and the “huge venture return i made in comscore” is in your imagination. i have not ever made any money personally on my comscore investment.

please don’t spew lies about me jason. with “friends” like you, who needs enemies?

and

hey Karl, i bet if you and i sat down and had a coffee or a beer and talked for a half hour or an hour you’d come away with a different perspective. if you get your data on me and my investments from Jason and his friend Mike Arrington, of course you are going to come away with an impression that isn’t correct. they like to sling mud at me and my investments. i am not going to get into a pissing match with them online. but i am “kinda sad” that you are getting the wrong impression. i don’t know where you live but if we are ever in the same town, give me that half hour and i bet you’ll have a different and better opinion.

Comscore’s CMO Linda Abraham also weighed into the argument on Posterous:

Jason:
You really need to get your facts straight.

1) First of all, we measure Unique People rather than Unique Cookies which web analytics systems erroneously can unique visitors. I would challenge you to find any kind of server side measurement system that measures people, not machines or cookies. To show you how absurd server side numbers are, AOL Inc. had about 259 MM Unique cookies which gives it over 125% reach compared to a true reach of 54%. The inflation is driven by cookie deletion, multiple browsers, multiple machines for the same users, multiple devices etc… Large companies do not complain about their numbers because they know their server side numbers are flawed as obviously evident by the AOL metrics, not because ‘comScore fixes your number”. This dynamic is less obvious with smaller sites—they don’t realize how inflated their numbers are until their reach starts exceeding 100%.

2) Our Hybrid measurement is not mere pixel tracking as you assert. Our panel, which allows us to distinguish people from cookies, is a central part of the system used to correct for the inflation of cookie based server-side measurement.

3) You are confused about our pricing, so let me explain it to you:

• We charge a one-time setup fee of $5,000 that enables us to audit the beacon implementation and make sure we are measuring everyone consistently. This means auditing beacons on every page to identify pages with multiple beacons that result in over-counting, and pages with no beacon that result in undercounting. We have found about 15% of sites have placed multiple beacons on a page, and over 30% of sites that have missed a number of pages on their site. This auditing function is crucial to protect the system from being gamed. Imagine what happens, if unchecked, sites start cross beaconing each other to inflate their audience. The ‘free’ services do not incur this cost because not much is expected of them. We have seen many sites where the Quantcast beacons ‘fire’ up to 7 times from a single page!
• The initial $5,000 setup fee pays for that audit and gives you access to our reports on comScore Direct $5K for 6 month period.
• The $10K annual price is for ongoing access to our comScore Direct reporting system. However, you don’t have to subscribe to continue being measured using the hybrid methodology. As long as you maintain your beacons we will measure you with our hybrid methodology FREE of charge.

4) You may be upset because you don’t get a free subscription to the reports. We make no apologies for charging for access to our reporting system. That is the only revenue source we have to cover our costs. In doing so, we make a ‘mafia like’ pre-tax margin of less than 9% . Google and Quantcast offer metrics for ‘free’ because they have an advertising supported model. They use the data they collect from users or publishers to sell targeted advertising. We chose not to have a business model based on selling advertising, because we do not want to compete with our clients who make a living selling advertising, and who need a neutral third party to provide audience data that is free from conflicts of interest.

5) As for the free trial offer we made you, you need to get your facts straight. When we rolled out this new hybrid system, we needed some sites to beacon with us early to test it out and get user feedback .This is a common practice you might have heard of—it’s called ‘free beta.’ You chose not to participate, which is fine. But there was no attempt to ‘buy your silence’ and we challenge you to prove otherwise.

We provide a valuable service and we are proud of it. We offer the most accurate 3rd audience measurement tools available which are paid for in real dollars by more than 1,200 companies who, unlike you, freely choose them despite available ‘free’ services.

It’s unfortunate that you were picked on as a child. It must have been difficult to you. But you’re an adult now. If you want to debate, please do so with facts, not just blind fury.

My take – Abraham is right. Comscore is by far the best analytics service available. Alexa, Compete and Hitwise are seriously flawed (I may dive into this more in a future post). Quantcast has its own issues and is subject to abuse, which we’ve seen directly. Comscore uses panels and statistical analysis to generate traffic estimates. The new product measure traffic directly off of website servers and should provide nearly perfect data.

And the fact is that the company probably does need to charge to do this properly, as Abraham argues. If a competitor can provide the same service for less (or free), God Bless Them and I’ll support them all the way. Until then, the market will bear what it can bear.

We always choose to use Comscore data first when its available, and will continue to do so. Here’s an example of how useful it can be.

So in this case I respectfully disagree with Jason on the merits of his argument. And I ask Fred Wilson to try to keep me out of his various fights.

Compete’s revenues in 2007 rose 50 percent to $15 million, but it lost $4.5 million. Compete offers Web traffic stats for free on its site Compete.com, and competes with Alexa, Quantcast, (both also free) and comScore (not free). According to Compete’s own stats, it attracts about the same number of U.S. visitors a month as Alexa (727,000 for Compete vs. 758,000 for Alexa), but Quantcast is the leader with more than double that (1.9 million uniques).

This is timely for the company, which competes directly with Amazon’s Alexa. Recenty, Statsaholic has been in a very public dispute with Alexa over use of its data, with both sides looking bad. That dispute recently went to litigation. As some services shy away from Alexa, either due to public perception or inflexibility over the Alexa APIs, Compete could grab additional market share.

Compete is using Mashery to handle the logistics and distribution of its API. We wrote about Mashery when they launched late last year. Our previous coverage of Compete is here.

However, some people argue that the page view is no longer a proper measure of a website’s heft. New web page design principles such as Flash and AJAX are making constant page requests obsolete. One of the most extreme examples of this phenomenon is Justin.TV where you can log on and never refresh the page. This is great news for web users, but it’s sowing confusion among advertisers over how to peg a site’s true advertising appeal.

Comscore, who’s currently looking to go public, has been evolving their metrics to keep up with the changes. They recently announced their “visit” metric after facing some heat by BusinessWeek over ranking MySpace above Yahoo’s in monthly page views last November. The visit metric was meant to gauge user engagement by counting the number of unique requests for a site at least a half hour from the last request. All those pesky MySpace page requests would be lumped into one visit, giving a fairer idea of how often each unique user was engaging with a website each month. It had the result they wanted, bumping Yahoo back on top.

Compete also has a visit metric. But today they also launched a new metric called “attention,” which argue see as a better measure of user engagement. Attention is the total amount of time U.S. users spend on a website as a percentage of total time spent on the Internet by all U.S. users. It’s analogous to Alexa’s reach metric, which tracks the number of visitors to a site as a percentage of total internet users. Compete’s attention metric is like airtime, whereas Alexa’s reach is more like audience size.

According to Compete, we spend about 1% of our internet time on YouTube. Compete also tracks the change in attention over time, called velocity, unique visitors per month, site visits, page views per visit, and average stay.