How Bad Did Analytics Bugs Fool Facebook Pages? Reach Looked Down 14% When It Wasn’t

Josh Constine

Josh Constine is a technology journalist who specializes in deep analysis of social products. He is currently a writer for TechCrunch. Previously, Constine was the Lead Writer of Inside Facebook, where he covered Facebook product changes, privacy, the Ads API, Page management, ecommerce, virtual currency, and music technology. Prior to writing for Inside Facebook, Constine graduated from Stanford University... → Learn More

Tuesday, February 26th, 2013
Comments
Facebook Reach Bugs Graphs

Months of mysterious Insights bugs duped Facebook Page owners into thinking their reach was plummeting. The median Page’s analytics showed it reached 14.39 percent fewer people than it actually did, according to early data and graphs from EdgeRank Checker, a Facebook analytics startup that gave us the first look. The bugs may have caused people to mistakenly undervalue their Pages, change strategies or buy ads to make up for “lost” reach.

Facebook announced the bugs last week. During its efforts to speed up its iOS and Android apps in August and December, it accidentally stripped out the markers used by Page Insights to count impressions. This and other bugs led several metrics to be underreported. Fixes and bug-prevention systems are now in place, and accurate data began flowing into Page Insights yesterday, making today the earliest opportunity to get a concrete sense of the glitch’s impact.

Facebook Reach Total Organic ViralEdgeRank Checker analyzed 1,000 Pages who posted both last week when Insights was still wrong, and yesterday with reach reported accurately. How much reach was underreported depends a bit on the Page’s fan count, as those with 500,000 to 1 million Likes were led farther astray.

A Page with 750,000 fans would have been reaching 47,000 fans per post (6.24 percent), according to the broken analytics, when in reality they were reaching 78,000 (10.35 percent). That means Insights was underreporting reach by a massive 66 percent. The impact was more subtle for a Page with 100,000 fans. The buggy Insights would have shown them reaching 10,000 of their fans per post when they were actually reaching 11,000, with reach underreported by 10.2 percent.

The most stunning part of the data was that the median Page’s viral reach according to Insights went up 275 percent after the fix. EdgeRank Checker‘s CEO Chad Wittman believes Facebook also fixed a bug that caused viral reach to exclude the impressions from posts re-shared by a Page’s fans.

Facebook Page Total Reach Change

Once there’s a whole month of accurate data, we’ll get a more precise look at the effect of these bugs. Facebook tells me it doesn’t calculate the aggregate impact, but didn’t dispute EdgeRank Checker’s findings. However, it did note the day-over-day data may look a little more extreme, and the data may even out a bit with time.

While in some ways Page admins should be relieved, some are certainly mad. Though Facebook snapped into action squashing the bugs as soon as they were discovered, it still didn’t disclose the issues for the three weeks while it fixed them.

Some admins have complained about wanting refunds for ads they bought during the bug period, because they wouldn’t have paid for them if they knew their real reach was higher than shown. If Facebook wants these Pages to keep filling it with content and buying its ads, it can’t have any more slip-ups like this.


Company: Facebook
Website: facebook.com
Launch Date: February 1, 2004
IPO: NASDAQ:FB

Facebook is the world’s largest social network, with over 1 billion monthly active users. Facebook was founded by Mark Zuckerberg in February 2004, initially as an exclusive network for Harvard students. It was a huge hit: in 2 weeks, half of the schools in the Boston area began demanding a Facebook network. Zuckerberg immediately recruited his friends Dustin Moskovitz, Chris Hughes, and Eduardo Saverin to help build Facebook, and within four months, Facebook added 30 more college networks. The original...

→ Learn more
Company: EdgeRank Checker
Launch Date: June 1, 2011

Facebook Analytics focusing on the effect of EdgeRank on Facebook Pages. EdgeRank has a drastic impact on Facebook Marketers, that’s where EdgeRank Checker has filled the void. Having helped over 200k Facebook Pages who influence over 3 billion fans, has helped marketers at all levels improve their visibility on Facebook. EdgeRankChecker.com is a web application built to help Facebook marketers succeed against EdgeRank. EdgeRank Checker provides a free EdgeRank Score for it’s users to better understand how EdgeRank affects their updates....

→ Learn more