Most businesses live or die on Google search rankings. If you can’t crack the top few results, it’s almost impossible to consistently drive organic traffic to your site.
But how does Google decide which sites appear at the top of the results list? It uses Googlebot, a proprietary automated software script to “crawl” every website on the Internet and create an index that becomes the basis for search results. The problem is that this script is vague and mysterious, making it impossible for website creators to truly optimize for Googlebot and other search engines’ crawlers.
Enter Botify, a cloud-based crawler that will examine your website and create a detailed SEO analysis with actionable feedback. The French-based company, which launched today onstage at Disrupt NY, has already raised $7.2 million and on-boarded big-name customers like eBay, BlaBlaCar and Expedia.
But a few things make Botify stand out from the hundreds of other SEO companies out there.
First, the crawler is strong. The startup jokes that it holds the “world record” for most URLs crawled from a single website — 150,000,000. Botify also will let site administrators specify the crawl speed — anywhere from 10 to over 200 pages per second. Why would a site want its content crawled slower? Because Botify can crawl so fast that it sometimes stresses a site’s servers.
Besides a strong crawler overall, Botify can also detect which pages on a site have been crawled by Google. When Google crawls a site, it leaves traces in a site’s server logs. Once the site provides these logs to Botify, they can compare it with their own crawl to see if Google had visited a certain page.
Screen Shot 2016-05-08 at 7.06.08 PM
Screen Shot 2016-05-08 at 7.06.50 PM
Screen Shot 2016-05-08 at 7.06.28 PM
Screen Shot 2016-05-08 at 7.06.39 PM
Screen Shot 2016-05-08 at 7.08.06 PM
Screen Shot 2016-05-08 at 7.08.42 PM
Adrien Menard, co-founder and CEO of the company, explained that when “a page is not crawled by Google, it can’t drive any traffic and therefore any revenue. Pages ignored by Google represent a huge potential of traffic and revenue.”
So, Botify takes its crawl data and creates a unique recommendation report for websites. But unlike most SEO consultants, Botify won’t tell sites to add more content. Instead, the startup tells sites how to “organize site content to make it more efficient.”
Specifically, this includes how to organize internal links, how to direct Google to crawl specific pages over others and how to optimize pages to decrease load times.
This level of intense SEO work is probably better suited for larger websites, which Botify’s pricing structure alludes to. The least expensive plan is over $500 a month for a bot to crawl 5,000,000 URLs, and enterprise plans can run over $10,000 per year.