Akamai released a new Bot Manager tool today designed to give companies finely tuned control over Bot traffic on their websites.
The numbers vary, but research has shown that about half of traffic on the web comes from bots. Bots are small programs that interact with a website and can be used for good or ill, depending on your perspective, says Renny Shen, senior marketing manager for the enterprise at Akamai.
It’s worth noting that some bots are seemingly harmless, while others could be trying to scrape your site information or be part of a coordinated Distributed Denial of Service (DDoS) attack. Whether a bot is friendly or not is sometimes in the eye of the beholder, Shen explained.
“Most think of a bot as an attacker. For me, a bot is just a piece of software written to automate the task of interacting with a website,” he said. That’s why he believes a binary approach of blocking or not blocking might not always be appropriate.
“Bot traffic is a fact of life,” Shen says. “Because every website is different and every customer is different, we put the bot traffic control in their hands instead of blanket blocking.”
The Akamai Bot Manager gives fine-grained control over the bot traffic, letting customers understand the nature of the traffic and taking whatever action they deem appropriate such as blocking, monitoring, slowing it down, fooling it or letting it through. What a customer chooses to do, depends on the goals of the company.
For example, an eCommerce site might let a price scraper bot through for a partner, but block it for a competitor who is simply trying to get the information to match or beat their prices. Some companies might go so far as to create a page with false prices to fool the bot.
Others might not want to deal with the nitty-gritty of bot management at all, while still getting the benefits of the tool, and Akamai is also offering this as a managed service for those customers.
The tool lets you see a high-level view of all bot traffic on your site, then dive down to get detailed information about an individual bot such as the host name, the source IP address, what activities it performed and the impact that had on your website performance.
Courtesy of Akamai.
Once a company is armed with this level of detail, it can decide how best to deal with a bot and determine whether it’s good, bad or somewhere in between — and it’s not always obvious. Shen pointed to the Chinese search engine Baidu as an example.
“It doesn’t respect robots.txt and comes in aggressively to crawl a site,” he explained. That could leave a tricky decision for some sites who want to be ranked in Baidu’s search rankings, but don’t want the bot to have an adverse impact on the site’s performance.
The Bot Manager lets the customer manage that bot’s behavior as it moves through the site, putting the site owner in control instead of simply blocking it or letting it through. A site manager might decide in the Baidu case to let it through, but slow it down to get the advantage of being ranked on the search engine, but reducing the impact it could have on performance.
While the product was announced today, it will be commercially available at the end of the month. As with all of Akamai’s product, the price for using this product is based on the amount of traffic processed by the service.