So whatcha gonna do, when they come for you? Because bots are coming and will continue to come, as they account for almost half of Internet traffic, coming in at 46 percent, with the remaining 54 percent made up of real live humans. Until very recently, bots made up the majority of Internet traffic, and it was only in 2015 when human traffic overtook bots.
Bad Bot Landscape Report
The “2016 Bad Bot Landscape Report: The Rise of Advanced Persistent Bots,” which is published by Distil Networks, Inc., a global leader in bot detection and mitigation, reveals the state of bot traffic around the world.
According to Distil Networks, “Bad bots are used by competitors, hackers and fraudsters and are the key culprits behind Web scraping, brute force attacks, competitive data mining, online fraud, account hijacking, data theft, unauthorized vulnerability scans, spam, man-in-the-middle attacks, digital ad fraud, and downtime.”
The CEO and co-founder of the company, Rami Essaid, said 2015 was particularly bad for Advanced Persistent Bots (APBs), which mimic human behavior, load JavaScript and external assets, tamper with cookies, perform browser automation, and spoof IP addresses and user agents. These bots are particularly hard to detect, while being able to distribute attacks over hundreds of thousands of IP addresses. The Bad Bot Landscape Report highlights how bots are now more sophisticated and capable of evading traditional bot detection techniques.
Some of the key findings in the report include:
- Medium-sized websites (10,001 to 50,000 Alexa ranking) are at a greater risk, as bad bot traffic made up 26 percent of all Web traffic for this group,
- Eighty-eight percent of all bad bot traffic has one or more characteristics of an Advanced Persistent Bot,
- Fifty-three percent of bad bots are now able to load external resources like JavaScript meaning these bots will end up falsely attributed as humans in Google analytics and other tools,
- Thirty-nine percent of bad bots are able to mimic human behavior, fooling tools such as WAFs, Web log analysis, and Firewalls, and resulting in huge amounts of false negatives,
- Thirty-six percent of bad bots disguise themselves using two or more user agents, and the worst APBs change their identities over 100 times,
- Seventy-three percent of bad bots rotate or distribute their attacks over multiple IP addresses and of those, a whopping 20 percent surpassed 100 IP addresses.
The country from which the largest number of bad bots originate is still the U.S, accounting for more than 39 percent of bot traffic. India comes in second, followed by Israel in third. Those countries moved up 8 and 11 places, respectively.
So What Are Internet Bots Anyway?
A bot, a word derived from robot, is software that performs and automates tasks over the Internet. Put simply, it takes over these time-consuming, routine, hard to accomplish or impossible tasks from people. So why is this bad? Actually they are not bad, but every good technology can be used maliciously.
Good Bots
With so many applications online, bots are used to ensure the services they are designed for are carried out efficiently. Everything from gaming to IM, as well as analytics, data gathering, Web indexing, live streaming and more is made possible with bots.
Without bots, it would take a very long time to access the information and services you need online. Some of the good bots include: spider bots that explore content on websites; trading bots that locate the best deals online; and media bots that deliver updates on a range of services such as news, sports and weather. Googlebot, Google Plus Share, Facebook External Hit and Google Feedfetcher are some of the good bots, to name a few.
Bad Bots
Hackers also use bots by installing malicious files to gain access into the personal computers of individuals as well as the networks of organizations. Some of the more popular malicious bots are:
- Spam Bots used to collect data by using illegitimate means;
- Hacker Bots used search for vulnerabilities to be exploited;
- Botnets used for denial of service (DoS) attacks; and
- Download Bots used to force the download of a page that wasn’t requested.
How Bad Bots Can Impact Your Business
If you are a large company with large volumes of sessions every day, referral spikes of a couple of hundred a month may not impact your Google Analytics account. On the other hand, a local small business owner that is being bombarded with spam referral traffic is going to find it extremely frustrating to effectively use analytics tools. That is because the legitimate traffic is being stifled with all the Spam Bots.
These bots also use your server resources, which slows down access to your website as well as jeopardizing the overall security of your digital presence.
A website is a rich target for hackers. It not only provides valuable data that can be stolen, but it can also be used as a vector to launch different types of attacks. Bots play a big role in this, and it is up to you to find the right security provider capable of finding and eliminating the threats bad bots pose on your site.
Aira Bongco
The problem with bots is that they can create fake hits that doesn’t necessarily convert into sales, pushing your conversion rate down. And it also doesn’t help much with tracking.
Michael Guta
Hi Aira,
That is why it is important to identify the good bots from the bad ones. Once you do that, you can better analyze which of the good bots you can count to accurately measure your metrics.