Rogue Web Bots Attacking the Web; Increasing Costs, Stealing Content New Tool from Stop Bots in Their Tracks

Share Article

Unwanted web bots are finding ways to steal website content, and inflate bandwidth costs. releases free beta tool allowing webmasters to block bots and take back control. has released a free beta tool giving website owners the ability to learn about and block rogue web bots and agents that can steal website content, inflate bandwidth charges and stop web penalties from duplicate content. The tool allows webmasters to pre-select from a list of unwanted web bots to block, then creates a digital file to be placed on their server which stops the bots in their tracks.

“I would guess that 99.9% of people who have websites or run their own server have no idea what unwanted bots are doing to their system, their search engine ranking and their bandwidth limits,” said Don Cramer, creator of the tool. “I would estimate that costs associated with stolen content and excessive bandwidth charges that these bots create runs into the millions each year, maybe per month, depending the vertical your website operates in.”

The tool allows webmasters or anyone else who has a website to generate a specially customized file they can place on their server, which stops certain bots from visiting their website(s). The main types of bots to block are as follows:

  • Site Downloading Bots: Bots that download all content found on the target site

This type of bot also known as an "Offline Browsing Bot" will download your entire site including all images and other media, possibly even areas you do not want accessed. This type of bot will drain valuable bandwidth, server resources and slow down your site which can interfere with your visitors’ experience

  • Media Harvesting Bots: Bots that seek and download all media files from target sites

Media bots will crawl your site and download all media types specified by the user. Users will never see your site as intended and will easily bypass any type of products or other revenue generating portions of your site. Media bots waste bandwidth and steal your content without any benefit to the site owner. Bandwidth overages may even result in the suspension of hosting services

  • Email Harvesting Spam Bots: Bots that seek any email address found on the target site for spam use

Email "Spam" bots crawl your web site in search of email addresses which can be used for spam. These bots are a waste of bandwidth and server resources offering no benefit to site owners.

“It’s not a small problem; it’s a HUGE problem that most people don’t realize,” Mr. Cramer continued. “Almost every day during my research I come across a new bot that is spreading its way across the Internet, collecting information that most people don’t want to give away and reusing it illegally, and/or eating up bandwidth and costing webmasters money inflating the costs of goods for the end user.”

But What About “Good” Web Bots?

“Good” bots, like the Googlebot or other search engine bots that put your website in the search engines are not the problem. “The tool focuses only on blocking bots that aren’t providing value. The problem is, most webmasters don’t know which bots are good, and which bots are bad. This tool is a first step in helping them figure it out, and giving them back control over which ones to block.”

You can read more about the beta tool at


Share article on social media or email:

View article via:

Pdf Print

Contact Author

Don Cramer