Hay it is a bot party on my web site

Categories: Web Site
Comments: 2 Comments
Published on: June 14, 2005

I am seeing a lot of traffic from the search engine bots the last few days, the usual suspects include googlebot msnbot, and a new one is gigabot, and from what I can find from a google search on gigabot, is that it is a new search engine that wants to have more of an index then google (good luck to them.) The gigabot one is smashing my site, it has itself downloaded almost a gigabyte of stuff in the last 2 alone, all the pictures and videos I have put up are also being read by that bot, why a bot cares about pictures and videos I do not know, but it is reading them also.

2 Comments
  1. Tim says:

    I think it’s a spammer, I’ve banned gigabot from my site. Most of the hits were coming from Hurricane Electric which I know isn’t interested in being a search engine.

  2. Tom Marshall says:

    Gigabot has been overloading my hosting providers server all week – just from trawling one site, and has crashed it several times already.

    It’s gotten far more aggressive recently, and seems to ignore robots.txt too.

Welcome , today is Friday, July 21, 2017