I am seeing a lot of traffic from the search engine bots the last few days, the usual suspects include googlebot msnbot, and a new one is gigabot, and from what I can find from a google search on gigabot, is that it is a new search engine that wants to have more of an index then google (good luck to them.) The gigabot one is smashing my site, it has itself downloaded almost a gigabyte of stuff in the last 2 alone, all the pictures and videos I have put up are also being read by that bot, why a bot cares about pictures and videos I do not know, but it is reading them also.
I think it’s a spammer, I’ve banned gigabot from my site. Most of the hits were coming from Hurricane Electric which I know isn’t interested in being a search engine.
Gigabot has been overloading my hosting providers server all week – just from trawling one site, and has crashed it several times already.
It’s gotten far more aggressive recently, and seems to ignore robots.txt too.