Friday, October 28, 2005

Crawl Your Own Site...

I am being inundated by bots. I am not sure if this is because of the change of web host or something else. One particular bot [], appears to be trying to index the forums board, but it is doing it in a manner that is very invasive. It started yesterday and I had over 1807 hits. I denied access the series of IP address associated with the domain name, yesterday around 8:00 p.m. This morning I have lots of error logs indicating this bot is still trying to crawl my site. I am not sure who this bot belongs to but it must be stopped.

The Alexa bot (IA Achiver, 1055 hits) ceased its rampage through my site once I re-installed the robots.txt files. What I find particularly irritating about the IA Achiver bot is that it belongs to Alexa and the ranking service will derail site rankings if there are too many bot hits. However it did honor the robots.txt file so that's a good thing.

What's normal -- GoogleBot has around 140 hits for the one week period and MSNBot has about 273 hits for the same period. This is even high for my file headers, but okay.

This new bot [] is not honoring my file headers nor the robots.txt file. I had to deny access. I don't care if my forums are not indexed for this search engine. Stop crawling my site.

Technorati Tags: , ,


Post a Comment