Monday, November 08, 2004

The Alexa Crawl...

In an earlier post ...Internet Advertising I indicated that I submitted to the Alexa Crawl program. I have never submitted to this program for any of my sites but I have noticed the iachiever bot in my log files. I recently found out this bot is from Alexa.

I do not like bots crawling my sites because it inaccurately boosts the number of hits. Be very careful when using this submit function. Because I have dynamic features [forums and this blog] on my domain, bots have to crawl to ensure proper indexing for search engines. My main domain and the other sub-domains are not dynamic so I do not allow bots to crawl more than once every four weeks.

I found this out back in June 2004 when I couldn't figure out why hits were no longer being recorded from people while bot hits were being recorded. In my case there was a direct correlation with the affiliate hits, which is what raised the red flag.

I requested that at least one company remove a particularly busy bot and then put features in to stop others from returning as often. This may have been why Alexa, shortly thereafter -- dropped my rank from 1.6 million to over 5 million [summer 2004 time frame]. This would suggest that you may want to have as many bots crawling as possible. I repeat this is not a good idea. It may mask other things.


2 Comments:

At 8:40 AM, Anonymous said...

Further bots do not increment page counters...

QiSoftware

 
At 10:26 AM, Anonymous said...

In addition, bots do not increment page counters...

QiSoftware

 

Post a Comment