Monday, November 08, 2004

The Alexa Crawl...

In an earlier post ...Internet Advertising I indicated that I submitted to the Alexa Crawl program. I have never submitted to this program for any of my sites but I have noticed the iachiever bot in my log files. I recently found out this bot is from Alexa.

I do not like bots crawling my sites because it inaccurately boosts the number of hits. Be very careful when using this submit function. Because I have dynamic features [forums and this blog] on my domain, bots have to crawl to ensure proper indexing for search engines. My main domain and the other sub-domains are not dynamic so I do not allow bots to crawl more than once every four weeks.

I found this out back in June 2004 when I couldn't figure out why hits were no longer being recorded from people while bot hits were being recorded. In my case there was a direct correlation with the affiliate hits, which is what raised the red flag.

I requested that at least one company remove a particularly busy bot and then put features in to stop others from returning as often. This may have been why Alexa, shortly thereafter -- dropped my rank from 1.6 million to over 5 million [summer 2004 time frame]. This would suggest that you may want to have as many bots crawling as possible. I repeat this is not a good idea. It may mask other things.

Alexa -- Now That's Amazing...

The Alexa ranking for as of November 7, 2004 is shown in the image to the right. You will remember it was hovering at just over 1 million the other day. If not see this post --> Internet Advertising. I also explained that as of June 2004 my Alexa ranking was around 2 million.

The charts below show the total amount of bandwidth for the month of June 2004 v. the current totals for November 2004 [through 11.06.04]. The combined total bandwidth for the first 6 days of November is more than the entire month of June yet my Alexa ranking has suddenly fallen (in a two day period) to about the same as then.

There is a problem with my cPanel homepage bandwidth report which should reflect the entire bandwidth expended for the account including sub-domains. It is only reporting a total of 38.999 MB of bandwidth expended as of 11.06.04. This total appears to be the amount for only the Forums sub-domain -- yes it says 45.48 on the report above but on the sub-domain report [which I did not provide here], the numbers agree. The 38.89 shown for the in the above diagram - actually shows 27.298 MB in main domain report.

I recently had some sort of upgrade to my account and it appears something was incorrectly set. I have asked my web hosting company to correct the problem. The problem began October 18, 2004 -- see this forum post --> LunarForums Post - Bandwidth Problem. This was a immediate red flag for me.

Could Alexa have misinterrupted the data? WOW... I am sure the programs used by Alexa to gather bandwidth information are smart enough to catch something like this.

The cumulative unique site visitors for November [1-6] 2004 are: <> 640 <> 167 <> 434

Last month I averaged about 120 unique site visitors daily on the main domain alone.

The only real domains maintained on are the main domain and the blog and forums sub-domains. The other sub-domains shown in the June and November reports are re-directs and I expect those bandwidth numbers to be low. See earlier post --> Webalizer on bandwidth and stats. Also note that the blog sub-domain was implemented the beginning of September 2004 and the Forums are new. I installed them on September 28, 2004 but did not implement the sub-domain until late October.

Well that is the lesson for today... What are we going to learn tomorrow? Never write a report on Internet Advertising? The Alexa numbers I can live with... I have seen Alexa do strange things without explanation. I would like my reports corrected.

<>Updated 11.08.04 -- late night