On Yahoo! Slurp, Googlebot, and msnbot

• April 16, 2006
• James Skemp
• StrivingLife, Internet

Being able to look at the statistics for a couple of different sites, with very different audiences, I've found one common thread, which is quite alarming. Before you go any further, it has to do with bots and search engines, so if you're not interested in either, you can probably safely skip this post.

It seems that the sites that I can view the statistics for are getting hit fairly hard by Yahoo!.

For The Framing Business, over 54% of bot traffic comes from Yahoo!. Over 24% comes from Google, and almost 6.8% from MSN.

However, if you look at where people are coming from, only 4% are coming from Yahoo! search. Over 30% are coming from the various Google search pages, and almost 1.2% are coming from MSN.

Obviously, if bandwidth is getting tight, we know who to block. Of course, if bandwidth is available, there's little reason to block these bots, even if they are sucking quite a bit (of bandwidth).

For StrivingLife.net, it's very similar to this. However, bot crawling is pretty even (although msnbot is crawling quite a bit more than the other two). However, if I look at where they are coming from, it's most definitely Google (by over 100 times for this blog specifically).

If you're a site owner who stumbled upon this post, what kind of traffic are you seeing from the big three bots, and from their search portals? Is crawling versus referring quite different, or is it pretty comparable?