Article Surfing Archive

Mozilla or Godzilla?

It zips across the internet at speeds faster then any that has came before it. It's smarter and emulates human behaviors. It will accomplish things only dreamed about in the recent past. The Mozilla Googlebot has been released into the World Wide Web and it has been tearing through it since its release.

This new Googlebot has the ability to index many more pages then the old Googlebot. It can visit you so fast and so often it creates denial of service like attacks that can shut down hosting servers. This little crawler is the beginning of the end for many black hatters.

This Googlebot can read CSS, JavaScript, Div, and it also gets served up a website as if it were a normal person/viewer in effect bypassing bot redirects and cloaked pages. If this new bot has visited you it will come with this ID:

Mozilla/5.0 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)

With the blazing fast speed of this new Googlebot, Google should have their data centers up and running sooner than we all thought. Sure there is some bugs still left in the system like crashing hosting providers but when you bring out this big of a change you can expect little else. Google should be praised over this monumental step into the future. With this new technology things on the net may change forever.

The effects of the Big Daddy data centers (be it good or bad) are starting to be felt by everybody. Many webmaster have reported better rankings but for every one of these we also hear of webmasters claiming they went from #1 to #6000 or not ranked at all. The search engine result pages (SERPS) have many more results for sites indexed, due to this new bot, to compete with. Just because someone had a keyword listed #1 out of 360,000 result doesn't mean they will continue to have #1 status with 1,150,000 results. In spite of trying to make currently relevant sites the most important, by adding more and more sites to compete with, these sites can be lost to better sites that haven't been indexed until Big Daddy.

One webmaster wrote that he signed up for Google adsense and was soon deep crawled by the new bot. Deep crawls were something that happened rarely and now Google deep crawls on demand? The bad part of this is that the new bot has shut down many servers and is being blocked out by these said servers. Google adsense uses the same bot so if the new Googlebot is denied access to your server don't expect to be able to hook up that new adsense account.

This Bot is the start of a new future and with the Big Daddy infrastructure Google will be able to make updates to the bot easier then they could have in the past. This bot is already doing things that have shaken the net, such as fill out and submit forms on websites to make sure the links work and click on live help links and initiate chats. It looks like the next step for Googlebot is going to be eyes and ears, time to watch movies and listen to .mp3's?