I can not tell you how many times I have answered this question in the forums, so I figured since many are asking, would be a great item.
First, we describe what we are talking about. A bot is a piece of software from a search engine that is designed to go through all the pages on your site, sort, and place it in a database.
Google has three bots are well known: the AdSense bot, the Freshbot and DeepCrawl it.
The AdSense bot, as you probably could guess, is used for publishers who have Adsense on their sites. As soon as a new page is created, the JavaScript code in the Adsense code sends a message to the AdSense bot, and will arrive within 15 minutes to index the page so you can serve more relevant ads.
But for this conversation we are only concerned about the DeepCrawl and Freshbot.
Freshbot tracks the most popular pages on your website. No matter if it’s one page or thousands. Sites like Amazon.com and CNN.com have pages that are crawled every ten minutes, since Google has learned that the pages have that amount of frequent changes. A typical site should expect to have a visit freshbot every 1 to 14 days, depending on the popularity of pages.
What happens to your site on a visit Freshbot is finding all the links deeper into your site. Put the links in a database so that when the DeepCrawl occurs, has a reference.
Once a month, the DeepCrawl robot will visit your site and all links found on the Freshbot. This is the reason why it can take up to a month for the entire site to be indexed in Google – even adding a Google site map.
Therefore, be patient and keep adding content to your site, and work on getting valuable backlinks to your site – Google will reward.
