Does Google crawl?

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View
When submitting to google they say that you only have to submit your
root directrory and assuming you have links, it should automatically
spider them during the next deep crawl.  However since google doesn't
appear to deep crawl anymore, is this still true?    I have a small
personal site that has my root directory indexed due to an inbound link
from a buddy of mine.  Googlebot visits every couple of days, but never
goes beyond the root.  Will google ever get around to indexing some of
the remainder of my site or does google now only deep crawl the more
important sites with high PR?

Re: Does Google crawl?

On Sun, 22 Feb 2004 19:11:59 -0500, Johnny Bela wrote:

Quoted text here. Click to load it

If your site is crawlable, Googlebot will crawl it.

Sometimes there can be a delay of almost a month between the time
Googlebot gets / and /robots.txt and the time Googlebot returns to
retrieve your actual content pages.

But... make certain your site is crawlable!  Googlebot is pretty bloody
smart, but it's not infallible.  It has trouble with some JavaScript
links, for example.

Test your site using Lynx, if you can.  If Lynx can use your site,
Googlebot should be able to.

Re: Does Google crawl? says...
Quoted text here. Click to load it

Thanks for the info.  I just checked my logs and it appears that
googlebot crawled the rest of my site this morning.  So for those that
are interested, my first visit from googlebot was on Feb 2 and today-
three weeks to the day, the rest of the site was spidered.  

Re: Does Google crawl?

Quoted text here. Click to load it

Appreciate that very useful tid-bit of information !
Can you say how often googlebot visited between Feb 2
and today and if googlebot did visit other times between
Feb 2 and today did it grab anything in addition
to / and /robots.txt ??


Re: Does Google crawl?

It used to before it learned how to walk.

Re: Does Google crawl?

Sahmmm wrote:

Quoted text here. Click to load it

:-) Someone's on form today!


Re: Does Google crawl? says...

Quoted text here. Click to load it

Based on a quick review of the logs, Googlebot visited every day with
the exception of 3 days as far as I can tell. Each time, it downloaded
only / amd robots.txt.  

Site Timeline