Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- Google only spiders the robots.txt
- Nico Schuyt
November 18, 2005, 3:57 pm
rate this thread
look at the robots.txt and leaves:
crawl-66-249-71-53.googlebot.com - - [15/Nov/2005:05:16:39 +0100] "GET
/robots.txt HTTP/1.0" 200 25 "-" "Googlebot/2.1
crawl-66-249-72-226.googlebot.com - - [16/Nov/2005:02:14:45 +0100] "GET
/robots.txt HTTP/1.1" 200 25 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
The same with MSNbot
The content of the robots.txt is:
Should I change the robots.txt to make Google visit the other pages?
Re: Google only spiders the robots.txt
__/ [Nico Schuyt] on Friday 18 November 2005 09:57 \__
I can count about 30 bytes. Why do you reckon this might be? Have you
changed it recently? I see that you still experiment as shown below.
Is the file robots.txt potentially malformed? It doesn't appear to be the
case. I also notice that your site has decent traffic and ranks. What hap-
pens if you remove robots.txt altogether? When did you register the site
or first observe such requests from search engines? Is there any new con-
tent to index? XML sitemaps?
I now see:
Roy S. Schestowitz | "I feed my 3 penguins with electricity and love"
http://Schestowitz.com | SuSE Linux | PGP-Key: 0x74572E8E
11:40am up 15 days 7:34, 4 users, load average: 0.41, 0.31, 0.42
http://iuron.com - next generation of search paradigms
- » Title search options-- Need recommendations for search-engine utilities.
- — Next thread in » Search Engines