Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- Missing robots.txt file
- Joe Blow
August 28, 2004, 11:41 pm
rate this thread
file. I receive on average 5-6 notifications per day letting me know
that the file was requested was missing, when in fact it is there.
Our server logs indicate that the file is also being accessed
successfully, but and wondering why I am still receiving notifications.
This morning so far:
* NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html ;
* msnbot/0.11 (+http://search.msn.com/msnbot.htm )
Can someone shed some light on what is happening?
The web site is www.wiavic.org.au
Re: Missing robots.txt file
Hate to answer a question with a question (or not answer it as the case may
be) but are spiders only supposed to look for robots.txt in the base
directory, or do they look for it at the entry point from which they start
crawling? Or do spiders check for a copy of robots.txt in every directory
that they crawl?