Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- What do i put in a ROBOTS.TXT file ?
November 29, 2004, 4:38 pm
rate this thread
I have been trying to find some information about Robots.txt files, what to
put in them and what they do.
I realise you can get spiders to look/not look at particular pages, and i
have read about the ROBOTS=ALL stuff which you can put in your html... but
what do you put in the Robots.txt file ?? (this information is quite scarce)
I see from my site log that they are looking for my robots.txt file quite
I have no pages i want to hide from the crawler/spiders
Any help would be wonderful
Re: What do i put in a ROBOTS.TXT file ?
Mr McN wrote:
If you want to learn how it all works, start here:
and here: http://www.robotstxt.org/wc/robots.html
If you just want all spiders to see everything, do nothing.
If you want spiders to see everything and you don't want your
logs to be filled with requests for a robots.txt that isn't there,
make a robots.txt file that has the following two lines in it...
...and save it in your root directory.
Or you can copy this one that has the above lines in it:
(If it looks like one line to you, that's because your browser
or text editor doesn't handle UNIX style newlines correctly.
Just copy the file without editing it - it is fine as is.)
BTW, it should be named robots.txt, not ROBOTS.TXT.
Guy Macon <http://www.guymacon.com