Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- Question about robot.txt files
- Henrique Viana
December 29, 2004, 3:33 am
rate this thread
have noticed that just about every single search-engine looks for a robot
file first, which results in a 404 page, that is ok with me because I have a
customized 404 page, on the other hand I've read that, if you don't want to
generate a 404 page every time search engines come a long is to create an
empty file called robot.txt.
But now I see the search engines some times look at the robot file and stop.
What is the best to deal with this matter?
Can one tell the search engines some thing else in the robot file besides
what pages not to be included?
Place 200 MILLION A-D-S for Fr^ee