Question about robot.txt files

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View
I normally don't have a robot.txt for the search-engines on my sites, but I
have noticed that just about every single search-engine looks for a robot
file first, which results in a 404 page, that is ok with me because I have a
customized 404 page, on the other hand I've read that, if you don't want to
generate a 404 page every time search engines come a long is to create an
empty file called robot.txt.

But now I see the search engines some times look at the robot file and stop.

What is the best to deal with this matter?

Can one tell the search engines some thing else in the robot file besides
what pages not to be included?

Henrique Viana

Place 200 MILLION A-D-S for Fr^ee

Re: Question about robot.txt files

Do you know the content of robot.txt for web site in your webserver.

Google it self suggests that you should have robots file.

Warm Regards,
Brijesh Kumar
(Affordable Web Solutions Company, Affordable web development)

Re: Question about robot.txt files

On 29 Dec 2004 21:05:05 -0800, wrote:

Quoted text here. Click to load it

You don't always want to blindly do what Google suggests. They aren't
on your side, you know.

         home of SEO that's shiny!

Re: Question about robot.txt files

Yes Big Bill,

I admit with you but the robot.txt is important. You agreed or not. I
know web server has default setting.

Warm Regards,
Brijesh Kumar
(E-commerce Web Design and Development)

Site Timeline