robot.txt - Page 2

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View

Re: robot.txt

David Graham wrote:
Quoted text here. Click to load it

As far as your implementation of the robots.txt file is concerned, it looks
to be the correct way to *ask* the spiders not to index the sefriendly

User-agent: *
Disallow: /sefriendly/

Most search engines seem to adhere to the rules but, as has been pointed
out, robots.txt doesn't present any barrier other than putting up a keep-out

If you don't have a link to the page from an already spidered site, your
sefriendly directory won't be found anyway - robots.txt or not.

And, if you really want to be safe, you could always password protect the
directory with .htaccess - dead easy and the spiders don't get past the
password protect.

If you feel that any of the above is incorrect,
inappropriate or offensive in any way,
please ignore it and accept my apologies.

Re: robot.txt

Quoted text here. Click to load it

Thanks to everyone. I wonder if you could give me a few pointers to how I go
about password protecting using .htaccess


Re: robot.txt

Quoted text here. Click to load it

not necessarily. there are numerous ways to block 'bad' spiders ~~

Re: robot.txt

Quoted text here. Click to load it
Thanks, I will read the links. I thought this robots.txt post would just be
a simple little matter - perhaps not!

Site Timeline