Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
As far as your implementation of the robots.txt file is concerned, it looks
to be the correct way to *ask* the spiders not to index the sefriendly
Most search engines seem to adhere to the rules but, as has been pointed
out, robots.txt doesn't present any barrier other than putting up a keep-out
If you don't have a link to the page from an already spidered site, your
sefriendly directory won't be found anyway - robots.txt or not.
And, if you really want to be safe, you could always password protect the
directory with .htaccess - dead easy and the spiders don't get past the
If you feel that any of the above is incorrect,
inappropriate or offensive in any way,
please ignore it and accept my apologies.