Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
June 1, 2007, 5:17 pm
rate this thread
Re: Do not want some of my web pagfes listed in yahoo sercah dir
In the old days when the major SE's crawled from a single bot and/or IP
range, even then there ocassional malfunctions.
Today with all the major bots utilizing a variety of bots and IP ranges
to support a variety of tools the likelihood of runaway bots and
compliance is more open to abuse.
A more effective solution for what you desire would be to place your
files in the private and NOT mention the folder in robots.txt (request),
while simultaneously implementing htaccess restrictions which will NOT
permit any bots access.