Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- help with syntax of robots.txt pls?
- Pete Bennett
August 13, 2003, 5:35 pm
rate this thread
- Kim AndrÃ© AkerÃ¸
August 13, 2003, 10:54 pm
Re: help with syntax of robots.txt pls?
Only well-behaving bots honor the rules set in the robots.txt file (and the
META tags). Also, the robots.txt file doesn't support the blocking of search
engines directly. Read up on the syntax of the robots.txt file here:
If your server runs on Apache, you can follow the documentation on "Access
control", which comes in handy using .htaccess files:
Most servers can deny access based on IP address and/or hostname; check your
Kim André Akerø
- ICQ: 92708098