Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- Robots.txt issue
January 13, 2005, 5:02 pm
rate this thread
- Klaus Johannes Rusch
January 18, 2005, 12:05 am
Re: Robots.txt issue
>>Is there something wrong with it? Yahoo Slurp still indexes content in
>>the auth folder...
> Disallow: /auth/*
The Robots Exclusion Protocol does not specify wildcards, some robots
support this as a proprietary extension.
The problem in the original post was probably the blank line between the
User-agent and the Disallow lines.
Klaus Johannes Rusch
- » How does a server know if your browser client supports frames?
- — Previous thread in » HTML Authoring Forum