Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- Robots.txt bocking user sites
- Rodrigo Lopez-Negrete
March 12, 2007, 1:24 am
rate this thread
I have a question regarding the use of the robots.txt file. I want to
allow acces to all folders and files of all the users except one.
Every user has a wwww.mysite.com/~user/ site, and I want to block
access to all robots to the site that belongs to a user called test.
How should my robots.txt be written??
could this work:
Thank you all,
Re: Robots.txt bocking user sites
The latter is correct. Since all pages are allowed by default, this line
So you can reduce your robots.txt to just this:
As usual when someone mentions robots.txt, I feel like I need to address
a common misconception (just in case): robots.txt *prohibits* entry but
doesn't *prevent* entry. Analogously, a "NO ADMITTANCE" sign on a door
prohibits entry, but you'll need to put a lock on the door to actually
prevent people from getting in.
Have you read the robots.txt Web site? It's short and can tell you what
you need to know.
Whole-site HTML validation, link checking and more