Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- Robots.txt ?
January 24, 2005, 10:21 pm
rate this thread
Re: Robots.txt ?
You could try generateing robots.txt dynamically. Based on the user
agent, just decide whether you wish to let it in today or not, and
based on that decision, output the user agent deny for it or don't.
It's good to know however that not all robots follow the robots.txt
(though the well behaving ones do). Another thing to note, is that
Google (for example) drops the pages it can't access from it's