|
Posted by Thor Kottelin on May 11, 2005, 1:49 am
Please log in for more thread options
socialism001@yahoo.com wrote:
>
> I have a folder that I want googlebot to index but I don't want any
> other bots to be able to index the files. How would I do this in the
> .robots file.
ITYM "don't want any other bots to index the files", and "the robots.txt
file".
This isn't HTML, but anyway:
User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /folder/
The robots.txt format is explained at
<URL:http://www.robotstxt.org/wc/norobots.html>.
If you do want to *prevent* other robots from visiting the directory in
question, you would have to use real access lists instead of relying on the
robots in question to present themselves in an honest fashion.
Thor
--
http://www.anta.net/OH2GDF
| 
XML Sitemap