Do you have a question? Post it now! No Registration Necessary. Now with pictures!
December 10, 2003, 2:34 am
rate this thread
Say I declare the Following Directory to be Disallowed in Spidering by a
Robot such as Google:
Now Say I place a link in Raw Text on the Home Page URL Pointing to a
specific Page within that directory like:
Will The Robot Exclude every other possible spider in that directory and
still include the sitemap.php to be Spidered?
Re: Robots.txt Question
No, it will not spider /phpsearch/sitemap.php.
"To exclude all files except one
This is currently a bit awkward, as there is no "Allow" field. The easy
way is to put all files to be disallowed into a separate directory, say
"docs", and leave the one file in the level above this directory:
Alternatively you can explicitly disallow all disallowed pages:
Wat wil jij leren? http://www.leren.nl