Robots.txt Question

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View

Say I declare the Following Directory to be Disallowed in Spidering by a
Robot such as Google:

User-agent: *

Disallow: /phpsearch

Now Say I place a link in Raw Text on the Home Page URL Pointing to a
specific Page within that directory like:


Will The Robot Exclude every other possible spider in that directory and
still include the sitemap.php to be Spidered?

Re: Robots.txt Question

Quoted text here. Click to load it
Quoted text here. Click to load it

No, it will not spider /phpsearch/sitemap.php.

"To exclude all files except one
This is currently a bit awkward, as there is no "Allow" field. The easy
way is to put all files to be disallowed into a separate directory, say
"docs", and leave the one file in the level above this directory:
User-agent: *
Disallow: /~joe/docs/

Alternatively you can explicitly disallow all disallowed pages:
User-agent: *
Disallow: /~joe/private.html
Disallow: /~joe/foo.html
Disallow: /~joe/bar.html"

René Pijlman

Wat wil jij leren?

Site Timeline