Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- noindex, follow
June 14, 2004, 12:33 pm
rate this thread
Re: noindex, follow
in a meta-tag named robots it means "dear crawler, please don't put this
single page into your database but follow links on this page so that you
might find more pages of which you may take some to put them into your
database". It's basically a possibility to let authors manage robot-behavior
without having access to the /robots.txt (which, of course, wouldn't let you
decide for something to be crawled for links but not indexed and something
to be indexed but not crawled for links).
Re: noindex, follow
If you mean what does "noindex, follow" mean, and if you found it in the
contents of a robots meta tag on a web page - it means that the page should
not be indexed by the search engine bot but that it should follow the links
to other linked pages, which themselves may have further robots meta
information. You might want to have a look at:
If you are asking because you are considering using it, I'd suggest the use
of robots.txt as a more reliable method. For more info, see:
It's also worth mentioning that any robots instructions are subject to the
bot's preparedness to honour them - if it's essential that the bots don't
index a page, you may want to consider other approaches.
If you don't use a robots.txt file or the robots meta tag, the default is
that the bots will index the page and follow the links - in other words, to
have your site fully spidered, you don't have to do anything about the
bots - indexing everything is their default.
Even if you want the default of "index everything", it may be worth having a
completely empty robots.txt file so that you don't end up with a load of 404
file-not-found messages in your server logs from when the bots request the
file to check it.
If you feel that any of the above is incorrect,
inappropriate or offensive in any way,
please ignore it and accept my apologies.