robots query

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View

Hi All

Might sound daft, but if I disallow one of my sub-folders in my robots.txt
file will this stop the spider/bot from indexing my paths, eg:

typical folders are pics/ and scripts/ and if I disallow them will this make
the file calls, eg:

<SCRIPT LANGUAGE="javascript" SRC="scripts/fred.js"></SCRIPT>

<IMG SRC="pics/fred.jpg">

etc... etc...

in my HTML not generate properly as far as the spiders/bots are concerned?

I put index.html files in these folders so that nobody can get directory
lists, but I don't want the spiders/bots going crazy by going into these
folders and keep getting bounced to the same URL in some kind of crazy
recursive loop so I want to disallow them from their search, but I don't
want to do it if it makes their site ride uncomfortable :0)

Rgds Laphan

Re: robots query

And lo, Laphan didst speak in alt.www.webmaster:

Quoted text here. Click to load it


Only if it's a good little spider/bot.  The evil ones will ignore your  
robots.txt file.


The technical axiom that nothing is impossible sinisterly implies the  
pitfall corollary that nothing is ridiculous.
- - Orca Search: Full-featured spider  
and site-search engine

Re: robots query

On Wed, 23 Nov 2005, GreyWyvern wrote:

Quoted text here. Click to load it

... and the *really* evil ones won't ignore your robots.txt file but,
instead, aim right for everything you disallow.  Some guess at abusable
resource names and try various combinations.  Whether you have a FormMail
script or not, expect to see a few attempts to access one in your cgi-bin
directory.  Here's a regex indicating the various combinations I have
seen tried one after another from the same source (if I have it right):
according to the daily error report I get.  The script used here has a
completely different custom name.  That doesn't stop people from trying
over and over to access scripts we don't have.

Norman De Forest
"> Is there anything Spamazon DOESN'T sell?
 Clues. The market's too small to justify the effort."
      -- Stuart Lamble in the scary devil monastery, Fri, 13 May 2005

Re: robots query

Quoted text here. Click to load it

I haven't *yet*, *quit*, cared enough to write something to scan for
that sort of thing and, when it catches it, block the source IP in the
firewall.  Maybe temporarily -- but is that half an hour, or half a
RKBA: < <
Pics: < <
Dragaera/Steven Brust: <

Re: robots query

Laphan wrote:

Quoted text here. Click to load it

Robots won't look at your images or scripts anyway. (Well, OK -- there are
a few image search engines now.)

Toby A Inkster BSc (Hons) ARCS
Contact Me  ~

Site Timeline