Click here to get back home

reply urgent
 Home | NewsGroups | Search | About
Login Password
Register Now!
 alt.internet.search-engines    Post an article   get this group's latest topics as an RSS feed add this group's latest topics to your My MSN content add this group's latest topics to your My Yahoo content
Subject Author Date
reply urgent CreativeMind 02-20-2008
Posted by CreativeMind on February 20, 2008, 3:40 am
Please log in for more thread options
 google spiders all pages of a website?? actually i use some pages for
just returning values or some specific purpose.They are not intended
to be spidered. While analyzing a site, do the optimizers analyze
individually all pages or only default or index page??
thx

Posted by teranews on February 20, 2008, 5:46 pm
Please log in for more thread options
> google spiders all pages of a website?? actually i use some pages for
> just returning values or some specific purpose.They are not intended
> to be spidered. While analyzing a site, do the optimizers analyze
> individually all pages or only default or index page??
> thx

You need to upload a robots.txt file to your server.

The content should be something like:-

User-Agent: *

Disallow: /images/

which I use on a site to stop Google and others spidering my images folder.

hth
--
Rich
http://www.richdavies.com
http://www.richdavies.com/tomtom.htm


--
Posted via a free Usenet account from http://www.teranews.com


Posted by Phil Payne on February 21, 2008, 9:30 am
Please log in for more thread options
> You need to upload a robots.txt file to your server.
>
> The content should be something like:-
>
> User-Agent: *
>
> Disallow: /images/

No.

The blank line after User-agent: delimits the record. It has to be:

User-Agent: *
Disallow: /images/

Posted by John Bokma on February 20, 2008, 6:05 pm
Please log in for more thread options

> google spiders all pages of a website?? actually i use some pages for
> just returning values or some specific purpose.They are not intended
> to be spidered.

The best option is to password protect such pages. There are plenty of
"spiders" that don't check robots.txt

--
John Bokma http://johnbokma.com/

Posted by canadafred on February 22, 2008, 9:14 am
Please log in for more thread options
> google spiders all pages of a website?? actually i use some pages for
> just returning values or some specific purpose.They are not intended
> to be spidered. While analyzing a site, do the optimizers analyze
> individually all pages or only default or index page??
> thx

Whatever you do, when a visitor arrives at a page that was not
intented to attract him, make sure he can easily go to one that is.
Avoid redirects. Use hypelinkes text or graphics.

Similar ThreadsPosted
urgent - SEO help please October 5, 2005, 8:14 pm
SEO Help! URGENT! April 19, 2008, 1:40 am
Hi SEO's (urgent) March 29, 2006, 7:53 am
urgent - Pls help - Pay per click May 23, 2006, 6:50 pm
urgent opening January 13, 2007, 4:03 am
URGENT: Time Sensitive! March 12, 2005, 11:57 am
URGENT OPENING-DELHI January 13, 2007, 3:53 am
urgent Question on Internet traffic June 22, 2006, 10:40 am
I need urgent review --www.googleadservices.info August 29, 2006, 10:30 am

Our other projects:

Art Dolls, Fairies and Mermaids - Sunnyfaces.net

Roy's Linux, Programming and Search Engines messages

1-Script XML SitemapXML Sitemap