Click here to get back home

how to stop google spidering https links

 HomeNewsGroups | Search

alt.www.webmaster - All aspects of webmastering business 

get this group's latest topics as an RSS feed add this group's latest topics to your My MSN content add this group's latest topics to your My Yahoo content
Subject Author Date
how to stop google spidering https links Sparticle 05-18-2009
Posted by Sparticle on May 18, 2009, 3:55 pm
Please log in for more thread options
Is there anything I can do to stop google indexing https links on a
site.

It seems that if you get an http:// and https:// link for EVERY link
when SSL is applied to the domain.

i.e. Can I put some code in the robots.txt file to disallow spidering
the secure links??

Thank you
Sparticle


Posted by Nathan Keel on May 19, 2009, 12:34 am
Please log in for more thread options
Sparticle wrote:

show/hide quoted text

Maybe set it so any HTTPS access read by an .htaccess file enacts the
robots.txt file to deny spidering, or just reject google's user agent
bot if it's port 443. It would be ideal if you could just have
separate documentroots or directories for secure access, so you could
have a separate robots.txt file for that area.

Posted by Sparticle on May 19, 2009, 6:12 am
Please log in for more thread options

show/hide quoted text

Thank you for the info.

The next site I build I'll apply the SSL to a subdomain and disallow
the directory. It's too late for the existing one.

At the moment I'm using ASP on windows 2003. A lot of stuff I see
online about manipulating .htaccess seems to be with mod_rewrite.
Isn't this an Apache thing?

I'm not sure how to apply this to a win 2003 server.

Thank you
Sparticle

Posted by Nathan Keel on May 19, 2009, 5:38 pm
Please log in for more thread options
Sparticle wrote:

show/hide quoted text

Yeah, it's just common for people to be using Apache. I don't know the
solution for ASP, other than maybe suggesting putting a check in the
scripts themselves, but there's probably a better way.

Posted by crankypuss on May 20, 2009, 5:00 am
Please log in for more thread options

show/hide quoted text

Have you tried putting the "nofollow" attribute in your links?

--
no aluminum siding offers today

Similar ThreadsPosted
How can I stop search engine spiders from spidering my site, if I don't own a domain? October 24, 2005, 7:56 am
How do I keep google from spidering? September 26, 2005, 10:45 am
Can I Redirect https://www.... to https:// without the www March 15, 2007, 7:04 pm
Google should stop archiving June 15, 2005, 4:15 pm
Re: Google should stop archiving June 16, 2005, 11:57 am
Re: Google should stop archiving June 16, 2005, 11:33 am
Re: Google should stop archiving June 16, 2005, 12:12 pm
Re: Google should stop archiving June 16, 2005, 12:13 pm
Re: Google should stop archiving June 16, 2005, 12:39 pm
Re: Google should stop archiving June 16, 2005, 12:40 pm

Our other projects:

Art Dolls, Fairies and Mermaids - Sunnyfaces.net

Driving a better car - Fuelzilla.com

Cabling site for homeowners and pros alike - Cabling-Design.com

Friends:

Roy's Linux, Programming and Search Engines messages

1-Script XML SitemapXML Sitemap
Privacy Policy