|
Posted by Don on February 21, 2008, 8:52 pm
Please log in for more thread options
>
>>
>>>
>>>>
>>>>> I'm using a robots.txt file to control what is and is not crawled
>>>>> by search engine bots but I'd like to block anything that isn't a
>>>>> known search engine bot doesn't get the file I'm feeding to
>>>>> google, yahoo and the others.
>>>>
>>>> Why?
>>>>
>>>> I can imagine that you want to block your entire site for any bot
>>>> that's known to be abusive though, but those probably don't check
>>>> your robots.txt anyway.
>>>>
>>>
>>> Perhaps I didn't say it right. I'm wanting to block the robots.txt
>>> that I'm feeding search engines from being given to anybody else.
>>
>> Why? If the reason is that you want to "protect" some folders: it's
>> not secure and bound to fail sooner or later. Remember that not all
>> bots honor the robots.txt, especially not the ones that you don't
>> want on your site in the first place.
>
> I want to keep certain humans from reading the robots.txt that I give
> to search engines because it's none of their bloody business what
> pages I tell SE's not to index and there are a few that might have
> mind enough to look at robots.txt They will not however expect to be
> handed a tailored version of it.
>
>>> I
>>> realize that they *could* spoof the SE's user agent or something,
>>> but my concerns are bright enough to look for robots.txt but not
>>> bright enough to expect to be handed a phoney
>>
>> You want to hide the key under the doormat which has in 5 languages
>> "The key is hidden nearby" written on top...
>
> Not really, or is it possible that they could also get my .htaccess?
> I didn't think that was possible. If they ask for a robots.txt and
> get one that's got nothing more than a pointer to a sitemap that will
> satisfy 'em.
The most effective way to do this is not allow the option of vieweing
robots.txt for denied IP ranges within htaccess.
As far as denying robots.txt to the entire general public?
It's a bad practice as the majority of the GP never even heard of
robots.txt
| 
XML Sitemap