Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- URL restricted by robots.txt
August 10, 2006, 12:54 pm
rate this thread
My site (blog) located at http://itly.blogspot.com
Presently search engines do NOT visit my blog.
How to rectify it?
Under Google Sitemaps it shows
"URL restricted by robots.txt"
But since I am using blogger I can NOT edit the robots.txt
But I can access the source code of my blog.
Is there anyway to enable search engines to crawl and index my blog?
Re: URL restricted by robots.txt
Hi Charles et al,
I'd be extremely surprised if any well-meaning robot/spider/crawler
didn't stop asking questions at robots.txt. One of the main points of
robots.txt is to allow Webmaster to keep bots from using bandwidth, and
if a bot ignores robots.txt and downloads a file to check to see if it
contains a META noindex, that'd severely undermine that feature of the
To the OP -- the META robots tag described on the page to which Charles
correctly pointed to you is mostly used in the negative (i.e.
noindex,nofollow) to keep bots out. The positive (index,follow) are the
de facto standard for bots visiting a page, so adding a robots META tag
with content="index,follow" will only tell them what they already know.
As other have suggested, if you want to get spidered, you'll have to
FWIW, if a page is verboten according to robots.txt, Nikita the Spider
doesn't download it to check the META tags.
Whole-site HTML validation, link checking and more
- » Cheap, Low Budget, Customized and Affordable Web Design Services
- — Previous thread in » Webmaster Forum