URL restricted by robots.txt

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View

My site (blog) located at http://itly.blogspot.com

Presently search engines do NOT visit my blog.

How to rectify it?

Under Google Sitemaps it shows

"URL restricted by robots.txt"

snapshot here

But since I am using blogger I can NOT edit the robots.txt


But I can access the source code of my blog.

Is there anyway to enable search engines to crawl and index my blog?

Please reply.

Re: URL restricted by robots.txt

If I have to put something in META tags

What is the exact code actually?

Re: URL restricted by robots.txt

fantababy wrote

Quoted text here. Click to load it

You can get it here:


Again, I suspect the robots.txt would over-rule it, not sure though.

Charles Sweeney

Re: URL restricted by robots.txt

Quoted text here. Click to load it

Hi Charles et al,
I'd be extremely surprised if any well-meaning robot/spider/crawler
didn't stop asking questions at robots.txt. One of the main points of
robots.txt is to allow Webmaster to keep bots from using bandwidth, and
if a bot ignores robots.txt and downloads a file to check to see if it
contains a META noindex, that'd severely undermine that feature of the
robots.txt standard.

To the OP -- the META robots tag described on the page to which Charles
correctly pointed to you is mostly used in the negative (i.e.
noindex,nofollow) to keep bots out. The positive (index,follow) are the
de facto standard for bots visiting a page, so adding a robots META tag
with content="index,follow" will only tell them what they already know.
As other have suggested, if you want to get spidered, you'll have to
move elsewhere.

FWIW, if a page is verboten according to robots.txt, Nikita the Spider
doesn't download it to check the META tags.

http://NikitaTheSpider.com /
Whole-site HTML validation, link checking and more

Site Timeline