Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- strange situation
June 3, 2004, 10:35 pm
rate this thread
something is wrong...
i published my website few weeks ago.. and.. i'm tracking some search engine
they come to my site, get robots.txt and they leave the website without
requesting internal pages or links.
anyone could please explain me why.
please take a look at the website
Togli l'antispam per rispondere in privato
Re: strange situation
1. It was an idea based on previous experience with unterminated lines.
Since there can be various parameters the software waits for and end of line
code before it can be sure it has all the parameters. If in doubt, type in
the whole robots.txt again, avoiding any mistakes in the typing and pressing
return after the last k You don't want hidden invalid non-displaying bytes
embedded in the file. If the software that reads the robots.txt thinks it
has a line like Disallow: / then all robots will be stopped.
Read this: http://www.searchengineworld.com/misc/robots_txt_crawl.htm
2. I have just tested your robots.txt with a validator at
It points out you have a capital letter A for agent. This is not a fatal
error for all search engines, but a "warning" and therefore may or may not
be acceptable to all search engine robots. Change this to lower case a to
be on the safe side.
Best regards, Eric.