Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- Issue with MSNBot
November 16, 2004, 9:45 am
rate this thread
It keeps requesting pages that do not exist within our site. The site
uses a CMS and pages are retreived by passing a page id in the
querystring to the default.aspx page eg: default.aspx?page=8.
MSNBot keeps making requests to the site with invalid query values like
default.aspx?12 ( no page= included ) or it requests page values that
no longer exist. I've noticed that their cached version of our site
contains a link to the non-existing pages but should it not now be
spidering off of the current site not the cached one.
I've also added a robots.txt file that blocks MSNBot's access to all
except one file contained within a folder, this file contains one link
to the root with default.aspx?page=0 in the hope that MSNBot will come
here first and then get redirected but this does not appear to work
Can anyone shed some light on why MSNBot might be behaving this way ie:
Sending requests for non existing pages.
Any feedback would be greatly appreciated.