Do you have a question? Post it now! No Registration Necessary. Now with pictures!
November 10, 2005, 3:32 pm
rate this thread
'gsa-crawler'. It appears this is the user-agent of a Google Search
Appliance. An @ google email address is listed as well.
I would like to disallow this crawler from my website, but do not want
to restrict the regular google crawler. Can anyone confirm that
gsa-crawler is definitley NOT the crawler for google's search engine.
It looks like they like they are attempting to outsource the actual crawling
You have the machine that works like a local sitebased searchengine after
which Google comes in and take 1 file with all info... kinda like the
sitemap.xml they are promoting
Google's update frequency is slow and the GSA for corporate and business
sites and the sitemap.xml for the common man might be a good way to speed
things up dramaticly if it's use increases. It sure goes faster to index 1
file compared to crawling 500.000!
You might see an evolution from the traditional crawling Google to an