Yahoo! Search Crawler, Slurp, is moving

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Yahoo! Search Crawler, Slurp, is moving

Anyone who has seen their web server logs, has seen the Yahoo! crawler
come by and leave its mark. It identifies itself as 'Yahoo! Slurp' and
its domain is For those of you who follow the
search space, you can easily guess that this is one of the last
historic remnants from our acquisition of Inktomi a few years ago.
Well, the crawler finally decided to move and find a new home. We are
moving our crawler from to

The user-agent will continue to be Yahoo! Slurp, hence you do not need
to make any changes to your robots.txt file. Nor are we changing the
actual IP addresses of the crawler infrastructure during this
transition. However, if you do any reverse DNS checks for the crawler
identity or have any network access rules to allow,
please also update them to allow for

The crawlers will be switched over in phases starting immediately.
You'll see an increasing number of Yahoo! Slurp visits from and fewer and fewer from during the
transition, which will take a few weeks.

Keep an eye on this space and the Yahoo! Search Blog for more details
on when the migration is complete and details on bot verification with
the new domain name.

Priyank Garg
Product Manager, Yahoo! Search

Site Timeline