Weather Report - The New Yahoo! Search Crawler (Slurp) Is Here!

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Comments: Weather Report - The New Yahoo! Search Crawler (Slurp) Is

Thank you for making this change! I can already see a lower crawl load
from Slurp, with minimal changes in the rankings for the sites that I

Brian M
Posted by Brian M at July 28, 2006 08:55 AM

I am seeing to many directories and .gov and .org, and russians sites
in the results. Right now its looks like a bad copy of Google it was
much better before.
Posted by gary at July 28, 2006 09:43 AM

I thought I'd noticed a reduction in the number of hits from Slurp
recently! Glad to see it wasn't Slurp deciding it didn't like my site.
Posted by Alden Bates at July 28, 2006 03:22 PM

The old results were better. The new cralwer has brought in lot of
trash results for atleast the industries that I follow.
Posted by Chris at July 28, 2006 04:12 PM

Posted by Joerg Battermann at July 29, 2006 01:59 AM

Tats really good. We need to thank yahoo for making this change. Hope
the new slurp is healthy and eats only fresh food and generates better
indexes and SERPs thereforth.
Posted by Thomas at July 29, 2006 04:05 AM

Good News for Web Owners.
Posted by Jim at July 29, 2006 09:36 AM

Thank Goodness! Slurp has really been humping away at my sites over the
past few months.
Posted by Peter at July 29, 2006 10:10 AM

Thanks for the updated crawler, as any update is appreciated by those
of us in this community. I just hope some of my links come back!

Posted by Rob at July 29, 2006 06:39 PM

Hi, Interesting changes ! Just one observation regarding my search term
"Edinburgh cricket"; why does come 2nd !? It's a
domain registration page - not a cricket site. I'm sure it's just a
glitch which will be ironed out soon. Cheers, AL.
Posted by AL at July 31, 2006 12:53 AM

One thing i would like to see is date when last cached information.
When i see my website in yahoo search results and click on the 'cached
copy' i don't see the date this cached data is from.
Is it possible to get this information through other means?

My website changed from second position to third recently and hope that
this is the result of new crawler.

Posted by Sridhar at July 31, 2006 07:26 AM

A step in the right direction. Nice to see the search results updating
much more frequently.
Posted by Squiggs at July 31, 2006 08:36 AM

I'm already noticing an improvement in the indexing (i.e. not indexing
pages that we give you a 404 on).

One remaining indexing issue that I see is the main root page is
getting listed second on a site: command, so we see something like this
on a command

Any reason why the root would show second, and without www?

Posted by Joe at July 31, 2006 10:28 AM

Will this new slurp stop covering error logs with SlurpConfirm404 's ?
Posted by Thomas Tallyce at July 31, 2006 10:47 AM

How many times does the bot HAVE to reqest the robots.txt file in a
day? Seems to do it like every few page requests.
Posted by Arubicus at July 31, 2006 11:50 AM

Does your new crawler read Javascript?

Posted by Benoit at July 31, 2006 11:55 AM

Well, new or old, it doesn't really matter. The fact is that there are
a lot of glitches with Yahoo's search result. I am not gona vent or
something but I believe that Yahoo has more spam sites than ever. Only
because a dude has all the time in the world to get inbound links, he
doesn't deserve top rankings. I am not happy at all, specially with
Yahoo's ageing issues! Not a happy camper I must say
Posted by Zeeshan Syed at July 31, 2006 03:24 PM

not happy with yahoo results. There is lot of spam and free sites
coming in top. Sites coming in top rankings does not have good Backward
links nor content, so why such sites are coming on top.

The aim of launching the new crawler might be anything but you should
show good results than Google.
Posted by Clara at July 31, 2006 10:01 PM

Ever since the new slurp came out we see it spider our site but no
updates to the pages and the pages are slowly disappearing from yahoo.

Seems like slurp hits the robots file an awful lot and then some pages
it just keeps hitting a few times a day, but no updated cache's on
yahoo and the pages are slowly disappearing.

The site is a clean site that never had any problems on yahoo before, I
don't know what is going on.
Posted by RKR at August 1, 2006 05:04 AM

Data set looks ok. not quite as good as before however the hits to
servers look like they have decreased. I guess that the next update to
the algo will repair any gaps in the logic.
Thanks for the Update
Posted by Mike at August 1, 2006 06:25 AM

The robots are reading the "disallow: /file/" and a ""
shows all the old pages are gone, a step forward.

The Directory Title and Descriptions are still overtaking the site meta
tag, this problem needs a permanent fix.

As a result of the update, we have lost about 40% of Yahoo traffic, do
to Spammers, Geocties, Tripod and the like.

Baby Steps keep it up.

Posted by Blog at August 1, 2006 10:26 AM

Whatz wrong with Yahoo result.It seems every active site, they lost
postion in yahoo search, I only find dead sites in first results.
Posted by john peterson at August 1, 2006 12:44 PM

I notice that our site is dropping out of the search engine now. It is
a clean site addressing skin condition and used to be on the first page
of yahoo search results, what happened??
Posted by jo at August 1, 2006 10:26 PM

I tried a lot of pharmacies out there but the one that gave me the best
service with the best price is you won't
believe how they changed my life. thank you pharmacy-medicine i love
Posted by John Shefer at August 1, 2006 10:50 PM

No Yahoo - this is not on - lost lots of rankings on my own as well as
client sites - all of a sudden. Stop trying to be like Google - just be
yourselves guys
Posted by Deon Coetzee at August 1, 2006 11:23 PM

Thanks for the Weather Report.

One oddity I've noticed since July 28 is as follows: For a certain two
word search with about 9 million results, there is one index that
returns MySite as #1 - which it was consistently for several months
prior to July 28. Another index has MySite 'missing' for this same
search (by missing I mean not in top 100 results). For a different two
word search, MySite is still first page on the later index, but dropped
to fourth page on the former. Yet both indices seem to return MySite
for a variety of other searches about the same as prior to July 28 - at
least not any substantive differences I've noticed. I recall in 2004
and maybe early 2005 noting that Yahoo seemed to have 3 distinct
indices, but in 2006 I don't recall seeing such violent differences in
search results. I don't know if it is cookies, IP address or phase of
the moon that causes the different indices to be be used. Are several
indices in the process of being merged or is this some kind of A/B
testing? Danke (Go ahead and tell us - we'll keep your answer a
Posted by Confused at August 2, 2006 11:12 AM

OMG - this has trashed some of my clients listings. What the hec did
you do? There's a load of trashy results in place of the relevant
listings that existed before hand. The results are no longer to be seen
in the index. Nowhere at all. Something has majorly gone wrong. Put it
Posted by TomD at August 3, 2006 09:51 AM

OMG - this has trashed some of my clients listings. What the hec did
you do? There's a load of trashy results in place of the relevant
listings that existed before hand. The results are no longer to be seen
in the index. Nowhere at all. Something has majorly gone wrong. Put it
Posted by TomD at August 3, 2006 09:51 AM

Great I lost all my rankings again. First the weather update now the
new crawler. It seems like everytime you guys try to do something new,
we see more spam in the serps. Keep up the good work.
Posted by Mikey at August 3, 2006 09:56 PM

Welp our site has lost tons of pages now -- Go figure. Oh well what can
you expect.
Posted by Arubicus at August 4, 2006 10:21 AM

I really hope someone can help here. It seems like the new bot just
hits the same pages over and over and we are losing pages in the yahoo
index. Then I discovered that when I log into yahoo site explorer it
says my site has one subdomain, we have never had a subdomain. Could
this all be connected somehow?

Site Timeline