A reasonable way in constructing web crawlers

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Hello, everyone
It was learned that only 1/3 of the entire web is indexed, the
traditional approach of crawling the web by a single force of
commercial crawlers dont address the task. As well as Grub, the
distributed crawler, cannot display what I expeced. Is there any better
idea can cover the web more efficiently?
And the revisiting strategy can only rely on propability analysis, are
there any better mechanism in addressing the task? Appriciated for

Site Timeline