Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Purely speculating, Wikipedia has a huge number of inbound links (likely many more than CNet or even than more popular sites) which crawler allocation might be proportionate to. Even if it only crawled pages that had a specific link from an external site, that would be enough for Google to get pretty good coverage of Wikipedia.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: