Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Nobody follows robots.txt, because every site's robots.txt forbids anybody that isn't google from looking at it.

Also, "hiding their identity" is what every single browser does since Mosaic changed its name.



Including extra, legacy agents isn't hiding because they include their distinct identifiers too.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: