yeah this shit is getting cancerous
some sites (and increasingly more) can't even be crawled because their admins disable the robot.txt