Embed Notice

@robin in other words, force Google and the other data barons to participate in the crawling of the web as a public utility, since it has been proven too valuable/powerful as proprietary property of any single company.

There’s also an efficient-resourcing argument to be made, because many sites are hit hard by swarms of crawlers, all collecting the same data. Frequently crawled websites would be considerably cheaper to run if mass-crawling was more regulated and operated shared infrastructure.