Dan LuuWhy not just not charge people for a trivial amount of crawler traffic? If the backend is decently fast, it should basically be free to eat the load and not charge customers for it (and yes, I know the 0.5 QPS was for one site and it will add up; even so). My CDN doesn't charge me for getting DDoS'd because that traffic isn't really "my fault" and that could easily be orders of magnitude more traffic than all crawler traffic for all crawlers across every discourse instance.
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.