Embed Notice

@louis "cheese NOT toast" does not yield any results, though simply looking for "cheese" yields those with no "toast" in sight, and more than "cheese AND toast".

As for the interactivity, I enjoy proper lispy REPLs as well, but to be fair, with less interactive languages one probably would make the state needed for resumption persistent, and probably have fewer runtime exceptions: it would be odd to keep restating large-scale crawling on each adjustment.

How much space does the crawled data take, by the way? If it is not too big, maybe you could also organize something similar to Common Crawl for Gopher, by publishing that as a single dump. Not sure how useful that would be though, but would have been useful for this project if somebody else did it earlier.