So, am hearing a lot of buzz about https://kagi.com and yeah, it seems good. (It is pretty well forgotten that one of the first good-ish search engines, Infoseek, had a pay-per-search model too.) Anyhow… where does Kagi get its data? I have seen many general statements that it has a crawler, but I see no evidence. In particular, I see nothing that (a) looks at my robots.txt and (b) has kagi in its name.