Conversation

Notices

1. Embed this notice
   Evan Prodromou (evan@cosocial.ca)'s status on Monday, 25-Mar-2024 00:25:40 JST Evan Prodromou

   • Bob Wyman

   @bobwyman can you help me out? I'm trying to remember the term of art for the opposite of real-time search. Is it just "static search"? I feel like there's a better term for it.

   • Embed this notice
     Evan Prodromou (evan@cosocial.ca)'s status on Monday, 25-Mar-2024 05:37:04 JST Evan Prodromou

     in reply to

     • Bob Wyman

     @bobwyman Until then, do you mind if I paraphrase for my ActivityPub book, with acknowledgment? "Prospective" and "retrospective" are way better terms than what I was using.

   • Embed this notice
     Bob Wyman (bobwyman@mastodon.social)'s status on Monday, 25-Mar-2024 05:37:05 JST Bob Wyman

     in reply to

     @evan The key to understanding the difference is to consider what is indexed. A retrospective system indexes "documents" which are usually just collections of static data. On the other hand, a query can contain ranges, boolean expressions, etc. which can't be indexed using the "inverted trees" and other methods typically used when indexing documents. Thus, true prospective systems, which index queries, require very different indexing structures than do retrospective systems.
     4/

   • Embed this notice
     Bob Wyman (bobwyman@mastodon.social)'s status on Monday, 25-Mar-2024 05:37:05 JST Bob Wyman

     in reply to

     @evan Because most people don't understand how to index queries, and because this problem is typically not discussed in schools or in books on search, most systems that should use prospective search are actually implemented by doing repeated prospective searches on incrementally built document indexes. While this is inefficient, it is easily hacked together and, given the power of machines today, most folk don't have enough queries to really notice the limitations of repeated retrospective.
     5/

   • Embed this notice
     Bob Wyman (bobwyman@mastodon.social)'s status on Monday, 25-Mar-2024 05:37:05 JST Bob Wyman

     in reply to

     @evan When the semantics of a prospective system are very simple, it is possible to use simple implementations. For instance, if the queries are limited to specifying a single string term (i.e. a user name) then one can build a simple, high-performing prospective system that extracts usernames from incoming posts and compares them to a hash-table of "queries" which are just usernames.

     A proper system would allow Booleans, ranges, etc., but that requires a dramatic increase in complexity.

   • Embed this notice
     Bob Wyman (bobwyman@mastodon.social)'s status on Monday, 25-Mar-2024 05:37:05 JST Bob Wyman

     in reply to

     @evan Someday, I'll write this stuff up in a form which is useful for general consumption. It's really not that hard if you recognize that prospective search requires approaches different from retrospective. I've always found it odd that proper prospective search technology isn't more widely used outside large platforms like Google, Yahoo, Facebook, Bloomberg, etc. Many applications could much better serve their users' needs if they did this properly.

   • Embed this notice
     Bob Wyman (bobwyman@mastodon.social)'s status on Monday, 25-Mar-2024 05:37:06 JST Bob Wyman

     in reply to

     @evan A prospective search compares each member of a sequence of one or more documents to a previously indexed set of one or more queries. In such a system a query asks: "Tell me whenever!" Because each query looks to the future, the queries are "prospective." Many notification systems rely on prospective search. Similarly, much of the internal operation of SocialWeb systems can be modeled as prospective searches. (i.e. copy post to my feed "whenever" I am mentioned in it.)
     3/

   • Embed this notice
     Bob Wyman (bobwyman@mastodon.social)'s status on Monday, 25-Mar-2024 05:37:07 JST Bob Wyman

     in reply to

     @evan I've found that "real-time search" isn't really as useful a term as people would expect... But, if you insist on using it, then the opposite of real-time is going to be either "periodic," "repeated," or "queued," depending on what you're trying to say.

     There are two kinds of search: Retrospective and Prospective.
     1/ (see following notes)

   • Embed this notice
     Bob Wyman (bobwyman@mastodon.social)'s status on Monday, 25-Mar-2024 05:37:07 JST Bob Wyman

     in reply to

     @evan A retrospective search compares a query to one or more documents that have been previously indexed. Such a query tells you what has been indexed in the past. Because the query looks to the past, it is "retrospective." You can process that query in "real-time," like Google, etc. do, or you can queue the query for processing once or more times in the future, but the query will always be looking to the past. Most "real-time" searches are implemented as repeated retrospective searches.
     2/