Conversation

Notices

1. Embed this notice
   Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 08-Aug-2023 02:52:07 JST Alex Gleason
   primal.net is apparently indexing the entirety of Nostr on a cluster of SQLite databases and serving it in microseconds. I'm intrigued by this, so I'm figuring out how to run Julia code to see it for myself. Also brewing some sen-cha. 🍵
   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Tuesday, 08-Aug-2023 02:56:58 JST Sexy Moon

     in reply to
     @alex > cluster of sqlite
     
     what are they doing, partitioning the keyspace across a bunch of nodes?
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 08-Aug-2023 02:57:13 JST Alex Gleason

     in reply to
     The main limitation of SQLite is that it locks up when you try to write too much to it at once. This makes it "not a great use-case for social media". Not even due to your posts, but because of all the likes and emoji reactions you guys do. I still haven't hit any of those limits in my own experience. But writing to multiple database files could help it a lot, I bet.
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 08-Aug-2023 02:58:31 JST Alex Gleason

     in reply to

     • Sexy Moon
     @Moon That's what I'm trying to figure out. One of the devs replied to me here pointing out the code that's doing it: https://github.com/PrimalHQ/primal-caching-service/issues/8#issuecomment-1667925661
     
     I think you could maybe segment particular events based on the first X characters of their hex ID. But I don't know how you query that.
   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Tuesday, 08-Aug-2023 02:59:44 JST Sexy Moon

     in reply to
     @alex what do you "query", like an entire timeline of someone, or is it full text search or something?
   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Tuesday, 08-Aug-2023 03:01:04 JST Sexy Moon

     in reply to

     • Sexy Moon
     @alex oh nvm reading now
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 08-Aug-2023 03:01:24 JST Alex Gleason

     in reply to

     • Sexy Moon
     @Moon The main problem I'm trying to solve is the home feed. So you would query events from all _users_ you follow. That means you would have to shard the databases by the author of the user rather than event ID... 🤔 Maybe that's it. But then what if you need the event by its ID?!
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 08-Aug-2023 03:06:29 JST Alex Gleason

     in reply to

     • MercurialBlack
     @MercurialBlack You can literally paste errors at ChatGPT and it will know exactly what's going on. https://chat.openai.com/share/5c7e1d2b-6a2b-4064-92f3-d7ee11a4fcd7
   • Embed this notice
     MercurialBlack (mercurialblack@pleroma.mercurial.blog)'s status on Tuesday, 08-Aug-2023 03:06:39 JST MercurialBlack

     in reply to
     @alex julia is easy
   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Tuesday, 08-Aug-2023 03:12:21 JST Sexy Moon

     in reply to
     @alex two databases lol
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 08-Aug-2023 03:20:54 JST Alex Gleason

     in reply to

     • Sexy Moon
     @Moon Well, here it is! Each one is a kv store holding blobs. I still don't totally get it.
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 08-Aug-2023 03:22:55 JST Alex Gleason

     in reply to

     • Sexy Moon
     @Moon This one has an "is_reply" 0 or 1, which is very interesting.
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 08-Aug-2023 03:23:03 JST Alex Gleason

     in reply to

     • Sexy Moon
     @Moon Forgot to attach the pic
     Sexy Moon likes this.
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 08-Aug-2023 03:24:32 JST Alex Gleason

     in reply to

     • Ademan
     @ademan I wouldn't use Postgres again after how much of a pain it was in Pleroma. The better question is why not just use LMDB if you're going to go through all this trouble. They're equally hard, but at this rate LMDB would be easier.
   • Embed this notice
     Ademan (ademan@thebag.social)'s status on Tuesday, 08-Aug-2023 03:24:33 JST Ademan

     in reply to

     Do you know what the motivation for this over just-using-postgres was? I’m confident you can beat postgres if you’re careful and thoughtful, but I’m also reasonably confident rolling your own system like this is at minimum going to be brittle, and probably have at least as many performance pitfalls at the end of the day.

   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Tuesday, 08-Aug-2023 03:27:40 JST Sexy Moon

     in reply to

     • Ademan
     @alex @ademan maybe they are like me and they never heard of it before you mentioned it just now, ha
   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Tuesday, 08-Aug-2023 03:30:34 JST Sexy Moon

     in reply to
     @alex I don't quite understand, the event ids are all 256 bits, right? so can't you just take 100 servers or whatever and break up the keyspace into 100 sections and when you get a new event you just send it to that shard and query it the same way? the ids are cryptographic so they should scatter roughly equally across the nodes.
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 08-Aug-2023 03:31:21 JST Alex Gleason

     in reply to

     • Sexy Moon
     • Ademan
     @Moon @ademan strfry uses it in the most genius way possible: https://github.com/hoytech/strfry
     
     I wish I was this smart, but it's going to take some time before I can figure out how to make range queries do what I want and be able to manage all that shit.
     
     It's a pure KV store but the keys are guaranteed to be in alphabetical order, so you have to build indexes and stuff manually around that concept and do "range queries" that seek all keys from a start til and end value.
     Sexy Moon likes this.
   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Tuesday, 08-Aug-2023 03:33:01 JST Sexy Moon

     in reply to

     • Ademan
     @alex @ademan I have to admit this is more interesting technically than activitypub json-ld fucking shit and just jamming json into a relational database and dumbly replicating.
     Alex Gleason and victor like this.
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 08-Aug-2023 03:34:23 JST Alex Gleason

     in reply to

     • Sexy Moon
     • Ademan
     @Moon @ademan Even the SQL version of Nostr is incredibly minimal. This is all you really need. Can't tell you if it performs well tho.
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 08-Aug-2023 03:34:35 JST Alex Gleason

     in reply to

     • Sexy Moon
     • Ademan
     @Moon @ademan You actually don't need "users", that's a Ditto thing.
   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Tuesday, 08-Aug-2023 03:38:17 JST Sexy Moon

     in reply to

     • Sexy Moon
     @alex oh I see, you need all kinds of searches of events not just by id.
     Alex Gleason likes this.
   • Embed this notice
     cjd (cjd@pkteerium.xyz)'s status on Tuesday, 08-Aug-2023 03:38:28 JST cjd

     in reply to

     • Ademan
     Cluster of sqlite databases sounds like they have enough partitions that there's no need to parallelize any single db and sqlite is just more light weight so they can run more of them on the same machine.
     Alex Gleason likes this.
   • Embed this notice
     :blank: (i@declin.eu)'s status on Tuesday, 08-Aug-2023 03:41:10 JST :blank:

     in reply to

     • Ademan
     @ademan @alex postgresql's process/server based arch just sucks for massively KV things, LMDB/RocksDB are definitely better
     
     looking over the code, there's a sqlite DB per thread so there's never a lock, and queries either go over all of them, or query the right shard/thread
     
     very elegant, uber performant
     Sexy Moon likes this.
   • Embed this notice
     Ademan (ademan@thebag.social)'s status on Tuesday, 08-Aug-2023 03:41:11 JST Ademan

     in reply to

     I was always under the impression that was more due to pleroma’s schema not postgres itself.

     LMDB does look interesting though…

   • Embed this notice
     victor (victor@crucible.world)'s status on Tuesday, 08-Aug-2023 03:46:57 JST victor

     in reply to

     • Sexy Moon
     • Ademan
     @Moon @alex @ademan The biggest selling point of Nostr to a developer is its simplicity. If you can open a websocket, send commands, and receive and parse responses in JSON format, then the rest is just protocol-ism and storing things as efficiently as possible.
     Alex Gleason likes this.
   • Embed this notice
     dadalo_admin (dadalo_admin@dadalo.pl)'s status on Tuesday, 08-Aug-2023 03:48:37 JST dadalo_admin

     in reply to

     @alex
     I think that is why u need kafka or sth like that before DB. There is no way to optimize it for peaks... currently I am working on the same problem and wasted few weeks for scaling optimizing, tests and the solution is one add this element to process data. I am also going to tokenize some data to process faster...

     I am learning this shit right now...

     Alex Gleason likes this.
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 08-Aug-2023 03:51:14 JST Alex Gleason

     in reply to

     • dadalo_admin
     @dadalo_admin > Kafka is an open-source distributed event streaming platform
     
     Seems like that could be useful to Nostr, which is based entirely on the idea of distributed event streaming.
   • Embed this notice
     dadalo_admin (dadalo_admin@dadalo.pl)'s status on Tuesday, 08-Aug-2023 03:58:22 JST dadalo_admin

     in reply to

     @alex
     Another advantage is that streaming module is separate and can be extended, optimised and also scaled... twitter used this as eventbus or sth like that...

     Alex Gleason likes this.
   • Embed this notice
     fiatjaf (3bf0c63fcb93463407af97a5e5ee64fa883d107ef9e558472c4eb9aaaefa459d@mostr.pub)'s status on Tuesday, 08-Aug-2023 06:42:45 JST fiatjaf

     in reply to
     I just wish they exposed their API in a standard Nostr relay format.
     Alex Gleason likes this.
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 08-Aug-2023 06:43:32 JST Alex Gleason

     in reply to

     • fiatjaf
     @3bf0c63fcb93463407af97a5e5ee64fa883d107ef9e558472c4eb9aaaefa459d They made it intentionally not incompatible:
     
     ["REQ", "amelx49c18", {"cache": ["net_stats"]}]
     ["CLOSE", "amelx49c18"]
     
     ["REQ", "p0xren2axa", {"cache": ["feed", {"pubkey": "64-hex digits of pubkey id"}]}]
     
     ["REQ", "vqvv4vc6us", {"cache": ["thread_view", {"event_id": "64-hex digits of event id"}]}]
     
     ["REQ", "ay4if6pykg", {"cache": ["user_infos", {"pubkeys": ["64-hex digits of pubkey id"]}]}]
     
     ["REQ", "2t6z17orjp", {"cache": ["events", {"event_ids": ["64-hex digits of event id"]}]}]
     
     ["REQ", "1uddc0a2fv", {"cache": ["user_profile", {"pubkey": "64-hex digits of pubkey id"}]}]
     
     Filter is extended to have an extra "cache" key. But I don't think they serve normal filters from this.