Conversation

Notices

1. Embed this notice
   Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 29-Aug-2023 04:06:23 JST Alex Gleason
   Diagram of Ditto's internal architecture so far. Not sure I've captured it perfectly, but should give the general idea. #ditto
   • Sexy Moon likes this.
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 29-Aug-2023 04:44:31 JST Alex Gleason

     in reply to

     • Microchimera
     @opphunter88 It's actually not super complicated, just requires a bit of bravery because we're going where no man has gone before.
   • Embed this notice
     Microchimera (opphunter88@gleasonator.com)'s status on Tuesday, 29-Aug-2023 04:44:32 JST Microchimera

     in reply to
     @alex This looks complicated. What are you going to do when you have to integrate Ditto into Revolver?
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 29-Aug-2023 04:46:53 JST Alex Gleason

     in reply to

     • ?? أحمد ??
     @ahmad I hate Postgres. It's a goal of mine to make this work with an embedded database.
     
     So far, I haven't optimized it. If performance is poor, there are lots of things I can try.
     
     Failing all that, the query builder I use should allow us to use any SQL compatible backend including Postgres.
   • Embed this notice
     ?? أحمد ?? (ahmad@bassam.social)'s status on Tuesday, 29-Aug-2023 04:46:54 JST ?? أحمد ??

     in reply to
     Are you ditching postgres? Or you'll be using both postgres and sqlite?
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 29-Aug-2023 04:47:59 JST Alex Gleason

     in reply to

     • gvs
     @gvs Ditto works with Soapbox through Mastodon API.
   • Embed this notice
     gvs (gvs@rebelbase.site)'s status on Tuesday, 29-Aug-2023 04:48:03 JST gvs

     in reply to
     And ditto's sqlite is browseable with Soapbox?
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 29-Aug-2023 04:48:37 JST Alex Gleason

     in reply to

     • Tassoman
     @tassoman Yes. The graph is made with Mermaid which kind of sucks and looks bad if you draw too many arrows.
   • Embed this notice
     Tassoman (tassoman@orwell.fun)'s status on Tuesday, 29-Aug-2023 04:48:39 JST Tassoman

     in reply to
     what about the output? It's just a reversed arrow?
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 29-Aug-2023 04:50:31 JST Alex Gleason

     in reply to

     • ?? أحمد ??
     @ahmad There are only 3 database tables. In the future there might only be 2. The whole codebase is built around NIP-01 filters, and the SQL part uses Kysely. It would be trivial to swap out SQLite for Postgres: https://gitlab.com/soapbox-pub/ditto/-/blob/develop/src/db/events.ts?ref_type=heads#L55-108
     
     And it would be possible to use a different backing store such as LMDB.
   • Embed this notice
     ?? أحمد ?? (ahmad@bassam.social)'s status on Tuesday, 29-Aug-2023 04:50:33 JST ?? أحمد ??

     in reply to
     I'm not a huge fan of postgres but I'm also afraid to use sqlite with such huge project.
     What about scaling? Is there an sqlite proxy and cluster software?
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 29-Aug-2023 04:51:10 JST Alex Gleason

     in reply to

     • verita84 :Debian_logo: :firefox: :bing: :android:
     • Microchimera
     @opphunter88 @verita84 Chad is my chart guy.
   • Embed this notice
     Microchimera (opphunter88@gleasonator.com)'s status on Tuesday, 29-Aug-2023 04:51:11 JST Microchimera

     in reply to

     • verita84 :Debian_logo: :firefox: :bing: :android:
     @verita84 @alex He needs to use that grant money to hire a chart guy
   • Embed this notice
     verita84 :Debian_logo: :firefox: :bing: :android: (verita84@poster.place)'s status on Tuesday, 29-Aug-2023 04:51:12 JST verita84 :Debian_logo: :firefox: :bing: :android:

     in reply to

     • Microchimera
     @alex @opphunter88
     
     Does not show how Client interacts with Ditto.
     maybe good to add a box or something around Mastodon API and Mastodon Streaming API or combine them?
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 29-Aug-2023 04:52:48 JST Alex Gleason

     in reply to

     • verita84 :Debian_logo: :firefox: :bing: :android:
     • ?? أحمد ??
     @verita84 @ahmad Looks cool, but I'm aiming for minimalism. Cassandra is maximalist.
     
     If I were to use a non-embedded DB, ElasticSearch is weirdly the best option, except that it isn't FOSS.
   • Embed this notice
     verita84 :Debian_logo: :firefox: :bing: :android: (verita84@poster.place)'s status on Tuesday, 29-Aug-2023 04:52:50 JST verita84 :Debian_logo: :firefox: :bing: :android:

     in reply to

     • ?? أحمد ??
     @alex @ahmad
     
     Cassandra =)
   • Embed this notice
     verita84 :Debian_logo: :firefox: :bing: :android: (verita84@poster.place)'s status on Tuesday, 29-Aug-2023 04:54:48 JST verita84 :Debian_logo: :firefox: :bing: :android:

     in reply to

     • ?? أحمد ??
     @alex @ahmad
     
     When I worked at ScyllaDB (C++ version of Cassandra), a customer used ES and it was terrible for them and they moved away from it.
     
     The Calkey users are looking at moving to/supporting Scylla for their Misskey clone
     Alex Gleason likes this.
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 29-Aug-2023 04:58:13 JST Alex Gleason

     in reply to

     • verita84 :Debian_logo: :firefox: :bing: :android:
     • ?? أحمد ??
     • :blank:
     @i @ahmad @verita84 Any DB that uses SQL can be made to work with this project.
     
     Originally I wanted the DB itself to be any Nostr relay, but the relay would need special characteristics (like indexes on particular tags) that I can't guarantee unless Ditto itself acts as the relay.
   • Embed this notice
     :blank: (i@declin.eu)'s status on Tuesday, 29-Aug-2023 04:58:14 JST :blank:

     in reply to

     • verita84 :Debian_logo: :firefox: :bing: :android:
     • ?? أحمد ??
     @alex @ahmad @verita84 apart from the forks that still are https://github.com/crate/crate
     
     there's always https://sqlite.org/hctree/doc/hctree/doc/hctree/threadtest.wiki on the horizon though
     Alex Gleason likes this.
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 29-Aug-2023 05:01:42 JST Alex Gleason

     in reply to

     • ?? أحمد ??
     @ahmad It's stored in the SQLite database. Everything is a Nostr event (pretty much), so it only needs one table for the events, and another table for indexing tags.
     
     Remote data is automatically pruned after 7 days. And since it's connected to the Nostr network, data can be refetched in realtime if you scroll back far enough. The experience will degrade when looking at remote content older than 7 days. Local content will load well and offer the best experience.
     
     It won't have the same problems as Pleroma with remote data making the database too huge.
   • Embed this notice
     ?? أحمد ?? (ahmad@bassam.social)'s status on Tuesday, 29-Aug-2023 05:01:44 JST ?? أحمد ??

     in reply to
     I think I'm missing something.
     Where does all the data get stored?
     
     User data, passwords, events, posts.... Etc
     
     If they are stored at the instance itself, we'll reach a point where the database is fre gigs big and it might need a load balancer.
     
     Am I missing something here?
   • Embed this notice
     🫡 (captain_berserker@berserker.town)'s status on Tuesday, 29-Aug-2023 05:03:42 JST 🫡

     in reply to

     Based 🫡

     Alex Gleason likes this.
   • Embed this notice
     ?? أحمد ?? (ahmad@bassam.social)'s status on Tuesday, 29-Aug-2023 05:16:30 JST ?? أحمد ??

     in reply to
     So things just going to be easier to maintain and better optimized that we don't need to scale the service in the future to use load balancers?
     
     Or things will be optimized but we might also reach a point where 32 cores aren't enough and we'll be needed 2 or 3 database servers which we should use MySQL for ot postgres so we can make it a cluster?
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Tuesday, 29-Aug-2023 05:16:30 JST Alex Gleason

     in reply to

     • ?? أحمد ??
     @ahmad I'm not building a system that's infinitely scalable. I'm building a system that's minimalist, low cost, and with account portability.
     
     It should be able to handle traffic from Poast, for example. I'm not designing it to handle Truth Social or Gab levels of traffic. But it doesn't matter, because users can move freely and seamlessly between servers, so if one Ditto server is struggling or offline, people can just use a different one with their same account.
   • Embed this notice
     Curtis Rock, SkD (curtis@social.teci.world)'s status on Tuesday, 29-Aug-2023 05:16:52 JST Curtis Rock, SkD

     in reply to

     • Tassoman

     @alex @tassoman I’m making graphs currently using Google Drawings

     SQLite also needs periodic vacuuming. The auto-vacuum option can provide temporary relief on file size, but not on defragmentation or compaction of partially filled database pages.

     https://www.sqlite.org/lang_vacuum.html

     Alex Gleason likes this.
   • Embed this notice
     meso (meso@the.asbestos.cafe)'s status on Tuesday, 29-Aug-2023 05:19:28 JST meso

     in reply to

     • ?? أحمد ??
     @alex @ahmad Ditto... good one
     Alex Gleason likes this.
   • Embed this notice
     Microchimera (opphunter88@gleasonator.com)'s status on Tuesday, 29-Aug-2023 07:39:49 JST Microchimera

     in reply to

     • ?? أحمد ??
     @alex @ahmad Even if it blew up and the big ones started to break someone could just make something like https://joinmastodon.org/servers and even stupid people should be able to figure it out.
     
     I think there are still video games where you manually pick a server based on latency and stuff like that, basically the same thing.
     Alex Gleason likes this.
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Wednesday, 30-Aug-2023 00:10:31 JST Alex Gleason

     in reply to

     • gvs
     @gvs Unusable, but moving quickly. Feels like I'm days away but realistically up to two weeks.
   • Embed this notice
     gvs (gvs@rebelbase.site)'s status on Wednesday, 30-Aug-2023 00:10:34 JST gvs

     in reply to
     Is ditto already usable or early development?
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Friday, 08-Sep-2023 01:16:11 JST Alex Gleason

     in reply to

     • Your New Marijuana Injecting Waifu :weed:
     @sjw Data is portable on Nostr, so I don't see a need to have a decentralized database. The Nostr network IS the decentralized database. If Ditto struggles to scale, the SQLite part could be replaced with a pool of dedicated Nostr relays.
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Friday, 08-Sep-2023 01:16:13 JST Your New Marijuana Injecting Waifu :weed:

     in reply to
     @alex it should use Cassandra instead
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Friday, 08-Sep-2023 01:23:55 JST Alex Gleason

     in reply to

     • Your New Marijuana Injecting Waifu :weed:
     @sjw CouchDB is cool, but I have a strong preference for embedded databases. I looked into PouchDB, but it's for something else.
     
     Also look at how good SQL is at turning Nostr filters into queries, it's phenomenal: https://gitlab.com/soapbox-pub/ditto/-/blob/develop/src/db/events.ts?ref_type=heads#L66-128
     
     Revolver doesn't exist, show me the code.
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Friday, 08-Sep-2023 01:23:56 JST Your New Marijuana Injecting Waifu :weed:

     in reply to
     @alex Then use CouchDB because it's the most comfy DB.
     
     Also what about Revolver?
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Friday, 08-Sep-2023 01:31:36 JST Alex Gleason

     in reply to

     • Your New Marijuana Injecting Waifu :weed:
     @sjw Looks good. If I swap out SQLite it would be for a key-value store like LMDB or bbolt as I've just learned. Main reason I haven't is because it's a skill issue.
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Friday, 08-Sep-2023 01:31:37 JST Your New Marijuana Injecting Waifu :weed:

     in reply to
     @alex what about bbolt?
     https://pkg.go.dev/go.etcd.io/bbolt#section-readme
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Friday, 08-Sep-2023 01:39:41 JST Your New Marijuana Injecting Waifu :weed:

     in reply to
     @alex there is also the original bolt which probably gets used the most and is feature complete
     https://github.com/boltdb/bolt
     Alex Gleason likes this.