Conversation

Notices

1. Embed this notice
   feld (feld@bikeshed.party)'s status on Monday, 03-Jul-2023 08:52:00 JST feld

   • Eric McCorkle
   > You then have a ZFS-style volume system, where you can specify things like "I want these objects replicated 2-3x, persistent / expiring, etc..."
   
   It's called CEPH
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Monday, 03-Jul-2023 23:31:26 JST feld

     in reply to

     • Eric McCorkle
     • prom™️
     IPFS doesn't fit the requirements of being able to accurately monitor and control the replication (will need a complex pinning service involved)
     
     Plus all the existing software speaks S3, not IPFS, so CEPH's S3 would be a perfect drop-in replacement
   • Embed this notice
     prom™️ (promovicz@chaos.social)'s status on Monday, 03-Jul-2023 23:31:28 JST prom™️

     in reply to

     • Eric McCorkle

     @feld @emc2 CEPH could work, but I don't think that its file system is meant to be run public. IPFS might be another option, although I'm not sure about its future ATM.

   • Embed this notice
     feld (feld@bikeshed.party)'s status on Tuesday, 04-Jul-2023 01:45:37 JST feld

     in reply to

     • Eric McCorkle
     • prom™️
     yeah but IPFS has a ton of other complexities that makes it suboptimal for this specific use case. It's really, really hard to know how many copies of things are out there...
   • Embed this notice
     prom™️ (promovicz@chaos.social)'s status on Tuesday, 04-Jul-2023 01:45:38 JST prom™️

     in reply to

     • Eric McCorkle

     @feld @emc2 S3 front-ends for IPFS also exist.

   • Embed this notice
     feld (feld@bikeshed.party)'s status on Tuesday, 04-Jul-2023 01:55:14 JST feld

     in reply to

     • Eric McCorkle
     • prom™️
     CEPH is already designed for distributed heterogeneous storage, though, which is one of the hardest things to get right. The question is whether or not it could work well with large number of cluster members spread out so far geographically.
     
     It already has the algorithm to make sure that changes require as few data moves/copies as possible.
     
     With any other system I worry that if some volunteer's data goes offline we're left in a dangerously degraded state // new storage comes online and the system stupidly starts an automatic replication of data to repair the lost redundancy or rebalance and it chokes the links between the nodes and causes the storage to have massive performance issues for the end users.
     
     CEPH is meant to be designed already to avoid that.
   • Embed this notice
     prom™️ (promovicz@chaos.social)'s status on Tuesday, 04-Jul-2023 01:55:16 JST prom™️

     in reply to

     • Eric McCorkle

     @feld @emc2 True. I don't think that either are a truly good fit. Architectures based on distributed hash tables could work to some degree, but some access latency would usually remain. I would also prefer something with crypto - because ActivityPub is already very insecure and brittle.

   • Embed this notice
     feld (feld@bikeshed.party)'s status on Tuesday, 04-Jul-2023 01:58:40 JST feld

     in reply to

     • Eric McCorkle
     • prom™️
     in either case, this is an incredibly hard problem to solve and I don't know any software other than CEPH or IPFS that would be a good starting point to build off of.
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Tuesday, 04-Jul-2023 02:01:56 JST feld

     in reply to

     • Eric McCorkle
     • prom™️
     That's why I suggest that it not be "public" as such. It shouldn't be some open system anyone can join without permission.
     
     It should be coordinated around a group of volunteers who know what they're doing, have access to the datacenter space required, and run as a non-profit.
   • Embed this notice
     prom™️ (promovicz@chaos.social)'s status on Tuesday, 04-Jul-2023 02:01:57 JST prom™️

     in reply to

     • Eric McCorkle

     @feld @emc2 Yes, but CEPH is not intended to be run in public - which basically means that you have internal adversaries.

   • Embed this notice
     feld (feld@bikeshed.party)'s status on Tuesday, 04-Jul-2023 02:04:40 JST feld

     in reply to

     • Eric McCorkle
     • prom™️
     Otherwise the best option we got is something like what Filecoin is trying to create on top of IPFS
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Tuesday, 04-Jul-2023 02:14:40 JST feld

     in reply to

     • Eric McCorkle
     • prom™️
     > but social issues tend to pop up at certain sizes.
     
     A DAO would be perfect but the fediverse is inherently allergic to anything from the crypto/web3 community
   • Embed this notice
     prom™️ (promovicz@chaos.social)'s status on Tuesday, 04-Jul-2023 02:14:42 JST prom™️

     in reply to

     • Eric McCorkle

     @feld @emc2 A fully distributed approach has different issues too, but I have seen it work several times.

     Non-profits are difficult in many countries. They can't be run base-democratic in the US. They can be in Europe, but social issues tend to pop up at certain sizes.

   • Embed this notice
     feld (feld@bikeshed.party)'s status on Tuesday, 04-Jul-2023 02:22:08 JST feld

     in reply to

     • Eric McCorkle
     • prom™️
     not all of us are so crazy 🥲
   • Embed this notice
     prom™️ (promovicz@chaos.social)'s status on Tuesday, 04-Jul-2023 02:22:10 JST prom™️

     in reply to

     • Eric McCorkle

     @feld @emc2 It is - but that's about culture. I can understand that very well - those crypto-hyperlibertarians don't seem to have social graces. They usually act like aggressive capitalists.