Conversation

Notices

1. Embed this notice
   Dean (dean@uwu.social)'s status on Thursday, 30-May-2024 18:52:12 JST Dean

   We're struggling to figure out a good way to migrate Mastodon media storage from one server to another... Mastodon stores files in millions of directories by hash (kinda like aaa/bbb/ccc/ddd) which means doing something simple like an initial rsync and then taking Mastodon down for a quick resync is impossible. We initially were going to go this route but after the initial file listing took more than 24h we cancelled it and gave up.

   So now we're looking at just copying the raw filesystem over, but if we want to do it without taking mastodon down for the entire sync we need to come up with a way of copying it and resyncing the changed blocks afterwards.

   One way could be to use overlayfs. Remount the old volume R/O, create a temporary upperdir and create an overlay between them. Then, copy the R/O image to it's new home, expand it or whatever, and apply the upperdir onto it. This way we only need to list the directories that actually had writes. Special care will need to be taken to ensure we delete any files that have overlayfs tombstones. IDK if anyone has ever done this before.

   Another way could be to use devmapper snapshots to create a new COW-backed volume, rsync the R/O underlying block device over and then apply the COW to the new volume with snapshot-merge. We tried testing this out and caused devmapper to die horribly and spit out kernel bug log lines, so we had to reboot and e2fsck for 2 hours.

   At this point it might be better to just take everything down for as long as it takes. I'm extremely annoyed at Mastodon's file structure making it impossible to move without major downtime. Their solution just seems to be "use S3 lol". It would probably take 24 hours (8TB at 1Gbps is roughly 17 hours). We could shrink it first since we don't use all the space, but resize2fs will take a while as well.

   If anyone has any tips or ideas for doing it with minimal downtime I'd like to hear them. Or if you're an uwu.social user and don't care about extended downtime I'd also like to hear your thoughts too.

   • Embed this notice
     :blobcathug: (jain@blob.cat)'s status on Thursday, 30-May-2024 18:52:10 JST :blobcathug:

     in reply to
     @dean how about purging remote media first?
   • Embed this notice
     :blobcathug: (jain@blob.cat)'s status on Thursday, 30-May-2024 18:57:17 JST :blobcathug:

     in reply to

     • :blobcathug:
     @dean i mean, i really dont get why mastodon saves media from remote instances... It doesnt really make sense and just blows up media storage...
     There is a maintenance job for that
   • Embed this notice
     :blobcathug: (jain@blob.cat)'s status on Thursday, 30-May-2024 19:05:00 JST :blobcathug:

     in reply to
     @dean :blobcatgoogly: are you sure you did or isnt the job done yet? Im asking because i cant imagine that the 8TB is just media from your users... Well it could be but it seems a bit high for the amount of MAU you have on your server
   • Embed this notice
     Dean (dean@uwu.social)'s status on Thursday, 30-May-2024 19:05:01 JST Dean

     in reply to

     • :blobcathug:

     @Jain yeah we purged media before we started, I don't think it speeds up the file listing though because the hash directories get left behind AFAIK :(

   • Embed this notice
     Dean (dean@uwu.social)'s status on Thursday, 30-May-2024 19:06:15 JST Dean

     in reply to

     • :blobcathug:

     @Jain the 8TB is the raw volume size if we were to copy the ext4 partition over instead of rsyncing each file over. Sorry if that wasn't clear 🙏

     :blobcathug: likes this.
   • Embed this notice
     :blobcathug: (jain@blob.cat)'s status on Thursday, 30-May-2024 19:18:33 JST :blobcathug:

     in reply to
     @dean i different way would be to use the database as your file list.
     Backup the DB, use the Backup to copy files, take down the instance, compare the stopped db with the backup, resync the difference and you are done...
     Altho this require more work...
   • Embed this notice
     Dean (dean@uwu.social)'s status on Thursday, 30-May-2024 19:21:20 JST Dean

     in reply to

     • :blobcathug:

     @Jain that's a good idea! Might look into this and see how feasible it is for us

     :blobcathug: likes this.