Notices by Emelia 👸🏻 (thisismissem@hachyderm.io)

@badnetmask @malte mmm, I've no idea about any of that 😅

I did use LUKS last time but managed to shutdown the machine & it became totally bricked needing OS reinstall, so I'm rolling out debian for it

@badnetmask @nevarro I'm pretty sure its just all the pipes in my head up there are f'ed — I've already had my nose completely rebuilt a few years ago

@badnetmask nah, we'd accidentally had pgbouncer configured for 2000-2500 connections, but the process had LimitNOFILESoft of 1024, so we were running out of sockets and accept() calls were failing when mastodon tried to connect to the database.

Issue only surfaced 2 days ago but we hadn't changed server configuration in 20 days, just upgraded system packages on non-database nodes in our infrastructure, so we thought we'd broken a dll linked library in ruby like libvips or libicu

we thought we'd fixed it yesterday, but had the wrong root cause. The error message was obtuse and the promtail log lines dropped the stack trace lines that followed, so we only had partial information & thought the error must've been from the upgrade, when in reality it was just a freak coincidence.

Finally figured it out earlier & deployed a fix. But it took a few tries to get it right.

We also now have alerts in place if pgbouncer errors again (we're in the process of rebuilding the primary database server & switching to pgcat which has much greater observerability)

The other option was that it was a networking error, but we could access the database node via the tailnet that connects our infrastructure.

@molly0xfff aha, this explains it, because there's absolutely no way the token has $20 billion backing it — even if all 77 million people who voted for trump bought it at 53$ it'd only be $4.08 billion

@mekkaokereke current estimates of yearly cost per account is $0.30 to $0.80 based on infrastructure, storage, etc. From what I've seen.

I'm pretty sure @esk or @dma worked out the numbers for Hachyderm too.

@skinnylatte another factor here is the oversupply of bootcamps and the differential between what they teach and what you'll actually need — I see a lot of these bootcamps in Germany which are really just a grift for government dollars (through providing "training"), they specifically target people who can't navigate the german system & also cannot challenge them within the german system.

I'm certain that also holds somewhat true for other countries, though maybe instead of a grift for government funding, it's a grift that saddles the person learning with bad debt or wage garnishments.

@hrefna @mxk yeah, we're doing review and all that before we make changes, but even then we can't sometimes be 100% sure everything is right, so we need to just double check everything's actually fine live.

@mekkaokereke this was added by Eugen and improved upon by various other people

@hrefna this wasn't about mastodon, but rather how developers work in general: the tech we use in the startup world is different to in enterprise world, which is different to that which a hobbyist might use.

@mekkaokereke I'd love to get some of those people involved in the ActivityPub Trust & Safety Taskforce, but as I don't have strong relationships with these people I'm not sure how best to invite them, and am also cognisant that standards work is an entire thing, and can be very laborious, so I don't wish to expect that of anyone from a marginalised background.

I'm currently making sure to cite articles by black & brown authors when I know of a relevant article to a specific work item.

@hrefna @DenialShown will be interesting to see what the ethics committee says about this

@molly0xfff do we even want to know what "fair democracy" is? It doesn't sound good as a company.

@mattly @graue "this has been rebased for nearly 2.6 years, that's kinda absurd" says it all really. I hope things improve soon, I really do.

@mattly @jenniferplusplus something that might interest you, which is a recent change I'm working on for Mastodon is expanding the moderation notes capabilities to more places. So, for starters, Instances: https://github.com/mastodon/mastodon/pull/31529

But also there is Appeals and maybe blocked email domains, hashtags, and IP rules. All tbd though.

@mekkaokereke I think the only "hard" one on that list for me, that I don't have a good answer for is “Composable moderation” — the only way this can work is through data sharing with third-parties, which can get sketchy without proper DPAs in place.

@hrefna on a possibly related note: is JSON-LD or RDF actually good for data sovereignty? (My gut feeling after 4 years is "no")

@mekkaokereke @Badgardener right now it's mostly having the time to both build, review and ship the features.

As mentioned in another thread I've been working on a bunch of these onboarding and moderation features we'll need in the coming months separately, along with others.

@mekkaokereke @Badgardener we should refuse to accept that.

But the reality right now is projects are underfunded. I've heard of at least a few that may need to shutdown within the next 6-12 months unless they can attract funding.

For 5 engineers in Europe, you're looking at a burn rate of at least €30,000 a month (that's €60-72k salaries), then there's operating expenses, taxes, server/infrastructure costs on top.

@mekkaokereke yeah, we're constantly fighting the fact that a lot of the trust & safety aspects in both Mastodon and ActivityPub in general were left languishing for so long.

Hopefully the new ActivityPub Trust & Safety Taskforce can make some forward improvements here through reports & recommendations.

Though funding for work is definitely also a factor here, it's obviously possible to do more faster with a budget of $30+ Million vs $600k/year

@hrefna @polotek @renchap definitely agree on that point.