Conversation

Notices

1. Embed this notice
   Kiwix (kiwix@mastodon.social)'s status on Monday, 09-Dec-2024 18:20:14 JST Kiwix

   • Hetzner

   So last week (on Sunday 1 December at 00:00), our server host canceled its service without warning.

   TL;DR: we do not recommend using @Hetzner_Online 's service

   Everyone else: a short 🧵

   • Kari'boka and 10π🍇 like this.
   • Rich Felker and Joachim repeated this.
   • Embed this notice
     Kiwix (kiwix@mastodon.social)'s status on Monday, 09-Dec-2024 18:58:26 JST Kiwix

     in reply to

     Murphy's law states that if things can go wrong, they will. Ideally in the worst possible way.

     For us, that meant having our servers disconnected at 00:00 on a Sunday.

     Our main storage backend became entirely unreachable. For the average user that meant not being able to access the library and download files, and for us that meant not being able to connect to it and see what was wrong.

     Michał "rysiek" Woźniak · 🇺🇦, Rich Felker and Anna e só and 4 others repeated this.
   • Embed this notice
     Kiwix (kiwix@mastodon.social)'s status on Monday, 09-Dec-2024 18:58:26 JST Kiwix

     in reply to

     Turns out that Hetzner has decided to cancel our account and terminate all servers. There was no warning (yes, we checked our spam folder), and nobody could be reached before Monday morning.

     When reached, they could not explain the reason for the cancellation:
     Them: - We sent you an email.
     Us : -We did not receive it, can you please resend?
     Them: - We don't have it
     Us: ಠ_ಠ

   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Monday, 09-Dec-2024 20:27:54 JST Rich Felker

     in reply to

     @kiwix The poor communication is bad, but this is the worst. If you're terminating service, data should be held for a reasonable time, at least a month, unless it was manually inspected and deemed illegal even to possess (i.e. CSAM), to allow customer to retrieve/migrate it. Immediate deletion is a huge red flag.

   • Embed this notice
     Kiwix (kiwix@mastodon.social)'s status on Monday, 09-Dec-2024 20:27:55 JST Kiwix

     in reply to

     In the meantime, all servers had been wiped already so no way to retrieve our data.

     If you are looking for a bad case of the Mondays, well, that was one.

     Haelwenn /элвэн/ :triskell: likes this.
     anban repeated this.
   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Monday, 09-Dec-2024 20:32:00 JST Rich Felker

     in reply to

     @kiwix Even if you don't care about customers terminated fir violation of ToS, immediate deletion for them means same could happen to any customer by technical glitch or employee error. That should not be possible in decent professional hosting.

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Aral Balkan (aral@mastodon.ar.al)'s status on Monday, 09-Dec-2024 21:34:10 JST Aral Balkan

     in reply to

     @kiwix Sorry you had to go through that. Please do keep us updated if they do provide any sort of explanation/apology/recourse.

     Kari'boka likes this.
   • Embed this notice
     Kiwix (kiwix@mastodon.social)'s status on Monday, 09-Dec-2024 22:18:47 JST Kiwix

     in reply to

     Luckily we have mirrors and these were not affected. We grabbed a new machine somewhere else (Scaleway ; if we name-and-shame the one we might as well name-and-greet the other) and immediately started re-importing our data to our new Master server.

     All in all, it still took about 48 hours to get these 8-ish TB back online.

     Haelwenn /элвэн/ :triskell: likes this.
     anban repeated this.
   • Embed this notice
     Kiwix (kiwix@mastodon.social)'s status on Monday, 09-Dec-2024 22:18:48 JST Kiwix

     in reply to

     If there is any silver lining to this, it is that we could see a few points of vulnerabilities as well as our ability to turn things around in a reasonably quick manner (here be kudos for the two heroes who manage our infra).

     Learning were made, and we will see in the coming weeks/months how we can implement new safegards within our resource constraints.

     /END

     Kari'boka repeated this.
   • Embed this notice
     Joachim (joachim_kreativ@literatur.social)'s status on Monday, 09-Dec-2024 22:18:50 JST Joachim

     in reply to

     @kiwix

     Huh, thanks for sharing. That was a though time, eh? :-(

     On slightly lighter note: most of your user should have had the content they wanted allready - localy and #offline. (That _is_ the point of #kiwix, right ? )

     anban likes this.
   • Embed this notice
     Skadi (skadi@bolha.us)'s status on Monday, 09-Dec-2024 22:37:36 JST Skadi

     in reply to

     @kiwix it's not the first complain I see in my feed and it's against the law, weird!

     Kari'boka likes this.
   • Embed this notice
     The Fedilore Otter 🦦 (fedilore@mastodon.social)'s status on Tuesday, 10-Dec-2024 02:23:57 JST The Fedilore Otter 🦦

     in reply to

     • Aral Balkan
     • Lenz Grimmer

     @aral @kiwix @lenzgr When it's fedi, the issue is usually the anti-porn rule in their TOS.

     In some cases, it's in good faith. For example, with the tenforward.social, they cited a lot of hardcore porn posted/boosted by the admin, on the admin's main account.

     In other cases, it seems like they don't really pay attention and are vulnerable to fake reports. For example, a series of small transfem instances were taken down over an anime picture of two clothed girls kissing.

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Jan Wildeboer 😷:krulorange: (jwildeboer@social.wildeboer.net)'s status on Tuesday, 10-Dec-2024 03:31:24 JST Jan Wildeboer 😷:krulorange:

     in reply to

     @kiwix Send them a request for information under #GDPR rules, forcing them to send you *all* data they have. Hetzner is based in Germany, so they have to comply.

     Kari'boka likes this.
   • Embed this notice
     Nemo_bis 🌈 (nemobis@mamot.fr)'s status on Tuesday, 10-Dec-2024 03:34:15 JST Nemo_bis 🌈

     in reply to

     • Rich Felker

     @dalias There's no shortage of false positives in that department at #Hetzner either. https://web.archive.org/web/20240423104405/https://thekinrar.fr/en/posts/xyz-suspension/

   • Embed this notice
     playit (playit@fediverse.dotslashplay.it)'s status on Tuesday, 10-Dec-2024 03:55:22 JST playit

     in reply to

     • Hetzner
     We got a slightly similar experience with @Hetzner_Online@social.cologne in 2021, following a bogus claim by a copyright troll ("DMCA Force", seemingly working for the game development studio Cyan Worlds).
     
     At no time did Hetzner try to check if the claim was legit, and forced us to take down all of our websites (at least they did not deleted them with no prior warning). They immediately sided with the copyright troll, against their legit paying customer.
     
     We managed to get an apology from Cyan Worlds, who promised they are going to be more careful in the future with who they work with, and made the copyright troll retract their claims. But nothing from Hetzner. Not even an informal apology. Not even an acknowledgement of the claims retraction.
     
     In our case we moved to self-hosting to ensure this would never happen again, but this might not be a good solution for you if you have much bigger hosting needs.
     
     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Julian Andres Klode 🏳️‍🌈 (juliank@mastodon.social)'s status on Tuesday, 10-Dec-2024 11:08:55 JST Julian Andres Klode 🏳️‍🌈

     in reply to

     • Rich Felker

     @dalias @kiwix Hetzner argued they sent a message you don't know when that happened, I have no reason not to believe them.

     Someone will have filed an abuse notice due to copyright violations, hetzner will have sent an email and then terminated the account after not receiving a response, that's quite normal, isn't it?

   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Tuesday, 10-Dec-2024 11:08:55 JST Rich Felker

     in reply to

     • Julian Andres Klode 🏳️‍🌈

     @juliank @kiwix No, there needs to be a large window between disabling of account and deletion of data.

   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Tuesday, 10-Dec-2024 19:39:24 JST Rich Felker

     in reply to

     • Julian Andres Klode 🏳️‍🌈

     @juliank @kiwix Um, no. They can disable access to the hosted server without wiping it.

   • Embed this notice
     Julian Andres Klode 🏳️‍🌈 (juliank@mastodon.social)'s status on Tuesday, 10-Dec-2024 19:39:25 JST Julian Andres Klode 🏳️‍🌈

     in reply to

     • Rich Felker

     @dalias @kiwix The Digital Services Act, article 6 requires a hoster in the EU (who like social networks are classified as a information society service):
     "upon obtaining such knowledge or awareness, acts expeditiously to remove or to disable access to the illegal content."

     In the case of a server where they don't control individual content, removing or disabling access to that content exclusively is not possible, so removing all data is the only thing they can do.

   • Embed this notice
     Lenz Grimmer (lenzgr@mastodon.social)'s status on Tuesday, 10-Dec-2024 22:00:37 JST Lenz Grimmer

     in reply to

     • Aral Balkan

     @aral @kiwix I'm afraid I have no insight into this incident. Our social media team posted a statement on Reddit here: https://www.reddit.com/r/hetzner/comments/1ha5qgk/comment/m1c3n7w/

   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Tuesday, 10-Dec-2024 22:53:04 JST Rich Felker

     in reply to

     • Julian Andres Klode 🏳️‍🌈

     @juliank @kiwix Sure there is. It admits fixing a mistake without a huge PR disaster.

   • Embed this notice
     Julian Andres Klode 🏳️‍🌈 (juliank@mastodon.social)'s status on Tuesday, 10-Dec-2024 22:53:05 JST Julian Andres Klode 🏳️‍🌈

     in reply to

     • Rich Felker

     @dalias @kiwix Yes but there's little point in denying kiwix access to their server and keeping it using resources and continue to bill them?

   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Tuesday, 10-Dec-2024 22:56:07 JST Rich Felker

     in reply to

     • Julian Andres Klode 🏳️‍🌈

     @juliank @kiwix They can act by turning off the server or disabling the routes to it. Not deleting it.

   • Embed this notice
     Julian Andres Klode 🏳️‍🌈 (juliank@mastodon.social)'s status on Tuesday, 10-Dec-2024 22:56:08 JST Julian Andres Klode 🏳️‍🌈

     in reply to

     • Rich Felker

     @dalias @kiwix I think this is where it gets silly because the regulation isn't written for three parties.

     To give an example, f you host a social network and have the issue, you delete the users content.

     But now if you don't host the social network yourself, copyright owners can just complain to your hoster, and then ask them to delete your social network.

     The provider has the option of forwarding this to you as their customer but oh well if you don't respond they need too act themselves

   • Embed this notice
     Julian Andres Klode 🏳️‍🌈 (juliank@mastodon.social)'s status on Tuesday, 10-Dec-2024 22:57:53 JST Julian Andres Klode 🏳️‍🌈

     in reply to

     • Rich Felker

     @dalias @kiwix Hetzner locked me out of my account and denies me access to my data while continuing to bill me for it isn't much better than just deleting it tbh.

     Like once the action is taken either way, there's no way to get the data back because if you did get your data back it would not be inaccessible

   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Tuesday, 10-Dec-2024 22:57:53 JST Rich Felker

     in reply to

     • Julian Andres Klode 🏳️‍🌈

     @juliank @kiwix Sure it is. You contact them, figure out what's up and if you can remedy it, get customer service to let you in to take your things and move out, or get back up and running if it was in error.

     Same principle as evicting landlord having to let you get your stuff, not being entitled to throw it away.

   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Tuesday, 10-Dec-2024 23:06:48 JST Rich Felker

     in reply to

     • Julian Andres Klode 🏳️‍🌈

     @juliank @kiwix "Block access" is about third party/public access to the data, not tenant's access to their own data.

   • Embed this notice
     Julian Andres Klode 🏳️‍🌈 (juliank@mastodon.social)'s status on Tuesday, 10-Dec-2024 23:06:49 JST Julian Andres Klode 🏳️‍🌈

     in reply to

     • Rich Felker

     @dalias @kiwix you can't let them in and move data out because you are required by law to not let them have access to the data, again, as stated, it must be deleted or rendered inaccessible.

     If you let them in and move it out you have just made it accessible again.

     You don't upload a pirated movie to a hosting site, and then get the right to download it again after it gets flagged.

     Here Hetzner is the hosting site, and the server disk contains the pirated content.

   • Embed this notice
     pixelschubsi (pixelschubsi@troet.cafe)'s status on Tuesday, 10-Dec-2024 23:24:07 JST pixelschubsi

     in reply to

     • Jan Wildeboer 😷:krulorange:

     @jwildeboer That's not how this works. @kiwix is registered as a Swiss company and thus is not a data subject in the sense of the GDPR (only natural persons can be data subjects).

   • Embed this notice
     Julian Andres Klode 🏳️‍🌈 (juliank@mastodon.social)'s status on Tuesday, 10-Dec-2024 23:28:20 JST Julian Andres Klode 🏳️‍🌈

     in reply to

     • Rich Felker

     @dalias @kiwix It's funny but it is what it is. Go complain to Brussels.

     But if you think about it you create a file sharing association and host a server, it's never public, you are always sharing the data amongst yourselves.

     The DSA still applies. Not to you as the file sharing service provider but to the hoster hosting you.

     I don't see how to make sure you can get your own content back without introducing a whole bunch of loopholes like that tbh

   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Tuesday, 10-Dec-2024 23:28:20 JST Rich Felker

     in reply to

     • Julian Andres Klode 🏳️‍🌈

     @juliank @kiwix No hosting provider is going to get in legal trouble for preserving the disk pending contact with a representative of the tenant and allowing nothing but a disk image transfer via control panel or similar if the site isn't reinstated.

     We're not talking about running a private warez server behind login on a live server with routed IP. Just single "moving out data".

     But the more important part is possibility to reinstate in the event of human error. This doesn't require any exfil

   • Embed this notice
     Julian Andres Klode 🏳️‍🌈 (juliank@mastodon.social)'s status on Tuesday, 10-Dec-2024 23:28:21 JST Julian Andres Klode 🏳️‍🌈

     in reply to

     • Rich Felker

     @dalias @kiwix No it's not, that's the crux of the issue.

   • Embed this notice
     cjd (cjd@pkteerium.xyz)'s status on Wednesday, 11-Dec-2024 02:35:14 JST cjd

     in reply to
     Guys.
     
     Hetzner
     Is
     Not
     A
     Reliable
     Company
     
     They've been kicking people off their service "because feelz" for YEARS.
   • Embed this notice
     Aral Balkan (aral@mastodon.ar.al)'s status on Wednesday, 11-Dec-2024 02:54:05 JST Aral Balkan

     in reply to

     • Lenz Grimmer

     @lenzgr @kiwix Thanks, Lenz.

     Kari'boka likes this.
   • Embed this notice
     🔗 David Sommerseth (dazo@infosec.exchange)'s status on Wednesday, 11-Dec-2024 04:24:22 JST 🔗 David Sommerseth

     in reply to

     @kiwix This is not the first time I've heard this about Hetzner. And I have a cruel experience myself.

     Many years ago I tried signing up with them, but after a few days waiting I was requested to send copies of identity confirmation and the credit card used. Being inexperienced with such services back then, I thought this was common and did so. After yet another few days my sign-up was rejected without any reason. I was going to use the setup for professional hosting, so I tried a bit to understand why but the communication went dead.

     I found other alternatives back then within the same price range and got started within an hour, with no issues at all.

     I am located in Europe and the alternative provider I went with was also a German company. That's when I decided to consider Hetnzer a scam company.

     They might feel they're too big to fail. But as these cases grows and the communities gets aware of it ... Hetzner is eventually entering into a never ending downwards spiral.

     Kari'boka likes this.