Kevin Beaumont
Like @Quinnypig, I’d love if Google went on the record about what happened with this one.
https://www.unisuper.com.au/contact-us/outage-update
Tl;dr is an entire org got wiped, and they’re trying to suggest it’s some kind of Google Cloud issue as the cause.
Kevin Beaumont
Google have apparently issued a statement via Unisuper.
Might be worth people following up with Google to see if they wrote this - as far as I know it’s the first case of a major cloud provider screwing up a customer so badly. Unisuper say in their update they’ve been able to restore data from a backup they had at another cloud provider.
Corey Quinn
@GossiTheDog holy crap.
Kevin Beaumont
GCP issue where they accidentally deleted a customer continues - Google Cloud aren’t challenging it. Customer was able to recover as they had backups in another cloud provider.
Kevin Beaumont
Gonna make some Killed By Google: Your Company T-shirts.
Jonathan Yu
@GossiTheDog I don't see anything in this article or anywhere else about deleting data, only a "misconfiguration," though it does stand to reason that the data was deleted since it has taken so long to restore. Did I miss something?
I guess offsite/cross-cloud backups were a saving grace in this case, though I wonder how many companies have this scenario as part of their disaster recovery planning
Kevin Beaumont
One of the absolutely wild, to me, things is learning that cloud providers have fans… who absolutely refuse to believe the cloud provider could do something wrong.
Who the fuck stans a cloud provider? Tech nerds, apparently.
J. B. Crawford
@GossiTheDog name one thing Oracle Cloud has done wrong
Kevin Beaumont
Google Cloud (GCP) CEO has gone on the record now and said directly they accidentally deleted a $125bn business cloud estate in error. https://qz.com/google-cloud-pension-fund-unisuper-1851472990
A config mistake at Google’s end led to their VMs being deleted.
Omar Jarjur
@GossiTheDog That article seems to just be regurgitating the statement from UniSuper. It doesn’t actually include any quotes from Google.
JP
@GossiTheDog @Quinnypig @jpm infrastructure as code is the the. Fastest way to fuck up at scale.
Marius Kießling
@GossiTheDog I already have it on my todo list for our next account meeting. It’s especially interesting because we basically eliminated the option ‘Google screws up so badly that a complete project or organisation suddenly disappears’ from our threat models. If that holds up, I guess we have to reconsider that.
Conor
@GossiTheDog wow, google can't even keep themselves to only killing their own products!
ocdtrekkie
@GossiTheDog The other fun fact is that a bunch of these companies have like 100,000 employees, and they get *really upset* if you point out on HN or the like that they are stanning their employer.
I have been accused of "doxxing" for merely pointing out a guy loudly justifying why Google was the best at everything was a manager on their Ads team.
Daniël Franke 🏳️🌈
This seems relevant: https://berlin.social/@ainmosni/112404348515843128
ocdtrekkie
@ainmosni @GossiTheDog Yep, one of my favorite "shills Google on HN in his free time" guys got laid off by them last year, and I had a good laugh about it.
Daniël Franke 🏳️🌈
The sad thing is, those big corps do try to brainwash their employees. They put so much money and effort in making their employees think they are all a big awesome family, and that being there makes them cooler and better than everyone else.
I used to work for a place that had events like that, and it worked for a while. I snapped out of it eventually, but the cult tactics do work.
ocdtrekkie
@ainmosni @GossiTheDog The free on-site amenities and events are huge. It's not just about keeping you on campus, it's about tying your whole life to the company. A few of the people I have argued with the loudest were "Googler with Googler spouse" which is just... absolute corporate success. The entire family is company-owned.
ocdtrekkie
@ainmosni @GossiTheDog There's nothing inherently wrong, of course, with meeting your spouse at work, but it happens so much more often in these companies because the employee's entire social circle *is* the company.
Daniël Franke 🏳️🌈
The events were such a big thing, especially if you hire a lot of non-locals. You end up with people, who's entire social circle is in the company, which makes leaving or even criticising the company difficult.
Thibault D.
@GossiTheDog I find it impressive. I don't use Google Cloud but I use Azure and can appreciate that it is impossible to get Microsoft's support to acknowledge even the smallest bug or error from their side, even when all evidence points at them without ambiguity.
So seeing Google acknowledge this is big for me.
clacke
@GossiTheDog If the insurance company placed their data with Google Cloud and Google Cloud wiped their data through no fault of the insurance company, that's a Google Cloud issue. The engineers and internal tools at Google Cloud form part of Google Cloud.
If an engineer at Google puts the wrong value in their provisioning tool, or encounters a corner case they weren't aware of in their provisioning tool, and that deletes your 1900 VMs, which seems to have been something like what was happening here, I see nothing wrong with saying that's a Google Cloud issue.
Jonathan Yu
@GossiTheDog Yeah, makes sense. I'm still kinda miffed about the vague "one of a kind issue" and "due to a software bug" language, it's sad that Google is playing it off instead of taking accountability. They've had a number of high profile incidents lately, it seems.
AWS seems more forthcoming in incident postmortems and the like, but maybe that just speaks to their maturity as a cloud vs Azure & Google
clacke
@jawnsy @GossiTheDog I wouldn't read that as something that weakens their culpability. It's to communicate the level of systematic risk to other customers.
"[Did we mess up and delete all your data? Yes. Was it entirely our fault? Yes. Are we liable? Yes. Are we likely to mess up in this exact way next week with someone else's data? No.]"
Kevin Beaumont
Google have issued a blog about the Google Cloud customer they accidentally deleted.
A Google staff member accidentally set a termination date on their account, effectively, which deleted several thousand of their VMs. https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.