Kevin Beaumont
If you want to know something crazy:
- This year TCS migrated their EDR to CrowdStrike
- Then they announced a strategic partnership with CrowdStrike
- Then they lost all their systems
- They’re just finishing recovery today, 6 days in
- Then they got a $10 Uber Eats voucher
- …which got cancelled due to Uber flagging CrowdStrike’s account as fraudulent
Richard
@GossiTheDog oh my god the $10 gift card is real? I honestly thought that was just a joke floating around. I feel like a $10 “sorry” is worse than nothing at all.
Kevin Burns
@GossiTheDog wait the Uber eats thing wasn’t a joke?
Kevin Beaumont
CrowdStrike are… having a week.
Kevin Beaumont
Questions for your EDR providers (do not assume they are experts in availability):
- What are your different update processes?
- How do you test them?
- Do you dogfood test them?
- Do you roll them out in waves? What are the details, eg what percentages and when?
- Do you monitor failures and roll back?
Kevin Beaumont
CrowdStrike staff members are selling CrowdStrike monopoly sets they were given on eBay.
Kevin Beaumont
CrowdStrike filed at 8-K with the SEC on July 22nd for a cybersecurity incident. https://www.board-cybersecurity.com/incidents/tracker/20240722-crowdstrike-holdings-inc-cybersecurity-incident/
Kevin Beaumont
Almost a week in, CrowdStrike say 97% of devices are back online. https://www.axios.com/2024/07/25/crowdstrike-97-percent-systems-online
Kevin Beaumont
Microsoft are talking about changes to Windows after the CrowdStrike incident. Good.
https://www.theverge.com/2024/7/26/24206719/microsoft-windows-changes-crowdstrike-kernel-driver
Chris Adams
@SecureWaffle @GossiTheDog They _must_ do that if they want to sell Windows in Europe, where they have a requirement that third-parties get the same access as their own security products do.
This is healthy in a way: if the kernel & Defender teams work out what it'd take for the latter to run in user space, I'd bet that MVP API would cover most other users with minimal additions since they all care about the same kinds of activities.
SecureWaffle🧇
@GossiTheDog
I wonder if Microsoft will move their own Defender/ATP out of kernel. Will they next remove their defender/ATP from needing to run in the Linux kernel?
Kevin Beaumont
There’s a really good discussion on @riskybusiness’s YouTube show about the CrowdStrike incident.
About the 3 minute mark @alex made me realise I was far too kind to CrowdStrike. He rightly rips them apart.
Kevin Beaumont
Delta are looking to sue CrowdStrike and Microsoft. HT @hrbrmstr
Kevin Beaumont
Re the Delta case - the lawyer they’ve hired successfully sued Microsoft previously on behalf of the US government, and the decision was upheld on appeal too. The ruling almost lead to the breaking up of Microsoft.
The following US government backed out of the case.
Bill Gates said at the time the lawyer was “out to destroy Microsoft”.
So there’s a chance here the CrowdStrike incident may end up having implications across vendor industry around warranties etc, we’ll see.
Kevin Beaumont
Jim Cramer does it again.
Norbert Kowallik
@GossiTheDog honestly, that’s what I expected. I‘m really looking forward to the first meeting with our crowdstrike technical advisors after the vacation. And I’m wondering what our management will decide about our future with crowdstrike, my guess is… we‘ll keep them and try to negotiate something regarding license renewal.
Jason Haar :laserkiwi:
@GossiTheDog @nkow14 migrate to what? They are pretty good...
Maxi 10x 💉
@GossiTheDog The PC emulator for iOS?!
Kevin Beaumont
Replacing an XDR platform at scale takes some time, so if you’re wondering what the translation of Elon’s tweet about Crowdstrike is:
Elon: can we replace Crowdstrike?
Somebody: yes, we’ll begin looking into it but..
Elon: job done
Kevin Beaumont
Delta’s CEO has confirmed they plan to take legal action against CrowdStrike after incurring a $500m loss https://www.ft.com/content/dba1cb7a-46b1-4f94-b596-432e7d899f8d
Kevin Beaumont
CrowdStrike shareholders are suing CrowdStrike https://www.bbc.com/news/articles/cy08ljxndr4o
Kevin Beaumont
CrowdStrike made a net loss of $845m between 2018 until this year.
Kevin Beaumont
Spirit Airlines in the US anticipates a $7.2 million hit to its third-quarter operating income due to operational disruptions caused by the CrowdStrike incident, which forced the carrier to cancel 470 flights.
Kevin Beaumont
Here's the Delta boss on his thoughts about the CrowdStrike incident.
They had 40k Windows Server boxes alone, all with BitLocker full disk encryption enabled, all of which wouldn't boot and weren't fixable without manually unlocking BitLocker. That had gone all in with CrowdStrike + Microsoft's most premium offerings.
He has a really good point about how tech companies have become obsessed with growth as their only metric of success, and customer satisfaction is not on the radar.
Kevin Beaumont
There's a really mad moment in that interview where they ask them what assistance CrowdStrike have offered, and he essentially says nothing, not even a lunch voucher.
What a time to be alive.
Glen Arrowsmith
@GossiTheDog 40k Window servers??? That doesn't make sense! 4000 would be hard to believe. Why do they need so many?
Justin Scholz
@GossiTheDog “free consulting” 🤣
Glen Arrowsmith
@GossiTheDog omg. Were they all sitting on <5% CPU utilization?
System Adminihater
@GossiTheDog Its true. Tbey dont give a fuck whether anything works or not.
Kevin Beaumont
CrowdStrike’s website then vs now
Kevin Beaumont
CrowdStrike complained to Cloudflare about a CrowdStrike parody site… and Cloudflare took it down. Without a court order. https://clownstrike.lol/crowdmad/
Kevin Beaumont
Additionally to loop this in, CrowdStrike submitted a takedown for a parody label (they’ve since rescinded it after being called out).
Ryan Castellucci :nonbinary_flag:
@GossiTheDog FrEeZe PeAcH
Interpipes 💙
@GossiTheDog does 40k servers pass the sniff test or do they mean 40k devices?
Because unless their servers should have had LOM, surely, and they only have ~100k staff total, many of whom would use shared endpoints (or not use one at all), so I can believe 40k endpoints.
Even the most egregious example of unintegrated acquisitions I know of was only at around 10k servers (estimated!) for 50k employees, and Delta doesn’t really have a similar acqui-problem, do they?
Kevin Beaumont
We’ve reached the part of the brand cycle where people are using CrowdStrike as an excuse https://www.theverge.com/2024/8/2/24212298/mrbeast-beast-games-crowdstrike
Kevin Beaumont
360 takes a look at the Crowdstrike kernel drivers - finds they implement an eBPF like system, contain a wide attack surface, don’t check validity of update files (eg no signing of updates) and claim they contain conditions for LPE and RCE vulnerabilities. https://mp.weixin.qq.com/s/uD7mhzyRSX1dTW-TMg4UhQ
Graham Sutherland / Polynomial
@GossiTheDog also just writing it off as "the Chinese" is straight up fucking racism, there's literally 1.4bn people living there and treating them like a homogenous unit is fully bullshit
Kevin Beaumont
Previously on Crowdstrike Falcon vulnerability research, check out this timeline where they tried to use NDAs to avoid disclosure, then fixed it without telling anybody. https://modzero.com/modlog/archives/2022/08/22/ridiculous_vulnerability_disclosure_process_with_crowdstrike_falcon_sensor/index.html
Kevin Beaumont
EFF are calling for antitrust action after the CrowdStrike incident https://www.eff.org/deeplinks/2024/07/crowdstrike-antitrust-and-digital-monoculture
Kevin Beaumont
Bloomberg report a vast majority of the CrowdStrike losses reported by customers will be judged by insurance as not covered by policies. https://www.bloomberg.com/news/articles/2024-08-02/billions-in-damages-from-crowdstrike-outage-to-go-uninsured
Kevin Beaumont
CrowdStrike are publicly threatening their customer, Delta. https://www.theverge.com/2024/8/5/24213521/crowdstrike-refutes-blame-delta-outage-litigation
Kevin Beaumont
I've written up a bit about CrowdStrike's latest bold strategy.
Kevin Beaumont
Microsoft have now queued up to try publicly throw their customer under the bus, claiming (without evidence) Delta’s CrowdStrike woes were due to non-Windows systems. The CrowdStrike issue only impacted Windows systems so I hope somebody at Microsoft knows what they are doing.
https://www.theverge.com/2024/8/6/24214371/microsoft-delta-letter-crowdstrike-response-comments
Kevin Beaumont
If anybody wants the subtext of what is happening here, CrowdStrike and Microsoft both really do not want to get sued by Delta and have it go to court as it would potentially be explosive for both orgs and the wider security industry.
The customers are always plebs to be milked, as is status quo.
Alex
@GossiTheDog i hope they get the book thrown at them.
loucovey
@GossiTheDog I was involved in another issue where an organization wanted to sue a couple of people for fraud. Their lawyers told them, "Well, the problem here is that your bookkeeping practices and approvals showed you knew exactly what was happening, so you would be held liable as well."
That' is what would happen with Delta if they sued. Theyr own negligence contributed to the scale of the problem, after others had resolved it.
Guelfo Alexander Ghibellini
@GossiTheDog Microsoft was forced by law to open that segment of code to third parties. now they'll close it again and CrowdStroke will be resident in the BIOS/UEFI. And if they succeed, they will be resident in the physical Power Button. So they can also sell hardware.
System Adminihater
@GossiTheDog Microsoft killed the Action Pack. That was the last straw. We ride at dawn.
Kevin Beaumont
CrowdStrike incident root cause analysis is out.
Overall, good… but.
It is very verbose but doesn’t say much. Some of the wording will confuse people - eg it talks about rings (waves) in a way which makes you think it is already implemented. It isn’t. They’re saying they plan to implement it later.
Channel updates weren’t tested on a real Windows PC prior to deployment, they relied on automated bespoke code testing. They don’t mention that and it’s the real reason.
Kevin Beaumont
Risky Business take on CrowdStrike root cause report is good.
You can see the confusion the report provides in this discussion I think, eg some of the things are talked about as being implemented - but they’re down as findings for improvement. It’s the way the report is worded, to make you believe certain things existed.. that don’t yet.
Kevin Beaumont
Really good piece about CrowdStrike (technically CSC) misusing DMCA takedown notices over trademark disputes.
CrowdStrike probably want to have a word with CSC about this and Cloudflare should tighten process as DMCA isn’t supposed to be used for this. I know CSC do it.. but they shouldn’t be.
Wider point: cyber industry abusing process in takedowns.
Kevin Beaumont
CrowdStrike have responded to two claimed vulnerabilities in CrowdStrike Falcon, including one made by a former staff member: https://www.crowdstrike.com/blog/tech-analysis-addressing-claims-about-falcon-sensor-vulnerability/
There may be more to come on this one..
Kevin Beaumont
CrowdStrike vs Delta vs Microsoft continues to play out in public, now SEC filings
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.