The issues have been linked to a CrowdStrike update.
This is absolutely fuckin ridiculous. Good luck IT teams
I don’t like to post links to reddit, but holy cow, is this ever a shitshow.
Weird, that sub has downvotes disabled it appears.
And what will happen once it’s resolved? Oh, a fine that can be written off? Sure, no problem. Just took out a few countries, governments, agencies and businesses worldwide, but don’t do it again 😉 Come contract renewal, you can increase the prices to pay off the fine and we’ll diligently sign it 😘
Are you implying they should somehow have perfect quality control? Generally the lost customer trust is enough of an incentive to not let things like this happen. Things slip through QA, and the only way to prevent that at least 99.99% of the time is to invest MASSIVE amounts of money that really aren’t justified for everything. Aviation does this, because there is significant risk of death if something goes wrong, so regulations force them to. Other industries arguably should (car manufacturers…), but a random security software? No.
Car manufacturers have ISO 26262 to regulate safety critical software development, whereas aviation is mainly based on RTCA DO178.
The concepts are pretty similar. Details differ.
Nothing is perfect, nothing is absolute, and yes that’s an oxymoron but you get the point. Anyway, there are ways to minimize risk
- A/B testing
- gradual roll out
- monitored roll out
- rollback
And not only on the side of Crowdstrike, there are things that can be done by their customers:
- OS rollback from weekly or monthly snapshots of the boot drive or system drive (probably shouldn’t change that often)
- if that isn’t possible with that OS, use another OS
- automated deployment (again, probably possible to fallback to a last known good deployment)
- investment in sysadmins
- investment in security staff
Probably lots more, but I’m not a sysadmin. I bet you though, that the hospitals, rail, and other governmental institutions simply don’t have enough money to invest in that because of budget cuts and austerity measures. Some hospitals still have Windows XP running.
Companies and governments don’t think IT and security are important until they are. It’s not about creating a perfect system, it’s about creating a system that can bounce back quickly.
Yeah, this absolutely smells like a corporate culture issue, not a one off glitch in QC. Fuckups of this magnitude shouldn’t be possible without multiple failsafes breaking and people ignoring protocol. Not to say that “perfect storm” events don’t ever happen, but it seems like the less-likely possibility to me.
