I could not resist noting this today:
Microsoft might need to get the basics right before bragging about them.
Thanks for reading this post.
You can share this post on social media of your choice by clicking these icons:
You can subscribe to this blog's daily email here.
And if you would like to support this blog you can, here:
It was nothing to do with Microsoft. It was due to a pushed update from CrowdStrike which obviously hadn’t been tested before being rolled out. It just happened that it brought down machines running Windows. It could just have easily been Mac OS or Linux.
But it didn’t
So it did have a lot to do with Microsoft
Hi Richard,
Sorry to be pedantic, but it really did not have anything to do with Windows. Other companies that used Windows but not Crowdstrike, were unaffected. That Crowdstrike was used by so many influential businesses, is the problem, along with, as others have commented, it is an outsourcing issue, rather than the OS specifically.
Regards
But in that case my point still stands
Microsoft whould have warned them of the risk of Crowdstrike and did not
Or it should have provided a better alternative
Of course it did.
Thank you and well said, Richard.
Bit busy at work today, but I will try to comment later as I worked with the Bank of England on such matters in 2021 and have done a bit this year.
With regard to Microsoft and Gates, this is how Gates buys silence: https://gript.ie/bill-gates-bankrolled-select-media-outlets-to-the-tune-of-319-million-including-the-uks-guardian-and-the-bbc/. The Gates Foundation*, a “tax efficient” investment vehicle pretending to be a charity, has placed many employees in media, government and academia. Hacks and other professionals want to work with or for Gates. US foundations are required to contribute 3% to charity annually.
Richard,
You are correct! To put this in “construction” terms. Microsoft is the General Contractor and Crowdstrike is the Sub-contractor. A general Contractor is responsible for the work of the Sub-contractor they engage.
If a plumbing contractor installs four perfect bathrooms and one sub-standard bathroom for a general contractor, the sub-contractor (plumber) is at fault but the responsibility is on the General Contractor.
Do not forget that the Crowdstrike update was a kernel module (thus powerful enough to do anything it liked to any computer on which it was installed) which was ‘signed’ as legitimate **by Microsoft**. With malicious intent the outage could have been far worse, and apparently Microsoft has no system in place to prevent that.
You have answered the question I asked about the relationship between the two companies. Clearly Micro$oft must share some of the blame.
There is a great deal of pedantry, special pleading and repressed legalism here. This is a consequence of the interconnectedness of complex, insecure systems; as a consequences of the innovation in Big Tech that demonstrates principally the innovators do not know what they are doing because there is insufficient restraint, built-in redundancy or testing (all of which is inevitable, because it is endlessly repeated – it isn’t new, the disasters simply become become bigger, and the disorder greater) What I see here is a defence built out of complacency. The law runs far, far behind innovation; and that will take decades to resolve – too late for everybody.
Adam Ferguson explained the problem in 1767. We never, ever learn: “EVERY step and every movement of the mul∣titude, even in what are termed enlightened ages, are made with equal blindness to the future; and nations stumble upon establishments, which are indeed the result of human action, but not the execution of any human design. If Cromwell said, That a man never mounts higher, than when he knows not whither he is going; it may with more reason be affirmed of communities, that they admit of the greatest revolutions where no change is intended, and that the most refined politicians do not always know whither they are leading the state by their projects”.
In the 21st century, replace the word “state” or “nation” or “community” in Ferguson, with “business” or “digital revolution” and you will realise that States now bend to technology, and are their servants, but we remain in exactly the same predicament as Ferguson observed, and gifted us with a prophetic insight into our present and future.
CrowdStrike does have versions for other operating system, and these have been causing problems too:
See: “CrowdStrike broke Debian and Rocky Linux months ago, but no one noticed”
https://www.neowin.net/news/crowdstrike-broke-debian-and-rocky-linux-months-ago-but-no-one-noticed/
My attitude would be that if you are foolish enough install proprietary software on a Debian system then you deserve anything that happens as a consequence. Asking around a bit, I found out that CrowdStrike messes with the Linux kernel. Software that is allowed to do that is absolutely verboten as far as I am concerned. Indeed the standard security tools one would run on a Debian system go to great efforts to ensure that such things cannot happen by accident.
I suppose the real question is what is the relationship between Micro$oft and CrowdStrike? Micro$oft obviously can’t physically stop users installing software on their operating systems. But do they actually recommended CrowdStrike? If not, I can’t see how they can share any of the blame for what happened.
Actually it did affect some Linux systems: https://www.theregister.com/2024/07/21/crowdstrike_linux_crashes_restoration_tools/
Linux even has a Blue Screen of Death now thanks to systemd…
“Microsoft whould have warned them of the risk of Crowdstrike and did not”
It would have come as a surprise to Microsoft as well. Crowdstrike is the name of a company that provides cybersecurity software. It isn’t part of MS. The fact that they didn’t do a controlled roll-out of the update, which would have highlighted issues sooner rather than later, is puzzling.
MS can’t be held responsible for third party’s incompetence.
I never thought, in over 40 years in the IT business, that I would be defending MS!
But Microsofty says it can do Cybersecurity
My whole point is that it obviously could not
What did I get wrong?
I see your point.
I also wasn’t aware that this was an update to the kernel which does raise a lot of questions as to why it wasn’t tested properly and rolled out in a phased manner.
I hope Crowdstrike have good insurance cover.
The Crowd Strike software update affected the kernel – a highly sensitive and protected part of the Windows operating system. Normal third party “user mode” software updates cannot access this part of the operating system unless system administrators in the companies installing Crowd Strike explicitly override the built in safety controls by giving the software “kernel level” access – which is admittedly common practice for security software. So like most disasters this comes down to insufficient risk management and change control on the part of companies sourcing Crowd Strike – with benefit of hindsight system admins should have evaluated the huge impact a faulty update could cause and implemented mitigations such as staggered update schedule or redundant systems that could be safely booted. Easy to say after the event. I personally do not think managing this risk was really MS responsibility once the system admins handed Crowd Strike kernel super powers. In my experience, after twenty years in the biz, I believe the software / IT industry is still professionally immature and we do not have the frameworks and institutions of other professions needed to manage the very significant risk of IT properly – let alone the emerging risks of AI!
Thanks
But it still says, MS did nit properly assess the risks to cybersecurity, and that is all I suggested.
“But Microsofty says it can do Cybersecurity”
Well, they would say that wouldn’t they?
It is partly Microsofts fault. The reason is that for Crowdstrike to work, it has to have privileged access to the underlying operating system (Windows), in order to monitor for anomalies. Obviously Microsoft doesn’t want to allow just anyone to distribute software with such access, so they cryptographically sign any such kernel extensions – in other words they effectively warrant that the 3rd party software is safe to install in Windows.
Microsoft should do enough testing internally on any such 3rd party kernel extension, rather than relying just on the 3rd party who obviously have commercial incentives to say everything is okay. It looks like the update was just a data file that was not handled by the existing code safely, causing the computers to crash with BSOD. I recall Apple phones had a similar problem previously with crafted text messages that contained data that hadn’t been tested for. It is easy to miss this stuff.
Still most of the blame must be on Crowdstrike who should never release updates without appropriate testing. I have heard suggestion it was a logo graphic or font change, so they didn’t feel the need to go through a full test cycle. That may not be the actual case, but it is clear that Crowdstrike have missed a test case (as well as Microsoft).
Thanks
I agree with what others have said, that crowdstrike is the root cause of the outage. But Microsoft does share some blame:
* That their product is so lacking that companies feel the need to install third party security and management software in the first place
* That their product is vulnerable enough that a single dodgy driver can collapse the entire system
But I would argue there is another entity that takes some of the blame: the market. We essentially have 2 major operating systems in commercial use. (I’m not counting iOS, MacOS, android, or chromeos, because practically no-one uses those in customer-facing kiosks or behind-the-scenes cloud systems). We have linux and windows. So whenever one of those faces a fault, half of the world’s systems go dark.
I’m not really sure what the solution to that is. My personal and political preference would be to see greater investment in Linux and other open source projects. If we’re going to have a duopoly, we might as well invest and tighten up security in the choice that we can actually use and modify freely, rather than continuing to support a company that wouldn’t exist were it not for its own intertia.