- IT security issues are being exacerbated by unregulated auto-update mechanisms.
- Systemic and fundamental change to a centralized, approval-based update system is necessary.
A simple rule of thumb for complex systems is that wherever simplicity is added, there is corresponding complexity added elsewhere. For instance, in early PC computing, only software updates were required when the latest version was bought. Bug fixes sent to existing users were exceedingly rare, as they required physical media. With the advent of the Internet, physical media was gradually shunted to the side as bandwidth increased. Bug fixes were suddenly available to anyone who wanted to download and install them. Then came auto updating. Software began to reach out on its own to check to see if it were up to date and, if not, updated itself. Bugs were eliminated and security enhanced. In turn, this enabled rapid iteration software development and the so-called ‘fail fast’ mentality for startups and app developers. After all, if the app was flawed, a patch would simply be applied as fast as the developer could make it.
Now we are seeing the corresponding complexity rearing its ugly head. Automatic software update functions are being exploited to inject malware, particularly ransomware. A great example of this would be the Kaseya incident that compromised over a thousand companies worldwide. The SolarWinds hack of 2020 was also spread via the auto-update mechanism, although the initial vulnerability was not in the update mechanism itself.
Modern IT departments are made of tens of thousands of separate pieces of hardware and software. Every vendor, and sometimes every sub-vendor, has their own update mechanism. IT can and does check updates before applying on critical systems… but many updates are still automatic, particularly on systems that are installed and operated by managed service providers. It is nearly impossible for an IT department to keep tabs on all of the updates and changes made by third parties.
For some, the answer would be to swing the door shut, not take any updates or only do those updates on a timed basis (i.e., weekly, monthly, quarterly, yearly). But these updates are not trivial, and the vast majority of them are not the result of ‘fail fast’ development mentalities. These updates provide real bug fixes and, most importantly, patches to security vulnerabilities. With the pace of zero-day vulnerabilities coming out, it is not reasonable to forgo patching even if considered only from a security standpoint.
Customers and the overall IT industry need to create a standardized update mechanism, or at least a standardized mechanism that allows customers to approve every update before it happens. This would be a huge change for vendors and more work for IT practitioners. This methodology could be substantially hardened, peer reviewed, open-sourced, and most important of all, standardized. The situation cannot continue with literally thousands of ways to update software and firmware, with each company running its own uniquely vulnerable system for accomplishing the task. This methodology has to encompass managed service providers as well, even so-called ‘white glove’ services. IT has been outsourcing to the cloud, MSPs, and other places for some time now, but in the light of the growing security issues deriving from mass software update distribution, IT needs a modicum of control back. A central update service would be a strong first step. There are a hundred caveats and complications to this approach, but before addressing them, the enterprise IT industry must admit that this is a problem that needs to be addressed.
For many, this proposal will be akin to tilting at windmills. Vendors are notoriously independent, MSPs want to provide hands-off service, and the IT department does not want another dashboard to monitor. But this problem is going to get worse unless the systemic, fundamental weaknesses in how software updates work today are addressed. The conversation needs to begin now.