Understanding what OT vulnerability management is isn’t difficult. Practicing it is what is difficult. So, let’s get the easy part out of the way!
What is a vulnerability?
A vulnerability is a system weakness that, if exploited, will result in a breach of system integrity, which could then lead to system malfunction. We are dealing with three types of vulnerabilities:
- Bugs that are documented as CVEs — I’ll get to that in a second;
- configuration vulnerabilities, such as default passwords, and
- technology that is insecure by design. Don’t worry, I’ll get back to that later.
For now, just consider that vulnerability management is oftentimes and for no good reason limited to dealing with CVEs. Let’s look at them first.
Common Vulnerabilities and Exposures
CVE is an acronym for “Common Vulnerabilities and Exposures”. It refers to a repository maintained by the US government where known vulnerabilities in software and hardware products are listed. This whole effort makes up the National Vulnerability Database, which you can access at nvd.nist.gov. You don’t need to worry that the National Vulnerability Database would be incomplete as far as OT products are concerned. Every vulnerability worth worrying about will end up in the NVD. Check it out, for example, by simply doing a search for “Rockwell”.
Note that each CVE affects a generic product, usually in a specific version. So if you match your installed products, be they operating systems or firmware versions, with the National Vulnerability Database, you know which vulnerabilities affect your installed base.
The remedy for pretty much all CVEs is to install a security patch, or a new firmware version. In rare cases, you may even have to replace hardware.
Security patches make things a little bit more complicated for Microsoft products because you may run a vulnerable OS version, but several CVEs have already been “patched”. Therefore, your computer may no longer be vulnerable for those CVEs. In order to figure out if that is the case, it must be determined which patches you have installed, and which CVEs are mitigated by those patches. The mitigated CVEs are then something that you no longer need to worry about. For practical reasons, this process is almost impossible to execute for a human being, but is a standard function of an asset management system. The asset management system does the NVD downloads, the CVE matching and patch info processing automatically.
Insecure by design technology
To a large extent, commonplace technologies used in OT are insecure by design. All the widely used OT network protocols, such as Modbus, Profinet, Ethernet/IP, don’t support authentication and authorization. And that means that once you are on the network, you can mess with the OT devices on that network pretty much all you want. The savvy hacker’s method of choice then becomes reading the product manual, rather than trying to identify exploitable bugs in the firmware.
This insight is of practical relevance. It means that it’s often a waste of time to fix CVEs in the firmware or OS of automation devices. A good example is embedded web servers in automation products, of which there are plenty. These web servers are notorious for buffer overflows and related crap which can be used for infiltration. However fixing those would be pretty much pointless.
The bottom line is: Before you devise your vulnerability management strategy, consider that the biggest vulnerabilities do not show up as CVEs. CVEs only cover bugs that can be patched, or neutralized with a new firmware version.
Prioritization
Sorry to break it to you, but there are so many vulnerabilities in OT that you will never be able to fix them all. Prioritization helps you to focus your limited resources to those vulnerabilities that are most important.
An average Windows computer in OT has well over thousand vulnerabilities. How come? Because these boxes are only patched anecdotally. For the vast majority of Windows installations in OT, there is no automatic patching, and for a good reason.
Setting aside that computers in OT often run 24/7 and cannot be rebooted when IT wants to, a security patch is a configuration change. And your OT application isn’t necessarily happy with that change. The vendor in question might not be happy with that change either and told you right away that your warranty will be void if you install unapproved patches.
Therefore, installing patches, updating operating systems, and updating firmware is very costly in OT. Patches and new software and firmware must be tested for compatibility before deployment. The installation is usually only possible during planned outage. And installation may require proximity to the affected asset, which could involve travel.
So in reality, patching in OT is a completely different game than in IT. It’s one of the most costly mitigation methods.
CVE severity and exploitability
So which vulnerabilities should you patch? The commonly used practice is to use the severity rating that is provided by the National Vulnerability Database. CVEs are ranked as critical, severe, medium, and low. Most organizations only tackle the “critical” vulnerabilities and ignore the rest. This is not the most rational approach in OT as you will be facing a lot of older vulnerabilities that are specified using CVSS versions lower than 3, where the “critical” rating wasn’t used yet. In other words, you can find some pretty nasty stuff that is rated “high” rather than “critical”. Hence, you may consider to focus on the base score instead on severity.
Another parameter that could be used for singling out the vulnerabilities worth patching is if there is already an active exploit for the CVE. For this information there is no official government database, but there are several private brokers of exploitability information, some even free of charge.
Or you could go all the way and throw in threat intelligence for an informed decision about what active hacker groups are currently exploiting. However, I don’t believe this is a particularly promising strategy as threat actor behavior can change much quicker than you will be able to respond. Hackers can employ new exploits over night, but you may need a year or more to fix a particular vulnerability across your installed base.
Asset contextualization
A different strategy for filtering your actionable CVE result set is to factor in asset context — since not every asset is equal when it comes to cyber risk. Some assets are more exposed than others and thus more prone to compromise.
Network exposure can easily be identified if you have a modern asset management system installed. You can either identify an asset’s exposed position in the network topology, or you can simply check data flow. In brief, assets that are hidden deeply behind DMZs and firewalls are less of a concern than those at the front lines.
As a caveat, don’t forget to consider assets that are exposed to nomad laptops, such as portable engineering stations brought in by contractors, and attached to assumedly isolated process networks.
Another way to prioritize by context is based on asset criticality, which implicates higher cost of consequence in case of successful breach. Unfortunately, criticality cannot be measured automatically but has to be assigned by experts who are familiar with an asset’s specific process function. Usually this is in the hands of control engineers.
A simple example is safety systems. A safety system has a higher criticality than a basic production system and should therefore be better protected. Having that said, in real life you would not necessarily focus on a safety controller, which is insecure by design anyway, but on the safety engineering station.
The big picture
A question that we frequently hear from people interested in our OT-BASE asset management system is: Will I get alerts on new vulnerabilities in real time? If anything, this indicates a substantially wrong idea of how OT vulnerability management actually works. Yes, you will see new CVEs as they come in. But you won’t be able to focus on those because of your huge backlog of vulnerabilities.
Here’s a simple question. What’s more important to fix: The twelve new critical CVEs that just dropped in, or the seventy five thousand vulnerabilities that you still didn’t manage to mitigate? If you think this is a trick question, you are mistaken. It’s OT vulnerability management reality. The highest priority is not today’s critical vulnerability, but the hundreds or thousands that you haven’t been able to fix for years.
You will find yourself in a tight spot between a huge backlog of vulnerabilities plus a steady stream of new vulnerabilities coming in every week. And you will soon realize that patching today’s critical vulnerability doesn’t make a difference. It doesn’t move the needle.
OT vulnerability management strategy
So what does move the needle then?
The fundamental objective of OT vulnerability management is to build up a cyber security capability that allows you to reduce your backlog of vulnerabilities while mitigating new CVEs quicker than they flow in.
This requires a strategic approach, and you shouldn’t expect dramatic success within weeks or months. OT vulnerability management is a long game, and the name of the game is incremental improvement.
On the tactical side, consider that patching is often not the first choice for mitigating CVEs. Consider network security, application whitelisting, and system hardening instead. Standardizing configurations for typical equipment such as HMI stations or network switches will go a long way toward that goal, and will also make auditing so much easier.
The biggest gains are to be made with a proactive approach. If you can get your vendors, contractors, and OEMs to deliver hardened and well protected machines, plant components, and software applications at the time of commissioning, you are on a trajectory to success. Even though that success is gradual and slow, it is real — unlike the anecdotal patching activities that aren’t just a waste of time but also create a false sense of security.
So, as I said in the beginning: Understanding OT vulnerability management isn’t difficult, but practicing it requires determination and patience.
Go for it!