The Simple Cyber Governance Program OT vulnerability management process
© 2018 Langner, Inc.
Using the content of this document in production environments, in the process of planning or designing operations technology systems and architectures, in product development or any other commercial activity requires a commercial SCGP license.
Vulnerability Management in SCGP is the process in which known cyber security vulnerabilities in hardware, firmware, and software products are identified, assessed, and eventually mitigated. While many vulnerabilities in the OT domain are more or less intentional product design / technology features and will never get “fixed” by product updates, vulnerabilities that are due to programming errors (“bugs”) are reported by various organizations and may be eliminated by installing security “patches” or other mitigations. This document outlines the procedure that is used within SCGP to deal with vulnerabilities that are considered “bugs”.
The SCGP vulnerability management module is written for cyber security experts who are responsible for identifying and assessing such vulnerabilities, as well as for defining appropriate mitigation strategies.
Vulnerability management is characterized by the fact that it is aiming at a moving target, with activities that are half scheduled, half event driven. The organization will check for new vulnerabilities that affect the installed base in regular intervals, and may sometimes see a need to quickly respond to vulnerabilities using substantial effort. In this regard, vulnerability management shares characteristics with cyber security incident management.
Since it is not practically possible to process vulnerability data manually, vulnerability management requires a mature asset management system. For this reason, it may be advisable for an organization to phase vulnerability management into the later stages of a cyber management program, when asset management is under control.
Reported vulnerabilities as they are the focus of the vulnerability management in SCGP are part of the SCAP, which is an acronym for Security Content Automation Protocol. This protocol was published first in 2010 in an effort to automate vulnerability management for large organizations such as the US government. Part of SCAP are vulnerability definitions (CVE), a vulnerability scoring system (CVSS), and product identifiers (CPE) that allow asset owners to check which vulnerabilities are reported for the products they are using.
Unfortunately, the automation part of SCAP is more wishful thinking than reality. Vulnerability management in respect to CVEs is still more of an art than a science and requires experienced human analysts.
Another reason why automation stops short of good results in vulnerability management is the fact that especially in operations technology, mitigation options are often anything but straightforward. A specific vulnerability is rarely associated with one specific security patch that would mitigate the vulnerability and could be applied just like that. For some vulnerabilities, no security patch is available. For other vulnerabilities where a patch is available, it may not have been approved by all relevant software vendors, or it may have known adverse side effects. For all those reasons, vulnerability management in the OT domain requires a carefully designed workflow that is executed by experienced human experts. This is what this module is all about.
New module in SCGP 18
1.1.1 The NIST CVE database
The US National Institute of Standards maintains a database of known vulnerabilities for public download in machine readable format. This database acts as the global reference for users of digital products.
The URL for downloading CVEs at the time of creating this document is:
1.1.2 When to download CVEs
Data contained in the NIST database is updated constantly and should be downloaded in regular intervals.
The download interval should depend on organization’s capabilities to mitigate the backlog of vulnerabilities that have already been identified and are scheduled for remediation. For example, checking for new vulnerabilities daily may not be the best strategy for an organization that has a backlog of several hundred vulnerabilities waiting for mitigation. For many organizations, monthly vulnerability definition downloads are appropriate.
Nevertheless, there may be a requirement for out-of-band downloads in the case of vulnerabilities of special importance or criticality. Examples would be high-profile vulnerabilities such as Heartbleed, Meltdown, and Spectre.
Procedure: In a regular interval, download vulnerability definitions from the NIST vulnerability database. Adjust this interval based on the organization’s capability to process CVEs.
Procedure: Perform out-of-band vulnerability updates for high-profile vulnerabilities that require prioritized processing.
1.1.3 Loading CVEs into the asset management system
Downloaded vulnerability definitions are then loaded into the vulnerability management system / asset management system.
1.2.1 Set Common Platform Enumerators for installed products
CVEs can be matched with installed products using the Common Platform Enumerator, or CPE. The CPE is intended to uniquely identify a specific product. A CPE also contains the vendor name and version information.
Unfortunately, there is no reliable algorithmic way how to associate CPEs with product and vendor names as they appear on or in the product itself. For this reason, associating installed products with their CPEs cannot be done fully automatically but requires a bit of manual labor. The upside is that once the proper CPE for a given product is set, this process usually doesn’t need to be repeated.
Procedure: For installed hardware and software products, make sure that Common Platform Enumerators (CPEs) are properly set.
In order to make this process most efficient it is suggested to focus on those hardware and software products which are installed most often.
1.2.2 Link installed products with CVEs based on Common Platform Enumerators
Once that CPEs have been set, the asset management / vulnerability management system in use should be able to link CVEs with the installed base.
Matching published vulnerabilities with installed products based on Common Platform Enumerators isn’t bulletproof. For example, some CVEs only affect specific sub-versions or product combinations which are not properly reflected in CPEs. For this reason, a subject matter expert will browse CVEs with CPEs that match the installed base and then flag CVEs that are irrelevant because the specific product combinations don’t completely match installed systems.
Procedure: Check if a specific vulnerability affects the installed base as expressed by CPEs, or if it is irrelevant due to side conditions that aren’t met by installed products. Based on the assessment, flag the CVE as relevant or irrelevant.
Prioritizing CVEs for mitigation is one of the more difficult tasks of vulnerability management, as there are usually much more vulnerabilities than what can actually be remediated with given resources. There will never be a shortage of vulnerabilities, but always a shortage of resources, and those limited resources need to be spent well — not just for remediating CVEs, but also for competing tasks such as network security.
2.2.1 Standard prioritization
NIST CVE definitions provide a multi-dimensional rating of vulnerabilities:
- CVSS vector
- CVSS base score
- severity rating.
Unfortunately this information cannot be used for prioritizing vulnerabilities in an OT environment without putting it into context.
2.2.2 Prioritization based on context
As an example why the standard prioritization can be misleading, potential safety and process/business impacts are not (and cannot) be reflected in the NIST scoring system. Therefore they have to be provided by a subject matter expert working for the asset owner. Based on such context, the SME may and will come to a prioritization that is not aligned with NIST scores.
Procedure: Prioritize known vulnerabilities based on the criticality of the affected devices.
2.2.3 Prioritization based on capacity
In real life, there will never be enough time to remediate CVEs to a satifying degree as so many other cyber security issues need to be addressed as well — often by the same people who are responsible for vulnerability management. For this reason, available resources are another major factor affecting prioritization.
Procedure: Prioritize known vulnerabilities based on the resources available for remediation.
Most published vulnerabilities can be mitigated by installing a security patch from the vendor in question — in theory. Reality is sometimes different:
- There might not exist a security patch for a given vulnerability. While this is rarely the case, it does happen.
- A security patch might not (yet) be approved by vendors of critical applications that are running on the affected system(s).
- “Patching” the affected systems might be not an option because it might interrupt the physical process.
- As a result of factors listed above, alternative mitigation strategies can often be preferred over installing a “patch”, even if one is available. Such an alternative strategy could be, for example, de-installing a vulnerable software application that isn’t required for the purpose of a given system; a simplified case of system hardening.
In other words, security patches are not the silver bullet when it comes to vulnerability remediation in OT environments. Even if a security patch is available, the asset owner may chose to not install it, remediate the vulnerability by other means such as network security controls, or “live” with the vulnerability.
Since the strategy for dealing with a given vulnerability that has been flagged as relevant cannot be determined by reading the vulnerability definition, the responsible SME should specify the recommended mitigation strategy along with the vulnerability. The mitigation strategy should be specified as detailed as possible, for example including links to security patches if the chosen strategy is to install such patches.
Procedure: For vulnerabilities flagged as relevant, define and document a recommended mitigation strategy as detailed as possible.
If the recommended mitigation strategy for a given vulnerability requires a software configuration change, such as installing a security patch, it is a good idea to turn the mitigation into a standard configuration by adding the patch to a configuration baseline. This way it can be assured that devices using a vulnerable product will be associated with the patch in a “standard” way. It also links vulnerability mitigation with compliance.
Creating or updating configuration baselines also makes tracking mitigation progress easier, and helps to assure that new devices using the vulnerable product will be using the patch as well, for example at the stage of Factory Acceptance Test.
Procedure: If the recommended mitigation strategy involves a software configuration change, check if existing configuration baselines should be updated, and/or if new configuration baselines should be defined.
In most cases, vulnerability remediation will involve configuration change. Therefore, it is a good idea to open change cases right away.
Procedure: If the recommended mitigation strategy involves a configuration change, check if a change case for implementing the configuration change can be opened right away, and open one if it can.
The final step for mitigating a known vulnerability is to execute the recommended mitigation and mark the specific vulnerability as fixed.
Procedure: Execute the recommended mitigation for all affected systems and mark the specific vulnerability as fixed for the respective systems.
OT-BASE doesn’t automatically update its CVE definitions, for the simple reason that you wouldn’t want the software to automatically download files from the Internet. However, updating CVE definitions is very simple:
- Go to the NIST web site (https://nvd.nist.gov/vuln/data-feeds#JSON_FEED) and download CVE definitions.
- In OT-BASE, go to Inventory/Import. Enter the CVE definition filename and click on “Import”. You don’t need to unpack ZIP files before import, OT-BASE does that automatically for you.
Imported CVEs may not show in CVE listing if they affect no installed products, based on CPEs. However, if you want to check if your CVEs are actually in OT-BASE, just enter the ID of a CVE that you have imported in the quick search field on the very top of the OT-BASE window, right next to the OT-BASE product name. If your CVEs were imported correctly, the CVE profile of this vulnerability will then pop up.
OT-BASE links CVEs automatically to your installed base, but it can only do so based on the Common Platform Enumerator (CPE). This means that you have to set CPEs for your installed products properly, or you won’t see CVEs. Unfortunately, this requires a bit of manual labor since there is no algorithmic way to reliably determine the CPE for a given product name and version.
In order to set CPEs for your installed products, go to Inventory/Software and select the tab “Products”. Then select the software product for which you want to set a CPE by clicking on the product entry. For your vulnerability management to be effective, you specifically want to set CPEs for products that you have installed many times, such as operating systems and firmware.
After selecting a specific software product, click on “Edit” to open the edit dialog. Click on the tab labeled “CPE”. In the lower part of the dialog box you see the OT-BASE CPE assistant. In the upper line it shows you the product name and version as it is listed in the OT-BASE database. In the lower line it shows you potential matches of any CPEs that it knows about. The CPEs are taken from all the CVEs you have loaded into OT-BASE. You can now drop down the various fields such as vendor and product name to find the best match. In some cases this is very easy, in other cases it requires a bit of experimenting. You can even manually clear the suggestions that OT-BASE provides you in the various fields and use other strings to compose the CPE. Once that you have picked a choice, click on “Generate CPE”, and OT-BASE will transform your inputs to the appropriate CPE format that will be shown on top of the CPE assistant. If you are happy with your choice, click on “Save”.
Repeat the same procedure for hardware products.
Workflows/CVE gets you to the CVE listing. Please note that the listing only contains CVEs that affect one or more of your devices. The list will be empty until you specify CPEs for at least some of your installed products.
The list can be sorted for different criteria by clicking on the column header. The default sort is by number of affected devices. Clicking on the column header of “Base Score”, for example, will sort the list by CVSS base score.
The filter opportunities in the column header allow for narrowing down the output set. For example, if one is only interested in CVEs from 2018, the “CVE ID” filter can be set to “CVE-2018” in order to achieve that. As another example, the CVSS vector field can be filtered to only show vulnerabilities with particular characteristics. In the following example, only vulnerabilities are shown where the access vector involves the network (AV:N).
Yet another way to reduce the output set is to limit the scope of affected devices. This is achieved by expanding the “Scope” dialog in the top region of the window. The dialog allows you to narrow the scope down in respect to location, OT product (DCS), physical process function, network, and lifecycle stage. Note how the statistics on top of the table are updated once that you apply a filter or limit the scope.
4.4.1 Overall assessment
After inspecting the CVE description and maybe also checking the various links that are usually provided as part of a CVE, a human expert needs to decide if the CVE is relevant or not, since this cannot be accomplished automatically. In a nutshell, “relevant” in real life means: This CVE needs to be mitigated. Indicating relevance in OT-BASE is simply achieved by leaving the “Relevance” setting in the CVE edit dialog at “Yes”.
Changing the “Relevance” setting to “No” means that the CVE will no longer be shown in device and product profiles. You can still see “irrelevant” CVEs in the CVE listing, and if you want to focus only on those, just filter the “Relevance” column using “No”.
Since you will rarely be able to mitigate all CVEs that affect your installed base, prioritization is important. Common wisdom suggests to prioritize vulnerabilities according to their CVSS severity rating, i.e. starting with “CRITICAL”, than working your way to “HIGH” and so on. In real life, priorities may be different because other factors such as potential safety impacts and device context play an important role as well.
For this reason, you can express your own CVE priority which is based on your assessment. This is done in the “CVE” tab under “assessment”, where you can specify a priority. By default, the priority is set to the CVSS severity rating.
While the CVSS severity rating is important, it doesn’t necessarily reflect the order in which you intend to mitigate vulnerabilities. Hence, after your individual assessment and prioritization, you just need to sort the vulnerability listing in the “Priority” column in order to display the mitigation order that you have selected. To sort the table, simply click on the column header.
4.4.3 Device-specific assessment
In a microscopic view, relevance of a vulnerability can be set for individual devices. This way you can indicate that while a given CVE may be relevant for some devices, it isn’t necessarily relevant for all devices. Again, “relevant” simply means: “Needs to be mitigated”. An example would be devices in an isolated network zone, or devices which aren’t networked at all, for which you decide to leave the CVE unmitigated.
In order to specify device-specific relevance, click on the tab “Affected Devices”. For each device listed, relevance can be set or cleared. By default, a CVE is relevant for all affected devices.
For devices for which you have declared a specific CVE as irrelevant, the CVE will not be listed in the “Known Vulnerabilities” section of the device profile.
Once that you have defined a mitigation strategy, it can be documented in the respective free-text field for the given CVE. You can use URLs in this field which are automatically hyperlinked by OT-BASE, so that a user who opens the CVE profile only needs to click on the link in order to open the target you are referring to. Hyperlinks can be used to point to vendor-specific details on how to mitigate a vulnerability or to web sites from where a patch can be downloaded.
Another way to help users with mitigation is to attach files to a CVE, which can be done in the tab labeled “Files”. Such files could be extended procedural documentation about the suggested mitigation. Certainly you can also attach security patches directly to a CVE.
Configuration baselines are maintained in the Workflow/Audit menu area. In the listing of existing baselines, select the baseline that needs to be updated in respect to vulnerability management, or create a new configuration baseline. In most cases, the updated baseline definition will include an entry for one or more security patches.
If the recommended mitigation involves a configuration change — which it usually will –, it is a good idea to initiate a change request for the devices selected for mitigation right away. In order to do this, go to Workflow/Changes and click on “Add”. Then, add the affected devices, fill in the other required information about the change, and click on “Request”.
If the mitigation for a vulnerability involves a configuration change and a change case was opened, you can see in the respective change case profile which devices have already been re-configured. If a baseline definition was updated indicating that specific software must be present (such as a security patch) or must not be present (such as a vulnerable software application that isn’t required in the first place), the baseline profile tells you right away which devices are compliant and which ones aren’t.
Finally, you should indicate that a CVE is mitigated for an affected device by checking the box “fixed” in the “Affected Devices” listing in the CVE edit dialog.
There are several reasons why this needs to be done manually. For example, mitigation may not involve the installation of security patches at all, because there are none, or because other security controls have proven more efficient. Another reason would be that a security patch may be revoked by the vendor after the fact. Yet another reason would be that remediation was achieved in a way that cannot be automatically identified by OT-BASE, such as disconnecting a device from the network. In any case, checking the “fixed” box manually indicates that an engineer has actually verified the mitigation.
Once that you have marked a specific vulnerability as “fixed”, it will no longer show in the “Known Vulnerabilities” section in the device profile. Also, the numbers in the CVE listing will be updated automatically, so that you get a realistic picture of the number of outstanding mitigations for specific vulnerabilities.