How to Define Industrial Security Systems Testing Criteria

Before buying a security product, service, or system, there are success criteria and standards that companies set in order to ensure that newly acquired tools meet company standards. Post-purchase, security systems should be periodically tested to verify that they continue to meet success criteria. 

The proof of value (POV) success criteria for security systems differ on a per company and per product category level. Each industry has distinct inherent security strengths and weaknesses. Each security product category (antivirus, firewall, intrusion detection system) has different objectives and limitations.

How to Define Testing Criteria for Industrial Security Systems

Planning Phase - Gather information needed for assessment execution, such as the assets to be assessed, the threats of interest against the assets, and the security controls to be used to mitigate those threats.

Execution Phase - Primary goals for the execution phase are to identify vulnerabilities and validate them when appropriate. This phase should address activities associated with the intended assessment method and technique. 

Assessment Phase - The assessment phase focuses on analyzing identified vulnerabilities to determine root causes, establish mitigation recommendations, and develop a final report.

Testing security systems in OT is complex. For one, OT environments comprise a multitude of devices including endpoints, controllers, servers, and other network devices. Additionally, OT networks differ from one another. You therefore need a test bed that resembles your environment, which is not a trivial matter.

While there are a multitude of known IT security testing tools and implementations, OT's multigenerational and software-hardware-machinery architecture have resulted in a lack of standard tools for testing cybersecurity in real-life environments. 

OT Lab Setup

Testing an OT network necessitates defining custom scenarios and testing them on the network. Running live attacks on an OT network is a non-starter because of the above-mentioned ramifications of ceasing operations. Therefore, an OT lab setup is the recommended way to simulate live attacks. 

There are many options - from low budget open source simulators to fully functional OT physical labs running real processes. The below image illustrates the spectrum of OT lab setups.

Image 1 - OT lab setup

OTORIO chose to design its testing lab based on the high end model, so it is flexible and capable of testing every type of OT environment. A detailed view of our lab setting can be found on the white paper that we recently posted.  

Attack Scenario Tools

The next step is to dive into attack scenario tools. The tools are divided into two categories - blue team and red team. Both blue and red team tools can assist in testing security systems, networks, and products.

Table 1 - A comparison of popular attack scenario tools

As you can see, there are a number of excellent choices when it comes to attack scenario tools. An interface option is suited for operations teams who may not be experienced in scripting. Almost all available attack scenario tools offer the ability to add procedures. Most work with Linux and/or Windows. 

We opted for Caldera (or Cyber Adversary Language and Decision Engine for Red Team Automation). Caldera is interface based, which is key for OT security as not every security tool operator is a cyber security expert. Caldera can be run with little or no technical background. In industrial environments, many operational professionals "wear different hats" and perform operations, business, OT, and cybersecurity tasks during their workday.

Matan Dobrushin
Head of Research

Idan Helzer
Cybersecurity Researcher