By the Numbers: Towards Standard Evaluation Metrics for Programmable Logic Controllers' Defenses

Our modern society relies on important utility infrastructures such as water treatment plants and electric energy distribution grids. These infrastructures are managed by Industrial Control Systems (ICS), which include devices such as sensors, actuators and Programmable Logic Controllers (PLCs). PLCs are a key component of ICS as they serve as a bridge connecting the cyber and physical worlds. A cyberattack on a PLC could have disastrous real-world consequences, such as longstanding energy blackouts. Researchers have produced a plethora of security defenses in order to safeguard PLCs from cyberattacks, e.g., PLC-specific Intrusion Detection Systems (IDS). However, most of these defenses report incomplete or no performance evaluation metrics. Worse, the defenses that do report metrics evaluate them in an, ad-hoc way without providing details. As a consequence, PLC defenses cannot be compared or built upon, which is one of the main ways science progresses. It also makes it difficult to assess the effectiveness of such defenses against attacks. In this paper, we propose a standard set of performance evaluation metrics designed specifically for PLC security defenses. We propose three types of metrics: security, overhead, and effectiveness metrics. We then lay out what are the challenges faced when collecting these metrics, e.g., the heterogeneity of PLC architectures, and provide recommendations on how these challenges can be addressed to obtain accurate metrics. Obtaining and reporting these metrics will enable researchers to move PLC security research forward ultimately improving the security of ICS and our critical infrastructure.