AST-SafeSec: Adaptive Stress Testing for Safety and Security Co-Analysis of Cyber-Physical Systems

Cyber-physical systems are becoming more intelligent with the adoption of heterogeneous sensor networks and machine learning capabilities that deal with an increasing amount of input data. While this complexity aims to solve problems in various domains, it adds new challenges for the system assurance. One issue is the rise in the number of abnormal behaviors that affect system performance due to possible sensor faults and attacks. The combination of safety risks, which are usually caused by random sensor faults and security risks that can happen during any random system state, makes the full coverage testing of the cyber-physical system challenging. Existing techniques are inadequate to deal with complex safety and security co-risks against cyber-physical systems. In this paper, we propose AST-SafeSec, an analysis methodology for both safety and security aspects that utilizes reinforcement learning to identify the most likely adversarial paths at various normal or failure states of a cyber-physical system that can influence system behavior through its sensor data. The methodology is evaluated using an autonomous vehicle scenario by incorporating a security attack into the stochastic sensor elements of a vehicle. Evaluation results show that the methodology analyzes the interaction of malicious attacks with random faults and identifies the incident caused by the interactions and the most likely path that leads to the incident.


I. INTRODUCTION
C YBER-physical systems (CPSs) are essential components of many critical infrastructure applications, such as transportation systems, electric power, telecommunications, industrial manufacturing, water utilities, and oil and natural gas distribution. CPSs involve the integration of cyber and physical components to perform a variety of required tasks. Since information and communication technologies (ICTs) are becoming ubiquitous, several emerging technologies such as cloud computing, wireless sensor networks, Internet of Things (IoT), and artificial intelligence (AI) are adopting CPSs at Manuscript  an increasing rate. With rising complexity and autonomy, unknown and undesirable emergent properties of CPSs come to the surface [1]. Security and safety are emergent properties of CPSs, arising from the complex interactions among system components [2]. Threat actors are constantly improving their strategies in order to exploit undesirable emergent system properties, causing serious harm to society. This becomes apparent from the exponentially growing number of cyber attacks against CPSs [3]. The Stuxnet virus that infected the industrial control systems in Iran caused significant physical damage to critical infrastructure [4], while cyber attacks against the Ukrainian power grid resulted in a system-wide collapse [5]. Another malicious behavior involved an attempt to add a dangerous amount of sodium hydroxide to a water treatment plant in Florida [6], [7]. It is also worth noting that increasing system complexity has led to several safety incidents without any malicious intent, illustrating how simple errors can cause a huge impact [8]. For example, the cases of fatal aviation accidents [9] involving Boeing 737 MAX aircraft have been linked to the malfunction of flight control systems due to erroneous sensor data. In another fatal incident, a female pedestrian was killed because a self-driving Uber car experienced failings of its autonomous system [10]. Therefore, to prevent further undesirable events from occurring, there is a crucial need to better understand how these smart CPSs behave while experiencing malicious attacks and random faults.
The majority of modern CPSs rely heavily on sensor data to make decisions and perform actions. Hence, sensor data can be a fruitful source of information on how to influence CPS behavior throughout changes that may occur due to possible faults and attacks. However, most of the current testing approaches, e.g., [11], do not distinguish whether potential disturbances are coming from random faults or malicious attacks. In addition, attacks can happen during any system state, including normal and failure states. Safety and security issues are threatening CPSs at an increasing rate, leading to potential new incident scenarios that create a major need to consider unintentional or random risk sources and intentional or malicious risk sources together rather than separately. In terms of autonomy and interconnectivity, the complexity of CPSs leads to an increasing number of ways that can be exploited. This suggests a significant gap in knowing how safety and security analysis, and testing can be integrated in order to attain a more rich verification of CPSs to find the full set of most likely trajectories that lead to incidents. Most importantly, this integration can help in understanding how malicious attackers could take advantage of possible sensor faults in order to deploy their hostile actions.
This study focuses on answering the following research question: How can cyber-physical system be tested to cover possible risks caused by a combination of malicious sensor inputs and random sensor faults? To answer this question, we propose the AST-SafeSec methodology, which enables the automated machine learning (ML)-based safety and security co-analysis of complex CPSs. Due to increasing CPS complexity, stress-testing methods can demonstrate that the system will not cause undesirable events under abnormal circumstances, such as unexpected input values or overload conditions [12]. For that, we need to have a clear map detailing the possible ways to generate these unexpected input values. Then, we need to execute these possible ways in a combined view of safety and security. Hence, we provide a generic approach that consists of two phases: (i) pre-testing phase for selecting the combined stress-test strategy to follow under both faults and attacks and (ii) stress-testing phase for executing the selected strategy and finding the most likely incident scenarios that may occur in the CPS under test.
In the stress-testing phase, we extend the adaptive stresstesting (AST) framework [13], which is a stress-testing approach that focuses only on safety analysis. The extension is an additional SafeSec layer on top of the reinforcement learning (RL) agent's policy of AST, which is used to adjust the original policy of random actions per each state according to the attack policy and its constraints identified in the pretesting phase. Teixeira et al. [14] defines that "for each attack scenario, the attack policy is designed according to the adversary's intent, namely the attack goals and constraints." The attack goals are related to the attack's impact on the system operation, while the attack constraints can determine the efficiency of the attack. For example, in terms of detectability, attacks can be constrained to remain stealthy, or the adversary can have attack resource constraints that limit the available information from the system. Therefore, the integrated stress-test strategy of combining malicious attacks with random faults in sensor data is achieved by finding the minimum change to the original random action such that the selected attack constraints are met.
To evaluate the AST-SafeSec methodology, we applied it to an autonomous vehicle scenario by incorporating attack constraints that aim to bound certain variables of the sensor's readings to the stochastic sensor elements of the vehicle. The evaluation results show that adding the SafeSec layer on top of AST could reveal the risk caused by interactions between malicious attacks and random faults, thus reflecting the possible impacts of the attack policy with its constraints and helping to discover the most likely path that leads to the incident in a combined view of safety and security.
The main contribution is that we propose a novel testing approach for a combined validation of safety and security properties aimed at producing an understanding of the interplay between safety and security. Results of the testing could help CPS tests identify emergent risks and mitigate them effectively.
The paper is organized as follows: In Section II, we provide an overview of the related work. In Section III, we present the research motivation and problem formulation. In Section IV, we introduce our AST-SafeSec approach. Section V describes the evaluation of AST-SafeSec in an autonomous vehicle scenario. Finally, in Section VI, we discuss our contributions, potential research, practical and ethical implications, and the limitations of our work. The conclusions and future work are presented in Section VII.

II. RELATED WORK
The introduction of emerging technologies in CPSs has led to a new layer of complexity for the security and safety analysis of such systems, especially when the security issues of modern CPSs are evolving, as new vulnerabilities are discovered continuously. Prevention is a key element of safety and security and works as the first line of defense, aiming to identify possible faults and attacks to decrease the likelihood of their occurrence [15]. Given the increasing complexity and risks of CPS behavior, testing is one method that can contribute significantly to preventing undesired actions before incidents occur. Several methods have been introduced to address the testing of CPSs' safety and security [11]. Different needs and challenges are involved in the testing of CPSs [16] since they interact with the physical world and have continuous states, and inputs are applied at different points in time.

A. Testing Methods for Ensuring Safety of CPSs
The safety aspects are related to unintentional and random risk sources. Testing based on formal specification methods, e.g., [17] and [18], have been employed for safety analysis. However, such methods cannot scale well to handle complex CPSs with large state spaces and autonomous agents. For the complex CPSs, simulation-based testing approaches are commonly used since they can be applied to large-scale systems [19]. Notably, in large complex CPSs, black-box simulation testing strategies have proven to be an optimal solution when a concrete system specification is not always feasible [20]. This mapping from input to output can be sampled under the falsification problem aimed at finding fault disturbances that cause the CPS to violate its safety property. To achieve this, one promising approach is the application of ML techniques that can accelerate the testing process and discover additional ways that a CPS may fail without being biased toward what humans think the CPS will fail. For instance, Lee et al. [13] presented the AST method that integrates the RL technique for safety validation of safetycritical systems.

B. Testing Methods for Ensuring Security of CPSs
Security aspects are related to intentional and malicious risk sources. Extensive work has been done in security testing for information technology (IT) systems through fuzz testing, e.g., [21] and [22], and penetration testing [23] compared to Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.
CPS security testing. Analogous to the falsification problem, attack injection can be utilized to investigate the effects of potential cyber attacks on CPS behavior. In the CPS context, a black-box security testing for security-informed safety was presented by Skoglund et al. [24], with the aim of showing that cybersecurity threats do not endanger safety and to withstand security threats that can affect the safe operation of automated driving systems. The security-informed safety approaches examine the influence of cyber attacks on the safety of physical processes. Moreover, various studies [11] focus on fault injection techniques for simulating attacks to evaluate their impacts on system safety and to find security vulnerabilities of target systems. Similarly, the application of ML techniques into the CPS testing process has proven to be a promising research direction to ensure CPS security. For example, Yan et al. [25] examined the vulnerability of smart grids under sequential topology attacks with the least attack efforts. The authors used a Q-learning approach for finding the most vulnerable attack sequence that can cause critical failure of the system.
All the existing approaches, e.g., [13] and [24], do not consider the possible combination of intentional and unintentional causes. As Zhou et al. [11] highlighted, a key research direction for future complex testing methods of CPSs is a combined schema for non-functional CPS testing. Notably, testing CPS that includes ML models, whose precise behavior is difficult to determine, highlights the need for a more rich verification of CPS. Both intentional attacks and random sensor faults and inference can result in unexpected inputs to CPSs. If we do not analyze safety and security together, we may (i) overlook new safety and security co-risks that can lead to serious incidents and (ii) take measures that are not appropriate to reduce the risks. Table I provides a comparison of our methodology focus with focuses of existing methodologies presented in the literature. Traditional testing methods often handle safety and security analyses separately. This segmented approach may overlook potential interconnected safety and security co-risks that could lead to new incident scenarios, for example, the security attack may occur when a CPS component is at a faulty state. In contrast, AST-SafeSec simultaneously considers safety and security, revealing vulnerabilities that might be unnoticed if the two areas were analyzed independently.
Furthermore, it is crucial not only to determine whether a failure can occur, but also to identify which adversarial path is most likely to occur to allocate the safety and security assurance budget. AST-SafeSec can automatically and efficiently discover the most likely adversarial path by testing the CPS using RL agent. The agent learns to discover the adversarial path maximizes the reward it receives. Finally, in comparison to existing methodologies, AST-SafeSec offers greater coverage in fault and attack scenarios. It ensures a more comprehensive exploration of potential threats to the system through the AST-SafeSec's systematic steps in the pre-testing phase, which allows the selection of a testing strategy.

III. MOTIVATION AND PROBLEM STATEMENT
In this section, we present our research motivation and define the problem we aim to address.

A. Research Motivation
The interconnectedness of various CPSs and the integration of autonomous capabilities make CPS cyber risks very difficult to handle [26]. Rapidly increasing system complexity and environment uncertainty are intrinsic to CPSs. To ensure reliable and safe operation of CPSs, security testing must be done in the presence of normal and failure system states. Thus, neither a pure safety approach nor a pure security approach can fully cover the risks posed to CPSs, which may cause harmful consequences for the physical infrastructure that the CPSs control or interact with [27]. Therefore, safety and security should be treated jointly for a more comprehensive risk assessment.
In addition, evaluating a CPS by considering the continuous interactions with its environment is essential. The CPS behavior at the macro level is the result of emergent non-linear interactions of its components at the micro level. This is related to the fact that CPSs often exhibit emergent rather than resultant outcomes [28], [29]. Thus, the combined study on safety and security analysis of CPS behavior when operating in a stochastic environment is a necessity. The concern is not limited to whether an incident event can occur, but also to discover which incidents are most likely to happen, which adversarial path is most likely to occur, and the consequences of attacks that happen at CPSs' normal and failure states.
To address this challenge, we need to consider randomattack disturbances. The random aspect represents the safety challenges of a CPS, meaning that many safety issues are caused by the random faults of CPS components' happening during any system states. The attack aspect represents the security challenges of CPSs, meaning the potential attack approaches that produce adversarial disturbances. The random-attack disturbances try to cover all risks caused by attacks that can start during any random states of a CPS. The random states include CPS states in normal and abnormal operations.

B. Problem Statement
To explain and demonstrate our idea of dealing with random-attack disturbances, in this paper, we limit our focus to one type of stochastic element, i.e., sensors, in the CPS. We consider randomness in sensor readings of a CPS since these are the main inputs perceived by the CPS that influence its actions. As data from various sensors can be used to perform an attack, and the attack may happen during any system states, we may have different starting points of the attack. As the attack approach and consequence of the attack can vary at different system states, we need to check if the selected attack approach tampering sensor data can succeed at random CPS states.
The precise formulation of the problem we want to address is as follows. We assume that an adversary has the ability to intercept sensor readings within a CPS. First, the random starting noise values can be placed for a set of variables from sensor readings. The sensor values will drive the CPS to a particular state. Then, the random-attack disturbances interfere with the sensors' readings and correspond to random starting points followed by attack approaches and constraints. Since an attack can happen at any random starting point, our goal is to search the possibilities of having this attack succeed in different states of the system, including normal and failure.
Several approaches [30] that focus mainly on the combined safety and security co-analyses of systems have been proposed. For example, STPA-SafeSec [31] aims to accumulate information on how security and safety constraints can be violated by components and their interactions in the control structure. The approaches, e.g., [30], could be part of the test planning process [32]. However, in order to ensure that undesirable events have been truly eliminated or controlled to an acceptable level of risk, testing methodologies are essential.

IV. THE AST-SAFESEC METHODOLOGY
In this section, we present our AST-SafeSec methodology to address the random-attack disturbances. Any scenario discovered by the AST-SafeSec methodology indicates a risk that needs to be mitigated to ensure the safety of the CPS under test.
Traditionally, CPSs are less complex with few interactions, making cause-and-effect relations visible because system outcomes are usually resultant and can be deduced from component-level behavior. As a CPS increases in complexity, the system behavior and outcomes become increasingly emergent, and the system behavior cannot be deduced from component-level behavior [33], [34]. Now, cyber risks are becoming more difficult to handle due to the creation of a complex stochastic space. Therefore, AST-SafeSec focuses on integrated safety and security aspects for security analysis to be considered under the intrinsic randomness of certain stochastic elements of CPSs. Figure 1 illustrates our focus.
To address our research challenge, we first need to have a clear map outlining the possible ways to generate these unexpected input values. Then, we need to brute force the execution paths of the CPS under test in a combined view of the safety and security aspects and execute the paths to check if the CPS under test reaches an incidental state. Hence, the AST-SafeSec methodology consists of two sequential phases: • Pre-testing phase for selecting the stress-test strategy to follow. The purpose is to provide a qualitative method of analyzing each sensor's faults and attacks stressors of a given CPS. This can enumerate a number of risks, showing that there are many paths to consider for a more comprehensive CPS verification.
• Stress-testing phase for executing the selected stress-test strategy. The purpose is to uncover adverse effects of CPS behavior over time, which emerge from possible random and deliberate changes in sensor data that might otherwise be overlooked. This can reveal adversarial capabilities that can help provide defensive strategies for future unknown threat scenarios to win battles against sophisticated adversaries.

A. Pre-Testing Phase
In the pre-testing phase, we propose a planner to aid analysts in selecting stress-test strategies customized to the CPS under test. The purpose of this planner is to help communicate the proposal of stress tests internally within the organization before making the decision to proceed with the actual stress testing. This helps to limit and prioritize the testing focuses. It also provides guidance on possible attack vectors and their potential interaction with sensor faults that defenders need to consider for more comprehensive CPS protection. The interaction between attack vectors and faults is often a non-linear conjunction of events from multiple components instead of a single component-based event. In general, humans tend to believe that a conjunction of two events is more likely to happen than one event happening alone, known as the "conjunction fallacy" [35].
The planner has a hierarchical tree structure for qualitatively analyzing each sensor's possible faults and attacks stressors of a given CPS. Tree structures have been used to enumerate risks [36], [37]. While developing a stress-test strategy, each phase belongs to different levels of the tree hierarchy under the goal, as illustrated in Figure 2. As the tree descends, we enumerate subsequent details on how one or multiple sensors failed or are being attacked. This tree-style representation can better depict the depth of complexity behind each phase of the strategy by showing that there are multiple paths to consider for a more consistent system verification. Each stresstest strategy corresponds to a tree that describes a potential way of generating random-attack disturbances by considering a series of fault and attack stressors to achieve the goal state, which can be utilized as a basis for the stress-testing phase.
1) Define Root Event: The E refers to a subset of state space, in which an incident event of interest could occur. An event in E refers to both accidental faults on sensors and/or to actions performed by threat actors against the sensors of the CPS. Therefore, the root node of a tree is the goal of compromising the CPS behavior (e.g., autonomous vehicle collision).
2) Identify Sensors That Can Be Faulty or Attacked: Since any incident event can be caused by faults and attacks on sensor data, the step in Level 1 identifies the sensors of interest (s 1 , s 2 , . . . , s n ) used by a CPS, i.e., limiting the focus to sensors with a high probability of being faulty or attacked. The reason for minimizing the possible ways of attacking a CPS through only its sensor data is to investigate the possible adversarial system effects under an attacker's limited knowledge of the CPS. Moreover, the manipulation of sensor data flows can be of great importance to an adversary since sensor data are the main input sources perceived by the CPS, which then decides the actions to take.
3) Enumerate Stressors: The purpose of Level 2 is to enumerate the proximate causes of sensor failures and possible attack strategies. We want to find out possible causes of the failure and/or discrete actions that must be performed to achieve the goal state (i.e., when the final state of a CPS is matched to the failure state of the event in E). In making the selection of faults and attack types of the incident event of interest, a description of the system environment is required.
The following are examples of sensor fault stressors: • local failed component that refers to, e.g., issues from software bugs or mechanical defects; Fig. 3. Reinforcement learning process [47].
• failed connection between parts that refers to, e.g., tight couplings; • external factors that refer to, e.g., network latency or extreme nature conditions. The following are examples of sensor attack stressors: • adversarial sensor attacks, e.g., attacks against ML-based models in a CPS through data poisoning [38], L p -norm bounded perturbation [39], or a semantic adversarial attack [40]; • adversarial sensor timing attacks, e.g., attacks against ML-based models in a CPS to deceive in frequency by changing the timing of communication and computation operations [39], [41]; • false sensor data injection attacks, e.g., those target state estimations in a CPS [42], [43], [44]. 4) Identify Stressor Tools: Layer 3 of the tree hierarchy represents how to simulate the random-attack disturbances using stressor tools. The random-attack disturbances refer to environment actions that will be used by the agent in the stresstesting phase.

B. Stress-Testing Phase
In the stress-testing phase, we execute the selected stress-test strategy derived from the previous phase to ensure that undesirable events will not occur. Our idea is inspired by the safe exploration strategies using RL to achieve safe behavior of a system [45], [46]. Figure 3 shows a general flowchart of a typical RL process [47], where there is an agent that interacts with an environment by taking actions. On each timestep t, the agent takes an action a t ∈ A in the state s t ∈ S and receives a reward r t ∈ R from the environment based on the action taken. Then, the environment moves to another state s t+1 ∈ S. The agent's goal is to find a policy µ : S → A, which defines the action a = µ(s) at each state to maximize the reward obtained. The µ θ is a parameterized policy. We choose the RL approach because an agent can learn which actions to take by trying to understand the consequences of such actions. To harness the potential for scaling to large environments, we utilized deep RL, which employs a feed-forward neural network to represent the policy. In our implementation, we leveraged the Deep Deterministic Policy Gradient (DDPG) algorithm [48] due to its suitability for continuous action spaces, owing to the ability of its policy network to output actions directly, instead of their probabilities. Different from analyzing safety alone using AST [13], the safety and security co-analysis requires us to take attack constraints into consideration. Attack constraints are important since they can determine the efficiency of an attack. For example, attack constraints can determine which potential adversarial examples are valid inputs to the ML model within a CPS [39]. Another example is that in certain networked control systems, the use of cryptographic primitives (e.g., message authentication codes) can prevent injection attacks from occurring only at irregular intervals due to resource constraints [49]. Hence, attackers can model such frequency constraints, allowing an attack to occur a certain number of times.
In Figure 4, the frame depicted by the blue-color background represents the general working flow of the stress-testing phase of the AST-SafeSec approach. The inputs to this phase are stress-test strategies resulting from the pretesting phase. We consider an adversary with the ability to intercept sensors within a CPS. 1 ⃝ The main system input data can be collected from various sensor sources, such as surveillance sensors, cameras, speakers/microphones, or textual interfaces [50]. In particular, the identified sensors of interest are the ones recognized in Level 1 of the pre-testing phase. 2 ⃝ The main scheme of an RL agent is to generate random-attack disturbances on sensor data in order to bypass any potential detection from a CPS's detectors (e.g., AI-based detection models, users, or bad data detectors) and lead the CPS to the incident event of interest. More precisely, the event of interest is the one identified in the first step of the pre-testing phase. Employing different stressors from prior works in the literature (e.g., [39]) helps to generate disturbances that will be used to update sensor readings. In particular, the sensor's faults and attacks stressors are the ones recognized in Level 2 of the pre-testing phase. 3 ⃝ Then, these random-attack disturbances (i.e., as simulated based on the identified stressor tools in Level 3 of the pre-testing phase) are used by the RL agent in order to check their effectiveness for producing negative effects on a CPS under test. A CPS under test models the system behavior and can be viewed as the environment that interacts with the RL agent. At each time step, sensors provide observation of the new system state by a feedback loop of information. 4 ⃝ The simulated CPS updates its state and produces an indication of the existence of an incident scenario and its likelihood. 5 ⃝ The reward function of RL converts the simulated CPS outputs into a reward. Based on the received rewards, the agent's actions are updated. By using RL, there is a process of learning from reward signals to check if the selected disturbances on sensor readings achieve adversarial objectives (i.e., events of interest). Through multiple interactions with the CPS under test, which is often a simulator of the system, the agent learns to discover which disturbances maximize the reward it receives. In this case, the reward function gives rewards to incident scenarios and high likelihood transitions (from initial state to the event state) in order to find the most likely scenario to happen. If the CPS defenders identify such potential incident scenarios over the simulated CPS responses, they can react appropriately by gaining a solid understanding of a potential adversary's capability and decision process.
As explained in Section I, AST-SafeSec extends AST [13] with a SafeSec layer. AST defines the problem of discovering the most likely system failure as a sequential decision-making problem as shown in Figure 5.
1) Adaptive Stress Testing (AST): Given a simulator S and a subset E of the state space in which a failure event of interest occurs, AST can be used to discover the most likely path (s 0 , . . . , s t ) that terminates in the subset E. The solver that represents the RL agent selects environment actions that will be forwarded to the simulator. Then, it optimizes a reward based on transition likelihood and whether an event has occurred.
The objective of AST is to discover the most likely overall path with the highest likelihood, subject to the constraint that the final system state is a failure event: where a t indicates the random disturbances over discrete time t ∈ [0, . . . , t end ]. The random disturbances a t = µ θ (s t ) are distributed with probability density p(a t | s t ). Maximizing the overall path likelihood aims to find the scenario that causes the final state of the system s t end to belong to the subset E of state space where the event of interest occurs. For instance, an event could be an autonomous vehicle colliding with an object, and the goal is to apply random disturbances from a range of values on the vehicle until encountering the collision state.
2) AST With SafeSec Layer: AST [13] aims to perform only safety validation by randomly generating unexpected random disturbances. The purpose of the SafeSec layer is to start adjusting the received random actions taken by the RL agent from the original action policy (to unsafe ones) when necessary for attack constraints to be satisfied. To achieve this, the three main parts of the RL agent shown in Figure 4 include the following: (i) an actor network, representing the policy that takes the state s as input and directly outputs random action µ θ (s) taken from the action space; (ii) then, on top of the policy, an additional SafeSec layer is placed with the role of adjusting the random action µ θ (s) to a new action µ ′ θ (s) that also satisfies the attack policy and its constraints; and, (iii) finally, a critic network, representing the Q-value network in RL that takes the state and new action as input and outputs the Q-value, which estimates how good it is to take the action at that state based on the resultant reward from the reward function.
The SafeSec layer, as shown in Figure 4, is added to the policy level and operates as an action modifier based on the selected stress-test strategy throughout the whole learning process. The action correction occurs per each state with each forward propagation. For calculating the correction, a policy optimization problem needs to be solved as shown in Equation (2) for finding the minimum change to the original action µ θ (s) such that attack constraints are met. In particular, Equation (2) was originated by the context of safe exploration strategy for continuous action spaces [46], and we use it in the context of security for correcting actions to satisfy attack constraints that can be imposed on the attacks. The solution to Equation (2) can be a closed-form analytical solution due to the linear projection of the original action µ θ (s) as described in [46] with its mathematical proof. More precisely, each SafeSec signal (that corresponds to the observations of the state that aims to be attack constrained) c i (s, a) is approximated with a linear model with respect to action a. According to the selected stress-test strategy, we set the corresponding attack constraints with the choice of C i to limit the attacked bounded region because attack objectives should be posed as constraints to determine the efficiency of the attacks [14].
The new deterministic action selected by the involvement of SafeSec layer is denoted by µ ′ θ (s). In other words, the random-attack disturbances correspond to the new action µ ′ θ (s) that the RL agent selects and replace the random disturbances a t = µ θ (s t ) in Equation (1) in order to discover the most likely path that leads to an incident in a combined safety and security process.

V. EVALUATION
This section presents a case study of applying the AST-SafeSec methodology to an autonomous vehicle scenario as an evaluation. The purpose of the evaluation is to show the effectiveness of the AST-SafeSec methodology to analyze the interactions of malicious attacks with random faults on sensor data and to discover the most likely path that leads to the incident event of interest. We limit the evaluation to a simple scenario of a system with a noisy sensor by incorporating an attack policy that aims to strategically add disturbances to the sensor readings, illustrating the applicability of our proposed approach. In addition, we want to measure the differences when applying AST only, comparing it to AST-SafeSec to show that our approach could reduce the search space and reveal safety and security co-risks. Hence, we choose a simple simulator that has interactions with its environment, including a sensor with noisy measurements that are filtered from a tracker or state estimation module and passed to the system model to decide the actions on how to control the system based on its observations as illustrated in Figure 6.

A. System Model
The simulator we chose is a simple autonomous vehicle (AV) scenario [51]. The AV relies on the intelligent driver model [52] that aims to safely follow the traffic flow. In the AV scenario, the intelligent driver model interacts with pedestrians by treating the nearest pedestrian in the road as a target vehicle that needs to follow in a safe distance. Even though the vehicle under consideration is not entirely autonomous because of its reliance on physics-based models, it includes certain self-driving features where the intelligent driver model is used to perform operations by means of adaptive cruise control. The reason behind this choice of simulator is, as the authors [51] mention, that it would be better to start by choosing a simple case study scenario to illustrate the applicability of the proposed approach and then as future work to be applied to other more realistic system models. We consider a scenario that involves perturbing stochastic sensor elements in the AV's environment until the AV is involved in a collision with a pedestrian. In particular, we stress test an AV with a noisy sensor that receives data about pedestrians in a scenario that involves approaching a pedestrian crosswalk, as shown in Figure 7. The initial condition space of the AV scenario refers to an AV starting 35m away from the crosswalk, and a pedestrian is crossing the crosswalk from south to north.

B. Pre-Testing Phase
In the pre-testing phase, we use the proposed planner to select the stress-test strategy customized to the system of interest (i.e., AV scenario) as follows.
• Define root event: We consider an incident event of interest in E to be a collision, where a vehicle collides with a pedestrian. The adversary aims to cause adverse effects to the operation of the physical process.
• Identify sensors that can be faulty or attacked: To achieve this, the adversary will be aware of the set of sensor data to understand how random and deliberate changes to sensor data influence the physical process. We consider the combination of both fault and attack types on the same sensor s 1 in the AV scenario that receives data about positions and velocities of pedestrians.
• Enumerate stressors: From the safety side, we consider that the sensor s 1 is failed due to a local failed component, which is represented by randomness on generated sensor readings. Hence, due to faults, we consider potential random modifications of the sensor's measurements of the pedestrian defined by the s ped = [v x , v y , x, y] comprising the x and y-axis of the pedestrian's velocity and position. From the security side, we consider that an adversary performs a false sensor data injection attack on the sensor. Fig. 8. The selected stress-test strategy based on the planner's output in the pre-testing phase.
• Identify stressor tools: Finally, we provide subsequent details on how the sensor s 1 fails and is attacked. From the safety side, the sensor s 1 can fail due to local failed component targets mechanical aspects. From the security side, the false sensor data injection attack on the sensor s 1 is based on an additive attack on sensor readings as described in [42]. Due to the absence of bad data detector mechanisms in the existing vehicle simulator, we do not consider the attack to remain undetected and stealthy, but instead, we apply a simple attack policy based on strategic randomness. We assume the attacker will strategically add false data on the sensor readings rather than adding false data randomly to cause the vehicle to collide with the pedestrian. For each attack, the attack policy is designed based on attack goals and constraints [14]. In our scenario, the attack policy aims to bring the vehicle as close as possible to a changing target pedestrian location. Therefore, the attack constraints refer to constraint the injected false data not to be random but to lead to the goal of collision. Figure 8 illustrates the selected stress-test strategy based on the output of the planner in the pre-testing phase of the AST-SafeSec methodology.

C. Stress-Testing Phase
In the stress-testing phase, we execute the selected stress-test strategy from the previous phase. This is done with the involvement of the RL agent that interacts with the environment representing the CPS under test (i.e., AV scenario). As explained in Section IV-B, the RL agent includes three main parts. The results of these parts in our studied AV scenario are as follows.
1) Actor Network: The actor network represents the policy that takes the state as input and outputs the random action taken from the action space. The action space refers to the fault type on the failed sensor s 1 with random modifications on the sensor's measurements of the pedestrian. It represents the fact that the agent receives the environment actions from a continuous random range of noise values to be injected into the measurements of the sensor s 1 . At each timestep, the environment action vector is [a x , a y , ϵ v x , ϵ v y , ϵ x , ϵ y ], representing the random action µ θ (s) selected by the actor network where • a x , a y are the x and y-axis of the pedestrian's acceleration, • ϵ v x , ϵ v y are the random noise to be injected into the x and y-axis of the pedestrian's velocity, and • ϵ x , ϵ y are the random noise to be injected into the x and y-axis of the pedestrian's position. 2) SafeSec Layer: We introduce the SafeSec layer to test if the CPS can reach the incidental state of collision under random-attack disturbances. To achieve this, the SafeSec layer adjusts the above random environment action vector, such that the selected attack policy on the sensor s 1 is defined by its attack constraints to be satisfied. The strategy for developing integrity attacks (i.e., false data injection attack) on control systems includes an additive integrity attack as shown in Equation (3) that adds the attack vector to the actual sensor measurements over the attack duration. The attack vector that represents the data corruption is denoted by y t . According to the attack duration T α , the attack vector y t denotes the injected disturbances to the sensor s 1 at time t. It aims to represent the random-attack disturbances under both safety and security aspects (i.e., adjusted random action such that attack constraints are satisfied). In our evaluation scenario, the attack duration refers to the whole simulation process. The received falsified measurements y t can be defined as follows: Attack Constraints: The way that the attack vector y t is constructed is based on the attack policy and its constraints. For this paper, we consider a simple attack policy and its constraints based on strategic randomness. In particular, the attack objective function (i.e., as shown in Equation (4)) begins adjusting the random environment action vector µ θ (s) to the desired one µ ′ θ (s) based on the attack policy and its constraints to drive the vehicle to the pedestrian as soon as possible to the unsafe region, in which a collision between the vehicle and pedestrian could occur. We consider the region unsafe if the mininum distance between the vehicle and the pedestrian in x-axis is less than Min Dist=2.5m. Therefore, in this work, in order to bound this unsafe region, we define attack constraints that represent lower and upper bounds that should never be violated. In other words, the agent will try to correct the random actions to perform the more efficient attack, i.e., the attack that follows the attack constraint. In the case of attack constraint violations, it means that the injected disturbances are too random and will not be efficient in leading the collision.
Hence, the attack objective function, i.e., the one in Equation (4), aims to generate the random-attack disturbances by strategically adjusting the random action vector µ θ (s) in order to never violate the attack constraints. A gap away from each lower and upper bound of the minimum distance is incorporated in the choice of C i . This gap helps notify the SafeSec layer to start adjusting the random disturbances once the x-axis of the pedestrian's position leaves the unsafe region as illustrated in Figure 9. Applying different, more realistic attack policies is an avenue for future work. The SafeSec layer aims to apply attack constraints on the random environment action vector. The resultant new action µ ′ θ (s) represents the adjusted original random action by finding the minimum change to it, such that the selected attack constraints to be met is as follows. Here, s ped is the state of the pedestrian that corresponds to the observations of the state that aims to be attack constrained.
3) Reward Function: The reward function is based on the AST work [51]. For finding a collision, the reward is 0, while for failing to discover a collision during the search, there is a usage of the probability of the taken action by applying the Mahalanobis distance M(a, µ a |s) between the action a and the expected action µ a given the current state s. We have defined a horizon T , which refers to the maximum number of timesteps to search a potential collision within one iteration. For reaching the horizon T without collision and encouraging a faster convergence, the penalty for failing to discover a collision is given by large negative constant numbers, e.g., -10000. In addition, the penalty at the end of not finding a collision is scaled by the distance D I ST between AV position p v and pedestrian position p p . This penalty helps to encourage the pedestrian to early end trials closer to the AV, leading to faster convergence.

D. Results
To evaluate the effectiveness of AST-SafeSec when applied to an AV scenario, we perform stress-testing in a scenario involving a single vehicle approaching a pedestrian crosswalk with one pedestrian attempting to cross, as shown in Figure 7. The chosen type of the attack approach is the false data injection attack. The performance indicators used to assess the method's effectiveness are the likelihood of the final collision trajectory output by the system and the simulation steps. The latter metric serves to compare the efficiency of the two agents, one with the SafeSec layer and one without it.
Regarding the specifics about simulation parameters, in the AV scenario, we experiment with horizon T of 100 timesteps as adequate length to cover the considered scenario. We use 1000 iterations, and the training is obtained using a batch size of 4000. We set the discount factor to 0.99 and the step size to 0.1. The deep RL approach is implemented based on [48].
The results show that both deep RL agents (with and without the SafeSec layer) are able to identify incident event of interest trajectories by producing situations where the AV collides with the pedestrian. According to [51], the AST framework can discover collisions by leveraging the intelligent driver model's decision to miss out any pedestrian who is not in the road. For this study, the deep RL agent involving the SafeSec layer is under a simple attack constrained policy based on strategic randomness. The results show that AST-SafeSec successfully converges to a solution within a few simulation steps. This reflects the algorithm's computational ability to discover a collision. However, since an agent with the SafeSec layer aims to perform combined safety and security validation, in contrast to an agent without the SafeSec layer that performs only safety validation, the results can be different depending on the attack constrained policy that was applied. Table II shows the numerical results for two different scenarios. We consider single pedestrian scenarios with different initial conditions. Scenario 1 reflects the pedestrian with the initial state s 1 ped = [0.0m/s, 1.4m/s, 0.0m, −2.0m], and Scenario 2 reflects the pedestrian with the initial state s 2 ped = [0.0m/s, 1.4m/s, 0.0m, −4.0m]. Across all scenarios, the agent with the SafeSec layer reflects the impact of the security attack. In the experimented attack strategy, false data managed to satisfy the adversarial objective by finding a more probable path sooner than without the use of the SafeSec layer. Each scenario leads to different collision trajectories.
By calculating the Mahalanobis distance of the most likely path to the incident event found at each iteration, we derive the highest reward on that path. The most probable path refers to the path that leads to finding the most likely collision. More precisely, in Scenario 1 (without the SafeSec layer), the most probable path has a length of 59 timesteps with a sum of rewards -843 and is found at the iteration number 890. In Scenario 1 (with the SafeSec layer), the most probable path has a length of 35 timesteps with a sum of rewards -602 and is found at the iteration number 372. Accordingly, in Scenario 2 (without the SafeSec layer), the most probable path has a length of 27 timesteps with a sum of rewards -324 and is found at the iteration number 744. In Scenario 2 (with the SafeSec layer), the most probable path has a length of 27 timesteps with a sum of rewards -440 and is found at the iteration number 250. In summary, the results show that an agent with the SafeSec layer can analyze the interactions of malicious actions with random faults on sensor data by satisfying the impacts of the attack policy with its constraints, with the overall aim to discover the most likely path that leads to the incident. Figure 10 demonstrates the last reward received in the final timestep from each agent for Scenarios 1 and 2, respectively. This reflects the performance of the agents (with and without the SafeSec layer) in finding collisions across the number of iterations. The agent with the SafeSec layer has the advantage of reflecting the impact of the attack policy with its constraints to strategically guide the injection of false data, thus finding the most likely path that satisfies the adversarial objective. When a trajectory ends without a collision, the last reward refers to the extra penalty that is scaled by the distance between the pedestrian and vehicle. When a trajectory ends with a collision, the last reward is 0, indicating the existence of a collision. In Scenario 1, both agents lead the vehicle toward the pedestrian to create a collision. However, an agent with the SafeSec layer reflects the impact of the attack policy with its constraints, being capable of discovering a more probable path sooner than an agent without the SafeSec layer. Compared to Scenario 2, which begins farther away from the vehicle, it increases the opportunities of being hit by the vehicle as shown in the corresponding diagrams for both cases. These results can support the involvement of the SafeSec layer in analyzing the interactions of malicious attacks and random faults, thus reflecting the possible impacts of the attack policy with its constraints in a combined view of safety and security.

A. Comparison With Related Work
Given the increasing complexity of modern CPSs in terms of multiple interconnections and autonomous functionality, risks are becoming more difficult to handle due to the creation of a complex stochastic environment. Hence, in order to ensure safe and secure CPSs, it is essential to analyze the implications of this increasing complexity under the interactions of malicious attack and random faults. This paper aims to provide an integrated method of combining random faults with malicious attacks according to the attack policy and its constraints. The proposed methodology allows end users to decide what they should be testing for a given CPS while considering more comprehensive tests. There is little research related to the assessment of combinations of safety and security risks that a CPS may fail. Testing methods, e.g., [24] and [25], for ensuring the security of CPSs examine only the effects of cyber attacks on the safety of physical processes. Our work contributes to a collective approach in co-analysis of emergent safety and security properties of CPSs, in contrast to other similar studies that we described in Section II, which were mainly focused on these issues separately. Our approach helps in identifying a more complete set of scenarios that may lead to system damages, as well as understanding how adversaries can leverage the faults within a system for deploying malicious activities.

B. Research Implications
The AST-SafeSec approach contributes to aligning each sensor's faults and attacks stressors of a given CPS, resulting in a clear map outlining the possible ways to generate disturbances. Then, these disturbances are used to discover the most likely path that leads to an incident event of interest in a combined view of safety and security. Therefore, the proposed methodology can be used as a basis upon which researchers can examine the impact of different strategies under both random and malicious attack strategies to understand risks to CPSs. Moreover, the translation of experimental results could be useful for risk analysts into improved models for risk analysis.
This research provides a basis for the exploration of other potential research as follows.
1) Different Attack Approaches: Several attack approaches can be adopted for more comprehensive testing and verification of CPSs. The RL technique is used to guide the testing process for discovering which incident scenarios are most likely to occur under different types of stressors. The potential for several attack policies to be adopted based on their corresponding attack constraints can help build the research stream for a multi-combination of attack strategies.
2) Other Data Inputs: Several types of data inputs (e.g., textual and audio data) can play an important role in providing CPSs with useful information. Therefore, it is necessary to investigate the effectiveness of certain data input types in sensor data deception on the whole CPS pipeline [50], [53]. The application of the proposed methodology can be used as a basis for other data inputs. Furthermore, except for sensor data, the focus could be on other data flows for later manipulation of the resulting digital data.
3) Cyber Deception for Defense Strategies: Most of the current defense practices focus on detective and reactive responses during and after attacks have happened. This gives an asymmetric advantage for adversaries to have enough time to learn how to influence a CPS and thus prepare to launch their sophisticated attacks. To address this issue, defenders need to also investigate strategies that will engage with adversaries in the early phases of the kill chain in order to learn their tactics and disrupt their progress. The use of defensive deception approaches has shown great promise to capture how intelligent adversaries attain knowledge about a target CPS and disrupt its performance [54]. One of the key aspects in the lifecycle of every cyber deception strategy is, as a defender, to observe and estimate the capability of an adversary. Our approach can be used to guide the development and deployment of deception-based defenses. This is especially crucial nowadays, when attackers use AI techniques to automatically manipulate target systems in a very conceiving manner [55]. In the face of the AI era, deception-based defenses can improve the overall security of systems. Incorporating deception-based defenses can improve our understanding of compromise attempts and thus better protect the systems over time.

C. Practical Implications
Our approach is goal-directed, qualitatively enumerating a number of risks showing that there are many paths to consider for a more comprehensive CPS verification. The proposed methodology can help provide inputs about whether an attack can happen or not, including normal and failure states of a CPS. Thus, the methodology can contribute to the identification of potential consequences of safety and security co-risks of a CPS that may lead to serious incidents. This kind of scenario-based defense aims to uncover risks that emerge from possible random and deliberate changes in sensor data with adverse effects on CPS behavior over time that might otherwise be overlooked, thus contributing to the creation of more effective responses during future challenges. Finally, ranking of the paths based on the most likely incident scenarios can help stakeholders prioritize defensive strategies. The CPS testing is challenging due to the involvement and interactions of many components; therefore, having approaches that can contribute toward choosing the most cost-effective defense is essential.

D. Ethical Implications
A primary element of a defense strategy is to discover adversarial capabilities, motives, and tactics in order to acquire enough information to amount better defenses against them [56]. Hence, the use of such preventive methodologies, e.g., AST-SafeSec, by defenders can contribute to the selection of suitable defensive measures to mitigate potential risks and their associated damages such that identified incident scenarios do not emerge. Several security approaches that are intended for defenders can also be used by malicious actors. Publishing the AST-SafeSec methodology is, however, not risky. The proposed AST-SafeSec methodology alone would be insufficient for use by potential attackers because they would need to have knowledge of the operations of a target CPS, and gaining access to a concrete model of one is usually difficult [57].

E. Limitations
By seeking more comprehensiveness in testing safety and security emergent properties of CPSs, this paper offers a first step in utilizing combined random-attack disturbances against CPSs. However, this work has certain limitations and suggests further investigation across different future research directions.
One of the limitations of this study lies in the choice of the RL algorithm used, which may affect the policy that we obtain. We experiment with DDPG algorithm [48] due to the ability of its policy network to directly output actions instead of their probabilities. During the learning phase in RL, the basic aims are to find a potentially optimal policy; however, this is very challenging to acquire, and further investigation across multiple RL algorithms could be useful. Following the choice of RL algorithm, the optimization of hyper-parameters for the algorithm may also influence potential improvements of its performance.

VII. CONCLUSION AND FUTURE WORK
Motivated by the advances in CPSs and the evolving cyber threat landscape, this paper proposes a methodology for a more comprehensive risk assessment, composed of two phases. In the pre-testing phase, we propose a planner that has a tree hierarchy structure for selecting the stress-test strategy to follow. In the stress-testing phase, we execute the selected stress-test strategy through a process of learning from reward signals, which are used to check if the adversarial system inputs (i.e., falsified sensor readings) aim at satisfying a set of adversarial objectives. We evaluate the methodology in a simple autonomous vehicle scenario, in which the new agent can converge to a more likely path that leads to the incident event of interest by analyzing the interactions of malicious attacks and random faults on sensor data.
In future work, we plan to apply a combination of attack strategies targeting different sensors within a more realistic CPS model. Additionally, we aim to broaden the evaluation of the AST-SafeSec methodology across various CPSs and explore its potential with complex AI/ML systems. Specifically, we are considering expanding our proposed methodology to incorporate ethical stressors, acknowledging the inherent socio-technical nature of many AI/ML systems. These ethical stressors could address aspects such as fairness and non-discrimination. This will enable us to provide a comparison between the results for a more comprehensive understanding of the methodology's efficacy.