Advanced Performance Metrics and Sensitivity Analysis for Model Validation and Calibration

High quality generator dynamic models are critical to reliable and accurate power systems studies and planning. With the availability of PMU measurements, measurementbased approach for model validation has gained significant prominence. Currently, the model validation results are analyzed by visually comparing real–world PMU measurements with the model-based response measurements, and parameter adjustments rely mostly on engineering experience. This paper proposes advanced performance metrics to systematically quantify the generator dynamic model validation results by separately taking into consideration slow governor response and comparatively fast oscillatory response. The performance metric for governor response is based on the step response characteristics of a system and the metric for oscillatory response is based on the response of generator to each system mode calculated using modal analysis. The proposed metrics in this paper is aimed at providing critical information to help with the selection of parameters to be tuned for model calibration by performing enhanced sensitivity analysis, and also help with rule-based model calibration. Results obtained using both simulated and real-world measurements validate the effectiveness of the proposed performance metrics and sensitivity analysis for model validation and calibration.


I. INTRODUCTION
H IGH quality dynamic model of generators are critical to reliable and economical power system operations and planning. Dynamic studies for various system disturbances, such as faults, generation loss, line trip, etc., is carried out using these models for both short and long term planning. These studies provide information on several aspects of power systems dynamic stability such as rotor angle stability, damping ratio of system modes, primary frequency response, system frequency and voltage recovery, etc., and identify contingencies that can result in system instability and stability constrained transmission paths. The accuracy of these studies heavily depends on the quality of dynamic models used, thereby making validation and calibration of generator dynamic models critically important. The need for accurate and up-to-date dynamic models for reliable and economical grid operations and planning was reinforced after the well-known 1996 western grid blackout. The planning Western System Coordinating Council (WSCC) model could not replicate the unstable system oscillations observed following the series of events that led to the system-wide outage [1]. After this This work is supported by the U.S. Department of Energy's Office of Electricity through CERTS program. Pacific Northwest National Laboratory is operated by Battelle for DOE under contract DE-AC05-76RL01830.
U. Agrawal, P. Etingov and R. Huang are with the Pacific Northwest National Laboratory, Richland, WA, 99352 USA E-mail: urmila.agrawal@pnnl.gov event, NERC now requires all generators having capacity of greater than 10 MVA to be validated every five years. Also, Reliability Standards MOD-026 [2] and MOD-027 [3] have been developed to provide guidelines for generator model validation.
Traditional methods for validating generator dynamic models include staged and standstill frequency response testing [4]. These methods involve physical testing of the generators and therefore generators to be validated remain unavailable for normal operations. Even though these methods provide high quality dynamic models, these methods are technically difficult and are expensive [4]. With the availability of synchrophasor measurements, measurement-based validation methods have become widely-accepted [5]- [11]. This method requires Phasor Measurement Units (PMU) to be installed at the point of interconnection (POI) of each generator to be validated. The PMU measurements recorded at these locations are then used as play-in signals to validate generator dynamic models as shown in Fig. 1 [12]. This approach of validating dynamic models is available in several power systems simulator such as GE PSLF, SIEMENS PTI PSSE, PowerWorld Simulator and TSAT [13]. Several tools have been developed to validate generator dynamic modele using play-in PMU measurements, such as power plant model validation (PPMV) tool by Bonneville Power Administration (BPA) and Pacific Northwest National Laboratory (PNNL) [13], generator parameter validation (GPV) tool by Electric Power Group [14] and Power Plant Model Verification (PPMVer) tool by ISO-New England [7]. Currently, the model validation in most of the available tools is carried out either by visually inspecting the difference between the simulated and actual measurements of real and reactive power [7] or using some simple metric such as root-mean square error used in [13], peak value and peak-time of the first swing and steady-state error used in [15]. These metrics do not provide any information on model inaccuracies associated with various aspects of generator dynamic response, such as oscillatory response. Ref. [16] describes the application of frequency-domain based metrics, first defined in [17] as Magnitude-shape similarity measure, for quantifying model validation results. This metric finds a weighted average of the similarity measure of the magnitude and phase spectra in frequency domain. However, the frequency-domain based metric defined in [16] is a generic one and does not specify anything with respect to generator oscillatory response or governor response. In order to automate the process of model validation and calibration, some advanced metrics system is needed that takes into consideration various aspects of generator dynamic response for quantifying model validation results.
In this paper, a new methodology is proposed to quantify the model validation results by separating the slow governor and fast oscillatory response. Separate metrics are developed for these two responses; phase and magnitude for generator oscillatory response [18], and delay, peak value, peak-value time, rise-time and steady-state value for generator governor response. This proposed set of metrics is further used for enhanced sensitivity analysis to help with the selection of specific parameters for model calibration similar to the trajectory sensitivity analysis described in [19]. The proposed sensitivity analysis is, however, more rigorous in nature and provide detailed information on the sensitivity of each paraemter to specific aspect of generator response. The proposed advanced metrics and sensitivity analysis can further be used for rule-based model calibration method as described in [20]. In [20], model calibration methodology is based on mismatches observed in maximum amplitude and damping ratio of oscillations. With the methodology proposed in this paper, model calibration can be performed by capturing not just amplitude and damping ratio of generator oscillatory response, but also other aspects of oscillatory response including governor response. A detailed methodology is included in the paper to obtain proposed metrics and perform sensitivity analysis for model validation and calibration.
The rest of the paper is organized as follows: Section II provides background theory required to develop the proposed methodology, Section III presents a detailed description of the proposed methodology, Section IV provides results and discussion for proposed metrics and sensitivity analysis, and Section V concludes the paper.

II. BACKGROUND THEORY
This section briefly describes the Prony method used for analyzing generator oscillatory response, and characteristics of the step-response of a system used for analyzing generator governor response.

A. Prony method
While any modal analysis method, such as Prony [21] and Matrix-pencil [22], can be used to obtain the metrics for generator oscillatory response, Prony analysis method is used in this paper to illustrate the methodology of the proposed metrics for oscillatory response. The Prony method consists of three steps as described in [21]. Let the N samples of measurements be given by y 1) In the first step, a discrete linear prediction model (LPM) is obtained, that fits the signal, by solving a linear leastsquares problem given by where a = a 1 a 2 · · · a n T , and n is the model order selected to obtain system mode estimates. The least squares solution of (1) is given bŷ where † denoted pseudo-inverse of a matrix. The n-th order polynomial equation is then given by 2) In the second step, mode estimates, given byλ i = ω i + jσ i for i = 1 to n, are calculated usinĝ where are the roots of the estimated n-th order polynomial equation given by (3), ∆T is the sampling time period of the measurements, ω i is the frequency of each mode in rad/sec, and σ i is the damping coefficient. The damping ratio of each i th mode is given bŷ 3) In the final step, the initial amplitude and phase of each mode, given by the phasor estimateB i , is calculated solving where The least squares solution of (6) is given bŷ Following steps 1 to 3, mode estimates are given by (4) and mode shape of each mode is given by (7).

Validation of mode estimates:
Only estimating modes is not sufficient unless the mode estimates are validated [23]. For validating mode estimates, the original signal is compared with the reconstructed signal given byŷ where n r is the number of real modes, n c is the number of pair of complex modes andŷ i [k] is the contribution of the i th mode to the signal given bŷ The goodness of fit metric is calculated using [21] where ||.|| denotes root-mean-square norm. Using (10), model order n is selected that gives the best fit between the analyzed and reconstructed signal.
Sorting of system mode estimates As described in [23], not all mode estimates represent actual modes. Some of the mode estimates are spurious ones and need to be discarded. One of the ways of distinguishing actual mode estimates and spurious ones is to rank mode estimates based on their energy given by The modes having insignificantly small energy as compared to the highest energy can be discarded.

B. Characteristics of the step-response of a system
Governor response of a generator can be represented by the step-response of a system as shown in Fig. 2. Step-response characteristics of a system The metrics for validating model-based governor response is based on the following step-response characteristics [24]: • Peak value: Given by the peak absolute value of the governor response. This paper proposes a new methodology for quantifying model validation results as shown in Fig. 3. The main objective of this paper is to develop a set of metrics that can not only help distinguish a good model from a bad one, but also provide detailed information on specific aspect of modelbased generator response that does not match with actual generator response, such as phase and magnitude of oscillatory modes observed in the actual generator measurements and characteristics of the governor response. A detailed description of the proposed methodology is discussed next. A.
Step-1: Separating governor and oscillatory response During system faults, generator dynamic response can usually be broken down into two components, one is the slow governor response and the other fast oscillatory response. The generator oscillatory response is determined by system modes and therefore the frequency range of this response lies between 0.1 and 2.0 Hz. Therefore, the slow governor response and the oscillatory response can be separated by passing the generator response through a high-pass filter having a cut-off frequency of less than 0.1 Hz as illustrated in the Fig. 4(a) and Fig. 4(b). The oscillatory response is then obtained by taking the difference of the generator response and the oscillatory response, and passing the resultant signal through median filter to smooth out any oscillatory components present in the signal. This is the first and the important step in calculating proposed metrics and performing sensitivity analysis.

B. Step-2: Calculation of performance metrics
In the second step, metrics is calculated for the separated governor and oscillatory response corresponding to the active power. Metrics proposed for each of these responses is described next.
1) Active power -Oscillatory response: The metric for validating generator oscillatory response is calculated based on the properties of the oscillatory modes observed in the PMU and simulated measurements. Two metrics are proposed in this paper for validating generator oscillatory response, one for magnitude and the other for phase of oscillatory modes. The metric for magnitude incorporates any discrepancy associated with initial amplitude, damping-ratio and frequency of system modes between the model-based response and actual response. The metric for phase calculates any phase difference between the two signal. The two metrics can either be combined as a weighted average or can be used separately. In this paper, the two metrics are used separately as this can provide information helpful for calibration, i.e., if the calibration should focus on phase shift or magnitude or both.
The metric for validating the magnitude component of the model-based oscillatory response is given by: where m,i = ( superscript 'a' corresponds to estimates using actual response and 's' corresponds to estimates using simulated response, m,i is the normalized error metric for each mode, w i is the weight factor for each mode given by its energy as defined in (11), p is the number of dominant modes selected out of n modes based on their energy, and |.| denotes absolute value of the quantity. Here, the reconstructed signal used for calculating this metric is obtained by discarding initial phase of the modes so that the error associated with the phase does not impact the magnitude metric. The metric for validating the phase component of the modelbased oscillatory response is given by: where p,i is the normalized phase error associated with the i th mode observed in actual and simulated measurements. The metric obtained for each mode is weighted with its energy to obtain a single metric. If any mode observed in the PMU measurement is not observed in the mode estimated using the simulated data, a error of 1 is assigned to both m,i and p,i for that mode.
The step-wise methodology to obtain the proposed metrics for validating model-based oscillatory response is as below: (i) Pre-process PMU and simulated measurements by using signal processing techniques, such as filtering, downsampling, etc., for modal analysis (ii) Obtain mode estimates and mode-shapes for both preprocessed PMU and model-based measurements using (4) and (7). In this step, selection of model order is carried out for both the signals by comparing pre-processed original and reconstructed signal. Also, dominant modes are distinguished from the spurious ones by calculating energy of mode estimates using (11). (iii) Calculate the two metrics to validate the model-based oscillatory response by comparing it with the actual oscillatory response using (12) and (16).
2) Active power -Governor response: Based on the stepresponse characteristics of a system, as shown in Fig. 2, several metrics are defined to validate the model-based governor response by comparing it with the actual governor response. Each metric looks into a specific aspect of the governor response, which are as follows: (i) Delay (G d ): Obtained by taking the difference of the time taken by the model-based and actual governor response to reach 10% of their respective peak value with respect to a common time-reference. the time taken by the model-based and actual governor response to change from 20% to 90% of their respective peak-value. A simple root-mean square error can also be used instead of these several metrics to validate generator governor response, however it will not be able to provide any information on errors in specific aspects of the governor response, such as delay in the response, which can be helpful for model calibration.
Ideally, the mismatch observed in the actual and modelbased generator response should be equal to zero. However, that is generally not the case. Therefore, certain thresholds need to be determined for each metric to validate the generator model. These thresholds should be determined based on the industry practices and is beyond the scope of the paper.

C. Step-3: Sensitivity analysis
Based on the calculated metrics, if it is determined that the model needs calibration then the next step will be to perform sensitivity analysis to identify parameters that needs to be calibrated. In this paper, sensitivity analysis is carried out to identify these parameters by analyzing the impact of each model parameter on a specific metric having error greater than the threshold, for example where S G d ,H provides information on the sensitivity of generator inertia (H) parameter on governor delay metric (G d ).
This analysis can help determine if a specific parameter can be calibrated to reduce error associated with a specific error metric.

IV. RESULTS AND DISCUSSIONS
Results were obtained using both simulated and real-world PMU measurements to illustrate the effectiveness of the proposed metrics and sensitivity analysis.

A. Simulated data
The simulated-data based example used in this paper is taken from the 12 disturbances set prepared by NASPI Engineering Analysis Task Team and NERC synchrophasor measurement subcommittee team for NASPI Technical Workshop on Model Verification Tools in 2016 [25]. Fig. 5 shows the active power measured at the POI of the generator, and the model-based response of the generator obtained using PPMV tool developed by BPA and PNNL. Fig. 6(a) and Fig. 6(b) show governor and oscillatory response obtained from actual and model-based active power response. The results obtained for oscillatory and governor response is presented next. 1) Metrics for oscillatory response: Using the methodology described in the earlier section, metrics were calculated for validating the model-based oscillatory response of the generator. Before performing modal analysis, the signals were downsampled to 5 samples/sec and also frequency components lower than 0.1 Hz were removed. Using this pre-processed measurements, system modes and mode shapes were estimated for both actual measurements and model-based response. The model order selection is very critical to the proposed method as it can significantly affect the metrics for quantifying the validation results. For both actual measurements and modelbased simulated data, model order of n = 22 was chosen that gave the best fit between the original and reconstructed data as shown in Fig. 7(a) and Fig 7(b). Table I and II give the mode estimates for the PMU and model-based simulated measurements. For metric calculations, mode estimates having energy less than 5% of the highest energy were not considered.   Using (12) and (16), metrics for validating model-based oscillatory response is given in Table III. Based on the metric calculated for magnitude component of the oscillatory response, it can be said that the dynamic model does not accurately represent the model that generated the PMU measurements and requires calibration. This is also illustrated in Fig. 8(a) and 8(b) that compare the contribution of two dominant mode estimates to the PMU measurements and simulated generator response. As seen in these figures, the contribution of the two modes to the PMU measurements and generator response do not have a good match. However, the phase component of the oscillatory response matched well based on the metric calculated. By performing sensitivity analysis, model parameters that affect the magnitude of the oscillatory response can be identified for model calibration.  2) Metrics for governor response: Using governor response extracted from actual and model-based response measurements, metrics were calculated comparing the actual and model-based governor response given in Table IV. Based on these metrics, it can be said that model parameters that can increase the peak-value of the governor response needs to be calibrated. These parameters were identified using sensitivity analysis as will be discussed later.

B. Real-world data
The proposed methodology was also implemented using real PMU measurements recorded in Western Interconnection. Due to the confidentiality agreement, only limited information is available about the data. Fig. 9 compares the active-power response corresponding to actual and model-based response. The methodology described earlier were used to obtain the metrics, which are shown in Fig. 10(a) and 10(b), along with the respective responses. For modal analysis, model order of n = 18 and n = 16 was chosen for actual and model-based response with GoF metric equal to 0.88 and 0.84 respectively. Table V provides  governor response, it can be seen that the mismatch between the actual and model-based response comes from delay, which could not have been known by using a simple root mean square metric. Based on metrics for oscillatory response, it can be said that the model parameters need to be calibrated to improve oscillatory response of the model. C. An example illustrating the application of proposed metrics to the sensitivity analysis and model calibration In this paper, simulated-data example is used to illustrate the application of proposed metrics for sensitivity analysis and rule-based model calibration. In this example, all model parameters were changed by specific % and the corresponding change in the error metrics were calculated with respect to the original error metric using where X is the error metric for which sensitivity analysis is carried out. The sensitivity analysis results for the generator machine model (GENROU) parameters are summarized in Table VI. Table VII provides sensitivity analysis results for the selected parameters of the governor model (GGOV1) that were significant. The change in the exciter and stabilizer model parameters did not results in any significant changes to the active power response and therefore the results are not included. Based on this sensitivity analysis results, model calibration was carried out using rule-based method as described in [20].   By increasing the value of inertia-constant (h), the magnitude component of the oscillatory response showed a better match with the actual oscillatory response. For improving governor response, first the integral gain (kigov) was increased and then the proportional gain (kpgov) was adjusted to minimize error associated with the peak-time. The model-based active power response obtained using this calibrated model is shown in Fig.  11 along with the error metrics. As seen in this figure, the error metrics is much improved as compared to the original model. One thing must be noted here that, the objective of this example is only to illustrate how these proposed metrics and sensitivity analysis can be helpful for model validation and calibration, and not model calibration itself. Model calibration will require the validation using multiple events, which is beyond the scope of the paper.  This paper proposes advanced performance metrics for quantifying model validation results in a rigorous manner by breaking down generator active power response into slow governor response and fast oscillatory response. The results obtained using both simulated and real-world data validate the effectiveness of the proposed metrics in distinguishing a good model from the one that requires calibration. These proposed metrics analyze several aspects of generator dynamic response as compared to other existing metrics, and therefore provides more accurate results. Furthermore, generator model parameters that can help improve specific aspect of generator dynamic response, as given by error metrics, can be identified by performing sensitivity analysis using proposed metrics. This has been illustrated in the paper using simulated test-case based example. The proposed metrics and sensitivity analysis can also be further used for rule-based model calibration. A simple example is used to highlight this application in this paper. To conclude, proposed metrics and sensitivity analysis can be very useful tool for model validation and calibration.