Security-Constrained Unit Commitment for Electricity Market: Modeling, Solution Methods, and Future Challenges

This paper summarizes the technical activities of the IEEE Task Force on Solving Large Scale Optimization Problems in Electricity Market and Power System Applications. This Task Force was established by the IEEE Technology and Innovation Subcommittee to first review the state-of-the-art of the security-constrained unit commitment (SCUC) business model, its mathematical formulation, and solution techniques in solving electricity market clearing problems. The Task Force then investigated the emerging challenges of future market clearing problems and presented efforts in building benchmark mathematical and business models.


I. INTRODUCTION
F ERC landmark Orders 888 and 889 gave rise to the for- mation of independent system operators (ISOs) in the late 1990s.The ISOs formed wholesale electricity markets to foster competition among generating resources through hourly and sub-hourly prices.Grid operators must balance the instantaneous generation and consumption of electricity over various timeframes through control, dispatch, commitment, and procurement of energy and ancillary service products.
Maintaining grid security and reliability is a critical component of all electricity markets.In addition to balancing supply and demand under various system conditions, ISOs need to maximize the efficient use of the region's resources and transmission network through day-ahead and real-time markets and additional reliability commitment processes.The core of the market clearing and reliability commitment processes is the security constrained unit commitment (SCUC) and securityconstrained economic dispatch (SCED) software.The market clearing optimization software enables ISOs to unlock billions of dollars of benefits to society [1].
The SCUC problem has been studied extensively in the literature.Mathematically, the SCUC problem is a nonconvex, large-scale, mixed-integer optimization problem with a large number of binary and continuous variables as well as a series of prevalent equality and inequality constraints.The objective is the maximization of social welfare.However, traditionally the demand side is mostly inelastic and the objective is usually formulated as minimizing production cost plus the constraint violation penalties.The production cost usually includes startup, no-load, and incremental energy costs represented by piecewise linear functions.The SCUC model of a large-scale ISO such as MISO and PJM may have 36 time intervals with over 1000 generators and 10000 monitored transmission elements.In addition, enforcing the N-1 security standard in market clearing makes the SCUC problem almost impossible to solve directly, often requiring a decomposition scheme or iterations between optimization solver and network security analysis software.
SCUC constraints include resource-level constraints and system-wide constraints.Resource-level constraints are used to represent the physical operating characteristics of generators and other resources.They include capacity limits, ramp rate limits, minimum run times, minimum down times, etc.These constraints may link binary and continuous variables across multiple time intervals.System-wide constraints mainly include power balance constraints, reserve requirement constraints, and transmission constraints.Transmission constraints are enforced to meet North American Electric Reliability Corporation (NERC) N-1 and other reliability standards.They are usually associated with continuous variables and are often decoupled by time intervals.
In the early years of the electricity markets, SCUC used in the market clearing process was mostly solved by the Lagrangian Relaxation (LR) method.With the development of more advanced commercial mixed-integer linear programming (MILP) solvers, it became practical to replace LR with MILP with the first deployment by PJM in 2004 [150].Since then, most ISOs have switched to using MILP solvers to solve SCUC problems.Nevertheless, in view of the increased complexity in the presence of intermittent renewables, distributed energy resources (DERs), and sub-hourly commitment, there is a renewed interest in the study of LR-based decomposition and coordination approaches.
Traditionally, SCUC solution efficiency is usually driven by the number of binary variables, the number of security constraints, and the presence of loosely coupled decision variables such as those for storage state-of-charge management and combined-cycle generation, etc.With rapidly evolving powergrid operations, the drastically increasing complexity of market clearing models requires a continuing focus on improving the performance of SCUC optimization models and methods.
This paper focuses on the deterministic SCUC problem that is currently used in the electricity market clearing.Sections II and III review the state-of-the-art of the SCUC business model, its mathematical formulation, and solution techniques.Sections IV and V focus on the emerging challenges and efforts in building benchmark mathematical and business models.Section VI is the conclusion.

A. Business Model
The electricity market provides a mechanism for market participants to buy and sell electrical energy at prices established through a competitive auction process designed to meet energy demands (i.e., loads) and system reliability requirements with the least-cost resources available, or through contractual bilateral transactions.The ISO administers day-ahead and real-time markets, resulting in a two-settlement process.Due to the greater number of hours that must be considered when solving day-ahead SCUC, this usually presents the most computational challenges.
r In practice, the SCUC problem includes two components: a MILP optimization problem and a security analysis problem [2].Theoretically, the two components can be combined into one multi-interval Alternating Current Optimal Power Flow (AC-OPF) problem.In practice, the industry has taken advantage of the advanced technologies for solving Direct Current Optimal Power Flow (DC-OPF) and MILP, arriving at the current structure as follows: r ISOs clear the market to optimize multi-interval active power schedules.Reactive power scheduling and voltage controls are mainly managed in a separate process outside of the market and applied as linear constraints on active power [3].Losses are included in the load forecast.The impact of losses on efficient dispatch is represented by "penalty factors", as a linear marginal loss function [4], [182].
r SCUC optimization and network analysis are solved itera- tively.The network analysis part is sometimes referred to as the simultaneous feasibility test (SFT).
r SFT identifies violated base case and contingency con- straints using power flow and contingency analysis.New constraints from SFT are linearized and reported using generation shift factors (GSFs).
r The new constraints are added to solve the next iteration of the SCUC optimization problem.Some applications may also update marginal loss factors using updated power-flow results [7].In addition, PJM and MISO also use SCUC for after-thefact performance analysis, where full-day sub-hourly SCUC is solved using actual data to benchmark production performance and provide guidance to operational practice under uncertainty.PJM has reported significant savings [18].
As the mix of resources has evolved in recent years, the traditional SCUC model has started to show limitations in representing emerging resources [8], [9], [10].Consequently, there is a tradeoff between the need for accurate resource modeling and the need for reducing computational burdens.Section IV discusses emerging challenges with the current MILP solver.

B. Basic Mathematic Formulation
At a high level, SCUC can be seen as minimizing the operating cost of the given set of resources, subject to the physical constraints of both the system as a whole and individual resources, as follows:

Subject To :
g∈G Here the set of equations (SCUC.3)abstractly represents the physical constraints of individual resources g, with continuous production variables p g , binary status variables u g , and associated cost c g .The equations (SCUC.2) abstractly represent the physical and/or operational constraints on the transmission system, e.g., those that the ISO determines.These include energy balance, reserve requirements, and transmission limits.(SCUC.1)minimizes the total cost to operate the system.
A 3-binary SCUC formulation is presented in the Appendix.

C. Modeling of Security Constraints
The security constraints in the context of SCUC refer to transmission limit constraints under both normal operating conditions and contingency conditions.There are primarily two formulations used in academic study and industry to formulate power flows and model transmission constraints.The first one, traditionally referred to as the "B-theta" formulation, directly models voltage phase angles and power flows.The second way to model transmission constraints is to use the power transfer distribution factor (PTDF) or GSF matrix.It is nearly impossible to model N-1 constraints in the phase-angle formulation.As algorithms (e.g., [19]) have been developed to effectively filter unnecessary security constraints and new PTDF-based formulations (e.g., [20], [172], [173]) are developed for modeling flexibility, there is little benefit to use phase-angle formulations anymore.Practical ISO applications mainly use the PTDF-based approaches.

A. Strengthening MILP Formulation for Performance Improvement
Most researchers formulate SCUC as a MILP model.Researchers have attempted to strengthen the MILP formulation of SCUC from the perspective of two metrics: compactness and tightness [22].
An important method for improving compactness is to remove redundant constraints and non-binding constraints [34], [35], [36], [37], [131].Another way to improve the compactness is to reduce the number of binary variables.Similar plants are grouped into clusters to reduce binary variables in [38].The work [39] further reveals the errors in clustering and provides near-optimal schedules on a plant level.
An ideally tight MILP formulation yields a linear programming (LP) relaxed feasible region that is identical to the convex hull of the feasible integer points.In this case, the optimal solution of MILP can be obtained simply via solving LP relaxation [21].The tightness is usually measured with the "integrality gap" (IntGap) [22], [23], [24].Another measurement is the integrality gap in the root node (RootIGap) [25].
The unit commitment polytope has been studied extensively and there are different formulation approaches for individual generators.The formulations are different depending on distinct modeling approaches for the binary logic restrictions, minimum up/down time formulation, ramping up/down constraints, startup costs, and piecewise linear production costs.
For instance, there are different ways to choose binary variables.A compact formulation using one binary variable (1-bin: on/off) is presented in [28].Based on these single binary decision variables, alternating up/down inequalities are presented in [31] to strengthen the minimum-up/down polytope by providing its convex hull description.In other papers (e.g., [22], [23], [30]), three or two binary decision variables for each generator are utilized.For the three-binary decision variable approaches, the convex hull of the minimum-up/-down time polytope is introduced in [40].When the studies are extended to provide strong formulations including ramping constraints, new families of inequalities are presented in [23], [41], and [42] to tighten the ramping polytope.Reference [29] replaces the on/off variable in 3-bin with one binary variable depicting the state transition.In [41], the convex hull descriptions for the two-period and three-period cases have been provided.When all constraints are considered, there are no convex hull descriptions for the general case setting by using the one-binary, two-binary, or three-binary decision variables.Probably the best current integral UC formulation for individual generators considering all the constraints plus piecewise linear objective function is provided in a higher-dimensional space shown in [43], which is formulated by introducing binary decision variables to represent the "on" and "off" intervals (instead of individual time periods), requiring O(T 2 ) binary decision variables and O(T 3 ) continuous decision variables.A few additional recent papers have analyzed the polyhedral aspects of the interaction of the system constraints with the generator's technical constraints [33], [44], [45].Finally, two recent papers attempt to assess the current state-of-the-art in unit commitment formulations [21], [46].
Besides the constraints, [45] and [47] report tightening the piece-wise linear formulations for the incremental energy cost function in the objective to achieve the convex envelope of the objective function.This is further incorporated in [43] in a higher-dimensional space.A tighter linear approximation of the nonlinear objective function of units is presented in [32].Most of the above works present tightened formulations, possibly with proofs.In [33], a systematic approach is developed to tighten formulations.
In practice, industrial-scale UC problems are today solved by utilizing off-the-shelf commercial MILP solvers such as CPLEX, GUROBI, or XPRESS.Such solvers employ a heavily embellished version of the LP-based branch-and-cut (B&C) algorithm.The compact formulations and cutting planes can be embedded in the B&C framework to speed up the algorithm for solving the problem.Hence, benchmarking on realistic and diverse test cases is essential to quantify the performance of a SCUC formulation with a given solver.For a comprehensive overview of modern B&C methods, see [159].

B. Decomposition and Coordination Methods
LR was traditionally used to solve UC problems by exploiting separability, with subgradient methods to update multipliers [48].Standard LR, however, suffers from difficulties such as significant computational requirements to obtain subgradients and zigzagging of multipliers.Major difficulties of LR have been overcome by the recently developed surrogate LR (SLR) [49], in which surrogate subgradient directions are obtained after solving one or a few subproblems subject to the simple "surrogate optimality condition" to ensure that the relaxed problem is sufficiently optimized.Computational effort and multiplier zigzagging are much reduced since only one or a few subproblems are solved at a time.SLR has been further improved by adding absolute value penalties (which can be exactly linearized) on constraint violations to accelerate convergence [50].

C. Other Decomposition Methods
The ways to decompose a UC problem mostly fall into two categories: temporal decomposition [51], [52] and geographic decomposition [53].As the modern MILP solvers have become very efficient in solving SCUC, especially with the help of well-designed algorithms, e.g., machine learning enhanced optimization [54], decomposition does not give much benefit to solving SCUC in terms of solution time.However, a decomposable SCUC is useful for decentralized resource scheduling, such as inter-market energy exchange and loop flow mitigation [56], [57].
For large-scale day-ahead market clearing problems, in addition to binary variables associated with resource commitments, large numbers of transmission constraints coupled with a large number of continuous variables (usually from virtual transactions) can also drive computational challenges.In [75], the heuristic binary reduction and transmission constraint decomposition methods are developed and implemented at MISO as a heuristic "backup" method for the rare instance that the MILP solver faces difficulty reaching its tolerance.In [76], this "polishing" heuristic method is further integrated with the optimization solvers as a warm start technique through "MIP start" and lazy constraint settings.
Several effective primal heuristic methods including the enhanced polishing method, variable fixing, and enhanced RINS (RINS-E) are developed under the High-Performance Power Grid Optimization (HIPPO) project funded by the Department of Energy's ARPA-E program [77], [78].The concurrent SCUC architecture built under HIPPO allows these approaches to be implemented concurrently with significant performance improvement.

IV. EMERGING CHALLENGES
The SCUC model mainly consists of resource and transmission constraints.With the evolution of portfolio, traditional generators are augmented by emerging resources.Each new resource type introduces certain unique challenges.This section discusses emerging challenges driven by future resource and grid changes such as demand response and massive DERs [11], short-time-interval scheduling [12], increasing binding transmission due to insufficient security margins [13], and diverse operational characteristics [16].The interdependency with the natural gas network and the market clearing price mechanism are also discussed.

A. Combined Cycle
Configuration-based combined-cycle gas turbines (CCGTs) make up the majority of all new generation capacity created over the last decade.The fast response time and operational flexibility of CCGTs make them valuable resources for mitigating the variability and uncertainty of intermittent renewable energy.Six CCGT models currently exist in academic research and industry practice: Aggregate Model [79], [80], Pseudo Unit Model [79], Component-Based Model [80], [81], Configuration-Based Model [47], [80], [82], [83], [84], [85], Edge-Based Model [86], [87], and Configuration-Component Based Hybrid Model [88].The configuration-component based hybrid model maps the relationship between the configuration-based model and component-based model without introducing additional binary variables, allowing more accurate constraint formulations on the configuration or component level.
In [89], a transition curve model is presented to reflect the transition process.The column aggregation methods in [47] and warm start methods in [76] are useful for SCUC problems in general, and particularly helpful for the configuration-based CCGT model.
In [91], the Surrogate Augmented LR (SALR), an earlier version of the Surrogate Absolute Value LR (SAVLR) [50], is developed for effective coordination and accelerated convergence.The method may outperform MIP solvers in finding optimal solutions within tolerance for stressed MISO cases.

B. Electric Energy Storage
FERC Order 841 [92] requires each ISO to establish a wholesale market participation model for electric storage resources.The Order does not specifically require ISOs to optimize the state of the charge (SOC).Recently, storage-plus-generation colocated hybrid resources have been increasing as a share of newly proposed projects and participants are seeking self-optimizing opportunities [93].
Some ISOs (e.g., PJM) optimize pumped storage hydro in the day-ahead market [94].Even though most ISOs do not currently optimize SOC for battery storage, the SOC is an essential aspect of the operating characteristics of storage.As discussed both theoretically and empirically in [95], the combination of mutually exclusive charging and discharging modes together with SOC limits and round-trip efficiency strictly less than one presents a computational challenge.This subsection focuses on the storage model in day-ahead SCUC.It requires a more sophisticated representation of the potential future value of stored energy that may be realized in the real-time market when the solution window is shorter than the storage duration.

1) Pumped Storage Hydro (PSH):
In some RTOs, the PSH may simply offer to generate power and bid to buy power for pumping, analogous to the participation of thermal generators, but typically with some additional features such as maximum daily energy for the generating mode to represent SOC limits.The maximum daily energy constraints are also used for other fuel-constrained resources [6].Some other RTOs (e.g., PJM) have partially integrated the representation of PSH characteristics into their day-ahead unit commitment model [94].PSH optimization in SCUC may introduce computational challenges.
Besides formulating and solving PSH directly with a MILPbased SCUC model, PJM optimizes PSH after the MILP-based day-ahead SCUC clearing engine using a customized non-MILP based software [7], [94].
2) Battery Storage: In general, even though battery storage can be smoothly dispatched across charging and discharging modes, battery storage formulations still require binary variables due to the mutual exclusiveness between charging and discharging modes.Formulations for battery storage are discussed in [106] and [107] with exact relaxation methods based on assumptions on the charging and discharging offers.The methods may not be general enough to cover all possible offers or when the SOC limits are binding.References [108] and [109] prove sufficient conditions under which the battery storage formulation can be relaxed to the convex form in the economic dispatch model.NYISO presented its modeling of storage in the dispatch-only model with binary variables introduced for battery storage [111].
Reference [110] investigates the impact of optimizing battery storage on computational performance and the potential economic benefit, with a case study based on MISO day-ahead SCUC.It investigates the impact of battery storage binary variables, valid inequality constraints from [95], and explicit representation of battery SOC on the performance of day-ahead SCUC.

C. Demand Side Participation and Distributed Energy Resource
Most ISOs allow demand side to participate in the energy and ancillary service markets.However, the participation has been relatively low.The policy discussions may be found in [191], [192].Dispatchable demand may be incorporated into the market clearing process by changing the objective to maximizing social welfare, and adding variables and constraints associated with dispatchable demands [193], [112], [113].
The power industry has been exploring effective approaches to facilitating the participation of DERs in the ISO market, including the recent FERC Order 2222 [11].However, under the current locational marginal price (LMP)-based electricity market practice, resources are usually only allowed to aggregate under a single elemental pricing node (Epnode) where LMP is calculated.DERs may be incorporated into SCUC formulation as aggregation of homogenous or heterogenous generation.This may introduce a large number of small resources with a large number of variables and increase computational complexity.Practical SCUC MILP solutions are usually solved with a tolerance of a non-zero MILP gap.MILP solvers may not solve the binary variables associated with a small resource to the optimal if the total cost of the resource is less than the MILP gap.MISO [75] has a "polishing" method to refine commitment variables for out-of-money units, and NYISO [6] has a method to fix commitment variables for large resources and solve the commitment of small resources to a smaller MILP gap.In addition, the work in [112] and [113] leads to the possibility to represent demand response (DR) assets or even DERs with convex polytope approximation, which can potentially address the small resource issue without considering transmission constraints.However, aggregating small-scale DERs across locations with different LMPs introduces conflicts with the fundamental principle of the nodal electricity market.
Currently, distribution factors [114] are used to describe the ratios of individual DERs to the total power of the aggregated commercial pricing node (Cpnode).The distribution factorbased approximation model for aggregated DERs may introduce inevitable errors to power flows of individual transmission lines, leading to market clearing solutions that are suboptimal or even infeasible to the actual physical system.Some suggest that state estimation results may be used to update distribution factors for the rolling real-time SCED (RT-SCED) calculations.However, this feedback strategy may induce oscillations of LMPs and dispatches of generators and DERs as observed in recent studies [115], [116].
The distributed transmission system operator (TSO) and distribution system operator (DSO) coordination approaches have been studied in the literature [117], [118], [119], [120].Alternatively, a feasible region projection-based approach is recently explored [121], [122] to study the integration of DER-penetrated distribution systems into ISO market operation.

D. Virtual Transactions
Virtual transactions are financial contracts awarded in the dayahead market and settled at the differences between day-ahead and real-time prices.Most ISOs include virtual incremental offers (INCs) that are similar to generation offers and virtual decremental bids (DECs) that are similar to demand bids.Some ISOs also implemented up-to-congestion bids (UTCs) (see detail in [123]).The mathematical formulation of a virtual INC or DEC is relatively simple, only involving a continuous variable defined for a specific time interval at a pricing node with a dispatch range between 0 and the maximum offered MW.However, a large number of virtual transactions may increase the number of decision variables and the time for a MILP solver to solve each LP [75].It has been shown in [47] that combining variables with the same coefficient in transmission security constraints (i.e., column aggregation) could improve computational performance.
ERCOT also allows the blocking of UTC bids that require binary variables [124].Inherently UTCs are a very low-risk transaction as explained in [125].The number of UTCs can be very large and may have a more significant impact on congestion than INC/DEC bids.They may cause slow convergence between SCUC and network security analysis SFT [124], [125].It is also debatable whether virtual INC/DEC bids and UTCs can cause line loss.UTCs often cause significant marginal loss convergence difficulties.
In general, the model of INCs and DECs can be applied as the simplest DER model.The experience with virtuals may provide insights on future computational impact from DERs.

E. Sub-Hourly Unit Commitment
Day-ahead SCUC is traditionally formulated and solved with one hour as a time interval.Increasing dynamics on the grid prompted the industry to consider whether SCUC with subhourly intervals may improve system flexibility and reliability.However, it is much more complex than hourly UC because 1) the increased number of periods leads to larger problem sizes, and 2) the much-reduced unit ramping capabilities per period result in more complicated convex hulls.Existing ramp constraint formulations, using a 3-binary SCUC formulation, usually capture the convex hull for only 2 or 3 periods [41].
In [77], the approach of concurrently solving the hourly interval SCUC to warm start 15-min interval SCUC is developed in HIPPO.The greedy method with sequential neighborhood search can effectively find high-quality incumbent solutions.
The recently developed SAVLR method [50] overcomes the major difficulties of traditional LR and has accelerated convergence through absolute-value penalty terms on constraint violations.A novel approach is further developed in [127] to only relax system demand constraints and keep system reserve and transmission capacity constraints as "soft constraints" following the approach in [130].Furthermore, the Ordinal Optimization (OO) concepts [128], [129] are applied to generate good-enough subproblem solutions quickly without calling B&C in most cases.The results demonstrate that SAVLR+OO+B&C obtains near-optimal solutions in a computationally efficient manner for multiple difficult sub-hourly UC cases, significantly outperforms B&C, and is robust.

F. Large Scale Transmission System
With the penetration of distributed resources and more binding transmission constraints, it is important to investigate better formulations for transmission constraints in SCUC modeling.The work in [14] presents a fast SCUC for large-scale power systems with heuristics.In [15], transmission-constrained UC is presented, which considers both electricity and district heating networks.However, it is still a difficult problem to formulate.DERs introduce new challenges to transmission and distribution coordination.
Existing research focuses on improving the transmission security constraint representation by identifying active constraints while excluding the redundant ones [131], [132].It has been shown in [47] that column aggregation could improve computational performance.References [133] and [134] further explore column-based aggregations for complicated transmission security constraints.
In [116] and [135], the column-based aggregation approach is used to aggregate variables of pricing nodes with similar sensitivity factors based on a pre-defined threshold.It can reduce variables in transmission security constraints thresholds while ensuring feasibility to the original network security constraints.
In [75], it shows that usually only less than 20% of the pre-identified "watchlist" transmission constraints are binding in MISO day-ahead SCUC.The Gurobi solver provides three different settings on the "Lazy" constraint attributes [136].Using the lazy constraint settings can reduce the computational burden while maintaining the original problem definition.In [76], a "warm start" method is developed to identify unlikely-to-bind transmission constraints and set them as "lazy".In [90], the screening method is combined with lazy constraints to leverage mathematical modeling and historical data.
In practice, SCUC is solved interactively with the simultaneous feasibility test (SFT).In traditional approaches, partial refactorization algorithms are used [137].The extremely fast PYSFT using Python open-source sparse matrix library implemented in HIPPO [78], [171] takes advantage of the fact that the base case and a contingency case only differ by a small number of elements.The Sherman-Morrison-Woodbury formulation [162] can be used to update the inverted low-rank contingency admittance matrices.Faster low-rank factorizations enable the SFT solver to be called in the process of solving MILP through MILP solvers like GUROBI and CPLEX via their callback API.The combination of low-rank factorization and concurrent approach drastically reduced computation time in experimenting on large-scale SCUC cases [78].

G. MW Dependent Ramp Rate
Traditionally, a unit's ramp rate is modeled as a constant, which means it does not change with respect to the output level of a generator.Unfortunately, this can sometimes be far from reality.For instance, a CCGT can be represented by one pseudo generator with an operating range covering different configurations.When the generator is dispatched from one configure to another, transition time is often required, indicating a very slow ramp rate.In addition, the ramp rates in different configurations are often different.
To improve the modeling accuracy, the MW-dependent ramp rate is introduced.In [138], the MW-dependent ramp rate curve is represented by a piecewise step function of the generation output level.A single-interval dispatch model based on special ordered set type 2 (SOS2) constraints is developed for the ISO-NE market.It can be expanded for the multi-interval SCUC problem with respect to both energy and reserve products.The formulation can be solved with the built-in SOS2 variables in a commercial MILP solver.However, the SCUC performance using the built-in SOS2 method can be degraded significantly as the number of ramp rate curves increases.An efficient formulation such as [139] and/or solution methods need to be further investigated.
NYISO models its MW-dependent ramp rate curve using binary variables to improve performance.A piecewise linear function can be easily modeled using binary variables [140].NYISO experiences show that performance is significantly improved with the binary variables.

H. Gas Network
With the increasing interdependency between the power system and other infrastructures, electricity market clearing is also impacted by the physical restrictions and market operations of those infrastructures.One such example is the interdependency of the electricity grid and natural gas network, and the interdependency is intensified by the increasing numbers of gas-fired generating units and flexible multi-energy users that participate in both electricity and gas markets [174], [175], [176].For instance, gas prices of the natural gas market influence prices of the electricity market through bidding strategies of natural gas units [177].Gas availability, which depends on the gas network, may compromise the operational security of the electricity grid [178].Also, interval optimization is employed in [179] to address uncertainty in the interdependent operation of both networks.Coordinating the two markets has thus attracted increasing attention in recent years and is believed to be able to derive more satisfactory solutions than optimized separately.
Various strategies have been discussed in the literature to coordinate the electricity grid and the natural gas system in terms of energy market design and pricing in addition to market scheduling [175], including: (i) incorporating natural gas network constraints in electricity market clearing models; (ii) incorporating dynamic gas consumptions of the electric power system in natural gas market clearing models; (iii) sequentially optimizing the electricity grid and the natural gas network; (iv) co-optimizing the electricity grid and the natural gas network; and (v) time alignment of electricity and natural gas markets.
Nevertheless, the coordination and integration mechanisms of these two markets are still at a preliminary stage, because electricity and natural gas pricing are currently settled in two markets that are organized separately.Moreover, an energy company participating in both electricity and gas markets usually has independent decision-making processes with two distinct optimization models, although strategic behaviors of gas producers/consumers in the gas market may also influence electricity market operations.Advanced energy management strategies and market interactions involved in the two markets should be further investigated to capturing their distinct physical restrictions and market operations such as speed of energy flows, capability of large-scale energy storage, and flexibility in network operations.

I. Market Clearing Price
Locational marginal pricing (LMP) has become the dominant pricing scheme in all major US and some European electricity markets [4], [17], [180], [181], [182].The concept is originally presented in [17], [180] to achieve optimal resource allocation, provide economic incentives and manage congestions.LMP is often derived from a convex security-constraint economic dispatch (SCED) problem, which takes the commitment solution from a SCUC problem.LMP provides supports for the SCED solution, but revenue from LMP may not cover the total cost including the commitment cost.It is shown in [183] that LMPs plus make-whole payments are equilibrium prices for generation resources.
Different pricing schemes for markets with non-convexities are presented.A comprehensive review is provided in [187].Convex hull pricing (CHP) [184] is a pricing scheme that minimizes uplift payments over all possible uniform prices.Various methods are presented to solve CHP [185], [186], [187], [188].Average incremental cost (AIC) pricing is presented to eliminate the make-whole payment [189], [190].
Renewable integration creates new challenges in the pricing of electricity due to low marginal costs of renewables.Improving price signal to manage increasing variability and uncertainty becomes an important research topic in recent years.This part will be covered in a future report.

A. Formulation
In its general form, SCUC is an NP-hard optimization problem [141].Therefore, benchmarking against realistic test instances is important in any computational approach to SCUC.Because SCUC is typically solved with off-the-shelf MILP solvers, improvements to the base formulation (SCUC.1)-(SCUC.3)have been of special interest.Many researchers attempt to scale up an instance of the SCUC problem by considering a basic set of generating units and replicating them several times [28], [23], [22], sometimes with an objective function perturbation [21].Such approaches introduce artificial symmetry into the SCUC instance, making the corresponding MILP problems much more difficult for MILP solvers, even those with sophisticated symmetry detection techniques [142].
More recently, more realistic and diverse sets of publicly available generator data have been easier to come by.Perhaps the first instance of this is the test system provided by FERC [143].The RTS-GMLC test system [144] includes transmission, reserve, load, and renewable generation for an entire synthetic year, however, it is small-scale, consisting of only 158 units and 73 buses.The Power Grib Lib -Unit Commitment [145] (pglib-uc) test library includes modified versions of both test systems, along with an additional test system based on publicly available data from California ISO, for a total of 56 cases.However, no transmission network data is included.Finally, the aforementioned test instances and additional cases with transmission network based on MATPOWER OPF instances are all available as part of the UnitCommitment.jlpackage [146].
Most papers consider a fixed set of formulations for SCUC.Perhaps the largest such set of formulations is compared in [46], which considers 12 base formulations against the 56 test cases from pglib-uc.Against its recommended formulation "Tight (T)", [46] undertakes a comprehensive analysis, swapping a single component of the formulation at a time for a competitor.Additionally, [46] introduces the idea of exploring the space of SCUC formulations.The different formulations explored in [46] are available as Pyomo [147] models as part of the Egret software package [148] -enabling the exploration of over 100000 different SCUC formulations.

B. UnitCommitment.jl
The IEEE Task Force on Solving Large-Scale Optimization Problems in Electricity Markets has started the development of UnitCommitment.jl[146], [149], an extensible open-source Julia/JuMP optimization package for the problem.The package contains four main components.
First, the package includes an extensible and fully documented JSON-based data format to describe the most important aspects of the problem.It can also be easily extended with new sections and new data fields.
Second, the package includes two classes of diverse collections of large-scale benchmark instances.The first class is composed of instances previously presented in the literature.Included are both randomly generated instances used in previous studies [21], [163], [164], as well as instances based on more realistic systems [46], [143], [144].The package also includes a second class of newly developed instances, originally based on realistic optimal power flow (OPF) test cases from MATPOWER and augmented through machine learning models trained on public real-world bid and offer data.
Fourth, to simplify the task of measuring the performance impact of new solution methods and/or formulations, UnitCommitment.jlincludes automated benchmark scripts to perform statistic tests.
We refer to the package documentation for more details [160].The package is being developed openly and collaboratively on GitHub [161], and the task force strongly welcomes comments and suggestions, as well as code and instance contributions from industry and academia.

VI. CONCLUSION
This paper summarizes the technical activities of the IEEE Task Force on Solving Large Scale Optimization Problems in Electricity Market and Power System Applications.It first reviews the state-of-the-art of SCUC business model, its mathematical formulations, and solution techniques in solving electricity market clearing problems.It then investigates the emerging challenges of future market clearing problems and presents efforts in building benchmark mathematical and business models.
In view of the rapid transformation being witnessed by the industry, the Task Force recommends the following focus areas for future research: r Emerging new resource modeling and formulation (e.g., DER, storage, multi-configuration resources), as well as distributed or hybrid (a mixture of centralized and distributed) architecture.
r New solution methods and techniques, including inte- gration with machine learning.Even though commercial MILP solvers are powerful and can generally work well with proper formulation, the research community needs to work closely with the industry to be prepared for emerging challenging problems.
r Efficient interaction between optimization and network security analysis, including transmission and distribution coordination.
r With the increasing participation of inverter-based re- sources, reactive power and voltage issues need to be investigated within the SCUC framework.
r Managing uncertainty and variability with renewable pen- etration is not discussed extensively in this paper.Subhourly interval, advanced reserve models, stochastic/robust optimization, and the interplay between forecasting models and SCUC are research areas to address these issues.

APPENDIX
There are many different formulations for SCUC.In this Appendix, a 3-binary formulation is used to illustrate the SCUC problem.This formulation includes three binary variables for each generator at each interval: startup, shut down, and on/off commitment.Table I   Constraints ( 2)-( 6) are explicit representations of the system operating constraints (SCUC.2) above, and constraints ( 7)- (19) are explicit representations of the operating constraints for individual generators (SCUC.3)above.Specifically, constraints (2) represent the total system energy balance, constraints (3) are the regulation reserve requirements, constraints (4) are the "spinning" or "on" contingency reserve requirements, and constraints (5) are the operating reserve requirements.The reserve requirement constraints (3)-( 5) are "stacked" in such a way that a higher-quality reserve product can be utilized to fulfill the requirement for a lower-quality reserve product.In addition to giving the optimizer more flexibility in meeting reserve requirements, this also ensures that the reserves, when priced, will be in the order of reserve quality.Constraints (6) describe the security requirements.Turning to the generator constraints, constraints (7) are the minimum up-time requirement, constraints (8) are the minimum down-time requirements, constraints (9) are the "logical" requirements linking "on" and "off" statuses with start-up and shut-down decisions, ensuring that a start-up (i.e., v gt = 1) switches the status of the generator from "off" to "on" and a shutdown similarly switches the generator status, constraints (10) enforce when a "hot" or "warm" start is permitted given the last shut-down period, constraints (11) force exactly one start-up type to be chosen when the unit starts, and constraints (12) impose the binary restriction on the indicator variables (on, start, stop, start-type).
Finally, the constraints involving the continuous generator variables are presented.Constraints (13) ensure that the total of both dispatch and reserve procurement does not exceed the generator's maximum capacity; constraints (14) similarly ensure energy and reserve dispatch not to go lower than a generator's rated capacity.Together, constraints (13) and (14) enforce that a generator only produces power when it is "on", i.e., u gt = 1.Constraints ( 15) and ( 16) are ramping constraints, which enforce that the power output of an individual generator does not change too rapidly between successive periods, including start-up and shut-down ramping when the generator is turned on or off.Constraints (17) and (18), along with the objective, model the convex piecewise linear cost curve.Finally, constraints (19) and (20) place operational bounds on the amount of reserve that can be procured when a generator is on, and constraint (21) limits the amount of reserve procured when a generator is off.
Authorized licensed use limited to: UNIVERSITY OF TENNESSEE LIBRARIES.Downloaded on April 18,2024 at 21:59:10 UTC from IEEE Xplore.Restrictions apply.
Authorized licensed use limited to: UNIVERSITY OF TENNESSEE LIBRARIES.Downloaded on April 18,2024 at 21:59:10 UTC from IEEE Xplore.Restrictions apply.
Authorized licensed use limited to: UNIVERSITY OF TENNESSEE LIBRARIES.Downloaded on April 18,2024 at 21:59:10 UTC from IEEE Xplore.Restrictions apply.
Authorized licensed use limited to: UNIVERSITY OF TENNESSEE LIBRARIES.Downloaded on April 18,2024 at 21:59:10 UTC from IEEE Xplore.Restrictions apply.
Authorized licensed use limited to: UNIVERSITY OF TENNESSEE LIBRARIES.Downloaded on April 18,2024 at 21:59:10 UTC from IEEE Xplore.Restrictions apply.
Authorized licensed use limited to: UNIVERSITY OF TENNESSEE LIBRARIES.Downloaded on April 18,2024 at 21:59:10 UTC from IEEE Xplore.Restrictions apply.
is the nomenclature.
Authorized licensed use limited to: UNIVERSITY OF TENNESSEE LIBRARIES.Downloaded on April 18,2024 at 21:59:10 UTC from IEEE Xplore.Restrictions apply.