Reliability Standard MOD-033-1

MOD-033-1 — Steady-State and Dynamic System Model Validation

A. Introduction
1.

Title: Steady-State and Dynamic System Model Validation

2.

Number:

3.

Purpose:
To establish consistent validation requirements to facilitate the
collection of accurate data and building of planning models to analyze the reliability of
the interconnected transmission system.

4.

Applicability:

MOD-033-1

4.1. Functional Entities:
4.1.1 Planning Authority and Planning Coordinator (hereafter referred to as
“Planning Coordinator”)
This proposed standard combines “Planning Authority” with “Planning
Coordinator” in the list of applicable functional entities. The NERC
Functional Model lists “Planning Coordinator” while the registration
criteria list “Planning Authority,” and they are not yet synchronized. Until
that occurs, the proposed standard applies to both Planning Authority
and Planning Coordinator.
4.1.2 Reliability Coordinator
4.1.3 Transmission Operator
5.

Effective Date:
MOD-033-1 shall become effective on the first day of the first calendar quarter that is
36 months after the date that the standard is approved by an applicable
governmental authority or as otherwise provided for in a jurisdiction where approval
by an applicable governmental authority is required for a standard to go into effect.
Where approval by an applicable governmental authority is not required, the
standard shall become effective on the first day of the first calendar quarter that is 36
months after the date the standard is adopted by the NERC Board of Trustees or as
otherwise provided for in that jurisdiction.

6.

Background:
MOD-033-1 exists in conjunction with MOD-032-1, both of which are related to
system-level modeling and validation. Reliability Standard MOD-032-1 is a
consolidation and replacement of existing MOD-010-0, MOD-011-0, MOD-012-0,
MOD-013-1, MOD-014-0, and MOD-015-0.1, and it requires data submission by
applicable data owners to their respective Transmission Planners and Planning
Coordinators to support the Interconnection-wide case building process in their
Interconnection. Reliability Standard MOD-033-1 is a new standard, and it requires
each Planning Coordinator to implement a documented process to perform model
validation within its planning area.

Page 1 of 11

MOD-033-1 — Steady-State and Dynamic System Model Validation

The transition and focus of responsibility upon the Planning Coordinator function in
both standards are driven by several recommendations and FERC directives (to
include several remaining directives from FERC Order No. 693), which are discussed in
greater detail in the rationale sections of the standards. One of the most recent and
significant set of recommendations came from the NERC Planning Committee’s
System Analysis and Modeling Subcommittee (SAMS). SAMS proposed several
improvements to the modeling data standards, to include consolidation of the
standards (that whitepaper is available from the December 2012 NERC Planning
Committee’s agenda package, item 3.4, beginning on page 99, here:
http://www.nerc.com/comm/PC/Agendas%20Highlights%20and%20Minutes%20DL/2
012/2012_Dec_PC%20Agenda.pdf).
The focus of validation in this standard is not Interconnection-wide phenomena, but
on the Planning Coordinator’s portion of the existing system. The Reliability Standard
requires Planning Coordinators to implement a documented data validation process
for power flow and dynamics. For the dynamics validation, the target of validation is
those events that the Planning Coordinator determines are dynamic local events. A
dynamic local event could include such things as closing a transmission line near a
generating plant. A dynamic local event is a disturbance on the power system that
produces some measurable transient response, such as oscillations. It could involve
one small area of the system or a generating plant oscillating against the rest of the
grid. The rest of the grid should not have a significant effect. Oscillations involving
large areas of the grid are not local events. However, a dynamic local event could also
be a subset of a larger disturbance involving large areas of the grid.
B. Requirements and Measures
R1.

Each Planning Coordinator shall implement a documented data validation process
that includes the following attributes: [Violation Risk Factor: Medium] [Time Horizon:
Long-term Planning]
1.1. Comparison of the performance of the Planning Coordinator’s portion of the
existing system in a planning power flow model to actual system behavior,
represented by a state estimator case or other Real-time data sources, at least
once every 24 calendar months through simulation;
1.2. Comparison of the performance of the Planning Coordinator’s portion of the
existing system in a planning dynamic model to actual system response, through
simulation of a dynamic local event, at least once every 24 calendar months (use
a dynamic local event that occurs within 24 calendar months of the last dynamic
local event used in comparison, and complete each comparison within 24
calendar months of the dynamic local event). If no dynamic local event occurs
within the 24 calendar months, use the next dynamic local event that occurs;
1.3. Guidelines the Planning Coordinator will use to determine unacceptable
differences in performance under Part 1.1 or 1.2; and

Page 2 of 11

MOD-033-1 — Steady-State and Dynamic System Model Validation

1.4. Guidelines to resolve the unacceptable differences in performance identified
under Part 1.3.
M1. Each Planning Coordinator shall provide evidence that it has a documented validation

process according to Requirement R1 as well as evidence that demonstrates the
implementation of the required components of the process.
R2.

Each Reliability Coordinator and Transmission Operator shall provide actual system
behavior data (or a written response that it does not have the requested data) to any
Planning Coordinator performing validation under Requirement R1 within 30 calendar
days of a written request, such as, but not limited to, state estimator case or other
Real-time data (including disturbance data recordings) necessary for actual system
response validation. [Violation Risk Factor: Lower] [Time Horizon: Long-term Planning]

M2. Each Reliability Coordinator and Transmission Operator shall provide evidence, such

as email notices or postal receipts showing recipient and date that it has distributed
the requested data or written response that it does not have the data, to any Planning
Coordinator performing validation under Requirement R1 within 30 days of a written
request in accordance with Requirement R2; or a statement by the Reliability
Coordinator or Transmission Operator that it has not received notification regarding
data necessary for validation by any Planning Coordinator.

Page 3 of 11

MOD-033-1 — Steady-State and Dynamic System Model Validation

C. Compliance
1.

Compliance Monitoring Process
1.1. Compliance Enforcement Authority
“Compliance Enforcement Authority” means NERC or the Regional Entity in their
respective roles of monitoring and enforcing compliance with the NERC
Reliability Standards.
1.2. Evidence Retention
The following evidence retention periods identify the period of time an entity is
required to retain specific evidence to demonstrate compliance. For instances
where the evidence retention period specified below is shorter than the time
since the last audit, the Compliance Enforcement Authority may ask an entity to
provide other evidence to show that it was compliant for the full time period
since the last audit.
The applicable entity shall keep data or evidence to show compliance with
Requirements R1 through R2, and Measures M1 through M2, since the last audit,
unless directed by its Compliance Enforcement Authority to retain specific
evidence for a longer period of time as part of an investigation.
If an applicable entity is found non-compliant, it shall keep information related
to the non-compliance until mitigation is complete and approved, or for the time
specified above, whichever is longer.
The Compliance Enforcement Authority shall keep the last audit records and all
requested and submitted subsequent audit records.
1.3. Compliance Monitoring and Assessment Processes:
Refer to Section 3.0 of Appendix 4C of the NERC Rules of Procedure for a list of
compliance monitoring and assessment processes.
1.4. Additional Compliance Information
None

Page 4 of 11

Table of Compliance Elements
R#

Time Horizon

VRF

Violation Severity Levels
Lower VSL

R1

Long-term
Planning

Medium The Planning
Coordinator
documented and
implemented a
process to validate
data but did not
address one of the
four required topics
under Requirement
R1;

Moderate VSL

High VSL

Severe VSL

The Planning
Coordinator
documented and
implemented a
process to validate
data but did not
address two of the
four required topics
under Requirement
R1;

The Planning
Coordinator
documented and
implemented a
process to validate
data but did not
address three of the
four required topics
under Requirement
R1;

The Planning
Coordinator did not
have a validation
process at all or did
not document or
implement any of the
four required topics
under Requirement
R1;

OR

OR

OR

The Planning
Coordinator did not
perform simulation as
required by part 1.1
within 24 calendar
months but did
perform the
simulation within 28
calendar months;

The Planning
Coordinator did not
perform simulation as
required by part 1.1
within 24 calendar
months but did
perform the
simulation in greater
than 28 calendar
months but less than
or equal to 32
calendar months;

The Planning
Coordinator did not
perform simulation as
required by part 1.1
within 24 calendar
months but did
perform the
simulation in greater
than 32 calendar
months but less than
or equal to 36
calendar months;

The Planning
Coordinator did not
validate its portion of
the system in the
power flow model as
required by part 1.1
within 36 calendar
months;

OR

OR

OR
The Planning
Coordinator did not
perform simulation as

Page 5 of 11

OR

OR
The Planning
Coordinator did not
perform simulation as
required by part 1.2
within 36 calendar

R2

Long-term
Planning

Lower

required by part 1.2
within 24 calendar
months (or the next
dynamic local event in
cases where there is
more than 24 months
between events) but
did perform the
simulation within 28
calendar months.

The Planning
Coordinator did not
perform simulation as
required by part 1.2
within 24 calendar
months (or the next
dynamic local event in
cases where there is
more than 24 months
between events) but
did perform the
simulation in greater
than 28 calendar
months but less than
or equal to 32
calendar months.

The Planning
Coordinator did not
perform simulation as
required by part 1.2
within 24 calendar
months (or the next
dynamic local event in
cases where there is
more than 24 months
between events) but
did perform the
simulation in greater
than 32 calendar
months but less than
or equal to 36
calendar months.

months (or the next
dynamic local event in
cases where there is
more than 24 months
between events).

The Reliability
Coordinator or
Transmission Operator
did not provide
requested actual
system behavior data
(or a written response
that it does not have
the requested data) to
a requesting Planning
Coordinator within 30
calendar days of the
written request, but

The Reliability
Coordinator or
Transmission Operator
did not provide
requested actual
system behavior data
(or a written response
that it does not have
the requested data) to
a requesting Planning
Coordinator within 30
calendar days of the
written request, but

The Reliability
Coordinator or
Transmission Operator
did not provide
requested actual
system behavior data
(or a written response
that it does not have
the requested data) to
a requesting Planning
Coordinator within 30
calendar days of the
written request, but

The Reliability
Coordinator or
Transmission Operator
did not provide
requested actual
system behavior data
(or a written response
that it does not have
the requested data) to
a requesting Planning
Coordinator within 75
calendar days;

Page 6 of 11

did provide the data
(or written response
that it does not have
the requested data) in
less than or equal to
45 calendar days.

did provide the data
(or written response
that it does not have
the requested data) in
greater than 45
calendar days but less
than or equal to 60
calendar days.

did provide the data
(or written response
that it does not have
the requested data) in
greater than 60
calendar days but less
than or equal to 75
calendar days.

D. Regional Variances
None.
E. Interpretations
None.
F. Associated Documents
None.

Page 7 of 11

OR
The Reliability
Coordinator or
Transmission Operator
provided a written
response that it does
not have the
requested data, but
actually had the data.

Application Guidelines
Guidelines and Technical Basis
Requirement R1:
The requirement focuses on the results-based outcome of developing a process for and
performing a validation, but does not prescribe a specific method or procedure for the
validation outside of the attributes specified in the requirement. For further information on
suggested validation procedures, see “Procedures for Validation of Powerflow and Dynamics
Cases” produced by the NERC Model Working Group.
The specific process is left to the judgment of the Planning Coordinator, but the Planning
Coordinator is required to develop and include in its process guidelines for evaluating
discrepancies between actual system behavior or response and expected system performance
for determining whether the discrepancies are unacceptable.
For the validation in part 1.1, the state estimator case or other Real-time data should be taken
as close to system peak as possible. However, other snapshots of the system could be used if
deemed to be more appropriate by the Planning Coordinator. While the requirement specifies
“once every 24 calendar months,” entities are encouraged to perform the comparison on a
more frequent basis.
In performing the comparison required in part 1.1, the Planning Coordinator may consider,
among other criteria:
1. System load;
2. Transmission topology and parameters;
3. Voltage at major buses; and
4. Flows on major transmission elements.
The validation in part 1.1 would include consideration of the load distribution and load power
factors (as applicable) used in the power flow models. The validation may be made using
metered load data if state estimator cases are not available. The comparison of system load
distribution and load power factors shall be made on an aggregate company or power flow
zone level at a minimum but may also be made on a bus by bus, load pocket (e.g., within a
Balancing Authority), or smaller area basis as deemed appropriate by the Planning Coordinator.
The scope of dynamics model validation is intended to be limited, for purposes of part 1.2, to
the Planning Coordinator’s planning area, and the intended emphasis under the requirement is
on local events or local phenomena, not the whole Interconnection.
The validation required in part 1.2 may include simulations that are to be compared with actual
system data and may include comparisons of:


Voltage oscillations at major buses



System frequency (for events with frequency excursions)



Real and reactive power oscillations on generating units and major inter-area ties

Page 8 of 11

Application Guidelines
Determining when a dynamic local event might occur may be unpredictable, and because of the
analytic complexities involved in simulation, the time parameters in part 1.2 specify that the
comparison period of “at least once every 24 calendar months” is intended to both provide for
at least 24 months between dynamic local events used in the comparisons and that
comparisons must be completed within 24 months of the date of the dynamic local event used.
This clarification ensures that PCs will not face a timing scenario that makes it impossible to
comply. If the time referred to the completion time of the comparison, it would be possible for
an event to occur in month 23 since the last comparison, leaving only one month to complete
the comparison. With the 30 day timeframe in Requirement R2 for TOPs or RCs to provide
actual system behavior data (if necessary in the comparison), it would potentially be impossible
to complete the comparison within the 24 month timeframe.
In contrast, the requirement language clarifies that the time frame between dynamic local
events used in the comparisons should be within 24 months of each other (or, as specified at
the end of part 1.2, in the event more than 24 months passes before the next dynamic local
event, the comparison should use the next dynamic local event that occurs). Each comparison
must be completed within 24 months of the dynamic local event used. In this manner, the
potential problem with a “month 23” dynamic local event described above is resolved. For
example, if a PC uses for comparison a dynamic local event occurring on day 1 of month 1, the
PC has 24 calendar months from that dynamic local event’s occurrence to complete the
comparison. If the next dynamic event the PC chooses for comparison occurs in month 23, the
PC has 24 months from that dynamic local event’s occurrence to complete the comparison.
Part 1.3 requires the PC to include guidelines in its documented validation process for
determining when discrepancies in the comparison of simulation results with actual system
results are unacceptable. The PC may develop the guidelines required by parts 1.3 and 1.4
itself, reference other established guidelines, or both. For the power flow comparison, as an
example, this could include a guideline the Planning Coordinator will use that flows on 500 kV
lines should be within 10% or 100 MW, whichever is larger. It could be different percentages or
MW amounts for different voltage levels. Or, as another example, the guideline for voltage
comparisons could be that it must be within 1%. But the guidelines the PC includes within its
documented validation process should be meaningful for the Planning Coordinator’s system.
Guidelines for the dynamic event comparison may be less precise. Regardless, the comparison
should indicate that the conclusions drawn from the two results should be consistent. For
example, the guideline could state that the simulation result will be plotted on the same graph
as the actual system response. Then the two plots could be given a visual inspection to see if
they look similar or not. Or a guideline could be defined such that the rise time of the transient
response in the simulation should be within 20% of the rise time of the actual system response.
As for the power flow guidelines, the dynamic comparison criteria should be meaningful for the
Planning Coordinator’s system.
The guidelines the PC includes in its documented validation process to resolve differences in
Part 1.4 could include direct coordination with the data owner, and, if necessary, through the
provisions of MOD-032-1, Requirement R3 (i.e., the validation performed under this
requirement could identify technical concerns with the data). In other words, while this
standard is focused on validation, results of the validation may identify data provided under the
Page 9 of 11

Application Guidelines
modeling data standard that needs to be corrected. If a model with estimated data or a generic
model is used for a generator, and the model response does not match the actual response,
then the estimated data should be corrected or a more detailed model should be requested
from the data provider.
While the validation is focused on the Planning Coordinator’s planning area, the model for the
validation should be one that contains a wider area of the Interconnection than the Planning
Coordinator’s area. If the simulations can be made to match the actual system responses by
reasonable changes to the data in the Planning Coordinator’s area, then the Planning
Coordinator should make those changes in coordination with the data provider. However, for
some disturbances, the data in the Planning Coordinator’s area may not be what is causing the
simulations to not match actual responses. These situations should be reported to the Electric
Reliability Organization (ERO). The guidelines the Planning Coordinator includes under Part 1.4
could cover these situations.
Rationale:

During development of this standard, text boxes were embedded within the standard to explain
the rationale for various parts of the standard. Upon BOT approval, the text from the rationale
text boxes was moved to this section.
Rationale for R1:
In FERC Order No. 693, paragraph 1210, the Commission directed inclusion of “a requirement
that the models be validated against actual system responses.” Furthermore, the Commission
directs in paragraph 1211, “that actual system events be simulated and if the model output is
not within the accuracy required, the model shall be modified to achieve the necessary
accuracy.” Paragraph 1220 similarly directs validation against actual system responses relative
to dynamics system models. In FERC Order 890, paragraph 290, the Commission states that
“the models should be updated and benchmarked to actual events.” Requirement R1 addresses
these directives.
Requirement R1 requires the Planning Coordinator to implement a documented data validation
process to validate data in the Planning Coordinator’s portion of the existing system in the
steady-state and dynamic models to compare performance against expected behavior or
response, which is consistent with the Commission directives. The validation of the full
Interconnection-wide cases is left up to the Electric Reliability Organization (ERO) or its
designees, and is not addressed by this standard. The following items were chosen for the
validation requirement:
A. Comparison of performance of the existing system in a planning power flow model to actual
system behavior; and
B. Comparison of the performance of the existing system in a planning dynamics model to
actual system response.

Page 10 of 11

Application Guidelines
Implementation of these validations will result in more accurate power flow and dynamic
models. This, in turn, should result in better correlation between system flows and voltages
seen in power flow studies and the actual values seen by system operators during outage
conditions. Similar improvements should be expected for dynamics studies, such that the
results will more closely match the actual responses of the power system to disturbances.
Validation of model data is a good utility practice, but it does not easily lend itself to Reliability
Standards requirement language. Furthermore, it is challenging to determine specifications for
thresholds of disturbances that should be validated and how they are determined. Therefore,
this requirement focuses on the Planning Coordinator performing validation pursuant to its
process, which must include the attributes listed in parts 1.1 through 1.4, without specifying the
details of “how” it must validate, which is necessarily dependent upon facts and circumstances.
Other validations are best left to guidance rather than standard requirements.
Rationale for R2:
The Planning Coordinator will need actual system behavior data in order to perform the
validations required in R1. The Reliability Coordinator or Transmission Operator may have this
data. Requirement R2 requires the Reliability Coordinator and Transmission Operator to supply
actual system data, if it has the data, to any requesting Planning Coordinator for purposes of
model validation under Requirement R1.
This could also include information the Reliability Coordinator or Transmission Operator has at
a field site. For example, if a PMU or DFR is at a generator site and it is recording the
disturbance, the Reliability Coordinator or Transmission Operator would typically have that
data.

Version History

Version

Date

Action

1

February 6,
2014

Adopted by the NERC Board of
Trustees.

1

May 1, 2014

FERC Order issued approving
MOD-033-1.

Change Tracking
Developed as a new
standard for system
validation to address
outstanding directives
from FERC Order No. 693
and recommendations
from several other
sources.

Page 11 of 11