Form one time one time State framework Metrics

State Review Framework (Revision)

2185.02metrics.xls

State Review Framework

OMB: 2020-0031

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 2020-0031 can be found here:

Document [xlsx]

Download: xlsx | pdf

Overview

CAA metrics
CWA metrics
RCRA metrics

Sheet 1: CAA metrics

CAA Metrics - Please answer the following four questions.

What is your name and contact information?

Name:

Organization:

Address:

E-Mail Address:

Phone Number:

What new metrics are needed? (Please also explain how the proposed metric would support your program evaluation.)

Which metrics should we consider removing? (Please provide a brief justification.)

Do you have any suggestions for improving the existing metrics? (Please list suggested improvements in the far right column of the table below ("Your Suggestions"). Please type each of your suggestions in the yellow shaded region next to the corresponding metric.)

Name

Description

Guidance Requirement or Goal

Universe

Combined Region/State or State-only Data?

Original Source of Data Provided by EPA

Selection Criteria Explanation

Data Pulled From

Metric Type

Notes

Your Suggestions

1. Degree to which state program has completed the universe of planned inspections/compliance evaluations (addressing core requirements and federal, state and regional priorities) is completed.

1a1. CAA major full compliance evaluation (FCE) coverage in last two complete fiscal years

Percent of CAA active major sources receiving full compliance evaluations by the state in the last two fiscal years. Data pull uses the current universe.

Active major sources should receive a full compliance evaluation every 2 years. If region and state negotiated less than 100% 2-year coverage, refer to 1r below.

AFS Majors *See note in Selection Criteria Explanation.

State-only (but separate combined % also provided)

AFS

Select all major CAA sources from AFS which have had at least one FCE action within the last two complete fiscal years. Compare to the universe of all active majors plus other inspected majors. *(Numerator counts any AFS major source with an FCE regardless of operating status/denominator counts all AFS major sources with operating status O,T, or I, plus any other inspected majors.)

IDEA

Goal

Consistent with CMS. The select logic for this metric could be improved in the future when AFS "CMS category" and "automatic unknown compliance flag" data are quality tested as a way to measure inspection commitments. Note re: universe counts: OECA researching option of capturing End-of-Year universe starting with FY2005 for use where universe counts are required. If sources included on the list were not active for the full two years, FCE is not expected.

1a2. CMS major full compliance evaluation (FCE) coverage in last two complete fiscal years

Percent of CMS major sources receiving full compliance evaluations by the state in the last two fiscal years. Data pull uses the current universe.

CMS sources should receive a full compliance evaluation every 2 years. If region and state negotiated less than 100% 2-year coverage, refer to 1r below.

CMS Majors *See note in Selection Criteria Explanation.

State-only (but separate combined % also provided)

AFS

Select all major CMS sources from AFS which have had at least one FCE action within the last two complete fiscal years. Compare to the universe of all active CMS majors plus other inspected CMS majors. *(Numerator counts any CMS major source with an FCE regardless of operating status/denominator counts all CMS major sources with operating status O,T, or I, plus any other inspected CMS majors.)

IDEA

Data Quality

During the time frame of the review, states/locals will continually refine the universe of affected CMS sources. Therefore, both the AFS and CMS universes should be used for comparison purposes. Utilizing both universes allows a QA/QC review to identify any major sources that have not been included in CMS. If sources included on the list were not active for the full two years, FCE is not expected.

CAA synthetic minor 80% sources (SM-80) FCE coverage

Percent of CAA currently active SM-80 sources with full compliance evaluations by the state in the last four <increasing to five in FY06> fiscal years. Metric will be a 5-year 100% goal when data are available.

Active SM-80 sources should receive a full compliance evaluation every 5 years. However, the CMS policy has only been effective since FY02, so a 2002-2005 metric is provided (80% goal) (increasing to 2002-2006 at 100%). If region and state negotiated less coverage, also see 1r.

CAA SM-80%s *See note in Selection Criteria Explanation.

State-only (but separate combined % also provided)

AFS

Select all CAA SM-80 sources from AFS which have had at least one FCE within four <increasing to five in FY06> complete fiscal years. Compare to the universe of all active 80% synthetic minors plus other inspected SM-80s. *(Numerator counts any CAA SM-80 source with an FCE regardless of operating status/denominator counts all CAA SM-80 sources with operating status O,T, or I, plus any other inspected SM-80s.)

IDEA

Goal

Consistent with CMS. Current NPG document does not discuss whether regions and states can combine to meet goal. The select logic for this metric could be improved in the future when AFS "CMS category" and "automatic unknown compliance flag" data are quality tested as a way to measure inspection commitments. Note re: universe counts: OECA researching option of capturing End-of-Year universe starting with FY2005 for use where universe counts are required.

1c1. FCEs and reported PCEs for currently active synthetic minors (coverage)

Informational measure provides data for all synthetic minors.

FCEs at federally reportable sources are required to be reported. PCEs that are part of a negotiated compliance monitoring plan are to be reported pursuant to the Compliance Monitoring Strategy (CMS).

AFS SMs *See note in Selection Criteria Explanation.

State-only (but separate combined % also provided)

AFS

Select all CAA SM sources from AFS which have had at least one FCE or reported PCE within the last four fiscal years <increasing to five in FY06>. Compare to the universe of all active AFS SM sources plus other inspected AFS SMs. *(Numerator counts any SM source with an FCE or reported PCE regardless of operating status/denominator counts all SM sources with operating status O,T, or I, plus any other inspected SMs.)

IDEA (Partially available in OTIS.)

Informational-only.

Although state PCEs that are not included in a negotiated CMS plan and FCEs at non-federally reportable sources are not required, they will be included for informational purposes when voluntarily reported. This metric helps provide a complete picture of state evaluation activity. Note re: universe counts: OECA researching option of capturing End-of-Year universe starting with FY2005 for use where universe counts are required. Note: state PCEs are not required but will be included for informational purposes if reported.

1c2. FCEs and reported PCEs for CMS synthetic minors (coverage)

Informational measure provides data for all synthetic minors.

FCEs at federally reportable sources are required to be reported. PCEs that are part of a negotiated compliance monitoring plan are to be reported pursuant to the CMS.

CMS SMs *See note in Selection Criteria Explanation.

State-only, (but separate combined % would also be provided)

AFS

Select all CMS SM sources from AFS which have had at least one FCE or reported PCE within the last four fiscal years <increasing to five in FY06>. Compare to the universe of all active CMS SM sources plus other inspected CMS SMs. *(Numerator counts any CMS SM source with an FCE or reported PCE regardless of operating status/denominator counts all CMS SM sources with operating status O,T, or I, plus any other inspected CMS SMs.)

IDEA

Informational-only

Although state PCEs that are not included in a negotiated CMS plan and FCEs at non-federally reportable sources are not required, they will be included for informational purposes when voluntarily reported. This metric helps provide a complete picture of state evaluation activity. Also, during the time frame of the review, states/locals will continually refine the universe of affected CMS sources. Therefore, both the AFS and CMS universes should be used for comparison purposes. Utilizing both universes allows a QA/QC review to identify any SM sources that have not been included in CMS. Note: state PCEs are not required but will be included for informational purposes if reported.

FCEs and reported PCEs at minor sources (coverage)

Informational measure provides data for all minor sources.

Although not all minor source activity or PCEs are required to be reported, this metric provides useful information on the overall health of the compliance monitoring program.

AFS minor sources *See note in Selection Criteria Explanation.

State-only

AFS

Select all CAA minor sources from AFS which have had at least one FCE or reported PCE within the last four FYs <increasing to five in FY06>. Compare to the universe of all active AFS minors plus other inspected AFS minors. *(Numerator counts any AFS minor source with an FCE or reported PCE regardless of operating status/denominator counts all AFS minor sources with operating status O,T, or I, plus any other inspected AFS minors.)

IDEA

Informational-only

Some state/locals have either negotiated or expressed interest in substituting minors for larger sources, or evaluate them in addition to the CMS sources. Although not all minor source activity is reportable, this metric provides useful information on the overall health of the compliance monitoring program. OECA will output the CMS flag, which states may use to reflect negotiated minors into their CMS plans. Note: state PCEs are not required but will be included for informational purposes if reported.

Investigations at CAA stationary sources

Informational measure provides information on this type of activity consistent with CMS.

Required pursuant to the current Air Facility System (AFS) Information Collection Request (ICR). Category of compliance monitoring activity recognized by CMS.

Active AFS sources

State-only

AFS

Select all active sources from AFS which have had at least one investigation conducted (e.g., achieved date) within the last four fiscal years <increasing to five in FY06>.

IDEA

Informational-only

Investigations are a category of compliance monitoring activity recognized by CMS. They are more resource-intensive than the other compliance monitoring activities, and provide useful information on the overall health of the compliance monitoring program.

Review of self-certifications completed

% of self certifications received by state in fiscal year that have been reviewed.

CMS recommends all self-certifications to be reviewed. Goal = 100%.

All active majors with Title V Air Program Code 'active', and cert due, rec'd or rev'd.

State-only

AFS

Select all active CAA Title V sources from AFS which have had at least one "Compliance Certification State Review" action within the fiscal year. Compare to the universe of active CAA Title V sources with any certification due/received or reviewed action within the FY.

OTIS CAA Extended Management Reports (one state at a time)

Goal

High percentages shown (that is 100%) may be significantly affected by incomplete reporting of certification due/received MDR actions by the regions or states (if negotiated). Percentage is also affected by the fact that some reviews are for certs received in the previous FY. All majors supposed to have Title V permit by 2006.

Number of sources with unknown compliance status

AFS is designed to convert compliance status to unknown based upon the FCE frequency negotiated between the region and state. Frequency is variable, thus this measure may more precisely track whether commitments are completed.

CMS provides for flexibility in FCE frequency, which is tracked using AFS.

All majors and SM-80s with CMS (refer to 12j)

Combined

AFS

Select all sources where the current source compliance status is "unknown" as automatically generated by AFS.

OTIS CAA query

Review Indicator

Will pull current data.

Percent of planned FCEs/negotiated PCEs completed

States/locals, with regional approval, can develop alternative CMS plans. Such alternative plans can vary with respect to the universe of sources covered, the minimum frequencies, and/or the type of evaluation (i.e., FCE, PCE or investigation). To the extent possible, AFS should be used to document and track progress with these alternative plans. If not possible, regions and states/locals must still provide sufficient documentation to track and evaluate progress with these plans.

2. Degree to which inspection (Compliance Monitoring) reports document inspection findings, including accurate description of what was observed to sufficiently identify violations.

Percentage of Compliance Monitoring Reports adequately documented in the files.

Evaluation of documentation in the file that is based on the elements in the EPA Compliance Monitoring Strategy.

State-only

Review Indicator

3. Degree to which inspection (Compliance Monitoring) reports are completed in a timely manner, including timely identification of violations.

Percentage of Compliance Monitoring Reports which identify potential violations in the file within a given time frame established by the Region and state, within 60 days.

Evaluation of documentation in the file that is based on the elements in the EPA Compliance Monitoring Strategy.

State-only

Review Indicator

4. Degree to which significant violations (e.g., significant noncompliance and high priority violations) and supporting information are accurately identified and reported to EPA national databases in a timely manner.

High priority violation discovery rate - (per FCE coverage at majors)

New HPVs identified in fiscal year by the state divided by the number of sources with state FCEs performed in the fiscal year.

File review recommended for regions/states below 1/2 of the national average. (HPV policy)

Majors

State-only (with Regional-only provided)

AFS/ IDEA

Select major sources with new HPVs identified in the FY. Compare to the number of major sources with State FCEs conducted (onsite or offsite) within the FY.

IDEA

Goal

Regional-only numbers will show the HPV discovery rate for regional inspections within each state. Accurate HPV identification is important for timeliness tracking under the HPV policy, the Watch List, award program screening, and public access to the data.

High priority violation discovery rate (per source universe - major)

New HPVs identified in fiscal year by the State divided by the number of major sources.

File review recommended for regions/states below 1/2 of the national average. (HPV policy)

Majors

State-only (with Regional-only provided)

AFS

Select major sources with new HPVs identified in the FY. Compare to the number of active major sources.

IDEA

Review Indicator

Assists region in selecting files for review when there is a possible issue.

No activity indicator - HPV

No HPVs identified by the state in the fiscal year.

HPV policy

All AFS

State-only

AFS

Select sources with new State HPVs identified in the FY.

OTIS Management Reports

Review Indicator

Assists region in selecting files for review when there is a possible issue. Region should examine state enforcement cases to determine whether HPVs were not identified.

HPV reporting indicator at majors (formal enforcement only)

Metric computes the % of formal actions taken at major sources during the FY that received a prior HPV listing, and benchmarks it to national average.

Targeted file review, in addition to SRF random selection, recommended for regions/states less than half of the national average.

Majors with formal actions

State-only

IDEA or AFS

Select the universe of major CAA sources that received a formal enforcement action within the FY. Compare to the subset of these that were designated as an HPV (source flag, not pathway) within the same FY or the two quarters prior to that FY.

IDEA

Review Indicator

Assists region in selecting files for review when there is a possible issue; for example, if a low percentage of actions received an HPV listing, the region should review files at facilities without such HPV listing.

5. The degree to which state enforcement actions include required corrective or complying actions (injunctive relief) that will return sources to compliance in a specific time frame.

Percentage of formal state enforcement actions that contain a compliance schedule of required actions or activities designed to return the source to compliance. This can be in the form of injunctive relief or other complying actions.

Evaluated based on file reviews protocol using EPA or equivalent state penalty policies for setting injunctive relief.

State-only

Review Indicator

Percentage of formal or informal enforcement responses that return sources to compliance.

Evaluated basd on file review.

State-only

Review Indicator

6. Degree to which a state takes timely and appropriate enforcement actions, in accordance with policy relating to specific media.

Timely action taken to address HPV sources

HPVs that were in unaddressed (no action) status for greater than 270 days. The region should evaluate how the state compares to the national average. Counts sources with at least one HPV month in the last full FY.

HPVs should be addressed with a formal action within 270 days of day zero. Region should determine possible concern based upon state data. (HPV policy)

Sources with HPVs within fiscal year (use last full fiscal year)

State-only (with Regional-only provided)

AFS

Select the total number of flagged unaddressed HPVS (not pathways) during the FY. Compare to the subset of those that have ten or more continuous months (>270 days) of unaddressed HPV status, starting with the first HPV month within the FY.

IDEA

Review Indicator

Timely action to address individual HPV pathways

% of HPVs pathways that exceed the 270 day timeliness threshold. This metric looks at each individual HPV entry (6a is source-based) from the AFS "653" report. The region should evaluate the percentage within the state and ensure that it is in keeping with the HPV policy.

HPVs should be addressed with a formal action within 270 days of day zero (HPV determination date). Region should determine possible concern based upon state data. (HPV policy)

HPV occurrences within fiscal year that have had at least 270 days for response

State-only (with Regional-only provided)

AFS 653 Report

Using the AFS 653 Report, select the total number of unaddressed HPV pathways during the FY. Compare to the subset of those pathways where "DAYS USED TO ADRS" is >270 or "UNADR DAYS" is >270.

AFS

Review Indicator

No activity indicator - actions

No actions were taken by the state in the fiscal year.

HPV policy

All AFS

State-only

AFS

Select all state formal enforcement actions taken during the FY.

OTIS Management Reports

Review Indicator

7. Degree to which a state includes both gravity and economic benefit calculations for all penalties, appropriately using the BEN model or similar state model (where in use and consistent with national policy).

Percentage of formal enforcement actions that include calculation for gravity and economic benefit consistent with applicable policies.

File review based on file review protocol using national program policy or applicable state penalty policy.

State-only

Goal

8. The degree to which penalties in final enforcement actions include economic benefit and gravity in accordance with applicable penalty policies.

No activity indicator - penalties

No penalties assessed by the state during the fiscal year.

HPV policy

All AFS

State-only

AFS

Select all state formal enforcement actions with penalties taken during the FY.

OTIS Management Reports

Review Indicator

Penalties normally included with formal enforcement actions at HPVs

Percent of enforcement actions taken at HPVs that carry any penalty. This metric does not measure appropriateness of penalties, but does flag when additional review is necessary.

Additional review necessary if state is below 80% actions against HPVs with penalty. HPV policy assumes it is an exception that an HPV receives no penalty.

All state actions at sources that were previously HPVs

State-only

AFS

Select joint, state, or undetermined-lead HPVs in last FY with state formal enforcement actions in last FY. Compare to those with a penalty amount greater than zero.

IDEA

Review Indicator

The 80% figure is based on reality. Guidance does not provide a set cut-off, but expects penalties unless there is an extenuating circumstance. This metric may not be useful if state has identified few HPVs.

9. Degree to which enforcement commitments in the PPA/PPG/ categorical grants (written agreements to deliver a product/project at a specified time), if they exist, are met and any products or projects are completed.

State agreements (PPA/PPG/SEA, etc.) contain enforcement and compliance commitments that are met.

Review of PPAs, PPGs, SEAs, or other documents that list enforcement and compliance commitments.

State-only

Review Indicator

10. Degree to which the Minimum Data Requirements are timely.

Integrity of HPV data (timely entry)

Percent of HPVs that are entered to AFS more than 60 days after the HPV designation (day zero). Measures the "lag" between the date of HPV designation, and the actual reporting of the HPV designation to AFS.

Timely and Appropriate Enforcement Response to High Priority Violations policy document states that the data should be entered promptly after the designation is made. HPV entry should not be withheld until the action is completed.

All HPVs

State-only

IDEA using AFS data

IDEA compares the new HPVs that are reported into AFS each month, and compares the date that these HPVs became known to EPA to the HPV designation date that is entered into AFS. Examines the last four quarters of data.

IDEA

Review Indicator

If a problem exists in this area, it negatively impacts timeliness tracking under the HPV policy, the Watch List, award program screening, and public access. For 2005 reviews, all HPVs newly entered after Nov. 2004 are counted; eventually this will change to measuring the last four quarters of data.

HPV being entered in a timely manner (normally 45 days from inspection/file review)

The average number of days between the activity that triggered HPV (inspection, review, etc.) and the HPV determination.

HPV determinations are normally required within 45 days, however, extensions are acceptable in certain circumstances. Region should determine if state significantly departs from national averages.

State-only

Informational Metric/Review Indicator.

Regions should evaluate what is maintained in AFS by the State and ensure that all mimimum data required fields are properly tracked and entered according to accepted schedules, not to exceed 90 days.

Based on quarterly reporting.

11. Degree to which the Minimum Data Requirements are accurate.

Indicator of accurate violation/noncompliance data entry

Compares the sources with violations to the sources with HPV. If more HPV sources appear, this is a strong indicator that violations are not fully tracked in system.

Data quality review triggered if the number of noncompliant sources is lower than the number of HPV sources, i.e., ratio > 100%. (MDRs)

Majors

Combined

AFS

Select all major CAA sources with violations within the FY. Compare to the number of sources: # HPVs / # sources in non-compliance

IDEA

Data Quality

IDEA does not support state-only pull. During file reviews, any violations noted in files should be checked against violation data in AFS.

Stack test results at federally-reportable sources

11b1. % of stack tests conducted and reviewed without pass/fail results code entered to AFS

0% (CMS/ICR/MDR)

All stack tests conducted at Fed-Rep Sources

State-only

AFS

Select Total Number of State Stack Tests for active majors, SM, and NESHAP minors in last FY per CAA Management Reports. Compare to the subset of stack tests without a result code of passed or failed.

OTIS Extended CAA Management Reports

Goal

11b2. Number of sources with stack test failures at federally-reportable sources

100% of failures should be reported to AFS.

Same as above

State-only

AFS

Select the Total Number of State Stack Tests (including OWNER/OPERATOR-CONDUCTED SOURCE TEST) for active majors, SM, and NESHAP minors in last FY with a result code of failed.

IDEA

Data Quality

Regions should ensure that states agree with the sources that failed. Regions may want to review HPV/compliance status of these sources to ensure proper data tracking has occurred in AFS. Stack test failures often may trigger an HPV determination (see HPV policy for details).

12. Degree to which the minimum data requirements are complete, unless otherwise negotiated by the region and state or prescribed by a national initiative.

For the following elements, the region should provide the counts to the state from the State Review Framework Web site and ensure the state agrees with these counts. If there is disagreement, additional evaluation should be conducted to determine the source of the discrepancy and to develop an action plan for making appropriate corrections.

Title V universe

12a1. AFS operating majors

Region and state should agree on universe count.

N/A

AFS

Select all operating major sources from AFS.

IDEA

Data Quality

12a2. AFS operating majors w/ air program code = V

Region and state should agree on universe count.

N/A

AFS

Select all operating major sources from AFS that have an Air Program Code for Title V (APC1 = V).

IDEA

Data Quality

The region and state should coordinate with Title V permit personnel to ensure accurate Title V universe counts. For TOPS, managed by OAQPS, see http://ttnwww.rtpnc.epa.gov/tops/home.cfm (for EPA Intranet users).

State agrees with source count for Major, SM, NESHAP minor sources

12b1. Major

Minimum data requirements are listed in the AFS Information Collection Request.

N/A

AFS

Select all active major sources from AFS.

OTIS Extended CAA Management Reports

Data Quality

12b2. Synthetic minor

Minimum data requirements are listed in the AFS Information Collection Request.

N/A

AFS

Select all active SM sources from AFS.

OTIS Extended CAA Management Reports

Data Quality

12b3. NESHAP minor

Minimum data requirements are listed in the AFS Information Collection Request.

N/A

AFS

Select all active NESHAP minor sources from AFS.

IDEA

Data Quality

Subprogram universe is accurate in AFS (NSPS, NESHAP and MACT).

12c1. Subpart designation per CAA - NSPS

N/A

AFS

Select all active, federally reportable NSPS sources from AFS based on their air program codes.

IDEA

Data Quality

Note: See subpart implementation schedule in the ICR.

12c2. Subpart designation per CAA - NESHAP

N/A

AFS

Select all active, federally reportable NESHAP sources from AFS based on their air program codes.

IDEA

Data Quality

12c3. Subpart designation per CAA - MACT

N/A

AFS

Select all active, federally reportable MACT sources from AFS based on their air program codes.

IDEA

Data Quality

Compliance monitoring counts complete

12d1 - Sources with FCEs in reporting period

Minimum data requirements are listed in the AFS Information Collection Request.

Operating Major, SM, and NESHAP Minor sources

State-only

AFS

Select state, relevant FY, and Full Compliance Evaluation Coverage (state only).

OTIS Extended CAA Management Reports

Data Quality

Operating sources are those with OPST=O,T,I

12d2 - Total FCEs completed in reporting period

Minimum data requirements are listed in the AFS Information Collection Request.

All Major, SM, and NESHAP Minor sources

State-only

AFS

Select state, relevant FY, and Total Number of State Full Compliance Evaluations (FCEs) at RECAP Universes (state only).

OTIS Extended CAA Management Reports

Data Quality

All sources are those with OPST=O,T,I,P,C,X

12d3 - Number of PCEs reported to AFS in reporting period

State PCEs are not required (unless negotiated) but will be included for informational purposes when reported.

All sources

State-only

AFS

IDEA

Informational-only

Historical non-compliance counts complete

Number of sources that had violations at any point during the reporting period

Minimum data requirements are listed in the AFS Information Collection Request.

all

Combined

AFS

Select CAA, states by Region, the relevant FY, and sources in violation

OTIS Management Reports

Data Quality

Notice of violation counts complete

12f1 - Number of NOVs issued in last FY

Minimum data requirements are listed in the AFS Information Collection Request.

all

State-only

AFS

For total NOVs, select CAA, states by Region, the relevant FY, and state notices of violation.

OTIS Management Reports

Data Quality

During the review, Regions may want to compare sources with NOVs to sources with HPV listings to assist in determining integrity of HPV data.

12f2 - Number of sources with NOVs in last FY

Minimum data requirements are listed in the AFS Information Collection Request.

all

State-only

AFS

For sources with NOVs, use OTIS CAA query.

OTIS CAA query

Data Quality

During the review, Regions may want to compare sources with NOVs to sources with HPV listings to assist in determining integrity of HPV data.

HPV counts complete

12g1 - Number of new HPVs (pathways) in last FY

Minimum data requirements are listed in the AFS Information Collection Request.

all

State-only

AFS

Select CAA, states by Region, the relevant FY, and # new State HPVs

IDEA

Data Quality

12g2 - Number of sources in HPV in last FY

Minimum data requirements are listed in the AFS Information Collection Request.

all

State-only

AFS

In OTIS MRs, select CAA, states by Region, the relevant FY, and # sources with new HPV (for a combined number only).

IDEA

Data Quality

Formal action counts complete

12h1 - Number of actions issued in last FY

Minimum data requirements are listed in the AFS Information Collection Request.

all

State-only

AFS

For total actions, select CAA, states by Region, the relevant FY, and state total formal actions.

OTIS Management Reports

Data Quality

12h2 - Number of sources with actions in last FY

Minimum data requirements are listed in the AFS Information Collection Request.

all

State-only

AFS

For source count, select CAA, states by Region, the relevant FY, and # sources with state actions.

OTIS Management Reports

Data Quality

Assessed penalties complete

Total dollar amount of assessed penalties in last FY

Minimum data requirements are listed in the AFS Information Collection Request.

all

State-only

AFS

Select CAA, states by Region, the relevant FY, and total state penalties

OTIS Management Reports

Data Quality

Number of major sources missing CMS policy applicability

Identifies major sources in AFS that have not been identified with CMS applicability. Without CMS categories and frequencies, these sources are not included in the automatic unknown compliance status generation.

The CMS policy indicates that all major sources be evaluated within a specified timeframe as negotiated with the state and region.

Active majors

Combined

AFS/ IDEA

Select all active major sources from AFS where the CMS Source Category (CMSC) field is blank.

IDEA

Review Indicator

Sheet 2: CWA metrics

CWA Metrics - Please answer the following three questions.

What new metrics are needed? (Please also explain how the proposed metric would support your program evaluation.)

Which metrics should we consider removing? (Please provide a brief justification.)

Name

Description

Guidance Requirement or Goal

Universe

Combined Region/State or State-only Data?

Original Source of Data Provided by EPA

Selection Criteria Explanation

Data Pulled From

Metric Type

Notes

Your Suggestions .

NOTE: Due to changing WENDB requirements, this table will need to be updated when data for new facility universes are available. Some measures will need to change as a result of ICIS-NPDES and the Wet Weather SNC Policy that is under development.

1. Degree to which state program has completed the universe of planned inspections (addressing core requirements and federal, state, and regional priorities).

Inspection coverage - NPDES majors

EPA has a goal of 100% annual inspection coverage at NPDES majors.

100%. If the region has negotiated a different percent with the state, the region should include the goal under 1r.

Active NPDES majors with individual and general permits

State-only (but separate combined % also provided)

PCS/ICIS-NPDES

Select active NPDES major permits as the universe. Count facilities that have received one or more state or EPA inspections within the reporting year. Reconnaissance inspections are not counted at primary industries or at approved pretreatment facilities.

OTIS State Review Framework Data Metrics Tool

Goal

Corresponds to the PIP National Management Report inspection coverage measure for major sources. Note re: universe counts: OECA researching option of capturing End-of-Year universe starting with FY2005 for use where universe counts are required. Also note that all references to fiscal year are federal; discrepancies between state and federal fiscal years may cause discrepancies in EPA and state numbers. Should a region/state want to look at a different time frame, use 1r below.

Inspection coverage - NPDES non-majors

Inspections at NPDES non-majors with individual permits.

This metric is for information purposes only. It provides inspection coverage for NPDES non-majors with effluent limits and DMRs. In some cases, states may agree to inspection percentages for non-majors in PPAs, PPGs, or may be meeting basic goals by "trading" major for non-major inspections. This provides information to the region.

Active non-majors with individual permits

State-only (but separate combined % also provided)

PCS/ICIS-NPDES

Select active non-majors with individual permits as the universe. Count facilities that have received one or more state or EPA inspections within the year. (PCS metrics measure inspections during the NPDES Review Year, while ICIS-NPDES metrics measure the federal fiscal year. OECA will update the PCS logic after the ICIS-NPDES interface is completed.) Reconnaissance inspections are not counted at primary industries or at approved pretreatment facilities.

OTIS State Review Framework Data Metrics Tool

Informational-only

Corresponds to PIP National Management Report inspection coverage measure for non-majors.

Inspections at NPDES non-majors with general permits.

Active non-majors with general permits

State-only (but separate combined % also provided)

PCS/ICIS-NPDES

Select active non-majors with general permits as the universe. Count facilities that have received one or more state or EPA inspections within the year. Reconnaissance inspections are not counted at primary industries or at approved pretreatment facilities.

OTIS State Review Framework Data Metrics Tool

Informational-only

Other inspections performed (beyond facilities indicated in 1a and 1b) (Coverage)

This metric is for information purposes only. It provides inspection coverage for NPDES permittees that do not have effluent limits and DMRs. In some cases, states may agree to inspection percentages for non-majors in PPAs, PPGs, or may be meeting basic goals by "trading" major for non-major inspections. This provides information to the region.

Active facilities without effluent limits and that do not regularly submit DMRs

State-only (but separate combined % also provided)

PCS/ICIS-NPDES

Select active major and non-major permits without effluent limits and that do not regularly submit DMRs (non-standard, non-general permits) as the universe. Count facilities that have received one or more state or EPA inspections within the reporting year. Reconnaissance inspections are not counted at primary industries or at approved pretreatment facilities.

IDEA

Informational-only

May be a precursor for eventual measurement of inspections at wet weather inspections when these universes are cleaned up in the database.

Regions can track yearly commitments or multi-year plans. Reserved for inspection plan targets negotiated between the region and state in PPAs and grant agreements.

Focus on the negotiated percent of non-majors and regional tracking of any "major for non-major" inspection trades. Also, discrepancies between state and federal fiscal years may cause discrepancies between EPA and state numbers.

2. Degree to which inspection reports and compliance reviews document inspection findings, including accurate description of what was observed to sufficiently identify violation.

Percentage of inspection reports that are adequately documented.

Evaluate based on file review. File Review Protocol should look closely at evaluating the processes used by states to implement the Violation Review Action Criteria (VRAC) which provides guidance on how Regions and delegated states should review all violations for potential enforcement response.

State-only

Review Indicator

3. Degree to which inspection reports are completed in a timely manner, including timely identification of violations.

Percentage of Inspection Reports which identify potential violations in the file within a given time frame established by the Region and state.

Review should also identify which state inspections identified violations, and determine whether those violations were entered into PCS (single-event violations).

State-only

Review Indicator

Single-event violations reported to national system (non-automated violations arising from inspections and compliance monitoring)

4a1. Single-event violations at majors. This measure assesses whether violations determined by means other than automated discharge to limits comparisons are being reported and tracked in PCS.

States should be entering violations arising from major facility compliance monitoring. Violations found at non-major facilities are not yet required to be reported, but are included as a separate measure as a placeholder. Review of how violation determinations from inspections or other means are being reported is required for regions/states that show zero or very few single-event violations in PCS.

Active majors

Combined (IDEA does not provide ability to separate EPA from state)

PCS/ICIS-NPDES

Select all major facilities with single-event violations identified within the fiscal year under review.

OTIS State Review Framework Data Metrics Tool

Review Indicator

Several regions and states requested more detailed guidance on how to enter and track single-event violations. OECA clarified this in a data entry guide released in September 2005 (Interim Single Event Violation Data Entry Guide for PCS). If no activity is listed at all, regions should discuss the state's capability to send this information into the database.

4a2. Single-event violations at non-majors. This measure assesses whether violations determined by means other than DMRs are being reported and tracked in PCS.

Active non-majors

Combined (IDEA does not provide ability to separate EPA from state)

PCS/ICIS-NPDES

Select all non-major facilities with single-event violations identified within the fiscal year under review.

OTIS State Review Framework Data Metrics Tool

Informational-only

This measure will eventually change to a review indicator after data requirements are in effect for a full year.

Frequency of SNC

4b1. Facilities in SNC during reporting year

The list of SNC facilities is accurate only if the permit limits with their DMRs and permit schedules and compliance schedules with their report dates are routinely entered into the national system. Such data are required. Minimum data requirements are listed in the 1985 PCS Policy Statement.

Majors

Combined (PCS does not provide ability to separate EPA from state)

PCS/ICIS-NPDES

Select and count the total number of NPDES majors in SNC during last full FY.Select the number of NPDES major facilities that have had at least one occurrence of D, E, S, T, or X in the last year.

IDEA

Review Indicator

Region can examine data in element 12 to help support this analysis.

4b2. SNC Rate

Percent of major facilities in SNC. The list of SNC facilities is accurate only if the permit limits with their DMRs and permit schedules and compliance schedules with their report dates are routinely entered into the national system. Such data are required.

Majors

Combined (PCS does not provide ability to separate EPA from state)

PCS/IDEA/ICIS-NPDES

Percent of major facilities in SNC; select the number of facilities that have had at least one occurrence of D, E, S, T, or X in the last year and divide by total major facilities.

IDEA

Review Indicator

Region can examine data in element 12 to help support this analysis.

Wet Weather SNC placeholder

Metric(s) to be developed in the future.

5. The degree to which state enforcement actions include required corrective or complying actions (injunctive relief) that will return facilities to compliance in a specific time frame.

Evaluated based on file review protocol using EPA or equivalent state penalty policy.

State-only

Review Indicator

Percentage of actions or responses other than formal enforcement that return source to compliance.

Evaluated based on file review protocol using EPA or equivalent state penalty policy.

State-only

Review Indicator

6. Degree to which a state takes timely and appropriate enforcement actions, in accordance with policy relating to specific media.

Timely action taken to address SNC

Guidance calls for enforcement action before two quarters of QNCR effluent violation at same pipe, same parameter.

Number of facilities without timely action does not exceed 2% of active major universe throughout the fiscal year, based on the QNCR Guidance Manual.

Majors

Combined

PCS/ICIS-NPDES

Sum all facilities that meet the criteria of consecutive SNC effluent violations with no action and other unresolved SNC violations in consecutive quarters: QNCR Codes E, X, or S for 2 or more consecutive quarters and SNC effluent violations of the same pipe and parameter for 2 or more consecutive quarters, and no formal enforcement action within one year from the 2nd consecutive QNCR quarter; OR three consecutive quarters of SNC-level effluent violations at the same pipe and same parameter, and no formal enforcement action within one year from the 3rd consecutive violation quarter; OR QNCR Codes E, X, or S for 2 or more consecutive quarters and 2 or more consecutive, unresolved (PCS RNC resolution code = 1 or A) SNC effluent violations at the same pipe and of the same parameter; OR QNCR Code = D (DMR Nonreceipt) for 2 or more consecutive quarters and no formal enforcement action within one year from the 2nd consecutive QNCR quarter; OR, QNCR Code = S or T (Compliance Schedule Violations) for 2 or more consecutive quarters and at least one current open compliance schedule violation. Current refers to the date of the data pull. Calculate whether this number is greater than 2% of the current major universe.

OTIS State Review Framework Data Metrics Tool

Goal

Though other media use a state-only metric, this is appropriately combined since DMRs can be addressed by EPA or state. Hard to separate data. Results also will show what 2% of the current major universe is.

No activity indicator (actions)

No actions taken by state in fiscal year.

All facilities

State-only

PCS/ICIS-NPDES

Select OTIS CWA Management Reports query: State Enforcement - Total Formal Actions for last full FY.

OTIS Management Reports

Review Indicator

7. Degree to which a state includes both gravity and economic benefit calculations for all penalties, using the BEN model or similar state model (where in use and consistent with national policy).

Percentage of formal enforcement actions that include calculation for gravity and economic benefit consistent with applicable policies.

File review based on file review protocol using national program policy and the BEN or equivalent models.

State-only

Goal

8. The degree to which penalties in final enforcement actions include economic benefit and gravity in accordance with applicable penalty policies.

No activity indicator (penalties) **

No penalties taken by state in fiscal year. If state is not entering penalties to PCS, regions should request information.

Data not required, but expected to be when ICIS-NPDES requirements are final.

All facilities

State-only

PCS/ICIS-NPDES

Select OTIS CWA Management Reports query: Penalties - Total State Penalties for last full FY.

OTIS Management Reports

Informational-only; will convert to review indicator once ICIS-NPDES Policy Statement is finalized.

**Penalty data not currently required to be reported to PCS/ ICIS-NPDES. Because penalty data provide important information about the health of the compliance and enforcement program, region may need to seek information from the state regarding penalty amounts - particularly during the file reviews. This indicator can be based on state-provided data until ICIS-NPDES requirements are final. If preliminary data shows "0" for this metric, states may replace PCS/ ICIS-NPDES data with their own data.

Penalties normally included with formal enforcement actions

Percent of enforcement actions that carry any penalty. This metric does not measure appropriateness of penalties, but does flag when additional review is necessary.

Data not required, but expected to be when ICIS-NPDES requirements are final.

All facilities with state actions

State-only

PCS/ICIS-NPDES

Select OTIS CWA Management Reports query: State Enforcement - % State Actions With Penalty for last full FY.

OTIS Management Reports

Informational-only; will convert to review indicator once ICIS-NPDES Policy Statement is finalized.

See Notes for 8a. If state is not entering penalties into PCS, region should request information from state. (There may be legitimate circumstances for facilities to make this list.)

9. Degree to which enforcement commitments in the PPA/PPG/ categorical grants (written agreements to deliver a product/project at a specified time) are met and any products or projects are completed.

State agreements (PPA/PPG/SEA, etc.) contain enforcement and compliance commitments that are met.

Review of PPAs, PPGs, SEAs, or other documents that list enforcement and compliance commitments.

State-only

Review Indicator

10. Degree to which the Minimum Data Requirements are timely.

Regions should evaluate what is maintained in PCS by the State and ensure that all minimum data elements are properly tracked and entered according to accepted schedules.

File review to assess timely entry of data into PCS.

State-only

Review Indicator

11. Degree to which Minimum Data Requirements are accurate.

Actions are linked to the violations they address

Number of enforcement actions with EVTP (enforcement violation type) codes entered.

The region should ensure that all enforcement actions are properly linked to violations noted in PCS. If a state is below 80%, the region should perform additional review of data quality.

All facilities with actions

State-only

PCS/ICIS-NPDES

Select all state formal enforcement actions within the FY. Compare to the subset of these that have an associated EVTP (enforcement violation type) code(s) entered. (Note: "ERFN restriction" [see metric 12h2 for explanation] is not applied to this metric.)

IDEA

Data Quality

Entry of these links helps create an accurate Watch List. Region can enter online data errors via OTIS for inaccurate data.

12. Degree to which the minimum data requirements are complete, unless otherwise negotiated by the region and state or prescribed by a national initiative.

Active facility universe counts accurate for all NPDES permit types

12a1. NPDES majors with individual and general permits

Region and state should agree on active universe.

State-only

PCS/ICIS-NPDES

Select and count the total number of NPDES active major dischargers.

OTIS Management Reports or OTIS CWA query

Data Quality

Required data

12a2. NPDES non-majors with individual permits

Region and state should agree on active universe.

State-only

PCS/ICIS-NPDES

Select and count the total number of NPDES active non-major dischargers with individual permits.

OTIS Management Reports or OTIS CWA query

Data Quality

Required data

12a3. NPDES non-majors with general permits

Region and state should agree on active universe.

State-only

PCS/ICIS-NPDES

Select and count the total number of NPDES active non-major dischargers with general permits.

OTIS Management Reports or OTIS CWA query

Data Quality

Required data

When data are required, permit components should be tracked here including CSO, CAFO, Pretreatment, POTW, Biosolids, Stormwater (small/large MS4, construction, non-construction).

Majors permit limits and DMR entry complete

12b1. Limits at majors

95% entry standard for majors

Active majors

Combined

PCS/ICIS-NPDES

Active major individual and general permits which have at least one correctly coded limit in PCS, where: the permit expiration date and at least one limit end date are after the review date, and the start date(s) for the same limit(s) is/are before the review date, OR where: the permit is expired or administratively continued (while the permit application is under review by the state) (permit expiration date is before the review date), and at least one limit end date is equal to or later than the permit expiration date.

IDEA

Goal

100% data entry ensures accurate SNC calculation and accurate ECHO reports. Required data. This metric evaluates data as a snapshot in time, based on the last day +1 of the FY for prior FYs, or the last day +1 of the current official quarter for the current FY. (For example, FY2004 is based upon the status as of 10-01-2004.)

12b2. DMR entry rate at majors. The number of DMR forms received from major dischargers and entered into PCS for the last quarter of the FY divided by the number of DMR forms expected for majors in that quarter.

95% entry standard for majors

Active majors

Combined

PCS/ICIS-NPDES

The number of DMR forms received from major dischargers (municipal+nonmunicipal) and entered into PCS for the most recent quarter divided by the number of DMR forms expected for majors in that quarter.

PCS

Goal

100% data entry ensures accurate SNC calculation and accurate ECHO reports. Required data. Data for the current FY will be the most recent completed quarter, as noted on the Results page. DMR Forecasting Submission Date- Both (SUDB) is lagged one month from the quarter start and end dates.

12b3. Rate of manual override of SNC to a compliant status

Active majors

Combined

PCS/ICIS-NPDES

Rate of manual override of SNC to a compliant status in the last full FY. The denominator counts facilities that are PCS active majors with a QNCR value of D, E, S, T, or X (SNC values) within the FY. The numerator is a subset of the denominator where the SNC value was manually changed within the same quarter to a value of P, R, or C.

IDEA

Data Quality

There are legitimate circumstances for manual overrides; however, when done excessively, activity should be reviewed.

Non-majors permit limits and DMR entry

12c1. DMR limit entry rate

This is not currently a required field - no review needed - information only.

Active non-majors

Combined

PCS/ICIS-NPDES

Active non-major individual and general permits which have at least one correctly coded limit in PCS, where: the permit expiration date and at least one limit end date are after the review date, and the start date(s) for the same limit(s) is/are before the review date, OR where: the permit is expired or administratively continued (while the permit application is under review by the state) (permit expiration date is before the review date), and at least one limit end date is equal to or later than the permit expiration date.

IDEA

Informational-only

Pending new data requirements. Information not required now. The information can be helpful in verifying Annual Quarterly Noncompliance Report and tracking progress toward meeting new data entry requirements (will help region/state understand workload challenges for non-major DMR entry). This metric evaluates data as a snapshot in time, based on the last day +1 of the FY for prior FYs, or the last day +1 of the current official quarter for the current FY. (For example, FY2004 is based upon the status as of 10-01-2004.)

12c2. DMR entry rate. The number of DMR forms received from non-major dischargers and entered into PCS for the last quarter of the FY divided by the number of DMR forms expected for non-majors in that quarter.

This is not currently a required field - no review needed - information only.

Active non-majors

Combined

PCS/ICIS-NPDES

The number of DMR forms received from non-major dischargers (municipal+nonmunicipal) and entered into PCS for the most recent quarter divided by the number of DMR forms expected for non-majors in that quarter.

IDEA

Informational-only

Pending new data requirements. Information not required now. The information can be helpful in verifying Annual Quarterly Noncompliance Report. Data for the current FY will be the most recent completed quarter, as noted on the Results page. DMR Forecasting Submission Date- Both (SUDB) is lagged one month from the quarter start and end dates.

Inspection counts complete

12d1. Number of facilities inspected in last FY

Minimum data requirements are listed in the 1985 PCS Policy Statement.

Active

State-only (with region-only provided)

PCS/ICIS-NPDES

Select and count the total number of state NPDES inspections at any permitee during last full FY.

OTIS Management Reports

Data Quality

Required data

12d2. Total number of inspections performed in last FY

Minimum data requirements are listed in the 1985 PCS Policy Statement.

Active

State-only (with region-only provided)

PCS/ICIS-NPDES

Select and count the total number of state NPDES inspections at any permitee during last full FY.

OTIS Management Reports

Data Quality

Required data

Percent of violations linked to activity that identified the violation

To be developed. Will be discussed in more detail in upcoming single event violation data entry guide.

Active

State-only (with region-only provided)

ICIS-NPDES

IDEA

Placeholder only at this time. Data not required before ICIS-NPDES.

Notice of violation counts complete

12f1. Number of facilities with state NOVs in last FY

Minimum data requirements are listed in the 1985 PCS Policy Statement.

Active

State-only (with region-only provided)

PCS/ICIS-NPDES

For facilities with NOVs, use OTIS CWA query.

OTIS CWA query

Data Quality

Required data for major facilities. Output should be sortable by major/non-major.

12f2. Total number of state NOVs issued in last FY

Minimum data requirements are listed in the 1985 PCS Policy Statement.

Active

State-only (with region-only provided)

PCS/ICIS-NPDES

For total NOVs, select CWA, states by Region, the relevant FY, and state notices of violation.

OTIS Management Reports

Data Quality

Required data for major facilities. Output should be sortable by major/non-major.

Quality of violation data at non-major facilities (that regularly submit DMRs)

12g1. Noncompliance rate in database at non-major facilities in last fiscal year

Data are not required for 12g1 and 12g3; however, this metric provides a general sense of the quality of the data in the system (particularly given these data are shown in ECHO). This measure will become more important when the complete set of minors limits and DMRs are reported.

Non-majors (that regularly submit DMRs)

Combined

PCS/ICIS-NPDES

Select all active minor facilities with standard permit types. Compare to the subset of these that have at least one QNCR violation within the FY.

IDEA

Informational-only

Data are not required for 12g1; however, this metric provides a general sense of the quality of the data in the system before ICIS-NPDES is functional.

12g2. Noncompliance rate reported to EPA under the Annual Noncompliance Report (ANCR)

Non-majors (that regularly submit DMRs)

Combined

ANCR

manual count

ANCR

Informational-only

While most data metrics show data for the last full fiscal year, for this metric, the data metrics Web site displays the last full calendar year that is available. Historically it has taken eleven to fifteen months after the end of a calendar year to complete the ANCR for that year.

12g3. Number of facilities in database with DMR non-receipt for three continuous years

Non-majors (that regularly submit DMRs)

Combined

PCS/ICIS-NPDES

Select all active minor facilities with standard permit types that have had a QNCR violation value "D" (DMR non-receipt) in every quarter for the past three years, starting with the most recent official data reported by PCS to IDEA.

IDEA

Informational-only

Data are not required for 12g3; however, this metric provides a general sense of the quality of the data in the system before ICIS-NPDES is functional.

Formal action counts complete

12h1. Facilities with formal actions

Minimum data requirements are listed in the 1985 PCS Policy Statement.

All facilities

State-only

PCS/ICIS-NPDES

Select and count any state formal enforcement action in last full FY.

OTIS Management Reports

Data Quality

Required data for major facilities. Output should be sortable by major/non-major.

12h2. Total formal actions taken

Minimum data requirements are listed in the 1985 PCS Policy Statement.

All facilities

State-only

PCS/ICIS-NPDES

For total actions, select CWA, states by Region, the relevant FY, and state total formal actions. (Note: The "ERFN restriction" is applied to this metric (ERFN = Enforcement Action File Number). When counting state formal enforcement actions (EAs) and/or penalties, if more than one EA (per RECAP) carries the same Case File/Docket number (i.e., are the result of the same enforcement response), then only one EA may be counted within the fiscal year in which they occurred, even if the EAs are recorded at different facilities and/or on different dates.)

OTIS Management Reports

Data Quality

Required data for major facilities. Output should be sortable by major/non-major.

Assessed penalties complete *(Data are not currently required from states)

12i1. Formal actions with penalties

This is not currently a required field - no review needed - information only.

All facilities

State-only

PCS/ICIS-NPDES

Select and count any state formal enforcement action that includes a penalty in last full FY.

OTIS Management Reports

Informational-only

Pending new data requirements. Also see 8a Notes. State may replace data.

12i2. Total state penalties

Select and sum the penalty value for any state formal enforcement action that includes a penalty in last full FY.

OTIS Management Reports

Informational-only

Major facilities with compliance schedule violations

12j1. Permittees with unresolved compliance schedule violations, as of the end of the fiscal year.

Compliance schedules are WENDB for major permits. Review of violation determinations is recommended for regions/states that show zero or very few compliance schedule violations in PCS based upon a current snapshot.

Active majors

Combined

PCS/ICIS-NPDES

Select all active, major permittees with unresolved, detected compliance schedule violations as of the end of the fiscal year, based on the compliance schedule date (DTSC).

IDEA

Data Quality

This measure assesses whether compliance schedule violations are being reported and tracked in PCS. Compliance schedule events and violations from prior fiscal years may be counted here if they meet all of the selection criteria. Compliance schedule violations at non-major facilities are not required until the ICIS-NPDES Policy Statement is final, so will not be measured until after that.

12j2. Permittees with compliance schedule events with a compliance schedule date prior or equal to the end of the fiscal year.

Select all active, major permittees with compliance schedule events, with a compliance schedule date (DTSC) prior or equal to the end of the fiscal year.

Major facilities with permit schedule violations

12k1. Permittees with unresolved permit schedule violations, as of the end of the fiscal year.

Permit schedules are WENDB for major permits. Review of violation determinations is recommended for regions/states that show zero or very few permit schedule violations in PCS based upon a current snapshot. (In PCS Permit Schedules are called Compliance Schedules, the same as those from EAs.)

Active majors

Combined

PCS/ICIS-NPDES

Select all active, major permittees with unresolved, detected permit schedule violations as of the end of the fiscal year, based on the compliance schedule date (DTSC).

IDEA

Data Quality

This measure assesses whether compliance schedule violations are being reported and tracked in PCS. Permit schedule events and violations from prior fiscal years may be counted here if they meet all of the selection criteria.

12k2. Permittees with permit schedule events with a schedule date prior or equal to the end of the fiscal year.

Select all active, major permittees with permit schedule events, with a compliance schedule date (DTSC) prior or equal to the end of the fiscal year.

Sheet 3: RCRA metrics

RCRA Metrics - Please answer the following three questions.

What new metrics are needed? (Please also explain how the proposed metric would support your program evaluation.)

Which metrics should we consider removing? (Please provide a brief justification.)

Name

Description

Guidance Requirement or Goal

Universe

Combined Region/State or State-only Data?

Original Source of Data Provided by EPA

Selection Criteria Explanation

Data Can Be Pulled From

Guidance

Metric Type

Notes

Your Suggestions

1. Degree to which state program has completed the universe of planned inspections (addressing core requirements and federal, state, and regional priorities).

Inspection coverage for operating Treatment, Storage, and Disposal Facilities (TSDFs)

Per RCRA, region/state should inspect all operating TSDFs within two years; however, if facilities included on the list were not operating for the full two years, inspection is not expected.

Operating TSDFs should be inspected every 2 years.

Operating TSDFs

State-only (but separate combined % also provided)

RCRAInfo

Select all operating TSDFs in RCRAInfo that have been inspected during the past two FYs. Count each facility once regardless of inspection frequency within the timeframe.

IDEA

NPG

Goal

Note re: universe counts: OECA saving End-of-Year universe starting with FY2005 for use where universe counts are required; researching possible enhancements to allow for this on Web site. Trade agreements that are negotiated under national guidance should be tracked using metric 1r below.

Annual inspection coverage - Active Large Quantity Generators

National measure guidance calls for 20% annual coverage. If the region/state falls into an exception based on regional commitments, the region should fill in reserved items under 1r to indicate actual commitments or agreements.

20% annual inspection coverage for LQGs.

Active LQGs

State-only (but separate combined % also provided)

RCRAInfo

Select all LQGs in RCRAInfo that have been inspected during last full FY. Count each site once regardless of inspection frequency within the timeframe.

IDEA

NPG

Goal

The denominator makes use of the RCRAInfo Version 3 activity status flag. The March 21, 2005 Site ID Guidance document (RCRA Subtitle C EPA Identification Number, Site Status, and Site Tracking Guidance), posted in RCRAInfo, contains detailed activity status information. Regions and states should have their RCRA active universe defined in RCRAInfo. Trade agreements that are negotiated under national guidance should be tracked using 1r below. Note re: universe counts: OECA researching option of using end-of-year universe where universe counts are required.

Five-year inspection coverage - Active Large Quantity Generators

National guidance calls for 100% inspection coverage of LQGs over 5 years.

All LQGs should be inspected within 5 years, though if below the 100%, some may be attributed to universe changes or other reasons.

Active LQGs

State-only (but separate combined % also provided)

RCRAInfo

Select all LQGs in RCRAInfo that have been inspected during past five FYs. Count each site once regardless of inspection frequency within the timeframe.

IDEA

NPG

Goal

Five-year inspection coverage - Active Small Quantity Generators

This is an informational metric showing the percent coverage.

None

Active SQGs

State-only (but separate combined % also provided)

RCRAInfo

Select all SQGs in RCRAInfo that have been inspected during past five FYs. Count each site once regardless of inspection frequency within the timeframe.

IDEA

Informational-only

Inspection presence at smaller sites is important, and this metric provides information to regional reviewers. The denominator makes use of the RCRAInfo Version 3 activity status flag. The March 21, 2005 Site ID Guidance document (RCRA Subtitle C EPA Identification Number, Site Status, and Site Tracking Guidance), posted in RCRAInfo, contains detailed activity status information. Regions and states should have their RCRA active universe defined in RCRAInfo. Note re: universe counts: OECA researching option of using end-of-year universe where universe counts are required.

Inspections performed at sites other than those listed in 1a-d in last five years

This is an informational metric showing the number of other inspections performed.

None

All sites other than those listed in 1a-d (using active universe)

State-only (but separate combined # also provided)

RCRAInfo

Select all sites other than those listed in 1a-d that have been inspected during the past five FYs. Count each site once regardless of inspection frequency within the timeframe.

IDEA

Informational-only

This metric helps provide a complete picture of state evaluation activity.

Percent of planned inspections completed

Regions can track yearly commitments or multi-year plans. Reserved for inspection plan targets negotiated between the region and state.

If states have better active universe numbers than EPA provides in 1a-1d, state-derived results can be put in 1r; however, the region should ensure that the states update and maintain the RCRAInfo active universe. Also, trade agreements that are negotiated under national guidance should be tracked here.

2. Degree to which inspection reports and compliance reviews document inspection findings, including accurate description of what was observed to sufficiently identify violation.

Percentage of inspection reports that are adequately documented in files.

Evaluated based on file review protocol.

State-only

Goal

3. Degree to which inspection reports are completed in a timely manner, including timely identification of violations.

Percentage of Compliance Monitoring Reports/Inspection Reports which identify violations in the file within a given time frame established by the Region and state.

Evaluated based on file review protocol. 150 days to ID SNCs.

State-only

Review Indicator

4. Degree to which significant violations (e.g., significant noncompliance) and supporting information are accurately identified and reported to EPA national databases in a timely and accurate manner.

SNC identification rate

This measure assesses whether the region/state are actively identifying SNC problems from evaluations. Numerator=new SNCs in last FY, Denominator=evaluated sites in last FY.

Further discussion recommended for regions/states below 1/2 of the national average.

All sites

State-only (but separate combined % also provided)

RCRAInfo

Select all sites identifed as new SNCs in last full FY. Compare as a percentage to the number of sites inspected during last full FY.

OTIS Management Reports

ERP

Goal

This is a pointer to increased file review. When below half of the national average, it is recommended to review previous enforcement actions and cases in the pipeline to determine whether SNC did occur but was not reported.

Timely SNC determinations

% of SNC determinations (SNY date) completed within 150 days of "Day Zero" (first day of inspection) in last FY.

Goal is 100%

All SNCs

State-only

RCRAInfo

Will be programmed after a year of data is in RCRAInfo Version 3 (FY07).

IDEA

ERP and Data Appendix

Goal

2003 RCRA ERP calls for all SNC deteminations to be made within 150 days of day zero. Data flow in RCRAInfo Version 3. EPA will develop analytic approach to handle SVs converted to SNCs, and discuss the possible need for RCRAInfo to require secondary violator designation.

No activity indicator - new SNCs

No SNCs identified by the state in the fiscal year.

If SNC determinations are not being made, the region should evaluate whether the ERP is being followed.

All sites

State-only

RCRAInfo

Select all sites identified by the state as new SNCs in last FY.

OTIS Management Reports

ERP

Review Indicator

Region should examine state enforcement cases to determine whether SNCs were not identified.

SNC reporting indicator (actions receiving SNC listing)

Metric computes the % of formal actions taken during the FY that received a prior SNC listing, and benchmarks it to national average.

The RCRA ERP discusses the criteria for listing SNC. SNC normally requires a formal action. Some formal actions are taken at non-SNC, so there is not an expectation that 100% of sites with actions had SNC. File review recommended for regions/states below 1/2 of the national average.

All sites with actions

State-only

RCRAInfo

This measure gives the percentage of state formal enforcement actions taken during the fiscal year at facilities that were in SNC in the fiscal year or in the six months prior. (After this info populated in RCRAInfo V.3, this can look for all enforcement actions without a populated "Addressed SNC Dated" field.)

IDEA

ERP

Review Indicator

Review is recommended for states below half of the national average, though other states appear to have problems in this area, yet are not at half below the national average. There may be legitimate circumstances for sites to make this list; for example, if states take formal actions against all SVs. Metric will assist region when selecting files for review. Corrective action excluded.

5. The degree to which state enforcement actions include required corrective or complying actions (injunctive relief) that will return facilities to compliance in a specific time frame.

Evaluated based on file review protocol using EPA equivalent state penalty for assessing injunctive relief.

State-only

Review Indicator

Percentage of actions or responses other than formal enforcement that return source to compliance.

Evaluated based on file review.

State-only

Review Indicator

6. Degree to which a state takes timely and appropriate enforcement actions, in accordance with policy relating to specific media.

Timely action taken to address SNC

% of enforcement action/referral to DOJ/AG that have been taken within 360 days of Day Zero. Measured as number of SNC sites NOT exceeding 360 days from day zero over number of total SNCs in state. Note that ERP policy allows 20% of SNCs to exceed 360 timeliness milestone.

Of all enforcement actions taken within the fiscal year at SNC sites, 80% of SNC sites should receive a formal action or referral within 360 days of Day Zero.

SNC sites

State-only (but separate combined % also provided)

RCRAInfo

Will be programmed after a year of data is in RCRAInfo Version 3.

IDEA

ERP

Review Indicator

Data provided in reviews after RCRAInfo Version 3 in production for one year (FY07). The ERP provides 150 days for informal action, 240 days for an initial formal action, and 360 days for final formal action or referral to the State AG. ERP is effective for all civil activities initiated after it went into effect in 2003. Corrective action excluded.

No activity indicator - formal actions

No formal actions taken by state in fiscal year.

All sites

State-only

RCRAInfo

Select all state formal enforcement actions in last FY.

OTIS Management Reports

ERP

Review Indicator

If this indicates no actions, then look to see if state takes other sanctions for these violations. Corrective action excluded.

Percentage of formal enforcement actions that include calculation for gravity and economic benefit consistent with applicable policies.

File Review based on national program policy and BEN model or state models in use and consistent with national policy.

State-only

Goal

8. The degree to which penalties in final enforcement actions include economic benefit and gravity in accordance with applicable penalty policies.

No activity indicator - penalties

No penalties taken by state in fiscal year.

All sites

State-only

RCRAInfo

Select and sum the dollar amount of all penalties (final assessed penalties and SEP credits) at state enforcement actions (in 200,239 or in 250,339 or in 350,539 or in 550,799) in last FY.

OTIS Management Reports

Required data field in RCRAInfo

Review Indicator

If this indicates no penalties, then look to see if state takes other sanctions for these violations. Corrective action excluded.

Penalties normally included with formal enforcement actions

8b1. % of formal enforcement actions that carry any penalty in last FY.

Program review necessary if state is below one half of the national average for % of actions with penalty.

All sites with state formal actions

State-only

RCRAInfo

Select all state formal enforcement actions (in 200,239 or in 250,339 or in 350,539 or in 550,799) with penalties (final assessed penalties and SEP credits) in last FY. Compare to the total number of state formal enforcement actions in last FY.

IDEA

Review Indicator

This metric does not measure appropriateness of penalties, but does flag when additional review is necessary. If no penalties, then look to see if state takes other sanctions for these violations. Note that under the RCRA ERP not all non-SNC-related actions require a penalty. The action may include other sanctions depending on the situtation and may not include a monetary penalty. Corrective action excluded.

8b2. % of final formal enforcement actions that carry any penalty in last FY.

Program review necessary if state is below one half of the national average for % of actions with penalty.

All sites with state final formal actions

State-only

RCRAInfo

Select all state final formal enforcement actions (in 300,339 or in 350,399 or in 500,539 or in 550,699) with penalties (final assessed penalties and SEP credits) in last FY. Compare to the total number of state final formal enforcement actions in last FY.

IDEA

Review Indicator

State agreements (PPA/PPG/SEA, etc.) contain enforcement and compliance commitments that are met.

Review of PPAs, PPGs, SEAs, etc.

State-only

Review Indicator

10. Degree to which the Minimum Data Requirements are timely.

Integrity of SNC data (timely entry)

Percent of SNCs that are entered to RCRAInfo more than 60 days after the determination. Measures the "lag" between the date of SNC determination and the actual reporting of the SNC determination to RCRAInfo.

ERP states that the data should be entered when the determination is made. SNC entry should not be withheld until the action is completed. EPA expects SNC data to be entered more quickly than 60 days, so this metric provides some "cushion".

All SNCs

State-only

IDEA using RCRAInfo data

When a new SNC is seen by IDEA, it is known that the data was entered since the previous monthly RCRAInfo extract (does not provide the exact date). For analytic purposes, it is assumed that newly reported SNCs were put into RCRAInfo on the date of the previous extract. This date is compared to the date entered into RCRAInfo as the SNC (SNY) determination date.

IDEA

ERP and Data Appendix

Review Indicator

If a problem exists in this area, it negatively impacts timeliness tracking under the ERP, the Watch List, award program screening, and public access. There may be legitimate circumstances for sites to make this list, but the region should examine this issue to determine these reasons are valid. Minor additional time lags may occur for translator states.

Percent of inspections, enforcement actions, or other compliance or enforcement related activities for which there is a Nationally Required Data Element, that are entered into RCRA INFO in a timely manner.

File review

Regions should evaluate what is maintained in RCRAInfo by the State and ensure that all nationally required data elements are properly tracked and entered according to accepted schedules.

State-only

11. Degree to which Minimum Data Requirements are accurate.

Integrity of SNC data (correct entry of SNC determination date)

11a1. Measures "closeness" between SNC determination and formal enforcement action. Shows the number of sites that were "SNC determined" on same day as formal action.

ERP states that the data should be entered when the determination is made SNC entry should not be withheld until the action is completed.

All SNCs that have reached formal action

State-only

IDEA using RCRAInfo data

This measure compares the date of the SNC with the date of the first formal action that came after the SNC. This method does make some assumptions (e.g., that the action taken doesn’t address a previous SNC).

IDEA

ERP and Data Appendix

Review Indicator

11a2. Measures "closeness" between SNC determination in last FY and formal enforcement action. Shows the number of sites with SNC determination within one week of formal action.

ERP states that the data should be entered when the determination is made SNC entry should not be withheld until the action is completed.

All SNCs that have reached formal action

State-only

IDEA using RCRAInfo data

IDEA

ERP and Data Appendix

Review Indicator

Longstanding secondary violations not "returned to compliance" or redesignated as SNC.

Number of sites in violation for greater than 3 years. This measure is designed to ensure that violations are given an end date.

The ERP states that sites designated as secondary violators should be re-designated as SNC if the violator does not return to compliance in 240 days.

All sites

Combined

IDEA using RCRAInfo data

Select RCRA sites in violation greater than three years, but not currently in significant noncompliance.

OTIS RCRA interface (select facilities in violation greater than 3 years and not SNC)

Mandatory Data Elements in RCRAInfo

Data Quality

The region should review the site list and determine whether data quality issues exist that need to be addressed or whether SNC designations need to be made. This uses 3 years because that matches the OTIS data pull; however, guidance has a tighter 240-day standard.

12. Degree to which the minimum data requirements are complete, unless otherwise negotiated by the region and state or prescribed by a national initiative.

For the following elements, the region should provide the counts to the state from the OTIS State Review Framework data metrics site and ensure the state agrees with these counts. If there is disagreement, additional evaluation should be conducted to determine the source of the discrepancy and develop an action plan for making appropriate corrections.

Active site universe counts accurate

12a1. Number of operating TSDFs in RCRAInfo

Region and state should agree on operating universe count.

State-only

RCRAInfo

Select all operating TSDFs in RCRAInfo. Uses the current RCRAInfo definition of operating TSDF.

IDEA

Required data field in RCRAInfo

Data Quality

12a2. Number of active LQGs in RCRAInfo

Region and state should agree on active universe count.

State-only

RCRAInfo

Select all active LQGs in RCRAInfo.

IDEA

Required data field in RCRAInfo

Data Quality

12a2-12a4 use the RCRA Version 3 activity status flag.

12a3. Number of active SQGs in RCRAInfo

Region and state should agree on active universe count.

State-only

RCRAInfo

Select all active SQGs in RCRAInfo.

IDEA

Required data field in RCRAInfo

Data Quality

12a4. All other active sites in RCRAInfo

Region and state should agree on active universe count.

State-only

RCRAInfo

Select active sites in RCRAInfo other than those listed in 12a1-12a3.

IDEA

Data Quality

Inspection counts complete

12b1. Number of inspections performed by state during reporting period.

Mandatory data elements are listed in the RCRAInfo documentation.

all

State-only

RCRAInfo

Select all inspections (HQCAC,HQCDI,HQCEI,HQGME,HQFRR, HQNRR, HQCSE, HQOAM, HQFCI) performed by state during FY.

OTIS Management Reports

Required data field in RCRAInfo

Data Quality

12b2. Total # of sites inspected during reporting period

Mandatory data elements are listed in the RCRAInfo documentation.

all

State-only

RCRAInfo

Select all sites inspected (HQCAC,HQCDI,HQCEI,HQGME,HQFRR, HQNRR, HQCSE, HQOAM, HQFCI) by state during FY.

IDEA

Required data field in RCRAInfo

Data Quality

Violation counts complete

Number of sites with violations during reporting period

Mandatory data elements are listed in the RCRAInfo documentation.

all

State-only

RCRAInfo

Select all facilities with secondary violations in FY.

OTIS Management Reports

ERP and RCRAInfo

Data Quality

This metric should count violations for out-of-state sites that are resolved by a state.

Notice of violation counts complete

12d1. sites with state NOVs (informal enforcement actions)

Mandatory data elements are listed in the RCRAInfo documentation.

all

State-only

RCRAInfo

Select sites with actions in 100,199 during FY.

OTIS RCRA query

ERP and RCRAInfo

Data Quality

12d2. total state NOVs (informal enforcement actions) issued

Mandatory data elements are listed in the RCRAInfo documentation.

all

State-only

RCRAInfo

Select actions in 100, 199 during FY.

OTIS Management Reports

ERP and RCRAInfo

Data Quality

SNC counts complete

12e1. # of new SNCs in last FY;

Mandatory data elements are listed in the RCRAInfo documentation.

all

State-only

RCRAInfo

Select sites with new state SNYs in FY.

OTIS Management Reports

ERP and RCRAInfo

Data Quality

12e2. # of sites in SNC in last FY

Mandatory data elements are listed in the RCRAInfo documentation.

all

State-only

RCRAInfo

Select sites with an SNY in FY, or an SNY in a previous FY and no SNN before the start of the FY.

IDEA

ERP and RCRAInfo

Data Quality

Some states have different practices regarding turning off the SNC designation in RCRAInfo.

Formal action counts complete

12f1. sites with formal actions.

Mandatory data elements are listed in the RCRAInfo documentation.

all

State-only

RCRAInfo

Select sites with formal actions in 200,239 or in 250,339 or in 350,539 or in 550,799.

OTIS Management Reports

ERP and RCRAInfo

Data Quality

Corrective action excluded.

12f2. total formal actions taken.

Mandatory data elements are listed in the RCRAInfo documentation.

all

State-only

RCRAInfo

Select formal actions in 200,239 or in 250,339 or in 350,539 or in 550,799.

OTIS Management Reports

ERP and RCRAInfo

Data Quality

Corrective action excluded.

Assessed penalties complete

Total amount of final (assessed) penalties

Mandatory data elements are listed in the RCRAInfo documentation.

all

State-only

RCRAInfo

Select and sum the dollar amount of all penalties (final assessed penalties and SEP credits) at state enforcement actions (in 200,239 or in 250,339 or in 350,539 or in 550,799) in last FY.

OTIS Management Reports

ERP and RCRAInfo

Data Quality

File Type	application/vnd.ms-excel
Author	oecatech
Last Modified By	Rick Westlund
File Modified	2007-09-27
File Created	2004-03-18