2185.02ss.v12b

INFORMATION COLLECTION REQUEST

SUPPORTING STATEMENT

FOR: State Review Framework

September 27, 2007

(Revised on October 12, 2007)

U.S. Environmental Protection Agency

Office of Compliance

TABLE OF CONTENTS

Part A

1. IDENTIFICATION OF THE INFORMATION COLLECTION

1(a) Title of the Information Collection

1(b) Short Characterization/Abstract

2. NEED FOR AND USE OF THE COLLECTION

2(a) Need/Authority for the Collection

2(b) Practical Utility/Users of the Data

3. NONDUPLICATION, CONSULTATIONS, AND OTHER COLLECTION CRITERIA

3(a) Nonduplication

3(b) Public Notice Required Prior to ICR Submission to OMB

3(c) Consultations

3(d) Effects of Less Frequent Data Collection

3(e) General Guidelines

3(f) Confidentiality

3(g) Sensitive Questions

4. THE RESPONDENTS AND THE INFORMATION REQUESTED

4(a) Respondents/SIC Codes

4(b) Information Requested

5. THE INFORMATION COLLECTED: AGENCY ACTIVITIES, COLLECTION METHODOLOGY, AND INFORMATION MANAGEMENT

5(a) Agency Activities

5(b) Collection Methodology and Management

5(c) Small Entity Flexibility

5(d) Collection Schedule

6. ESTIMATING THE BURDEN AND COST OF THE COLLECTION

6(a) Estimating Respondent Burden

6(b) Estimating Government Burden

6(c) Bottom Line Burden and Cost Table

6(d) Reasons for Change of Burden

6(e) Burden Statement

Part B

1. SURVEY OBJECTIVES, KEY VARIABLES, AND OTHER PRELIMINARIES

   1. Survey Objectives

   2. Key Variables

   3. Statistical Approach

   4. Feasibility

2. SURVEY DESIGN

   1. Targeted Population and Coverage

   2. Sample Design

   3. Precision Requirement

   4. Questionnaire Design

3. PRETESTS AND PILOT TESTS

4. COLLECTION METHODS AND FOLLOW-UP

   1. Collection Methods

   2. Survey Response and Follow-Up

5. ANALYZING AND REPORTING SURVEY RESULTS

   1. Data Preparation

Part A

1. IDENTIFICATION OF THE INFORMATION COLLECTION

1(a) Title of the Information Collection

ICR: State Review Framework (EPA ICR Number 2185.02)

OMB Control Number: 2020-0031

AState Review Framework@

1(b) Short Characterization/Abstract

The existing ICR clearance under ICR 2020-0031 covers The State Review Framework (AFramework@). The Framework is an oversight tool designed to assess state performance in enforcement and compliance assurance. The Framework=s goal is to evaluate state performance by examining existing data to provide a consistent level of oversight. It is a uniform mechanism by which EPA Regions, working collaboratively with their states, can ensure that state environmental agencies are consistently implementing the national compliance and enforcement program and meeting agreed-upon goals and standards. Furthermore, the Framework is designed to foster dialogue on enforcement performance with states, which will enhance relationships and increase feedback, in turn leading to improved program management and environmental results.

The purpose of this ICR submission is to amend the existing approval to include the collection of information that will allow EPA to evaluate the full implementation of the State Review Framework.

The questionnaires are designed to collect relevant information from the states and territories that underwent the SRF reviews over the past three years. The first questionnaire will address the issues of the consistency of the reviews, whether the process was followed, the fairness of the review process, and then ask what improvements to the Framework are needed. The second questionnaire will address the issues of the metrics used to assess the states during the review. The third questionnaire will address resources issues so that we will be able to find efficiencies in conducting these reviews in the future.

2. NEED FOR AND USE OF THE COLLECTION

2(a) Need/Authority for the Collection

The original purpose of this collection was to assess state performance in enforcement and compliance assurance by examining existing data to provide a consistent level of oversight and develop a uniform mechanism by which EPA Regions, working collaboratively with their states, could ensure that state environmental agencies are consistently implementing the national compliance and enforcement program in order to meet agreed-upon goals. It is important to note that all data requested by the original collection is currently in EPA=s or the state=s databases and enforcement and compliance files.

This amendment to the ICR will allow EPA to collect information to support an evaluation of the recent three year implementation of the SRF, which is set to end on September 30, 2007, in order to assess its effectiveness and efficiency. These data will come from surveys of personnel in the EPA Headquarters and Regional Offices and the state environmental agencies. There are three survey instruments to be used under this request. The first is a general questionnaire to be administered to groups of EPA and state enforcement personnel and not on an individual basis. Questions are designed to be open ended, to allow the respondents to provide candid responses. The responses to the questions will be used by EPA and state managers to draw conclusions about specific aspects of the Framework implementation and will not be used to determine statistical significance or draw statistical inferences. The second is a questionnaire to obtain feedback from EPA and states about the data metrics used during the first round of reviews. The third survey is a questionnaire about the resources used by the respondents during the first round of state reviews. Again, no additional monitoring or sampling will be required by this ICR.

The Agency is permitted to review the states= Clean Air Act, Stationary Source program, the Solid Waste Disposal Act, Subtitle C program, and the Clean Water Act, National Pollutant Discharge Elimination System permit programs to ensure adequate performance. The Agency=s oversight authorities for these programs are:

(1) Clean Air Act, Stationary Source program:

Section 114 allows collection of information from states. Specifically, the collection of the requested information is authorized by 40 CFR 70.4(j)(1), which states that A[a]ny information obtained or used in the administration of a State program shall be available to EPA upon request without restriction and in a form specified by the Administrator, including computer-readable files to the extent practicable,@ and 40 CFR 70.10(c)(1)(iii), which addresses EPA oversight of State and local agencies= compliance and enforcement efforts for major sources under Title V operating permit programs.

(2) Resource Conservation and Recovery Act Subtitle C program:

The Act refers to activities at companies that generate hazardous waste. At '3007, the Agency is permitted to have access to and request records regarding hazardous waste generating activities. Additionally, 40 CFR 271.17(a) authorizes EPA, upon request without restriction, access to A[a]ny information obtained or used in the administration of a State program.@

(3) Clean Water Act, NPDES program:

The Act refers to activities involving the discharge of materials into waters of the United States. At '308, the Agency is permitted to review records to determine compliance with effluent limitations or treatment performance standards. Further, the NPDES state program regulations provide that A[a]ny information obtained or used in the administration of a State program shall be available to EPA upon request without restriction." 40 CFR 123.41. Also, 40 CFR 123.43 requires states to provide EPA with information on NPDES program implementation.

The information collected through this ICR will aid the Agency in achieving EPA=s Strategic Plan goal of focusing on results rather than activities within organizational units to increase compliance and environmental stewardship. This goal was developed in response to the 1993 Government Performance and Results Act and is described in EPA=s 2003 Strategic Plan, Goal 5, ACompliance and Environmental Stewardship.@

2(b) Practical Utility/Users of the Data

EPA will use the data obtained from the collection to determine the effectiveness and efficiency of EPA and states in implementing the State Review Framework and to draw inferences about the national compliance and enforcement program.

3. NONDUPLICATION, CONSULTATIONS, AND OTHER COLLECTION CRITERIA

3(a) Nonduplication

The information to be obtained under this ICR has not been collected by EPA or any other federal agency.

3(b) Public Notice Required Prior to ICR Submission

On June 14, 2007, EPA published a pre-ICR Federal Register Notice announcing its intent to submit an ICR to OMB regarding the State Review Framework. EPA received no comments from the public. EPA consulted with the following state organizations:

(1) Environmental Counsel of the States (ECOS)

(2) Association of State and Territorial Solid Waste Management Officials (ASTSWMO)

(3) Association of State and Interstate Water Pollution Control Administrators (ASIWPCA)

(4) National Association of Clean Air Administrators (NACAA)

EPA also consulted with individuals at these organizations and individuals from certain states:

(1) Name, phone number, affiliation

Name	Title	Affiliation	Phone Number
Lisa Jackson	Director, New Jersey Department of Environmental Protection (Co-chair, ECOS Compliance Committee)	ECOS	609-292-2885
Melanie Foster	Arkansas, Department of Environmental Quality (Chair ASTSWMO Compliance Committee)	ASTSWMO	501-683-0069
Paul Davis	Director, Water Pollution Control, Tennessee DEC (Co-chair, ASIWPCA Permitting and Compliance Task Force)	ASIWPCA	615-532-0625
Felicia Robinson	Administrator, City of Indianapolis Office of Environmental Services (Chair NACAA Local Agency Compliance Committee)	NACAA	317-327-2271

These state contacts and the membership of their respective organizations have worked in partnership with OECA on the State Review Framework and the design and implementation of the upcoming evaluation of this process. EPA has consulted widely with the membership of these associations on the Framework process and on the upcoming evaluation. Consultation with the states on the Framework has been extensive and ongoing since mid-2004. This included developing the logic model used for designing the evaluation. The consultation for this current questionnaire was conducted using input from a limited number of states who participate on the State Review Framework Work Group. OECA developed the questionnaire is based on the questions asked in the logic model. OECA shared the draft questionnaire with the states, via email, and EPA Regional participants on the Framework work group for their comments through conference calls. OECA received only a few but constructive comments from the states, which advised us to add a couple of critical questions and asked that the questionnaire be streamlined to eliminate redundancy. OECA accepted these comments and was able to streamline and shorten the questionnaire. The revised questionnaire was shared again with the work group for further discussed on a subsequent work group call, at which time the questionnaire was accepted by the group.

3(d) Effects of Less Frequent Data Collection

Each respondent will report only one time for each of the survey instruments

3(e) General Guidelines

This information collection is consistent with OMB guidelines contained in 5 CFR 1320.5(d) (2).

3(f) Confidentiality

Any information submitted to the Agency for which a claim of confidentiality is made will be safeguarded according to the Agency policies set forth in Title 40, chapter 1, part 2, subpart B - Confidentiality of Business Information (see 40 CFR 2; 41 FR 36902, September 1, 1976; amended by 43 FR 40000, September 8, 1978; 43 FR 42251, September 20, 1978; 44 FR 17674, March 23, 1979).

3(g) Sensitive Questions

The collection in this ICR does not contain any sensitive questions.

4. THE RESPONDENTS AND THE INFORMATION REQUESTED

4(a) Respondents and SIC Codes

The respondents for this ICR will be the following:

For the each of the survey questionnaires, there will be approximately 660 respondents from 50 states and 4 territories, and 20 local agencies. There are no SIC codes for the Respondents. For the resource information, there will be up to 50 state, four territory, and 10 local agency responses.

4(b) Information Requested

(i) Data items, including recordkeeping requirements

This evaluation requests information to assess the effectiveness and efficiency of the State Review Framework. There are no recordkeeping requirements associated with this collection.

The evaluation questions are attached.

(ii) Respondent Activities

Respondent will engage in the following activities during the Framework evaluation process:

Respond to evaluation survey questions regarding the effectiveness and efficiency of the State Review Framework.

Survey instrument will be administered to groupings of respondents using conference calls as the vehicle for reaching the respondents.

5. THE INFORMATION COLLECTED: AGENCY ACTIVITIES, COLLECTION METHODOLOGY, AND INFORMATION MANAGEMENT

5(a) Agency Activities

Administering the survey instrument.

5(b) Collection Methodology and Management

The Agency will provide respondents with the survey instruments. The Agency may be assisted by the state associations and an outside contractor in administering the survey instrument and compiling data.

5(c) Small Entity Flexibility

Small entities will not be affected as the collection will only be completed by States and territories.

5(d) Collection Schedule

The respondents will complete the evaluation survey instrument within a two to three month period of time. This will take place at the outset of the evaluation, which is scheduled to take no more than six months to complete.

6. ESTIMATING THE BURDEN AND COST OF THE COLLECTION

6(a) Estimating Respondent Burden and Costs

The estimated total hour burden per respondent is 8.59 hours. This burden hour estimate translates to a cost of $142 per respondent that voluntarily completes the survey resulting in the total of $72,512. There is no recurring respondent burden associated with this ICR. No capital or operations and maintenance costs are incurred by respondents under this ICR.

Table 1. Respondents= Burden and Costs

Survey Questionnaires for states and territories

The labor costs in the following table are based on the following average labor rates:

	Information Collection Activity	Total # of Responses (Note 1)	Total # of Respondents (Note 2)	Legal w/110%indirect cost rate $122/hr	# of Mng't Hrs	# of Mng't Respondents	Mng't w/110$ indirect cost rate $97/hr	# of Technical Hrs	# of Technical Respondents		Tech w/110% indirect cost rate ($65/hr)	Hr	Re	Clerical w/110% indirect cost rate ($42/hr)	Totals
1	General SRF Survey Questionnaire
	Conduct survey of 4 respondents in 54 States through the associations as groups. (Note 3)	54	216	0	2	54	$10,502	1	162	$20,928		0	0	0	N/A
	Conduct survey of up to 20 Local Agencies as groups. (Note 4)	20	40	0	2	20	$3,890	1	20	$2,584		0	0	0	N/A
2	SRF Metrics Survey Questionnaire
	Conduct survey of 4 respondents in 54 States through the associations as groups.	54	216	0	0.5	54	$7,876	2	162	$15,696		0	0	0	N/A
	Conduct survey of up to 20 Local Agencies as groups.	20	40	0	0.5	20	$2,917	2	20	$1,938		0	0	0	N/A
3	SRF Resource Survey Questionnaire
	Conduct survey of 54 states and territories.	54	108	0	0.5	54	$2,625	0.5	54	$1,292		0	0	0	N/A
	Conduct survey of up to 20 Local Agencies.	20	40	0	0.5	20	$972	0.5	20	$1,292		0	0	0	N/A
	Totals	222	512 (Note 6)	0	6	222	$28,783	7	438	$43,729		0	0	0	N/A
	Total # of Mng't Hrs					1,332
	Total # of Tech Hrs								3,066
	Total Estimated Burden Hours (Note 5)														4,398
	Total Estimated Cost (including 110% indirect cost rate)						$28,783			$43,729					$72,512
	Estimated Average Hours/Respondent														8.59
	Estimated Average Cost/Respondent														$142
	Note 1: Number of responses is the number of entities (state and locals) responding to the survey.
	Note 2: Number of respondents is the number of managers and technical staff responding to the surveys.
	Note 3: Survey questionnaire 1 will be shared with four entities in each of the 54 states and territories. They are: 1) environmental commissioners; 2) air administrators; 3) water administrators; & 4) waste administrators. Each questionnaire will be administered by the state association for commissioners (ECOS) and the media associations for the air (NACAA), water (ASIWCA), & waste (ASTSWMO). It is expected that the survey will be completed by the senior managers in each agency, with assistance from technical staff.
	Note 4: Survey questionnaire 2 Survey questionnaires 1 will be shared with four entities in each of the 54 states and territories. They are: 1) environmental commissioners; 2) air administrators; 3) water administrators) & 4) waste administrators. Each questionnaire will be administered by the state association for commissioners (ECOS) and the media associations for the air (NACAA), water (ASIWCA), & waste (ASTSWMO). It is expected that this survey will be completed by the technical staff for air, water, and waste in each state agency.
	Note 5: Total estimated burden is the sum of the total management cost and the total technical cost.
	Note 6: The 54 state and 20 local managers will be responding to all three of the questionnaires. Therefore, to avoid double counting, they are only included once in this count (660 - 148 = 512). The technical respondents for each of the questionnaires could be different since each instrument will require different expertise.

Legal: $54.65

Managerial: $44.20

Technical: $29.36

Clerical: $19.07

Indirect cost rate is calculated at110%. This is based on EPA standard multiplier in the ICR Handbook, Revised 11/05.

These rates are from the United States Department of Labor=s, Bureau of Labor Statistics, “Table 1. National employment and wage data from the Occupational Employment Statistics survey by occupation,” May 2006 http://www.bls.gov/news.release/ocwage.t01.htm.

6(b) Estimating Agency Burden

EPA Regions I through X and OECA will participate in the State Review Framework process and the following table details the hour and cost burden per region.

Table 2. Agency Burden and Costs

	Information Collection Activity	Legal $81/hr	# of Mng't Hrs	# of Mng't Respondents	Mng't $69/hr	# of Mng't Hrs	# of Mng't Respondents	Technical $51/hr	Clerical $28/hr	Totals
1	General SRF Survey Questionnaire
	Conduct survey of 10 EPA Regional Offices.	0	2	10	20	2	10	20	0
2	SRF Metrics Survey Questionnaire
	Conduct survey of 10 EPA Regional Offices.	0	1.5	10	15	1.5	10	15	0
3	SRF Resource Survey Questionnaire
	Conduct survey of 10 EPA Regional Offices.	0	0.5	10	5	1.5	10	15	0
	Total number of hours				40			50	0	90
	Total number of respondents			30			20			50
	Total costs		4		$2,759.68	5		$2,559.20	0	$5,318.88
	Total Estimated Cost	$5,318.88
	Total Estimated Hours	90.0
	Estimated Average Hours/Respondent	1.80
	Estimated Average Cost/Respondent	$106.38

Legal $50.93 (GS-14, Step 5)

Managerial $43.12 (GS-13, Step 5)

Technical $31.99 (GS-12, Step 1)

Clerical $17.31 (GS-6, Step 3)

These rates are from the Office of Personnel Management (OPM) AJanuary 2007 General Schedule@ and include the locality payment for the Washington D.C. area.

Indirect cost rate is calculated at 60%. This is based on EPA standard multiplier in the ICR Handbook, Revised 11/05.

6(c) Bottom Line Burden Hours and Cost Tables

Table 3. Total Estimated Respondent Burden and Cost Summary (over 3 years)*

	No. of Respondents	No. of Activities	Total Hours	Total labor Costs	Total Capital and Start up costs	Total annual O&M costs
Respondents for original collection (2185.01)	40*	9	15,367.20*	$507,103.60	$0.00	$0.00
Respondents for new collection (2185.02)	512	3	4,398	$72,512	$0.00	$0.00
Totals	552	12	19765.20	579615.60	$0.00	$0.00

*(For the 2185.01 original ICR the annual Respondent number is 13 and the annual Hours number is 5122. This makes the Total annual respondents 579 and Total annual Hours 6234)

Table 4. Total Estimated Agency Burden and Cost Estimate

	No. of Activities	Total hours	Total labor Costs	Total Capital and Start-up costs	Total Annual O & M costs
Respondents for original collection (2185.01)
Regions	9	3055	$95,178.00	$0.00	$0.00
Headquarters	6	193	$5,804.50	$0.00	$0.00
Respondents for new collection (2185.02)
Headquarters and Regions	3	90	$3,130.90	$0.00	$0.00
Totals	16	3338	$104,113.40	$0.00	$0.00

6(d) Reasons for Changes in Burden

The reason for the change in the burden increase for this request is to allow as many managers and staff of the state and local environmental agencies throughout the country to participate in the evaluation of the State Review Framework. The original request was to gather information in state enforcement and compliance files. The current request is to conduct an evaluation of the State Review Framework, which will require that we approach many more individual respondents and ask that they respond to one or more survey instruments.

6(e) Burden Statement

Burden Statement: The annual public reporting and recordkeeping burden for this collection of information is estimated to average 8.59 hours per response. Burden means the total time, effort, or financial resources expended by persons to generate, maintain, retain, or disclose or provide information to or for a Federal agency. This includes the time needed to review instructions; develop, acquire, install, and utilize technology and systems for the purposes of collecting, validating, and verifying information, processing and maintaining information, and disclosing and providing information; adjust the existing ways to comply with any previously applicable instructions and requirements; train personnel to be able to respond to a collection of information; search data sources; complete and review the collection of information; and transmit or otherwise disclose the information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control numbers for EPA's regulations are listed in 40 CFR part 9 and 48 CFR chapter 15.

To comment on the Agency's need for this information, the accuracy of the provided burden estimates, and any suggested methods for minimizing respondent burden, including the use of automated collection techniques, EPA has established a public docket for this ICR under Docket ID Number EPA-HQ-OECA-2007-0466, which is available for online viewing at www.regulations.gov, or in person viewing at the EPA Docket Center, EPA West, Room 3334, 1301 Constitution Avenue, NW., Washington DC 20460. The EPA Docket Center Public Reading Room is open from 8:30 a.m. to 4:30 p.m., Monday through Friday, excluding legal holidays. The telephone number for the Reading Room is (202) 566-1744, and the telephone number for the EPA Docket Center is (202) 566-1514. An electronic version of the public docket is available at www.regulations.gov. This site can be used to submit or view public comments, access the index listing of the contents of the public docket, and to access those documents in the public docket that are available electronically. When in the system, select “search,” then key in the Docket ID Number identified above. Also, you can send comments to the Office of Information and Regulatory Affairs, Office of Management and Budget, 725 17th Street, NW, Washington, D.C. 20503, Attention: Desk Officer for EPA. Please include the EPA Docket ID Number, EPA-HQ-OECA-2007-0466, and OMB Control Number, 2020-0031, in any correspondence.

Part B of the Supporting

1. Survey Objectives, Key Variables, and Other Preliminaries

1. Survey Objectives

The key study questions for the evaluation that the surveys will address are:

1. Has the SRF program improved the consistency across state compliance and enforcement programs?

2. Has the SRF program improved the consistency of EPA oversight of state compliance and enforcement programs?

3. Were the SRF reviews implemented in a collaborative fashion? To assess this we will look at:

4. What can be done to improve the efficiency and effectiveness of the SRF review process including data and file metrics and resources used to conduct the reviews?

5. What value have states, EPA regions, and OECA derived from the SRF reviews and approach, e.g., improvement to enforcement programs, benefits to states, understanding of existing flexibility, Element 13?

6. How can we begin to utilize differential oversight in future oversight strategies, now that the first round of oversight has been conducted and national baselines established?

2. Key Variables

The key variables are consistency, efficiency, effectiveness, and fairness of the Framework implementation. The evaluation is not using statistical methods to evaluate this activity. Thus no statistical method is being employed to analyze the information.

3. Statistical Approach

No statistical approach is being used to analyze the information.

4. Feasibility

The surveys are designed to utilize open ended questions about the Framework process. Respondents might face the obstacle of not having first hand knowledge of a particular question. The surveys are being administered to personnel at different levels of each organization, so if a respondent does not know answer, another person in their organization can be of assistance. This may be more the case regarding the metrics, which will require more technical expertise.

There are sufficient funds to complete the survey as designed. OECA has received $110,000 in contract support from OPEI to conduct this evaluation. We also have the support of ECOS and the media associations (NACAA, ASIWPCA, and ASTSWMO) who will help to facilitate the distribution and completion of the surveys.

The survey results will be ready in time to help us to make the decisions we need in order to assess the Framework implementation and make improvements for the next three-year cycle.

2. Survey Design

1. Targeted Population and Coverage

To ensure that everyone who underwent a State Review Framework review in the states, territories, and local agencies has an opportunity weigh in on the process, each survey is designed to reach each of the potential members of this population. The respondent pool is the universe of state, territory, and local environmental agencies that underwent a review based on the State Review Framework. That would include 50 states, four territories, and approximately 10 local agencies. Each entity will have an opportunity to participate in the surveys. In each of the state and territorial agencies, there the respondents will include the state commissioner and representatives from each of the three media program offices that were reviewed (air, water, and waste).

The first survey questionnaire will be administered to each of these groups in a series of conference calls that will be set up by media. ECOS will administer the survey to the environmental commissioners. NACAA will administer it to the air managers. ASIWPCA will administer it to the water managers, and ASTSWMO will administer it to the waste managers. Each association will develop, in coordination with OECA a specific methodology for doing this. On the main, they will have a series of call with no more than five respondents in order to obtain the best possible participation. The contractor will be on each call to help facilitate the survey and to record the responses.

The other two surveys, one about resources and one about metrics, will be distributed to each of the states, territories, and local agencies to complete. The results of the resource survey will be analyzed by the contractor and OECA. The results of the metrics survey will be analyzed by a work group of EPA and state personnel working as part of the evaluation process to improve the metrics.

2. Sample Design

The sample design is to reach each of the agencies that underwent a State Review Framework review.

3. Precision Requirement

The analysis of the information gained from the surveys will not utilize statistical methods.

4. Questionnaire Design

The questionnaires designed for this evaluation were designed to specifically address the study questions identified in section 1 a above.

The first questionnaire identifies a series of question that will help us to answer the overarching questions. This links up with all six of the overarching study questions on a one-to-one basis.

The second questionnaire for the metrics is designed to address the issues in study question 4. This instrument asks about the effectiveness of each of the metrics.

The third questionnaire will collect resource information and also addresses issues in study question 4. This instrument asks for factual information on the use of agency resources during the review process.

Each of these questionnaires was developed in coordination with the association mentioned above in order to ensue that the questions are not duplicative and that the help us to address the right issues and to ensure their reliability.

3. Pretests and Pilot Tests

The survey questionnaires were not pre-tested as such. The draft questions were shared with the members of the work group, which is made up of EPA and state representatives from the state associations, who are the respondent population. They provided useful comments that lead to sharpening and reducing the number of questions in the questionnaires.

4. Collection Methods and Follow-Up

1. Collection Methods

In order to ensure that the data collected is accurate and useful to the evaluation, the persons who will be responding to the questionnaires will be the persons who participated in the State Review Framework reviews in each of the states, territories, and local agencies.

The first questionnaire will be responded to by the senior managers at the commissioner level in those agencies who oversaw the review process, and the technical managers for each of the three media programs (air, water, and waste).

The second questionnaire will be responded to by the technical managers for each of the media programs.

The third questionnaire will be responded to by the senior managers and the technical managers.

2. Survey Response and Follow-Up

The response rate is expected to be 100%. This may very, but because of the methodology of having conference call and the interest of agencies that were reviewed under the State Review Framework, it is anticipated that we will reach this target. There will be provisions for any agency that is not able to participate on these calls to receive the questionnaires by email and given an opportunity to respond. Each of the state associations will assist in following up with their membership to ensure the maximum possible follow-up.

5. Analyzing and Reporting Survey Results

1. Data Preparation

There will be no electronic data processing. The information derived from the questionnaires will be qualitative data and will be arrayed in spreadsheets for the contractor and the work group members to analyze and assess.

2. Analysis

As noted above, the data will be arrayed in spreadsheets. There will be no regression analysis or other statistical methods employed. Further analysis of the data will be employed once the data has been collected.

16