NOAA B-WET SS 102512 Revised 020213 Part A

SUPPORTING STATEMENT

NOAA BAY WATERSHED EDUCATION AND TRAINING (B-WET) PROGRAM NATIONAL EVALUATION SYSTEM

OMB CONTROL NO. 0648-xxxx

A. JUSTIFICATION

1. Explain the circumstances that make the collection of information necessary.

This request is for a new information collection.

The NOAA Office of Education’s Bay Watershed Education and Training (B-WET) program seeks to contribute to NOAA’s mission by immersing participants in Meaningful Watershed Education Experiences (MWEEs) in order to create an environmentally literate citizenry with the knowledge, attitudes, and skills needed to protect watersheds and related ocean, coastal, and Great Lakes ecosystems (http://www.oesd.noaa.gov/grants/bwet.html). B-WET currently funds projects in seven regions: California, Chesapeake Bay, Great Lakes, Gulf of Mexico, Hawaii, New England, and the Pacific Northwest.

In keeping with Executive Order 12862, Setting Customer Service Standards, B-WET proposes to create a cross-region, internal evaluation system to monitor program implementation and outcomes on an ongoing basis. An on-going collection is proposed because the intent of the evaluation system is to support on-going program improvement. Based on a review of annual evaluation system results, B-WET will make adjustments to its Federal Funding Opportunities (FFOs) and activities. On-going data collection is proposed to enable assessment of the benefits of continuous improvements and, thus, support adaptive management of the program. This effort is related to the NOAA Education Strategic Plan 2009-2029. See in particular Outcome 1.1 on page 16, “Evaluating Education.”

To meet this need, B-WET seeks to answer the following questions as part of the proposed evaluation system:

1. To what extent do regional B-WET programs support grantees in implementing Meaningful Watershed Educational Experiences (MWEEs)?

2. How are MWEEs implemented by grantees and teachers?

3. To what extent do B-WET-funded projects increase teachers’ knowledge of watershed science concepts, their confidence in their ability to integrate MWEEs into their teaching practices, and the likelihood that they will implement high quality MWEEs?

4. To what extent do B-WET-funded projects increase students’ knowledge of watershed concepts, attitudes toward watersheds, inquiry and stewardship skills, and aspirations towards protecting watersheds?

B-WET grantees and teacher-participants in the grantees’ professional development will be asked to voluntarily complete online questionnaires to provide evaluation data. One individual from each grantee organization will be asked to complete a questionnaire once per year of the award, and the teacher-participants will be asked to complete one questionnaire at the close of their professional development (PD) and one after implementing MWEEs with their students (before the end of the following school year). A database will be used to collect and store these data, as well as to automatically generate results in the form of aggregate descriptive statistics.

The proposed evaluation system will be maintained by B-WET staff with occasional assistance from an external contractor.

2. Explain how, by whom, how frequently, and for what purpose the information will be used. If the information collected will be disseminated to the public or used to support information that will be disseminated to the public, then explain how the collection complies with all applicable Information Quality Guidelines.

Program Improvement

The proposed evaluation system, influenced by the principles underlying utilization-focused evaluation (Patton, 2008), was specifically designed by a team of researchers from the University of Michigan (UM) and the Institute of Learning Innovation (ILI) to meet users’ information and decisions needs. The primary users of the proposed evaluation system will be the B-WET staff members who administer the B-WET grant program, and its national coordinator. These individuals will review the evaluation system’s results annually to determine what changes may be necessary to the grant program to maximize benefits for K-12 teachers and students. The system will automatically generate results in the form of aggregate descriptive statistics (at the national and regional level) to inform decisions about the program at both of these tiers.

Secondary users will be staff in the NOAA Office of Education who will also annually review a synthesis of findings from B-WET’s evaluation to determine what they may suggest for improving its other education programs. Evaluation findings will also be used at the national level to report on agency performance measures and respond to other Administration data collection activities, such as the annual Committee on Science, Technology, Engineering, and Math Education (CoSTEM) survey. Tertiary users will be grant recipients who will also be provided with access to a synthesis of findings so that they may identify ways to improve their respective environmental science and education programs.

Public Dissemination

It is also anticipated that a synthesis of findings will be disseminated to the public or used to support publicly disseminated information. The proposed data collection will therefore ensure that the Information Quality Guidelines of utility, objectivity, and integrity are met.

Utility:

The proposed evaluation system is designed to answer the questions described earlier in Question 1, to primarily meet B-WET’s decision needs. To answer these evaluation questions, the ILI-UM team of researchers first identified relevant constructs (based on B-WET’s logic model and MWEE characteristics). Next, they adapted and adopted items to measure these constructs from existing valid and reliable indices and scales or developed new ones (when existing ones were not available). As a result, only data which has a necessary purpose for answering the system’s evaluation questions and, thus, meeting B-WET’s information needs, will be collected. Please refer to the proposed evaluation system metrics matrix illustrating the connections between evaluation questions, constructs, and items included in the proposed instruments (posted as a supplementary document).

Objectivity:

Presentation: The descriptive statistics (e.g., frequencies) that will be automatically generated based on the online data collected from respondents will be accurate, clear, complete, and unbiased. In addition, only aggregate statistics at the national and regional level will be reported. Thus, individual sources of data will not be disclosed and study participants will remain anonymous.

Substance: The items included in the questionnaire, as well as the questionnaires themselves, which respondents will be asked to voluntary complete, were developed by the ILI-UM team based on best social science research practices. The majority of items included in the questionnaire, for example, were adopted or adapted (with respective researchers’ permission) from existing studies, including the “Evaluation of National Oceanic and Atmospheric Administration Chesapeake Bay Watershed Education and Training Program” and Executive Summary (data gathered under OMB Control Number: 0648-0530), a recent exploratory study of the benefits of Meaningful Watershed Education Experiences (Zint, In Process), and a range of other relevant science and environmental education studies published in peer-reviewed journals. New items were developed only when existing measures for a construct were not available. The face and content validity of all of the items in the proposed questionnaires were established through reviews by the nine internal NOAA B-WET Advisory Group (BWAG) members, three B-WET grantees, three evaluation specialists, and two watershed science researchers. Face validity is established by showing the questionnaire to a group of experts (e.g., researchers, practitioners) and asking them for feedback on whether the measures look like they’ll measure the constructs. We established face validity with review by B-WET, evaluators, grantees, and teachers. For content validity, we consulted with these experts and also did an extensive literature review.

Based on exploratory factor analyses conducted with SPSS and M+ by Zint (In Process), the scales that will be used by the proposed evaluation system are expected to have good to excellent reliability (i.e., Cronbach Alpha range: .70 to .90) (Nunally & Bernstein 1994; Carmines & Zeller, 1979). Similarly, because the respective factors explained a substantial amount of variance (i.e., range: 40% to 90%) in the Zint (In Process) study, there exists additional support for the validity of the measures to be used by the proposed evaluation system.

Integrity:

NOAA’s Office of Education will retain control over the information and safeguard it from improper access, modification, and destruction, consistent with NOAA standards for confidentiality, privacy, and electronic information. See response to Question 10 of this Supporting Statement for more information on confidentiality and privacy. The information collection is designed to yield data that meet all applicable information quality guidelines. Prior to dissemination, the information will be subjected to quality control measures and a pre-dissemination review pursuant to Section 515 of Public Law 106-554.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological techniques or other forms of information technology.

The proposed evaluation system will be electronic. Study participants (i.e., B-WET grantees and teachers who participate in their professional development) will receive email prompts to complete the online instruments accessed through a Web site data entry portal. The portal will be designed in a questionnaire format with built-in “logic” that will prompt respondents to complete

only items relevant to their experience. Data will be stored as part of an associated database that will also automatically generate descriptive statistics. The proposed data collection process will minimize costs, while also being sensitive to issues of respondent burden, accuracy, and efficiency. It is assumed that most respondents will have access to the Internet at work, home, on a smartphone, or at a public institution such as a local library. Individuals who do not have easy access to the Internet can contact their B-WET regional coordinator (name and contact information will be provided) to obtain a paper copy of the questionnaire and a return envelope.

4. Describe efforts to identify duplication.

No other NOAA programs are surveying B-WET grantees or the teachers that participate in their respective Meaningful Watershed Educational Experience professional development.

5. If the collection of information involves small businesses or other small entities, describe the methods used to minimize burden.

The proposed study will ask individuals working for small non-profit organizations and some small businesses, as well as teachers in schools, to participate by completing the proposed questionnaires. The study will minimize burden on respondents because completion of the proposed questionnaires is voluntary. In addition, an iterative item review process was used to eliminate any non-essential questions, thus keeping the questionnaires as streamlined as possible while ensuring that sufficient data will be collected to answer the evaluation questions. Should they choose to complete the proposed questionnaires, grantees should be able to complete their questionnaire within 30-60 minutes (depending on the nature of their program) and teachers, within 30 minutes.

6. Describe the consequences to the Federal program or policy activities if the collection is not conducted or is conducted less frequently.

The proposed evaluation system will contribute to ensuring that federal funding is used in an effective and efficient manner to educate teachers and students about watershed science and environmental issues. B-WET will have scientific data to assess the effectiveness of their grant funded programs (i.e., B-WET-funded teacher professional development and student MWEEs). The results of the proposed study will also provide insight into how to design improved watershed education programs.

If the study were not conducted, B-WET would not have access to scientifically assess the effectiveness of its program/MWEEs and/or to scientifically determine how best to improve its program/MWEEs.

Continuous data collection, as proposed through the evaluation system, will allow on-going monitoring of outcome results and thus, on-going program/MWEE improvements.

7. Explain any special circumstances that require the collection to be conducted in a manner inconsistent with OMB guidelines.

The collection will be conducted in a manner consistent with OMB guidelines.

8. Provide information on the PRA Federal Register Notice that solicited public comments on the information collection prior to this submission. Summarize the public comments received in response to that notice and describe the actions taken by the agency in response to those comments. Describe the efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported.

A Federal Register Notice published on June 5, 2012 (77 FR 33194) solicited public comments. No comments were received.

During the development of the B-WET evaluation system, the ILI-UM team solicited input from a range of individuals including B-WET grantees, evaluation experts, watershed scientists, and statisticians on all aspects of the proposed evaluation system. Their suggestions informed the design of the proposed study (e.g., type of data collection, frequency and timing of data collection, reporting formats, etc.). Their feedback was also used to improve the questionnaire items and led to confirmation of their face and content validity.

In addition, the grantee and teacher questionnaires will include several measures at the end of the respective instruments to allow respondents to comment on the data collection process and content. This feedback can be used to improve both the data collection process and instruments over time.

9. Explain any decisions to provide payments or gifts to respondents, other than remuneration of contractors or grantees.

Incentives, in the form of financial compensation or material gifts, are known to increase response rates (Dillman et al., 2009; James & Bolstein, 1990). Because NOAA is a federal agency, however, it cannot offer such an incentive to grantees. Therefore, NOAA B-WET will encourage grantees to ask teachers to complete the surveys as part of their other professional development responsibilities. For example, if the grantees provide stipends to their professional development teachers, they could include a requirement that teachers complete the questionnaire to receive the payment.

10. Describe any assurance of confidentiality provided to respondents and the basis for assurance in statute, regulation, or agency policy.

An assurance of confidentiality will not be provided to respondents. B-WET grantees and teachers who respond to the questionnaires will remain anonymous to B-WET and NOAA.

Anonymity will be guaranteed in the following ways:

• Neither B-WET grantees nor teacher respondents will be asked to provide information that can identify them as individuals as part of the questionnaire.

• Information that is needed to link data, that is (1) award numbers to link data provided by grantees with teachers participating in their professional development and (2) teacher-generated codes to link responses to their initial and subsequent questionnaires, will not be associated with any of the other data they provide.

• Email addresses, used to (1) invite prospective participants to participate in the study with a link to the questionnaire and (2) track response rates and prompt non-respondents, will not be associated with any of the data provided by respondents.

• Results will only be presented in aggregate form (across all grantees and teacher respondents), not by individual or grant.

Finally, the database is also being designed to meet Federal Information Security Management Act (FISMA) security guidelines to ensure all the data provided by respondents will be secure.

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private.

No questions of a sensitive nature will be asked.

12. Provide an estimate in hours of the burden of the collection of information.

Table 1 provides estimates of the time and cost burden for the proposed information collection. The numbers of possible respondents indicated in the table are estimated from B-WET’s FY2010 budget, which was the year of the maximum level of funding historically received by the program. Future numbers of respondents will vary based on annual program funding and the resources grantees are able to leverage.

Table 1: Estimate of Burden Hours for Information Collection

Informant	Number of possible respondents annually	Response frequency	Expected number of responses	Average time per response (hours)	Total respondent time (hours)	Estimated hourly wage (dollars)	Estimated labor cost burden to respondents (dollars)
Grantees	125	1	113b	1.0	113	45.75f	5,170
PD teachers	4,000	1	3,200c	0.5	1,600	38.83g	62,128
PD teachers nonresponse		1	800d	0.1	80	38.83g	3,106
MWEE teachersa	3,200	1	1,280e	0.5	640	38.83g	24,851
MWEE teachers nonresponse		1	1,920d	0.1	192	38.83g	7,455
TOTALS	7,325		7,313		2,625		102,710

^a Some of the PD teachers may respond as MWEE teachers in the same year, but because that number is unknown, the highest possible number of respondents is used to calculate burden hours.

^b Assumes 90% response rate.

^c Assumes 80% response rate.

^dAssumes 100% response rate for nonrespondents to calculate maximum possible burden hours

^e Assumes 40% response rate.

^f U. S. Department of Labor, Bureau of Labor Statistics. May 2011. National Compensation Survey: Occupational Earnings in the United States, 2010. Table 5: Full-time State and local government workers: Mean and median hourly, weekly, and annual earnings and mean weekly and annual hours: Education administrators. http://www.bls.gov/ncs/ocs/sp/nctb1479.pdf

g U. S. Department of Labor, Bureau of Labor Statistics. May 2011. National Compensation Survey: Occupational Earnings in the United States, 2010. Table 5: Full-time State and local government workers: Mean and median hourly, weekly, and annual earnings and mean weekly and annual hours: Secondary School Teachers. http://www.bls.gov/ncs/ocs/sp/nctb1479.pdf

13. Provide an estimate of the total annual cost burden to the respondents or record-keepers resulting from the collection (excluding the value of the burden hours in Question 12 above).

There are no direct costs to participants. The only costs are the opportunity costs of respondents’ time required to provide information as explained in Question12 above. No capital equipment, start-up, or record maintenance requirements are placed on respondents.

14. Provide estimates of annualized cost to the Federal government.

The estimated cost to the federal government of implementing the NOAA B-WET National Evaluation System is based on the government's contracted cost for yearly maintenance of the data collection database, periodic study and analysis activities, and personnel cost of government employees involved in oversight and/or analysis. For the data collection activities for which OMB approval is currently being requested, the overall cost to the government is $267,000 over a three year period. This includes:

• $25,000 annualized( total over three years $75,000) for contracted activities including evaluation contractor oversight of initial data collection, ongoing database development, analysis of data, and report writing;

• $24,000 annually ($2,000/month, $72,000 over three years) for government personnel costs in hosting and maintaining the evaluation database; and

• $40,000 annually ($120,000 over three years) for government personnel costs in overseeing the evaluation activity.

Annualized cost: $25,000 + $24,000 + $40,000 = $89,000.

It is expected that the annual government personnel costs would need to be sustained over the duration of the use of the evaluation system, with periodic contracted work to analyze data and produce evaluation reports. These estimates are based on the evaluation contractor's previous experience managing other research and data collection activities of this type.

15. Explain the reasons for any program changes or adjustments.

This is a new program.

16. For collections whose results will be published, outline the plans for tabulation and publication.

For the primary stakeholders and users of the proposed evaluation system, i.e., the internal NOAA B-WET staff members who administer the B-WET grant program, the database will automatically share results as aggregate descriptive statistics (at the national and regional levels). For each question, the system will indicate how many individuals responded, the frequency with which a particular response option was selected. Should funding be available for more sophisticated analysis of the data (i.e., inferential statistics), a contractor will be hired who will produce a traditional research report and/or article for publication in a peer reviewed report consisting of introduction, methods, results, and discussion/recommendation sections.

Depending on the availability of the necessary funding, annual syntheses of the main findings as related to the questions the evaluation system was designed to answer (see #1 above) will be prepared to meet the needs of different stakeholder groups. These stakeholders include the NOAA Office of Education which seeks information to improve its other education grant programs and external stakeholders such as B-WET grantees and teacher participants seeking ways to improve their MWEE practices, as well as tertiary members of the public. B-WET (potentially with the help of a contractor) will prepare these syntheses, ensuring that they meet respective stakeholders’ needs both in terms of content and presentation. These syntheses will be made available online.

17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons why display would be inappropriate.

NA.

18. Explain each exception to the certification statement.

NA.

10