Supporting Statement (3145-0136)

REQUEST FOR CLEARANCE

 

National Science Foundation

Directorate of Education and Human Resources

Division of Undergraduate Education

Robert Noyce Teacher Scholarship Program

Monitoring System (Noyce)

 

 

Attachment F

Section A

Introduction

This request for Office of Management and Budget (OMB) review asks for a renewal of clearance for the monitoring data collection for the  Robert Noyce Teacher Scholarship Program (Noyce) that is part of the National Science Foundation's (NSF) Directorate for Education and Human Resources (EHR) Generic Clearance (OMB 3145-0136), which expires on March 31, 2011. The Noyce program is administered by the EHR Division of Undergraduate Education (DUE).

A.1. Circumstances Requiring the Collection of Data

The National Science Foundation Authorization Act of 2002 (P.L. 107-368) stated that the “[NSF] Director shall carry out a program to award grants to institutions of higher education (or consortia of such institutions) to provide scholarships, stipends, and programming designed to recruit and train mathematics and science teachers. Such program shall be known as the ‘Robert Noyce Scholarship Program’.” The legislation requires award recipients to provide monitoring information to the NSF: “An institution of higher education (or consortium thereof) receiving a grant under this section shall, as a condition of participating in the program, enter into an agreement with the Director to monitor the compliance of scholarship and stipend recipients with their respective service requirements.” The Noyce program solicitation states: “…In addition to the project-specific evaluation, all projects will be expected to cooperate with an NSF third-party monitoring and evaluation of program impact that will require annual data collection.”

 

Track I of the program provides funding to institutions of higher education to provide scholarships, stipends, and programmatic support to recruit and prepare STEM majors and STEM professionals to enter and complete teacher credentialing programs and become K-12 mathematics and science teachers. Scholarship and stipend recipients are required to complete two years of teaching in a high-need school district for each year of support. The program seeks to increase the number of K-12 teachers with strong STEM content knowledge.

 

In addition, Track II of the program supports STEM professional who enroll as NSF Teaching Fellows in master’s degree programs leading to teacher certification or licensure by providing academic courses, professional development, and salary supplements while they are fulfilling a four-year teaching commitment in a high-need school district. This new component also support the development of NSF Master Teaching Fellows by providing professional development and salary supplements for exemplary mathematics and science teachers to become Master Teachers in high-need school districts.

 

 

The Noyce monitoring system is the primary source of data on project activities. The monitoring system consists of a Web-based instrument that is completed by the principal investigator (PI) of each Noyce award (also called project or grantee).

 

The data collection instruments are included in appendices A1 and A2.

A.2. Purposes and Uses of the Data

The information collected in this task is required for effective administration, communication, and program and project monitoring; for meeting reporting requirements; for measuring attainment of NSF’s program, project, and strategic goals as laid out in NSF’s Strategic Plan; and as a baseline for future program evaluations.

 

The primary purpose of this collection is to provide data and information for effective program management and monitoring of program activities. This data collection activity is designed to track the extent to which Noyce awards meet the objectives of the program. This information is used to administer and monitor the progress of the program in the various institutions. Those findings are used to recommend, among other things, administrative changes in program functions; level of scholarship, stipend, and fellowship support; individual program focus and emphasis; and recruiting efforts.

In recent guidance from the Director of OMB, M-10-32, the need for rigorous evaluations and the objectives of evaluations of programs were clearly outlined, including the use of evaluation resources. Because the collection of data contained in this monitoring effort contributes to the formal evaluation of the program and provides regular measures of program performance by accumulating operating information from each project in the program, this guidance is particularly pertinent to this request.

“Improving and coordinating the use of existing evaluation resources. In addition to the voluntary evaluation initiative, agencies should continue to carefully assess, report on, and allocate the base funds and resources that the agencies have for conducting evaluation. Agencies are encouraged to share information beyond what is requested in guidance and consult with OMB’s Resource Management Offices (RMOs) to coordinate and improve the design, implementation, and utilization of evaluations.”

These directives establish an ongoing need for NSF to engage in an interactive process of collecting information and using it to improve program services and processes.

The Noyce program also uses the data to fulfill reporting requirements. As a part of its performance assessment activities, NSF relies on the judgment of external experts to maintain high standards of program management. Directorate and Office advisory committees (ACs) meet twice a year, while Committees of Visitors (COVs) for divisions or programs meet once every three years. Data collected in the Noyce monitoring system may be used to report to these committees on program activities. In addition, NSF is required to measure the attainment of its program, project, and strategic goals as identified by the President’s Accountable Government Initiative, the Government Performance and Results Act (GPRA) Modernization Act of 2010, and the NSF’s Strategic Plan. Data collected in the system help NSF management examine their progress towards the Foundation’s goals and respond to these reporting requirements.

Finally, the data can also be used as a preliminary step in more detailed future evaluation efforts. EHR makes these data available to NSF staff, EHR contractors with responsibility for the collection, and DUE program managers and their staff and contractors. Information collected may also be disseminated, in aggregate form, to current and prospective applicants to the Noyce program and to the broad science, technology, engineering, and mathematics education community upon request to and approval from the NSF.

Under the Noyce monitoring system, each PI of a Noyce award is required to provide annual data using the Web-based data collection system. The following is an overview of the types of information collected:

 

Data on scholarship, stipend, and fellowship recipients include:

·         Individual data including name, institution, gender, ethnicity, race, disability status, date of birth, previously held degrees (stipend and fellowship recipients only), previous occupation and year in previous occupation (stipend and fellowship recipients only)

·         Educational information, including activities, intended career level as a teacher, intended subject area(s)/discipline(s) of teaching, level of support in dollar amount, number of academic terms of support, anticipated date of completion of program, anticipated date of completion of master’s degree program (teaching fellowship recipients only), anticipated date of certification or licensure, cumulative grade point average (GPA), actual date of completion of program, and major

·         Additional information, including date of certification or licensure, grade level(s) and subject area(s) as listed on teaching certificate or license, teaching placement (including grade level(s) and subject area(s)/discipline(s) of courses teaching/taught), number of years of teaching (master teaching fellowship recipients only), number of years of teaching in a high-need school district, school, level of salary supplement in dollar amount (fellowship recipients only), recognition as a master teacher (master teaching fellowship recipients only)

Regarding other Noyce project participants the system collects:

 

·         Data on the discipline of each higher education faculty member involved in the project during the reporting period

·         Data on each of the post-secondary institutions involved in the project, including institution name and state, number of STEM majors who graduated and prepared to become K-12 mathematics and science teachers during the reporting period, number of STEM post-baccalaureates who completed the teacher preparation program and prepared to become K-12 mathematics and science teachers during the reporting period, and number of STEM career changers who completed the teacher preparation program and prepared to become K-12 mathematics and science teachers during the reporting period

·         Data on the programmatic activities conducted by the participating institutions, including mentoring, field experiences, and peer tutoring

·         Aggregated data on applicants to individual projects, including number of scholarship/stipend/fellowship applicants and cumulative GPA at the time of application (scholarship and teaching fellowship applicants only)

·         Data on each school district involved in the project, including name, city and State, and number of schools involved in the project

·         Data on each nonprofit organization involved in the project, including name, city and State, and activities

·         Project baseline for each post-secondary institution, including number of STEM majors who graduated and prepared to become K-12 mathematics and science teachers one year prior to the award, number of STEM post-baccalaureates who completed the teacher preparation program and prepared to become K-12 mathematics and science teachers one year prior to the award, and number of STEM career changers who completed the teacher preparation program and prepared to become K-12 mathematics and science teachers one year prior to the award

 

As is required under OMB’s Terms of Clearance (TOC) for requests made under the EHR Generic Clearance, a summary crosswalk has been prepared to demonstrate how the requested information conforms to the scope of the EHR Generic. The crosswalk for Track I (appendix B1) and the crosswalk for the new Track II (appendix B2) provide the complete list of data elements. In addition, appendices A1 and A2 provide screenshots of the Track I and Track II versions of the Web-based instrument.

A.3. Use of Information Technology To Reduce Burden

EHR tends to favor Web-based systems because they can facilitate respondents’ data entry across computer platforms. One innovative feature of many of the individual Web systems is the thorough editing of all submitted data for completeness, validity and consistency. Editing is performed as data are entered. Most invalid data cannot enter the system, and questionable or incomplete entries are called to respondents’ attention before they are submitted to NSF. Web-based data collection instruments employ user-friendly features such as automated tabulation, data entry with custom controls such as checkboxes, data verification with error messages for easy online correction, standard menus and predefined charts and graphics. All these features facilitate the reporting process, provide useful and rapid feedback to the data providers, and reduce burden.

 

The data for this monitoring effort are collected by 508-compliant Web-based data collection instruments. The instrument structure allows respondents to move between a menu screen and a screen addressing individual topics. The question format is primarily quick-response checkboxes, with text boxes provided for the addition of specific, outstanding examples. Respondents may enter and leave the instrument as often as they desire and continue to change their responses until they complete their authorized final submission. Respondents have access to online help material, including a glossary, a list of frequently asked questions, a list of STEM disciplines, and screen-specific help to assist them in their data entry. Additionally, since the collection is Web-based, minor changes in wording and displays can be easily made in response to user feedback. See appendices A1 and A2 for a copy of the data collection instruments.

A.4. Efforts To Identify Duplication

This system does not duplicate other NSF efforts. Comparable data are not currently being collected on an annual basis for the Noyce program. In addition, the collection is coordinated with the NSF FastLane Project Reports system (OMB 3145-0058) to ensure that the two collections do not collect similar data. Additionally, aggregate data are being shared with NSF-funded researchers as appropriate, thereby minimizing the possibility that other researchers will duplicate these efforts in their own future collections.

A.5. Small Business

No information is to be collected from small businesses.

A.6. Consequences of Not Collecting the Information

Without this information, NSF would be restricted in managing and reporting on the activities of awards in the Noyce program. Without this feedback, NSF would have no way of making systematic modifications to the program (e.g., adequacy of funding amount, duration of award, and institutional supports needed). These data will assist NSF in making informed decisions about future directions of the Noyce program. The information requested here is not available elsewhere.

Additionally, without this information NSF would find it difficult to meet GPRA and OMB reporting requirements and would be unable to comply fully with congressional and presidential mandates that the Foundation asses its STEM education programs, and with the specific statutory requirements of the Noyce program.

A.7. Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6

The data collection will comply with 5 CFR 1320.6.

A.8. Consultation Outside the Agency

The notice inviting comments on the EHR Generic Clearance (OMB 3145-0136) was published in the Federal Register November 9, 2010, Volume 75, Number 216, pages 68829-68830. No comments were received.

During the initial system development five PIs of Noyce awards reviewed the questions; their responses were taken into account in the development of the system. Additional changes in the system are informed by ongoing consultations with the respondents and ICF Macro (the contractor that designed the Web interface and database system), and user comments submitted during the collection period and expressed during the annual PI meetings are also taken into consideration.

A.9. Payments or Gifts to Respondents

No payments or gifts will be provided to respondents.

A.10. Assurance of Confidentiality

Respondents will be advised that any information on specific individuals will be maintained in accordance with the Privacy Act of 1974. Data collected are available to NSF officials and staff, evaluation contractors, and the contractors hired to manage the data and data collection software. Data are processed according to Federal and State privacy statues. Detailed procedures for making information available to various categories of users are specified in the Education and Training System of Records (63 Fed. Reg. 264, 272 January 5, 1998). That system limits access to personally identifiable information to authorized users. Data submitted will be used in accordance with criteria established by NSF for monitoring research and education grants and in response to Public Law 99-383 and 42 USC 1885c. The information requested may be disclosed to qualified researchers and contractors in order to coordinate programs and to a Federal agency, court or party in a court, or Federal administrative proceeding, if the government is a party.

 

The opening screen of the Noyce system states the following:

 

The Federal Government has a continuing commitment to monitor its awards to identify and address any inequities based on gender, race, ethnicity, or disability of the principal investigators/co-principal investigators, support recipients, or other participants.

 

Information from this data collection system will be retained by NSF, a Federal agency, and will be an integral part of its Privacy Act System of Records in accordance with the Privacy Act of 1974 and maintained in the Education and Training System of Records 63 Fed. Reg. 264, 272 (January 5, 1998). All individually identifiable information supplied by individuals or institutions to a Federal agency may be used only for the purposes outlined in the system of records notice and may not be disclosed or used in identifiable form for any other purpose, unless otherwise compelled by law. These are confidential files accessible only to appropriate NSF officials, their staffs, and their contractors responsible for monitoring, assessing, and evaluating NSF programs. Only data in highly aggregated form, or data explicitly requested as “for general use,” will be made available to anyone outside of the NSF for research purposes. Data submitted will be used in accordance with criteria established by NSF for monitoring research and education grants, and in response to Public Law 99-383 and 42 USC 1885c. Date of birth will be maintained in accordance with the requirements of the Privacy Act of 1974.

A.11. Questions of a Sensitive Nature

The Noyce monitoring system collects the name, telephone number, and e-mail address of each PI, and information on students including name and date of birth. These data are collected in order to monitor the award sites, to provide continuity, and to assess the success of the award programs. Respondents have the option of not providing information that they consider privileged, such as disability status, by marking the “not reported” option on the form. All information will be maintained in accordance with the requirements of the Privacy Act of 1974. Individualized data are provided only to Noyce program staff and to contractors authorized by NSF. Any public reporting of the data is in aggregate form.

A.12 Estimates of Response Burden

A.12.1. Number of Respondents, Frequency of Response, and Annual Hour Burden

The total average number of annual respondents is 274 and the average annual burden hour total is 3,836. The Web-based collection is an annual activity of the Noyce program. There are currently 234 Noyce awards and data is collected from each award site; the respondent will be the PI of the award. We anticipate that new awards will be added, but that older awards will also expire; thus, on average, the number of respondents will only increase by about 20 each year.

 

The burden estimate is outlined below:

 

Clearance Year	Type of Respondent	No. of Respondents	Burden Hours per Respondent	Total Annual Burden Hours
One	PI	254	14	3,556
Two	PI	274	14	3,836
Three	PI	294	14	4,116
Three-Year Total		822		11,508
Average Annual		274		3,836

A.12.2. Hour Burden Estimates by Each Form and Aggregate Hour Burdens

The Noyce system consists of one form for Track I awards and one form for Track II awards. The number of data elements for each is roughly equal and is expected to take the same amount of time to complete. Respondents will be project PIs. The estimated total annual response burden is 3,836 hours. The annual burden by form was calculated as follows:

 

Clearance Year	Form	No. of Respondents	Burden Hours per Respondent	Annual Burden Hours
One	Noyce annual data collection	254	14	3,556
Two	Noyce annual data collection	274	14	3,836
Three	Noyce annual data collection	294	14	4,116
Three-Year Total		822		11,508
Average Annual		274		3,836

A.12.3. Estimates of Annualized Cost to Respondents for the Hour Burdens

The overall annualized cost to the respondents for the Noyce data collection is estimated to be $153,440. The following table shows the annualized estimates of costs to PI/institution leader respondents, who are generally university professors. The estimated hourly rate is based on a report in the April 16, 2010, edition of The Chronicle of Higher Education (2010). (“What Professors Earn.” The Chronicle of Higher Education, 56(31), A10, Washington, D.C.: The Chronicle of Higher Education, Inc.) According to the report, the average salary of an associate professor across all types of doctoral-granting institutions (public, private, church-related) was $83,511. When divided by the number of standard annual work hours (2,080), this calculates to approximately $40 per hour.

 

Clearance Year	Type of Respondent	No. of Respondents	Burden Hours per Respondent	Average Hourly Rate	Estimated Annualized Cost
One	PI	254	14 hrs.	$40/hr.	$142,240
Two	PI	274	14 hrs.	$40/hr.	$153,440
Three	PI	294	14 hrs.	$40/hr.	$164,640
Three-Year Total		822			$460,320
Average Annual		274			$153,440

A.13. Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs to Respondents or Record Keepers

There is no overall annual cost burden to respondents or record-keepers that results from the distance monitoring of the Noyce program other than the time spent responding to the data collection instrument.

 

It is usual and customary for individuals involved in education and training activities in the United States to keep descriptive records. The information being requested is from records that are maintained as part of normal educational or training practice. Furthermore, the majority of PIs are active or former grantees or participants in programs or projects once funded by NSF. In order to be funded by NSF, institutions must follow the instructions in the NSF Grant Proposal Guide (GPG) that is cleared under OMB 3145-0058. The GPG requires that all applicants submit requests for NSF funding and that all active NSF awardees do administrative reporting via FastLane, an Internet-based forms system. Thus, PIs who are the respondents to the Noyce data collection make use of standard office equipment (e.g., computers), Internet connectivity that is already required as a startup cost and maintenance costs under OMB 3145-0058, and free software (e.g., Microsoft Explorer or Mozilla Firefox) to respond. The information requested is typical of educational and research portfolios and would be maintained as part of normal practice. Thus, there are no capital and startup costs or operation and maintenance costs to respondents or record-keepers.

A.14. Estimates of Costs to the Federal Government

Computing the annualized cost to NSF for the Noyce data collection was done by taking the budgets for three years and calculating the costs for each of the following operational activities involved in producing, maintaining, and conducting the Noyce data collection:

 

 

The annualized cost was computed as one-third of the total three-year costs; thus, the annualized cost to NSF for the Noyce data collection is $252,583.

A.15. Changes in Burden

The previously reported total annual estimated response burden for this collection was 1,050 hours for 75 respondents; the current request for 3,836 hours for 274 respondents is an increase of 2,786 hours. This increase is due to the increase in the number of awards due to growth in the program.

 

Although a different version of the data collection instrument will be used for Track II awards, the type and amount of information is equivalent to the Track I data collection instrument, and the amount of time expected to complete data entry is expected to be roughly the same for Track II awards as for Track I awards.

A.16. Plans for Publication, Analysis, and Schedule

This is a recurring study. Data collection is conducted annually, beginning in October and ending in December. Extensions are granted by NSF program officers as necessary. Agency staff can access the data through the online system as needed.

 

Like many agencies, NSF is reducing its reliance on formal (i.e., traditional) publication methods and publication formats. ICF Macro, the contractor that conducts this data collection on behalf of NSF, is forbidden contractually from publishing results unless NSF has made a specific exception. In short, all products of the collections are the property of NSF and NSF is the exclusive publisher of the information being gathered. Often it is only after seeing the quality of the information collected that NSF decides the format (raw or analytical) and manner (in the NSF-numbered product Online Document System (ODS) or simply a page on the NSF Web site) in which to publish.

 

The data from this collection will be used for internal review purposes and to monitor the Noyce awards, as well as for reporting to Congress and OMB. Reports to NSF management, PIs, and Congress dealing with characteristics and performance of the Noyce program will include statistical tables and charts generated from the database. At this time NSF has no set timeline for publishing interim reports from this study.

A.17. Approval to Not Display Expiration Date

Not applicable

A.18 Exceptions to Item 19 of OMB Form 83-I

No exceptions apply.

Section B

Introduction

B.1. Respondent Universe and Sampling Methods

The sample size is the entire universe of respondents. Over the three-year clearance period, we estimate that there will be an average of 274 respondents per year.

 

 

B.2. Information Collection Procedures/Limitations of the Study

This data collection uses a Web-based instrument. Each Noyce award is required to provide project information each year during the duration of their NSF funding. The data for post-support information, which relates to participants whose scholarship or stipend support has ended but who continue to participate in the project (i.e., have not yet met their service teaching requirement), will be collected throughout the duration of the award and will continue to be requested following the award period. The solicitation states that awardees are responsible for tracking recipients to ensure that they complete their teaching requirement, so these data will be maintained by the respondents and are needed to determine whether the requirements of the program have been met. The latest program solicitation for the Noyce program can be found here.

NSF understands the limitations of this data collection, particularly in terms of using the data to determine program effectiveness. Data collected through the Noyce system are not used to determine the ultimate effectiveness of its STEM educational interventions, but are used in program planning and management, to report on agency activities and goals, and to lay the groundwork for future evaluations.

B.2.1. Statistical Methodology for Stratification and Sample Selection

This data collection is a census, so no sampling is required.

B.2.2. Estimation Procedure

Not applicable

B.2.3. Degree of Accuracy Needed for the Purpose Described in the Justification

Not applicable

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

Not applicable

B.2.5. Use of Periodic (Less Frequent Than Annual) Data Collection Cycles

Not applicable

B.3. Methods for Maximizing the Response Rate and Addressing Issues of Nonresponse

The collection is part of reporting required of awardees. Past collections have had 100 percent response rates, and NSF expects that the rate will remain the same. Considerable effort is made for follow up for non-response and incomplete responses. Biweekly during the collection, the contractor sends e-mail messages and calls institutions that have not logged into the system. The contractor also notifies all institutions that are still entering data when the collection reaches its midpoint and when the system closing date is one week away. Examples of e-mail messages sent to the Noyce respondents are included in appendix C.

B.4. Tests of Procedures or Methods

The questions in this collection were reviewed by Noyce PIs during the development of the system. In addition, many of the items and response categories follow formats that are already in use within other tasks, such as the IGERT monitoring system, also cleared as part of the EHR Generic Clearance (OMB 3145-0136), including demographic questions, items on educational history, and questions about level of scholarship, stipend, and fellowship support.

B.5. Names and Telephone Numbers of Individuals Consulted

Agency

 

Joan Prival, National Science Foundation, (703) 292-4635

Connie Kubo Della-Piana, National Science Foundation, (703) 292-5309

 

Contractors

ICF Macro of Bethesda, MD will be responsible for data collection and analysis under the direction of Lea Mesner, (301) 657-3070.