Supporting Statement (3145-0136)

Request For Revision of Clearance: National Science Foundation, Directorate of Education and Human Resources, Division of Graduate Education

Distance Monitoring of Integrative Graduate Education and Research Training (IGERT) Program

 

Section A

Introduction

This request for Office of Management and Budget (OMB) review asks for a revision of clearance for the distance monitoring data collection for the Integrative Graduate Education And Research Training Program (IGERT) that was most recently cleared in Janurary 2005 under the National Science Foundation's (NSF) Directorate for Education and Human Resources (EHR) Generic Clearance, OMB 3145-0136. The EHR Generic Clearance includes collections of information about NSF's education and training (E&T) activities. This task is being submitted for revision due to the addition of several data elements, described in Section A1, that will increase consistency in monitoring and provide NSF with the data needed to manage the program.

A.1. Circumstances Requiring the Collection of Data

 

New Data Elements

 

A revision is requested because several new data elements were added during preparation of the survey for the 2007 collection and some existing data elements were revised. Specifically, principal investigators (PIs) of IGERT awards are now asked to provide information on up to three trainee achievements, up to three research achievements, and up to three education achievements. Previously PIs were asked to provide a single highlight each in the research, education, and trainee achievement areas. PIs are now asked for up to three barriers to implementation of the IGERT program and the response made to each of these barriers. A single barrier and response were previously collected. A section on outreach activities has also been added. Finally, PIs are asked to provide Highlights (one or more) of the IGERT program. The data collected through these new questions will allow program staff to better manage the IGERT program and to meet NSF's reporting requirements.

 

The new items are indicated on pages 18, 26 and 27 of Appendix B, the crosswalk of data elements in the IGERT system, and the revised items are indicated on pages 24, 25, 26. The crosswalk also identifies several questions that are being removed from the IGERT system for the 2007 collection.

 

 

Program Information

 

The IGERT program was initiated in 1998. IGERT awards are made to academic institutions in the United States and its territories that grant the Ph.D. degree and have research and training programs in science and engineering. The awards to each institution may be up to $500,000 per year for 5 years, with an additional $200,000 available for appropriate instrumentation or research materials during the first year of the award. The majority of the funds must be used for doctoral student stipends, training and educational activities, and related expenditures, such as student travel, publication costs, and recruitment.

The IGERT program is specifically mentioned in the NSF FY 2004 Performance Plan as supporting the agency's Government Performance and Results Act (GPRA) goal of building "a diverse, internationally-competitive and globally engaged workforce of scientists, engineers, and well-prepared citizen" by promoting "the integration of research and education" (p.17) (http://www.nsf.gov/pubs/2004/nsf04202/fy2004finalplan.doc). The information collected by the distance monitoring survey informs how IGERT is contributing to the NSF plan; IGERT reporting is specifically cited in the Performance Plan as a measurement of NSF goals (p.11).

NSF established an OMB-approved distance monitoring system in the first year of the IGERT program. The IGERT monitoring collection is the sole source of data on the extent to which programmatic outcomes are being achieved. The distance monitoring system includes two Web-based survey forms: the principal investigator (PI) survey and the trainee survey.  The PI survey collects basic information on program participants and program implementation and management. The IGERT trainee survey collects information on trainee demographics, activities and accomplishments during the funding period, as well as information that will allow NSF to conduct future follow-ups with trainees. In order to encourage frank responses to questions, trainees have sole access to their own surveys, although PIs can monitor trainees' completion of each section of the survey via a management screen. Together these surveys allow for a comparison of the project as designed and implemented by the PI and faculty, and the project as it is experienced by the participant trainees.

Data collection instruments are included in appendix A.

A.2. Purposes and Uses of the Data

The primary purpose of this collection is program management, also known as program monitoring.  This monitoring information will be used for analytical and policy support of the EHR directorate.

All information collected will be used by the EHR Division of Graduate Education for program monitoring and by the EHR Division of Research, Evaluation and Communication (REC) to support their program impact assessment and evaluation needs and to provide analytical and policy support to EHR. REC expects to make these data available only to REC staff, REC contractors with responsibility for database management or program impact assessment and evaluation, and Graduate Education program managers and their staff.

The data collection serves multiple purposes.  The data allows NSF to monitor specific currently funded projects in relation to NSF's program objectives. Serious lapses in adherence to program guidelines or administrative problems are flagged and can be addressed immediately (e.g., failure to recruit students or recruitment of ineligible students (support is restricted to U.S. citizens and permanent residents)). Additionally, the data are used to identify features of the program that promote or inhibit successful outcomes. The data are used to ensure that barriers can be eliminated and supports strengthened, and that the policies and structure of the IGERT program can be modified as needed.  Lastly, the data is used as a baseline for the program evaluations conducted by Abt Associates (OMB# 3145-0812).  The Web-based monitoring system was designed to collect information for these purposes. 

Under the IGERT monitoring system, each IGERT PI and IGERT trainee is required to provide annual data using the Web-based data collection system (see appendix A). The following is an overview of the types of information collected: 

• Project Characteristics: The PI provides basic information about the program: what NSF disciplines are supported in the project, what institution faculty are advising students, if any consortial arrangements have been established and what additional funding has been received.
• PI Assessment of Outcomes: The PI assesses the level of success in meeting the goals for the trainees as a group and provides some thoughts on why some goals have been implemented more successfully than others.   Additionally, the PI assesses what impacts the program has had on the primary institutions (e.g., institutionalized programs, faculty development). 
• Project Features: The PIs provides detailed information about activities developed and used by the program (e.g., recruitment strategies, research training, training for future faculty). They also comment on the effectiveness of all planned activities.
• Trainee Data:  The PI provides a list of trainees and some basic information about their progress towards a Ph.D and employment after leaving the program.  The trainees themselves provide demographic data, information on educational background, achievements during the award period, and detailed information about activities related to training goals. 
• Trainee Commentary: Trainees are given the opportunity to comment of the value of specific training activities and the IGERT trainee program as a whole.  PIs cannot access the trainee surveys and cannot read trainee comments.

See Appendix B for detailed list of data elements.

A.3. Use of Information Technology To Reduce Burden

EHR tends to favor Web-based systems because they can facilitate respondents' data entry across computer platforms.  One innovative feature of many of the individual Web systems is the thorough editing of all submitted data for completeness, validity and consistency.  Editing is performed as data are entered.  Most invalid data cannot enter the system, and questionable or incomplete entries are called to respondents' attention before they are submitted to NSF.  Web based surveys employ user-friendly features such as automated tabulation, data entry with custom controls such as checkboxes, data verification with error messages for easy online correction, standard menus and predefined charts and graphics.  All these features facilitate the reporting process, provide useful and rapid feedback to the data providers and reduce burden. 

The data for this monitoring effort are collected by 508-compliant Web-based surveys. The survey structure allows respondents to move between a menu screen and a screen addressing individual topics. The question format is primarily quick-response checkboxes, with text boxes provided for the addition of specific, outstanding examples. Respondents may enter and leave their surveys as often as they desire and continue to change their responses until they submit their surveys. Respondents have access to an online glossary to assist them in understanding the specific meaning of terms in the context of these surveys. Additionally, since the collection is Web-based, minor changes in wording and displays have been easily made in response to user feedback.

A.4. Efforts To Identify Duplication

This system does not duplicate other NSF efforts. Comparable data are not currently being collected on an annual basis for the IGERT program.  In addition, the collection is coordinated with the NSF FastLane Project Reports system (OMB 3145-0058) to ensure that the two collections do not collect similar data. As much as possible, data from other NSF monitoring collections are used to pre-fill IGERT items, further minimizing overall response burden. Additionally, aggregate data are being shared with NSF-funded researchers as appropriate, thereby minimizing the possibility that other researchers will duplicate these efforts in their own future collections.

A.5. Small Business

No information is to be collected from small businesses.

A.6. Consequences of Not Collecting the Information

Without this information NSF will be unable to document the effectiveness, impact or outcomes of the IGERT program.  The Foundation will be unable to disseminate information to other projects and institutions about successful approaches to the integration of research and education and graduate student training.  Additionally, without this feedback NSF would have no way of making systematic modifications to the IGERT program (e.g., adequacy of funding amount, duration of award, institutional supports needed).  Moreover, NSF will be unable to comply fully with the congressional mandate that the Foundation evaluate its science, technology, engineering, and mathematics (STEM) education programs.

A.7. Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6

The data collection will comply with 5 CFR 1320.6.

A.8. Consultation Outside the Agency

The notice inviting comments on the EHR Generic Clearance (OMB 3145-0136) was published in the Federal Register June 7, 2004, Volume 69, Number 109, page 31846. No comments were received.

During the initial system development principal investigators (PIs) from IGERT awards reviewed the system; their responses to the PI survey and their assessments of the trainee survey were taken into account in the development of the system.  Changes in the system since initial development are informed by ongoing consultations with the respondents, Macro International Inc. (the contractor that designed the Web interface and database system) and Abt Associates, Inc. (the contractor that performs program evaluations).  ORC Macro currently maintains the surveys and survey databases and provides technical support to respondents as needed.

A.9. Payments or Gifts to Respondents

No payments or gifts will be provided to respondents.

A.10. Assurance of Confidentiality

Data collected under this task are only available to the respondents, NSF, and the firms hired to manage the data and data collection software. Data are processed according to Federal and State privacy statutes. To protect privacy, only composite data or graphical representations will be released to the public.

For the collection covered by this clearance request, when respondents are presented with the first screen of the survey, they are additionally instructed as follows: "Information from this data collection system will be retained by the National Science Foundation, a federal agency, and will be an integral part of its Privacy Act System of Records in accordance with the Privacy Act of 1974 and maintained in the Education and Training System of Records 63 Fed. Reg. 264, 272 (January 5, 1998). These are confidential files accessible only to appropriate National Science Foundation (NSF) officials, their staffs, and their contractors responsible for monitoring, assessing, and evaluating NSF programs. Only data in highly aggregated form, or data explicitly requested as "for general use," will be made available to anyone outside of the National Science Foundation for research purposes. Data submitted will be used in accordance with criteria established by NSF for monitoring research and education grants, and in response to Public Law 99-383 and 42 USC 1885c. The Social Security number (SSN) and date of birth will be maintained in accordance with the requirements of the Privacy Act of 1974. Submission of the SSN is voluntary. It is used for survey quality control, program evaluation, and for matching with other data sets maintained in the Education and Training System of Records 63 Fed. Reg. 264, 272 (January 5, 1998)."

A.11. Questions of a Sensitive Nature

In some cases, instruments request information from respondents including name, address, Social Security number (SSN), date of birth, and grade point average.  These data are collected in order to monitor the award sites and assess the success of the award programs.  Information of this nature is also used to track recipients of funding and training. The IGERT survey requests that each trainee provide his or her name, SSN, phone number, a personal e-mail address (e.g. aol.com or hotmail.com), and contact information for a person likely to know how to reach him or her in three years. These data are collected in order to ensure consistent monitoring and to permit follow-up studies that examine the long-term effect of the IGERT program on individuals' success. The IGERT survey also collects the name, telephone number, e-mail address, fax number, disability status, and citizenship of each PI, co-PI or advisor. Additionally, trainee Graduate Record Exam (GRE) scores are collected. Respondents have the option of not providing information that they consider privileged, such as disability status, by marking the "not reported" option on the form or by leaving their SSN. Because the program requires that all IGERT trainees be U.S. citizens or permanent residents, the question of IGERT trainee citizenship is directly addressed both in the PI survey (by asking the PI to verify the IGERT-required citizenship/residency requirement for each trainee) and in the trainee survey (by not providing a "not reported" option for citizenship). All information will be maintained in accordance with the requirements of the Privacy Act of 1974. Individualized data are provided only to IGERT program staff and to contractors from Abt Associates, Inc. conducting site evaluations authorized by NSF. Any public reporting of the data is in aggregate form.

A.12 Estimates of Response Burden

A.12.1. Number of Respondents, Frequency of Response, and Annual Hour Burden

The total number of annual respondents is 1,700 (100 project PIs and 1,600 IGERT trainees and associates) and the total annual person-hours is 7,200. The Web-based collection is an annual activity of the IGERT program. There are currently 100 IGERT awards and data is collected from each award site. PIs complete the PI survey; all IGERT trainees and associates are required to complete the IGERT trainee survey. We anticipate that new awards may be added, but at about the same rate that active awards expire; thus, on average, the number of respondents will remain constant over time.  The annualized burden was computed by taking the number of respondents from the current survey cycle and estimating their response burden, based on a question in the Web-based data collection asking how long it takes respondents to complete the survey.  The burden estimates for each type of respondent are outlined below:

A.12.2. Hour Burden Estimates by Each Form and Aggregate Hour Burdens

As mentioned above respondents will be project PIs and IGERT trainees.  The estimated total annual response burden is 7,200 person-hours. There is a different Web-based form for each respondent.  The annual burden by form was calculated as follows:

Form Type	Respondent Type	Number of Respondents	Burden Hours Per Respondent	Total Person- Hours
Principal Investigator survey	Project PIs	100	40 hours	4,000
Trainee Survey	Trainees	1,600	2 hours	3,200
Total		1,700		7,200

A.12.3. Estimates of Annualized Cost to Respondents for the Hour Burdens

The overall cost to the respondents is estimated to be $164,000. The following table shows the annualized estimates of costs to respondents.  These estimated hourly rates are based on a report in the April 23, 2004 edition of The Chronicle of Higher Education.  According to the report, the average salary of an associate professor across all types of doctoral-granting institutions (public, private, church-related) was $68, 640.  When dived by the number of standard work hours (2,080), the average hourly wage is $33.00 per hour.

Respondents	Number of Respondents	Hours per Respondent	Average Hourly Rate	Total Annual Costs
Project PIs	100	40 hours	$33	$132,000
IGERT trainees and associates	1,600	2 hours	$10	$32,000
Total estimated costs				$164,000

^[1]2004. "What Professors Earn." The Chronicle of Higher Education, 50(33), Washington, D.C.: The Chronicle of Higher Education, Inc.

A.13. Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs to Respondents or Record Keepers

There is no overall annual cost burden to respondents or record-keepers that results from the distance monitoring of the IGERT program other than the time spent responding to the data collection instrument.

It is usual and customary for individuals involved in education and training activities in the United States to keep descriptive records. The information being requested is from records that are maintained as part of normal educational or training practice. Furthermore, the majority of PIs are active or former grantees or participants in programs or projects once funded by NSF. In order to be funded by NSF, institutions must follow the instructions in the NSF Grant Proposal Guide (GPG) that is cleared under OMB 3145-0058. The GPG requires that all applicants submit requests for NSF funding and that all active NSF awardees do administrative reporting via FastLane, an Internet-based forms system. Thus, PIs who are the primary respondents to the IGERT data collection make use of standard office equipment (e.g., computers), Internet connectivity that are already required as a startup cost and maintenance cost under the NSF GPG.  The information requested of trainees is typical of doctoral student educational and research portfolios and would be maintained as par of normal practice.

A.14. Estimates of Costs to the Federal Government

Computing the annualized cost to NSF for the IGERT data collection was done by taking the budgets for 3 years and calculating the costs for each of the following operational activities involved in producing, maintaining, and conducting the IGERT data collection:

Operational Activities	Cost Over 3 Years
System development (includes initial development of the database and Web-based application and later changes requested by the program, e.g., increased reporting tools, additional validations)	$315,000
System maintenance, updates, and technical support (system requires updates each year before opening the collection; maintenance is required to keep the system current with technology, e.g., database servers, operating systems)	$165,00
Data collection opening and support (e.g., online and telephone support to respondents and contacting respondents to encourage completion of the questions), reporting (as defined by the Division of Graduate Education), and followup activities (e.g., providing data to other consultants)	$160,000
3-Year Total for All Operational Activities	$640,000

 

 

 

The cost for the remaining one year of the clearance was computed as one-three of the total 3-year costs; therefore, the annualized cost to NSF for the IGERT data collection is $213,333.

 

A.15. Changes in Burden

While the total number of respondents in this collection remains the same, a total of 1,700 PIs (sometimes called evaluators) and trainees annually, the hourly burden is expected to rise from 6,600 annual person hours (and overall cost burden of $144,200) to 7,200 annual person hours (or an overall cost burden of $164,000). This is due to an increase in the time the PIs spend completing the survey; their burden is expected to rise from 34 hours to 40 hours. This change reflects the addition of the new data elements, including the request for PIs to provide an NSF highlight. No change is expected in the burden on the IGERT trainees, which remains at 2 hours.

A.16. Plans for Publication, Analysis, and Schedule

Data collection is scheduled to begin in late April, and award sites will have 90 days to enter data; extensions are granted by NSF program officers as necessary. Once the data collection has been completed, agency staff can access the data through the on-line system as needed.

Like many agencies, NSF is reducing its reliance on formal (i.e., traditional) publication methods and publication formats.  ORC Macro, the contractor that manages the data collection Web site and database, is forbidden contractually from publishing results unless NSF has made a specific exception.  In short, all products of the collections are the property of NSF and NSF is the exclusive publisher of the information being gathered.  Often it is only after seeing the quality of the information collected that NSF decides the format (raw or analytical) and manner (in the NSF-numbered product Online Document System (ODS) or simply a page on the NSF Web site) in which to publish.   

The data from this collection will be used for internal review purposes and to monitor the IGERT projects, as well as for reporting to Congress (e.g., the GPRA Annual Performance Plan). Reports to NSF management, PIs, and Congress dealing with characteristics and performance of the IGERT program will include statistical tables and charts generated from the database. At this time, NSF has not set timeline for publishing interim reports from this study. 

A.17. Approval to Not Display Expiration Date

Not Applicable

A.18 Exceptions to Item 19 of OMB Form 83-I

No exceptions apply.