Request for Clearance of Data Collection for the Evaluation of Historically Black Colleges and Universities – Undergraduate Program
Current OMB Control No: 3145-0204
Funded by the National Science Foundation
4201
Wilson Boulevard
Arlington, VA 22230
Program name: Historically Black Colleges and Universities – Undergraduate Program (HBCU –UP)
This request for Office of Management and Budget (OMB) review asks for a revision in the clearance of data collection for the Evaluation of the Historically Black Colleges and Universities – Undergraduate Program (HBCU – UP) which is funded by the Directorate for Education and Human Services (EHR) at the National Science Foundation (NSF).
The NSF funds research and education in science and engineering. It does this through grants, contracts, and cooperative agreements to more than 2,000 colleges, universities, and other research and/or education institutions in all parts of the United States. The Foundation accounts for about 20 percent of Federal support to academic institutions for basic research. EHR is the directorate within NSF that is responsible for health and continued vitality of the Nation’s science, technology, engineering, and mathematics (STEM) education and for providing leadership in the effort to improve education in these areas.
This package was originally granted clearance in October 2006 (Control No: 3145-0204) based on one data collection—a survey of recipients of STEM undergraduate degrees from institutions awarded an HBCU-UP grant from NSF. Due to programmatic changes, the Urban Institute (UI) will now also collect data from STEM faculty at those institutions. The following package reflects the entire data collection.
Overview of Program: Historically Black Colleges and Universities – Undergraduate Program (HBCU – UP)
The Historically Black Colleges and Universities – Undergraduate Program focuses on strengthening STEM education and research programs at HBCUs at the undergraduate level. HBCU – UP is one of the many programs at the NSF contributing to broadening participation in the nation’s STEM workforce. The program funds efforts “to build the science, technology, engineering, and mathematics (STEM) education and research capacity at Historically Black Colleges and Universities (HBCUs) as a means to broaden participation in the Nation's STEM workforce.”
Support is available for Implementation Projects, Planning Grants, Education Research Projects, and Targeted Infusion Projects. Implementation projects include comprehensive institutional approaches to strengthen STEM education and research programs. The activities of individual HBCU – UP projects are governed by differences in the characteristics of specific localities such as institutional resources and capabilities, long-term goals, and organizational mission. Individual projects may encompass a variety of activities including curriculum and course enhancement, faculty research and professional development, undergraduate research, academic enrichment, use of technology to enhance STEM instruction, and collaborations with other institutions and industry. Research projects funded are those with potential to strengthen the STEM education and research programs at HBCUs. Targeted Infusion Projects provide support to implement activities that will result in the achievement of a well defined goal within one STEM department over one to two years. Planning grants support institutional STEM self-analysis activities, which should inform the preparation of a proposal for an implementation project.
HBCU- UP was first implemented in Fiscal Year 1998 in response to a Congressional mandate within the VA HUD and Independent Agencies Appropriations Bill. Since 1998, funding has increased from $6 million to $25 million in FY 2005. Since 1998, sixty-three different HBCUs have received HBCU – UP funding. A total of sixty-seven implementation projects and eighteen planning grants have been awarded to HBCUs.
Overview of the Evaluation's Research Questions, Conceptual Framework, and Study Design
NSF has contracted The Urban Institute to conduct a three-year evaluation of the HBCU – UP program. In response to the NSF's Request for Quotation, UI proposed to conduct an evaluation using a mixed-methods approach to answer the main evaluation questions (listed below).
Main Research Questions
What has been the overall impact of HBCU – UP?
How has the program evolved since it’s inception?
In what ways have the set of projects contributed to the scholarly body of work about effective practices and strategies to address diversity in the STEM workforce?
Conceptual Framework
The conceptual framework that guides the evaluation is depicted in Figure 1. The conceptual framework outlines how we will carry out both a process and a summative evaluation and how each of these evaluation components responds to each of the evaluation questions. As the figure shows, process and summative evaluations will be conducted for each model identified.[1]
Study Design
This study has two components—a process and a summative (outcomes) evaluation (see Figure 1).
Process Evaluation. The goal of the process evaluation is to describe the characteristics of recognizable models among the HBCU — UP projects; point out which strategies accelerated or inhibited attainment of program goals; focus on the mix of strategies that promote linkages among projects and resources; and ascertain which additional areas the program has influenced. As part of the process evaluation, we will document how the national HBCU — UP program has evolved since its inception. We will also review the relevant literature on effective strategies for strengthening S&E programs at HBCUs and conduct a survey of faculty at participating institutions. The process evaluation, by documenting how the different models within the broader HBCU — UP program are being implemented, will help the evaluators to link strategies to outcomes, identify crucial components of a specific model, and contribute to the construction of general theories to guide future initiatives to increase the diversity of the STEM workforce.
Summative Evaluation. The summative evaluation will focus on the extent to which the HBCU — UP program has produced outcomes that meet its stated goals. Central to this component is a study of student outcomes. Data to measure student outcomes will be collected through a survey of graduates of HBCU-UP programs. In addition, outcomes data collected for HBCU-UP participants in each of the models identified will be compared across models to gauge the success of each model in producing these student outcomes. These data will also be aggregated for all the models in order to determine overall Program success.
[1] Please see Figure 1 in the Appendix.
The HBCU – UP program was initiated in 1998. This program has not been evaluated previously by any agency or individual, so data on the extent to which programmatic outcomes are being achieved are not available except from the current and proposed data collection activities.
Moreover, NSF is committed to ensuring the excellence and effectiveness of the programs it supports. NSF’s Division of Research on Learning in Formal and Informal Settings (DRL), (formerly called the Division of Research, Evaluation, and Communication (REC) is responsible for the evaluation and assessment of the impact of education and training programs throughout the NSF. In addition, NSF is required by the Government Performance and Results Act (GPRA) to report annually on the impact of the projects funded. The "HBCU – UP" is one of the programs featured in the NSF 2001-2006 Strategic Plan as helping implement the agency's GPRA goal of "a diverse, internationally competitive and globally engaged workforce of scientists, engineers, and well-prepared citizens” (see http://www.nsf.gov/pubs/2001/nsf0104/app7.htm).
The HBCU-UP Program also contributes to NSF's strategic plan, National Science Foundation - Investing in America's Future: Strategic Plan FY 2006-2011 (NSF 06-48). In the report, please see Appendix A: Expert Evaluations and Assessments, particularly section D. For a copy of the plan visit the NSF website: http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf0648.
The primary purpose of the current collection and this additional data collection is program evaluation. The data will be used to assess achievement of program goals.
In compliance with OMB directives, whenever possible, electronic paperless data collection will be used to administer surveys. More specifically, the faculty survey will be web-based; respondents will be sent an email with a URL embedded to complete the survey online. After an email follow-up, non-respondents will be sent a letter in the mail with instructions to respond to the survey on the Internet, by telephone (through a toll free telephone number where interviewers will administer the questionnaire), or by mail (a copy of the survey will be included). Non-respondents will be called via the telephone for a Computer Assisted Telephone Interview (CATI). Hence, respondents will have the option to respond by Internet, telephone, or mail. A combination of response methods will also be used for the graduate survey and the course revision form. The latter will be sent electronically to all institutions, and collected during the grantees’meeting to be held at NSF. The major advantages of multiple response methods are to increase convenience for respondents and to reduce data collection time. Electronic submissions will be emphasized and encouraged, however, to provide higher levels of standardization in the collected data.
It is important to add that for respondents to the faculty and graduate surveys who choose the telephone option (whether because they call in or project staff call them to administer the survey), responses will be entered directly into the project database. A major advantage of this option and of electronic submission is that data are checked automatically for completeness, validity, and consistency. Most invalid data cannot enter the system, and questionable or incomplete entries are called to attention before they are submitted. The different survey (and data entry) modes should facilitate the reporting process and reduce burden.
The Evaluation of the HBCU-UP does not duplicate other NSF efforts. The data being collected for this evaluation have not and are not being collected either by NSF or other institutions.
No information is to be collected from small businesses.
If the information is not collected, NSF will be unable to document the effectiveness and outcomes of the HBCU-UP program. It also will not be able to meet its accountability requirements, as it will be unable to assess the degree to which the program is meeting its goals. This lack of information may hamper program management. Moreover, NSF will be unable to comply fully with the Congressional mandate that the Foundation evaluate its science, technology, engineering, and mathematics (STEM) education programs.
The data collection will comply with 5 CFR 1320.6.
Two notices have been published soliciting comments from the public. The first notice was published in the Federal Register on March 3, 2006 (Volume 71, Number 42), when the first clearance package was submitted for the graduate survey (later granted clearance, OMB Control No. 3145-0204). Since new components have been added to this study that require new data collections, a second notice was published in the Federal Register on March 14, 2007 (Volume 72, Number 49). This package follows this second notice. A copy of the text of both notices is attached in the Appendix. No public comments were received in response to either notice.
The evaluation design was developed in consultation with NSF staff from the Division of Human Resource Development (HRD), which is the division within the Directorate for Education and Human Resources (EHR) that funds the HBCU-UP program.
A team of experts made up of experts in program evaluation, STEM education, and higher education will provide advice and counsel to the evaluators throughout the study. Expert team meetings will be held annually to review evaluation activities and provide advice on plans for activities during the following year. The experts will also offer feedback on the progress of the study. Robert Santos, who was originally from NuStats but has joined the Urban Institute, also provided expertise in the design of the study and instrument development.
No payment or gifts will be provided to respondents.
Respondents will be advised that any information on specific individuals will be maintained in accordance with the Privacy Act of 1974. Data collected are available to NSF officials and staff, evaluation contractors, and the contractors hired to manage the data. Data are processed according to Federal and State privacy statutes. Detailed procedures for making information available to various categories of users are specified in the Education and Training System of Records (63 Fed. Reg. 264, 272 January 5, 1998). That system limits access to personally identifiable information to authorized users. Data submitted will be used in accordance with criteria established by NSF for monitoring research and education grants and in response to Public Law 99-383 and 42 USC 1885c. The information requested may be disclosed to qualified researchers and contractors in order to coordinate programs and to a Federal agency, court or party in a court, or Federal administrative proceeding, if the government is a party.
Participants in this evaluation will be assured that the information they provide will not be released in any form that identifies them as individuals. Evaluation findings on the projects will be reported in the aggregate in both the annual reports and the final report. The contractor, The Urban Institute, and the subcontractor, NuStats, have extensive experience collecting information and maintaining the confidentiality, security, and integrity of survey data.
In accordance with The Urban Institute’s Institutional Review Board (IRB) requirements, the following confidentiality and data protection procedures will also be in place:
Evaluation team members (including subcontractors) will sign a staff confidentiality pledge and will be informed of the sensitive nature of materials and data.
Each survey respondent will be assigned an identification number so that respondents’ names will not appear in the database. Personal information (name, addresses and phone numbers) is being collected in order to identify survey recipients and contact them. That information will be kept separate from the data collected through the surveys, and will be used to assign a unique ID number to each respondent.
The file containing personal information and ID numbers will be stored in a CD Rom kept in a locked cabinet.
The database itself will contain no identifiers (no names, addresses or phone numbers) as these are only needed to survey respondents, not for analyses.
Access to the database will be limited to Beatriz Chu Clewell (PI), Clemencia Cosentino (Co-PI), and only those researchers who are granted access from the PI or Co-PI. No others will be authorized such access. Hard copies of the surveys will be kept in a locked file in a locked office. Access to the hard copies will again be restricted to authorized staff members only.
The Urban Institute’s Office of Information Technology will be informed of which computers have confidential data to ensure that those hard drives are not backed up.
All listings, forms, and completed surveys containing identifiable data will be shredded as soon as the need for the hard copies no longer exists.
All basic computer files will be duplicated on backup disks to allow for file restoration in the event of unrecoverable loss of the original data. These backup files will be stored under secure conditions and access will be restricted to authorized project personnel.
The faculty and the graduate (student) survey instruments request demographic, employment, and professional information from respondents—including name, age, gender, ethnicity, occupational status, and family information. These data are collected in order to conduct analysis by these categories. Reporting of this information is on a voluntary basis. Respondents may choose not to provide information that they feel is privileged. Any individualized data that are collected are provided only to the evaluation team and consultants conducting studies using the data as authorized by NSF. Neither sensitive nor personal data are collected in the course revisions form. Lastly, any public reporting of data will be in aggregate form.
This study relies on a survey of graduates and faculty members from STEM undergraduate programs in HBCUs. The survey instruments used in data collection for this evaluation appear in the Appendix.
This study conducts three one-time surveys—a survey of recipients of STEM degrees, a survey of STEM faculty members at HBCU-UP grantee institutions, and a survey of course revisions at participating institutions. These surveys will only be administered once. The total number of respondents is estimated to be 4155, with a total burden, across all three instruments and respondents, estimated at 1074 person hours.
Instrument Type |
Respondent Type |
Number of Respondents |
Burden Hours Per Respondent |
Total Person Hours |
Surveys |
Recipients of STEM undergraduate degrees and STEM faculty / staff from institutions awarded an HBCU-UP grant from NSF. |
4155 |
15 - 20 minutes |
107 |
The above section, A.12.1, provided the total estimated burden across all data collection instruments. This section provides burden estimates and related information for each individual data collection form.
Graduate Survey
Respondents to this survey are recipients of undergraduate STEM degrees at HBCUs awarded an HBCU-UP grant from NSF. There are 20 programs with an estimated total of 5,000 graduates that meet selection criteria for inclusion in this study (more details are provided in Section B). The expected response rate of 75% will yield no more than 3750 completed surveys. The estimated total burden is 938 hours (15 minutes per survey for the expected 3750 respondents). Respondents provide answers to the instrument just once over the three-year period.
Faculty Survey
The faculty survey population is about 1000 faculty members at the 20 institutions awarded HBCU-UP grants in cohorts 2 and 3. The sample will consist of a stratified sample of 450 faculty members (more details regarding sampling provided below in Section B). This is expected to produce 350 completed surveys of faculty members (a 78% response rate). The estimated total burden is 117 hours (20 minutes per respondent). Respondents participate once in this survey over the three-year study; there are no additional follow-up surveys under our proposed design.
Course Revisions Form
Created in response to a request from the American Competitiveness Council (ACC) conveyed to the evaluation by NSF, this form will be completed once by each of the institutions who have received an HBCU-UP grant. The number of respondents, therefore, will be 55. On average, it should take about 20 minutes to complete this form, resulting in a total burden of 19 hours.
The combined burden for all surveys was calculated as follows:
Instrument Type |
Respondent Type |
Number of Respondents |
Burden Hours Per Respondent |
Total Person Hours |
Survey Questionnaire |
Recipients of STEM undergraduate degrees from HBCUs awarded an HBCU-UP grant from NSF |
3750 |
15 minutes |
938 |
Survey Questionnaire |
STEM faculty at the participating HBCU-UP institutions |
350 |
20 minutes |
117 |
Course Revision Form |
STEM faculty at the participating HBCU-UP institutions |
55 |
20 minutes |
19 |
Total |
All respondents |
4155 |
15 - 20 minutes |
107 |
The overall cost to the respondents is estimated to be $22,931.
For the graduate survey, this figure is derived by multiplying the time burden (15 minutes per respondent) by the average hourly wage of a college graduate in STEM ($20). The estimated hourly wage for respondents is based on information from the National Science Foundation (http://www.nsf.gov/statistics/infbrief/nsf06303/nsf06303.pdf). For the faculty survey and course revisions form, this figure is derived using the same method—multiplying the time burden (20 minutes per respondent) by the average hourly wage of a STEM faculty professor ($31.84/hr). The estimated hourly wage for the faculty survey and course revision form respondents is based on information from the DAS Data System, National Center for Educational Statistics (http://nces.ed.gov/surveys/nsopf/).
All survey respondents will only be asked to complete the survey only once. The table below shows the estimated total cost to respondents.
Respondent Type |
Hourly Salary Estimate |
Burden Hours per Respondent |
Total Number of Respondents Across Sites |
Total Burden Hours Across Sites |
Estimated Annualized Costs |
|
|||
Survey Questionnaire |
|
|
|||||||
Recipients of undergraduate degrees in STEM from HBCUs awarded an HBCU-UP grant from NSF |
$20/hr |
15 min. |
3750 |
938 |
$18,760.00 |
|
|||
Survey Questionnaire |
|
|
|||||||
STEM faculty at the participating HBCU-UP institutions |
$31.84/hr |
20 min. |
350 |
117 |
$3,725.00 |
|
|||
Survey Questionnaire |
|
|
|||||||
STEM faculty at the participating HBCU-UP institutions |
$31.84/hr |
20 min. |
55 |
19 |
$446.00 |
|
|
||
Total |
$20 - $31.84/hr |
15 - 20 min. |
4155 |
1074 |
$22,931.00 |
|
|
||
The only burden to respondents is the time spent responding to the survey instruments. There is no additional burden to respondents or record keepers.
The estimated cost to the government of all data collection, analysis, and reporting activities associated with the survey used in this study is $672,986.00. The data are collected only once; hence the cost is incurred only once.
Estimated Cost to the Federal Government
Personnel |
|
Staff Cost (including Fringe Benefits) |
$228,505 |
Other Direct Costs |
|
Staff Travel and Per Diem |
$1,562 |
Advisory Panel Fees |
$5,868 |
Subcontracts |
$329,304 |
Subcontract Fee |
$14,939 |
Communication and Supplies |
$10,604 |
Indirect Costs |
|
G & A and Fixed Fee |
$82,204 |
Total Costs |
$672,986 |
This package contains a reduction in burden from 1250 hours to 1074 hours. Although the Urban Institute is adding two new data collections, the burden estimate decreases. This is because the original burden calculation for the graduate survey was extremely conservative and it did not take actual response rate into account. The new calculation is based on actual expected responses for all three surveys.
Timeline for data collection and analysis. The Evaluation of the HBCU-UP is a three-year evaluation that covers October 1, 2005 – September 30, 2008. Collection of information for this evaluation begins with a thorough literature review, review of HBCU- UP project documents, and identification of models. It is followed by the survey of graduates, a survey of faculty, and a short survey of course revisions. Data collection will last for six to eight months. Data entry, validation checks, and subsequent analyses will take place in 2007. A progress report will be submitted to NSF in October 2007. The final report will be submitted in October 2008.
Background
Review of Literature and Project Documents - 2005-2006
Data Collection and Analysis
Graduate Survey – October 2006 (immediately after clearance was granted)
Faculty Survey – about two to six months after receipt of clearance
Course Revision Form – immediately upon receiving clearance
Data Entry, Validation Checks, and Analysis – October 2006 – 2008
Reports
Interim Report – October 2007
Final Report – October 2008
Publications. Like many agencies, NSF is reducing its reliance on formal (i.e., traditional) publication methods and publication formats. The Urban Institute, which is conducting this third-party study of HBCU-UP on behalf of NSF, is forbidden contractually from publishing results unless NSF has made a specific exception. In short, all products of the collections are the property of NSF. After the products are delivered, NSF determines whether the quality of the products deserves publication verbatim by NSF, i.e., NSF is the exclusive publisher of the information being gathered. Often it is only after seeing the quality of the information delivered by the study that NSF decides the format (raw or analytical) and manner (in the NSF-numbered product Online Document System (ODS) or simply a page on the NSF Web site) in which to publish.
NSF plans to publish at least one analytical report in the ODS for the HBCU-UP within two years of the study's conclusion, which is estimated to be 9/30/2008. This means that the report will be available on the NSF Web site within 2 years of the conclusion of all data collection. NSF classifies formal publications as reports, not statistical reports.
Not applicable.
No exceptions apply.
| File Type | application/msword |
| File Title | Supporting Statement |
| File Modified | 2007-10-11 |
| File Created | 2007-10-11 |