FINAL DPC_Support_StatementA_Reinstate_with_changes

This application request is to continue an evaluation of the NIH Common Fund’s “Enhancing the Diversity of the NIH-funded Workforce Program” in response to feedback from the OMB that this initiative is not exempt from the Paper Work Reduction act and should go through the Reinstatement with Change process. The initiative described in this application has two five-year funding periods. The initiative has completed Phase I and is currently in the final phase and is a national consortium, comprised of three integrated initiatives: the (1) Building Infrastructure Leading to Diversity (BUILD) Initiative, (2) National Research Mentoring Network (NRMN), and (3) Coordination and Evaluation Center (CEC).

The data collection efforts are needed in order to meet the requirement for formal evaluations of the BUILD Phase I and II and NRMN Phase I initiatives under the NIH-funded cooperative agreement awarded to CEC at the University of California, Los Angeles (UCLA).

The CEC conducted evaluations of BUILD and NRMN include assessment of agreed-upon consortium-wide Hallmarks of Success at the student, faculty, and institutional level. The Hallmarks of Success represent key factors at each stage of a biomedical research career that contribute to development of a successful career in the biomedical research workforce. The data collection strategy described below was designed to support the evaluation of the overall effectiveness of BUILD and NRMN efforts and lessons learned will be disseminated to the broader biomedical research community.

A.1 Circumstances Making the Collection of Information Necessary

The data to be collected are responsive to helping fulfill the requirements of several relevant authorizing mandates:

Executive Order 12862, “Setting Customer Service Standards,” which directs Agencies to continually reform their management practices and operations to provide service to the public that matches or exceeds the best service available in the private sector; and
The March 3, 1998 White House Memorandum, “Conducting Conversations with America to Further Improve Customer Service,” which directs Agencies to determine the kind and quality of service its customers want as well as their level of satisfaction with existing services.
Executive Order 13450 - Improving Government Program Performance, which calls for evaluation as outlined at: https://www.whitehouse.gov/sites/default/files/omb/assets/performance_pdfs/eo13450.pdf
42 U.S. Code § 284n, “Certain demonstration projects” section, which was enacted as part of the National Institutes of Health Reform Act of 2006 to allocate funds for NIH institutes and centers to make grants for the purpose of improving public health through demonstration projects for biomedical research at the interface between the biological, behavioral, and social sciences and the physical, chemical, mathematical, and computational sciences [specifically, authority for NIH Office of Director that make the collection of information necessary as part of this initiative is based on the following: U.S. Code Title 42, Chapter 6A, Subchapter III, Part B - § 284n; “certain demonstration projects” at (https://www.law.cornell.edu/uscode/text/42/284n) and (http://www.law.cornell.edu/uscode/html/uscode42/usc_sup_01_42_10_6A_20_III_30_B.html)].

The Enhancing the Diversity of the NIH-funded Workforce Program¹ (also called the Diversity Program Consortium) was established in FY 2014 in response to a set of recommendations provided by the NIH Advisory Committee to the Director (ACD) Working Group on Diversity in the Biomedical Research Workforce (see Attachment 21 for the original member list). The ACD Working Group reviewed a study conducted by Ginther et al., (2011), presenting data regarding significant racial disparities in funding success of NIH R01awards.² In response, the ACD working group provided a set of 13 recommendations in four broad areas: data collection/evaluation; mentoring/career preparation and retention; institutional support; and research and intervention testing. These recommendations were the genesis of the development of Funding Opportunity Announcements for the NIH Building Infrastructure Leading to Diversity (BUILD) Initiative (Attachments 1 and 2), National Research Mentoring Network (NRMN) (Attachment 3) and the Coordination and Evaluation Center for Enhancing the Diversity of the NIH-Funded Workforce Program (Attachments 4 and 5).

The Diversity Program Consortium is funded by the Office of Strategic Coordination (OSC), located in the Division of Program Coordination, Planning and Strategic Initiatives (DPCPSI) at the Office of the Director (OD), and managed by the National Institute of General Medical Sciences (NIGMS). This initiative is a critical investment in biomedical student, faculty and mentoring, and infrastructure development that is necessary to achieve the NIH and the nation’s goal of enhancing diversity in the biomedical and health professional workforce as a key strategy for improving the health of the nation.

The Diversity Program Consortium supports complementary efforts to achieve its overarching goal of enhancing the diversity of the NIH-funded biomedical workforce. The 10 BUILD awards were granted to a diverse set of institutions nationwide with high levels of low-income students and limited previous NIH funding. The BUILD initiative supports the design and implementation of innovative programs, strategies and approaches to transform undergraduate research training and mentorship. The BUILD program elements at funded institutions include interventions at the student, faculty, and institutional level (see Attachment 5). Institutions funded through the BUILD initiative were encouraged to partner with other academic institutions as well as industry to provide a wealth of diverse training opportunities for their students. The support given to BUILD institutions is expected to directly impact undergraduate students and faculty as well as strengthen institutional research infrastructure (e.g. student recruitment, research training opportunities, faculty tenure and promotion, curriculum development, etc.) in biomedical disciplines. The programs aim to produce knowledge to promote culture change in the fields of biomedical education and training.

Complementing the BUILD initiative, the NRMN Phase I initiative provided mentoring resources and training to a nationwide consortium of individuals from diverse backgrounds at all career and training levels in biomedical research fields.

The CEC is responsible for the development and implementation of the required consortium-wide evaluation for bringing together the awardees to work collaboratively as a consortium along with the NIH to achieve the stated goals. The CEC is tasked with conducting a comprehensive evaluation of the Diversity Program Consortium activities (FY2015-FY2024). This request is for OMB clearance for an additional 3 years to continue the efforts. OMB clearance for the final year(s) will be sought in future years. BUILD and NRMN outcome evaluations will support the assessment of the Diversity Program Consortium’s overall effectiveness, produce knowledge that can be widely disseminated to transform research training and mentoring nationwide, and inform NIH’s training and diversity efforts. Data and knowledge developed from this evaluation will be directly responsive to the recommendations set forth by the ACD Working Group to the NIH Director. The transformative potential of this program will be realized as successful models of training, mentoring, and institutional development are disseminated and adopted across NIH and nationwide.

Evaluation of the BUILD program includes assessing the outcomes described in the Hallmarks of Success using the data elements described in this application (Attachment 7) and producing robust evaluation products (Attachment 8). The process may be visualized using the BUILD Logic Models (Attachment 10).

BUILD Student Outcomes include –

High academic self-efficacy
High self-efficacy as a researcher
High science identity
Satisfaction with quality of mentorship
Perceived sense of belonging within the university
Perceived sense of belonging within the research community
Intent to pursue a career in biomedical research
Entry into an undergraduate biomedical degree program
Persistence in biomedical degree or other formal research training program
Frequent receipt of mentoring to enhance success in the biomedical pathway
Participation in mentored or supervised biomedical research
Evidence of competitiveness for transitioning into the next phase in the biomedical career pathway
Participation in academic or professional organizations related to biomedical disciplines
Evidence of excelling in biomedical research and scholarship
Strong academic and professional networks
Completion of biomedical degree or other formal training program
Application and acceptance to a subsequent research training program in a biomedical discipline
Entrance into a subsequent research training program in a biomedical discipline

BUILD Faculty/mentor outcomes include –

High self-efficacy as an instructor in a biomedical field
High self-efficacy as an instructor to a diverse group of biomedical students
High self-efficacy as a mentor to biomedical research trainees
High self-efficacy as a mentor to a diverse group of biomedical research trainees
Frequently mentors students, post-docs, and/or more junior faculty on biomedical related issues
High self-efficacy as an independent biomedical researcher
High self-efficacy in the ability to secure external funding
Engaged in activities to secure research or research training funding
Securing research or research training funding
Evidence of scholarly productivity
Evidence of professional recognition and service
Strong academic and professional networks
Advancement to next career stage
Advancement to leadership positions in biomedical research and research training
Evidence of receiving training in areas to foster inclusive research training environments
Strong self-efficacy to act as a change agent to enhance diversity in biomedical research and research training environments
Uses evidence-based practices in teaching and mentoring

BUILD Institutional outcomes include –

Commitment to efforts that create, enhance, and/or maintain diversity and inclusion at all levels of the institution
Evidence of creating, enhancing, and/or maintaining diverse, inclusive, and culturally appropriate research and research training environments
Demonstrated institutional commitment to creating, enhancing, and/or maintaining the diversity of the biomedical faculty on campus by recruiting a diverse pool of potential applicants
Implementation of sustainable institutionally supported intra-institutional activities to achieve positive outcomes related to biomedical research capacity building and faculty development
Enhanced inter-institutional collaborations to achieve positive outcomes related to biomedical research, research training, and faculty development
Implementation of sustainable institutionally supported activities to achieve positive outcomes related to biomedical research training
Enhancing or maintaining the diversity of students, e.g. those from nationally underrepresented groups, who pursue degrees in biomedical fields
Demonstrated institutional commitment to efforts that sustain the interest of trainees from all backgrounds pursuing degrees in biomedical fields that increase persistence
Employs evidence-based approaches to establish and attain goals for graduation rates, time-to-degree, and the ability to transition to biomedical graduate and professional degree programs for students from all backgrounds
Demonstrated institutional commitment to implementing and sustaining mentoring practices that promote the development of research-oriented students from all backgrounds
Institutional infrastructure to track regular reporting of student demographics and outcomes with respect to biomedical fields

Evaluation of Phase I of the NRMN program includes assessment of the following outcomes using the Phase I NRMN Logic Model (see Attachment 11):

Matching and Linking Core:

Greater career persistence in biomedical sciences among under-represented groups
Greater self-efficacy in ability to succeed in biomedical careers

Training Core:

Increased skills, knowledge and self-efficacy (confidence) in grant writing
Increased skills, knowledge and self-efficacy (confidence) in creating and maintaining effectiveness of mentoring relationships
Greater advocacy for mentorship
Increased commitment to cultural awareness in promoting diversity in biomedical research

Referring Core:

Increased pool of high quality, audience-valued, targeted resources available to diverse individuals across career stages pursuing biomedical careers

Promoting Core:

Influence on institutional climate and structural barriers to creating an environment supporting diverse populations in the biomedical career pipeline
Recognition of the value of mentoring for diverse workforce at all career stages at colleges/universities
Commitment by institutions nationwide to promote diversity in biomedical sciences

A.2 Purpose and Use of the Information Collection

The various sources of data needed for the proposed BUILD and NRMN evaluations are described below. These sources include on-line surveys (paper versions are available if needed) administered to students/mentees and faculty/mentors, requests for institutional data from BUILD sites, as well as qualitative data to be collected during BUILD site visits and case studies.

SURVEYS

For BUILD, the purpose of the surveys is to collect baseline data and track subsequent outcome data to permit evaluation of primary programmatic objectives and outcomes outlined in A.1.

For NRMN, CEC-specific follow-up data collection is conducted through on-line surveys of NRMN participants. Baseline data for NRMN Phase I participants was collected by the NRMN cores via the NRMN website, and pre/post-tests conducted by NRMN investigators during their trainings. Follow-up CEC-administered surveys provide information on longer-term outcomes of the NRMN Phase I training and mentor activities, including mentor competency, perceptions of mentor self-efficacy, and increased scientific productivity and successful submission of NIH grants.

BUILD Institution Undergraduates

CIRP HERI Freshman Survey (see Attachment 12) – A survey administered to a sample of 500 “entering students” at each BUILD institution annually, including freshman as well as transfer students who enter at higher levels. The sample of 500 includes all students admitted to the BUILD program itself as well as a sample from similar students not in the BUILD program. As noted in B.1.1, sampling includes oversampling on the basis of declared major (80% biomedical vs 20% other) and race/ethnicity (oversampling African Americans and Hispanic/Native Americans to account for their known lower survey response rates). The initial survey has as its core the “The Cooperative Institutional Research Program (CIRP) Freshman Survey” items administered by the UCLA Higher Education Research Institute (HERI) (see Attachment 12). NOTE: As discussed further in sections A.3 and B.2.1, for all surveys, the planned primary mode of administration is on-line; however, paper versions are available as needed.

The CIRP HERI “entering student” survey is an existing source of data collection that has been fielded for 50 years by hundreds of two-year colleges, four-year colleges and universities. The CEC works with all BUILD institutions to ensure that these surveys are completed at minimum by the needed sample of 500 students. Use of the CIRP HERI survey as the core of the proposed baseline for undergraduates allows for institutional and planned national comparisons of data obtained from BUILD students to students from similar institutions that do not have the BUILD program but do administer the CIRP HERI survey to incoming freshmen. These surveys provide a baseline or pre-intervention level for each student. Items on the survey include student demographic characteristics, academic preparedness, expectations of college, interactions with peers and faculty, student values and goals, scientific identify, research self-efficacy, and student concerns about financing college.

Student Annual Follow-Up Survey (see Attachment 13) – BUILD students are contacted annually after their baseline enrollment and asked to complete an on-line survey. This survey seeks to update their academic progress and obtain information regarding Consortium Hallmarks of Success. Topics include students’ research and mentoring experiences, self-assessments of science identity, research self-efficacy and participation in professional conferences and publications.

College Senior Survey (see Attachment 14) – Administered to graduating seniors, this survey allows reflection on the undergraduate experience, degree attainment, and assesses future career and graduate education plans. Several items can be compared with the entering student survey described above.

BUILD Faculty

HERI Faculty Survey (see Attachment 15) – Baseline data for a sample of 50 faculty in biomedical disciplines at each BUILD institution (total N=500) are collected through an on-line survey administered in 2016. All faculty involved in BUILD program activities are included unless there are more than 25 in which case a subsample of 25 are randomly drawn to represent no more than 50% of the total faculty sample of 50. Faculty who have not participated in BUILD activities are randomly sampled to complete the total desired sample size of 50 at each institution. The core of the survey is the HERI Faculty Survey in order to allow for comparisons with faculty from non-BUILD institutions for whom secondary data is available from the national data collection waves. The survey asks about scholarly productivity, teaching load, patterns of interaction with students, perceptions of institutional support for research, and perceptions of diversity at the institution.

Faculty Annual Follow-up Survey (see Attachment 16) – Participants who completed the baseline HERI Faculty Survey are asked to provide updated information annually about their mentoring and research activities as well as their scholarly productivity.

NRMN Participants

NRMN Baseline Surveys – Based on the program’s second phase restructuring of the NRMN initiative, participants will no longer be asked to complete surveys that were required during the first phase. For this reason, Attachments 15 through 22 from the original submission are not included in this application (see Section A.15 for explanation of this adjustment).

NRMN Annual Follow-Up Survey (see Attachment 17) – Follow-up surveys for NRMN participants to document academic output and career progress. Follow-up data supports the CEC outcome evaluation with respect to outcomes of the training and professional development programs as well as outcomes relating to general academic output and career progress of NRMN participants. In the original submission, these were Attachments 17 and 18. For Phase II the follow-up surveys are combined into a single survey (see Attachment 17 and Section A.15 for explanation of this adjustment).

SITE VISITS AND CASE STUDIES

BUILD Site Visits and Case Studies (Attachment 18) - As part of the BUILD mixed methods evaluation design, the CEC and NIH conduct site visits to the 10 sites in order to understand the context and conditions at the BUILD sites. Site visits are an occasion for sites to showcase the defining features of their programs as well as to discuss any challenges related to program implementation and evaluation. The site visit is a three-way exchange of information among the BUILD site, the CEC and NIH that allows for critical face-to-face learning to transpire.

In addition to the site visits, the CEC conducts periodic in-depth case studies of the BUILD sites. The purpose of the case studies is to provide a holistic description of each BUILD program that goes beyond the information captured in the survey data. Case studies provide a clearer understanding of each BUILD site’s successes and challenges and enhance the validity of the consortium evaluation. Case studies offer the opportunity to provide a narrative description of the BUILD site’s inputs, activities, and outcomes. The BUILD case studies focus on the processes and procedures that build capacity and infrastructure to enhance diversity in the biomedical research workforce.

NRMN has undergone a restructuring for Phase II and as such the CEC is no longer conducting site visits or case studies. For this reason, Attachment 24 from the original submission was eliminated.

ADMINISTRATIVE RECORDS AND NATIONAL DATABASES

BUILD Institutional Records & Program Data (Attachment 19) – Additional student and faculty data is annually requested from BUILD institutions (e.g., activity rosters, courses, grades, majors).

BUILD Implementation Reports – These reports are no longer needed in Phase II because the sites have implemented their training and mentoring interventions. For this reason, Attachment 26 from the original submission has been eliminated.

Use of the Information Collected:

The purpose of this large-scale initiative is to provide generalizable knowledge about what factors are important to success and in what settings so that the broader scientific community may adapt and employ these evidence-based approaches to enhancing diversity in the biomedical research workforce. Details of the Priority Consortium Evaluation Products are listed in Attachment 8. The comprehensive approach will also allow for statistically robust testing of certain hypotheses that are not achievable at the site-level (see Attachment 26 for examples). Ultimately, without the data collection, NIH will not have information necessary to evaluate the effectiveness of the funding investment or to produce robust findings to advance the goals of enhancing diversity in the biomedical workforce.

To enhance and facilitate broader dissemination, the consortium will continue to generate compelling products that promote effective practices to stakeholders, policy-makers, educators, and others. The bullets below outline the multiple venues that have been, and will continue to be used for disseminating the study findings:

Briefs - small topical reports that highlight key outcomes and how they can be used to better inform diversity programs, specifically the impact of BUILD and NRMN.
Presentations at national conferences – talks and poster presentations of results at widely attended conferences to maximize the impact of the findings.
Journal articles – robust scientific findings subjected to the peer-review process and published in widely disseminated journals. These articles provide the evidence base for successful training and mentoring interventions conducted by BUILD and NRMN. The findings to date are curated on the NIGMS Diversity Program Consortium publication webpage.
Releases and brief notes – these items are to gain the attention of broader audiences through outlets such as the Chronicle of Higher Education, and news outlets.

A.3 Use of Information Technology and Burden Reduction

For BUILD, the surveys for all students and faculty are provided online to facilitate completion by respondents at times and locations convenient for them, thus reducing burden. Where possible, the CEC relies on collection of existing annual institutional data to further reduce the time needed to collect data directly from BUILD participants.

As noted earlier, baseline data for NRMN Phase I participants has been collected and is stored with the NRMN awardees. Annual NRMN online follow-up surveys hosted by the CEC are designed collect data from NRMN participants (students, postdoctoral scientists, junior faculty, senior researchers, and program leaders) on scientific productivity and career progress.

For all on-line surveys, participants receive a link to the online survey and a user code to access it. As appropriate, the online surveys use skip-patterns so that each respondent is only presented with questions relevant to his or her specific situation. In addition, most questions on the surveys are multiple choice or closed-ended to reduce burden on respondents. Informed consent procedures are incorporated into the survey administration to allow respondents to complete both activities during a single session.

All Diversity Program Consortium survey data are stored and can only be securely accessed at the CEC at UCLA. The Diversity Program Consortium data system is compliant with UCLA policies (e.g., Policy 404 - http://www.csg.ucla.edu/documents/2009/Policy404DeanVCfinal.pdf) on the protection of personally identifying information. In consultation with the Chief Information Officer of the UCLA David Geffen School of Medicine (DGSOM), the CEC follows internal Risk Assessment procedures for all systems (see Attachment 20: Coordination & Evaluation Center (CEC) Tracker Security Overview for details).

A.4 Efforts to Identify Duplication and Use of Similar Information

The data collection from BUILD undergraduate students and faculty, from NRMN participants, and from institutional sources do not duplicate other data collection efforts. The NIH has convened numerous discussions both across NIH (e.g., Office of Extramural Programs, Training and Advisory Council, and the Office for Scientific Workforce Diversity) and from individuals outside of the NIH (see Attachments 21, 22, and 23) to discuss the needs and objectives of the program and ensure that the knowledge needed could not be answered using existing data collection efforts. There is no similar information that has been collected across a multi-site, multi-year program aiming to develop training, mentoring, and institutional development to enhance diversity in the biomedical workforce. As noted above in some detail, where possible, the Diversity Program Consortium uses existing, secondary data from national surveys, such as those administered by HERI, or data collected by the NRMN or BUILD initiatives or the BUILD institutions themselves in order to leverage activity already underway at many institutions. Through these efforts, the CEC has minimized additional survey collection for evaluation of the Diversity Program Consortium.

A.5 Impact on Small Businesses or Other Small Entities

Small businesses are not involved in this study.

A.6 Consequences of Collecting the Information Less Frequently

The data collection schedules are designed to provide information needed for a rigorous longitudinal program evaluation. Efforts have been made to reduce the burden on participants by collecting data annually rather than on a more frequent basis. Programs are designed to annually enroll students and to assess factors before and after exposure to the intervention activities and to obtain consistent data for each student cohort. In order to assess outcomes, it is necessary to collect data for a significant period of time to see long-term changes (e.g., in degree attainment and matriculation to research graduate programs). It is also necessary to maintain contact with participants after they exit their institutions/programs for tracking of their biomedical research career outcomes. The timeline for data collection allows for adequate tracking of changes in the desired outcomes at the individual student, faculty, and institutional level in the short-, medium-, and long-term.

If the BUILD and NRMN surveys for students and faculty are not conducted or are conducted less frequently, NIH will not be able to determine whether the Diversity Program Consortium produced the desired programmatic outcomes and whether any changes in outcomes could be attributed to the program interventions. Also, if the surveys are not conducted, the program activities which produce the desired outcomes could not be identified and disseminated to the biomedical research training community. If information is not collected from BUILD institutions, NIH will not be able to assess how the interventions are being implemented, the impact of the interventions, the progress made in achieving programmatic goals and achieving sustainability and institutionalization of program interventions and activities. Further, NIH will not be able to take steps to improve implementation, sustainability, and institutionalization of the highly effective training and mentoring interventions.

A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

This project fully complies with all guidelines of 5 CFR 1320.5.

A.8.1 Comments in Response to the Federal Register Notice

The 60-day Federal Register notice for the original application was posted on December 3, 2019. Federal Register Notice Vol. 84 No. 232, Pgs. 66207-66209. Federal Register Notice: https://www.federalregister.gov/documents/2019/12/03/2019-26087/proposed-collection-60-day-comment-request-evaluation-of-the-enhancing-diversity-of-the-nih-funded

One public comment was received (see Attachment 27).

A.8.2 Efforts to Consult Outside Agency

To address underrepresentation in biomedical and behavioral research, NIH Director Dr. Francis Collins charged the Advisory Committee to the NIH Director (ACD) to form a Working Group on Diversity in the Biomedical Research Workforce (WGDBRW) to examine the findings and implications of the Ginther et al. (2011) study results.³ The vast majority of WGDBRW members were external to NIH. Dr. Collins charged the WGDBRW with providing concrete recommendations toward improving the recruitment and retention of underrepresented minorities (URM), people with disabilities, and people from disadvantaged backgrounds across the lifespan of a biomedical research career from graduate study to acquisition of tenure in an academic position or the equivalent in a non-academic setting. The WGDBRW met 13 times in person at the NIH’s Bethesda campus or by telephone and used a variety of means to gather information based on existing data and efforts, leading to a report and set of recommendations to the NIH Director.⁴ These recommendations led to the creation of the Diversity Program Consortium. (see Attachment 21: Original Advisory Committee to the Director – Working Group on Diversity in the Biomedical Workforce Membership).

In preparing this original application, the CEC consulted with other members of the Diversity Program Consortium, both generally and through the Diversity Program Consortium Executive Steering Committee (ESC) as well as NIH scientists involved in the Consortium to determine (a) exactly what information was needed to implement the required consortium-wide evaluation, and (b) what measures and data sources were recommended for collection of each required piece of information. Decisions regarding the data required from BUILD and NRMN participants and comparison groups, the required frequency of that data collection as well as evaluation of potential sources of data (i.e., what required primary data collection and what could be obtained from existing data sources) was informed by Consortium member’s extensive experience in implementation of educational interventions for university students and faculty and the methodologies and the type and frequency of data sources required for successful evaluation of those efforts.

Specific consultations included: (1) Group discussions with BUILD and NRMN investigators at the 2014 Consortium Kickoff meeting, (2) CEC orientation visits to each BUILD site and attendance at 2 NRMN leadership meetings, where evaluation and data collection were discussed with site investigators, faculty, and institutional representatives, (3) Working group conference calls with BUILD and NRMN representatives regarding evaluation plans and data collection (bi-monthly for 6-8 months) (4) CEC 1-day planning retreat with the NIH leadership team, and (4) ESC meetings with representatives from each BUILD and NRMN site and the NIH to develop, discuss, and approve proposed Consortium Hallmarks of Success.

The table below provides the names and affiliations for the BUILD PIs and NRMN leadership involved in the original consortium consultations.

BUILD & NRMN leadership consulted in Grant Year 01
	Name	Affiliation	Organization	Email
1	Laura Kingsford	BUILD	Cal State University Long Beach (CSULB)	Laura.kingsford@csulb.edu
2	Crist Khachikian	BUILD	Cal State University Northridge (CSUN)	Crist.khachikian@csun.edu
3	Gary Kuleck	BUILD	University of Detroit Mercy (UDM)	kuleckga@udmercy.edu
4	Farin Kamangar	BUILD	Morgan State University (MSU)	farinkamangar@gmail.com
5	Carlos Crespo	BUILD	Portland State University (PSU)	ccrespo@pdx.edu
6	Leticia Márquez-Magaña	BUILD	San Francisco State University (SFSU)	marquez@sfsu.edu
7	Karsten Hueffer	BUILD	University of Alaska Fairbanks (UAF)	khueffer@alaska.edu
8	Philip Rous	BUILD	University of Maryland Baltimore County (UMBC)	rous@umbc.edu
9	Lourdes Echegoyen	BUILD	University of Texas El Paso (UTEP)	Lourdes@utep.edu
10	Gene D’Amour	BUILD	Xavier University	gdamour@xula.edu
11	David Burgess	NRMN	Boston College	David.burgess@bc.edu
12	Chris Pfund	NRMN	University of Wisconsin, Madison	cepfund@wisc.edu

The initiative continues to be closely monitored by members of the community external to NIH. Current external input comes from the ACD Working Group on Diversity (see Attachment 22) as well as the ACD Working Group on Diversity, Diversity Program Consortium subcommittee (see Attachment 23).

A.9 Explanation of Any Payment or Gift to Respondents

To be responsive to the Funding Opportunity Announcements (see Attachments 4 and 5), the CEC must collect comprehensive, longitudinal data to assess the impact of interventions across the consortium. To accomplish this, the CEC must recruit students and faculty participants from groups with varying degrees of involvement with the interventions – ranging from heavily involved to those with no involvement. The latter are quite difficult to recruit as they have no direct connection to the Consortium’s activities. The CEC must secure robust participation for all groups in order to have data that will allow the CEC to draw valid conclusions from the program evaluation. The target student and faculty populations include hard-to-reach underrepresented groups that have traditionally participated in research studies such as this at lower rates.⁵ Additionally, once individuals are recruited into the project, the CEC must retain participants over the longitudinal follow-up to derive valid conclusions regarding the long-term impact of the interventions on primary outcomes, including rates of graduation and progression to graduate school for BUILD students and career progress with respect to research, publications and general career advancement for BUILD and NRMN faculty and trainees. Failure to achieve adequate response rates has the potential to result in non-representative samples, thereby undermining conclusions drawn from the data. Though the burden to participants for the individual surveys is minimal, it is essential that the CEC maximizes the willingness to respond repeatedly over time to requests that they complete an annual survey.

The current recruitment/retention strategies include non-monetary approaches known to improve response rates (e.g., using influencer emails and providing respondents information about the contribution they will be making to the understanding of the important issues on which the project focuses to enhance their intrinsic motivation to participate)⁶; however, Phase I analyses of response rates confirmed that successful recruitment/retention efforts require that the CEC offers monetary incentives through individual incentives. Individual incentives will include up to a $25 gift card to each faculty or student respondent for each survey that they complete. The choice of a $25 incentive is based on prior evidence from the literature⁷ and from experiences of members within the Diversity Program Consortium, which confirm that the incentives are crucial for recruiting a representative group, especially among individuals from underrepresented groups, who are key for this study.

A.10 Assurance of Confidentiality Provided to Respondents

The UCLA Office for Protection of Research Subjects (PRS) reviews the evaluation and its associated data collection protocols, along with the corresponding human subjects review committees of the lead intervention institutions, as noted below.

The NIH Office of Human Subjects Research (OHSR) was consulted and because each site has IRB approval and because the NIH will not be collecting or analyzing the data, there is no need for the office to review the package. The NIH Privacy Act Officer has reviewed the package (Attachment 24). The Consortium Data Sharing Agreement Phase II (see Attachment 25) describes the requirements for data collection, integrity, storage, security, confidentiality, use, sharing, ownership, rights, and responsibilities.

Participants in the Consortium-wide evaluation are informed that their responses to the data collection efforts are only to be disclosed to authorized users for analysis and reporting. The Coordination and Evaluation Center (CEC) at UCLA is responsible for ensuring the security of the data. Authorized users include: (1) the CEC implementing the evaluation, (2) authorized staff at the NIH, and (3) PIs and staff for the BUILD institutions and NRMN network that are involved in the local evaluation. The data is shared with institutions to avoid duplication of efforts and to reduce burden by avoiding the completion of multiple surveys with similar questions. Given the nature of the study, participants are given the assurance that their information will be protected and secured to the extent permitted by law. To that end, data may only be disseminated in an aggregated, de-identified form to the public in order to inform the research community of the results of the study, while protecting the identity of individual respondents.

The CEC conducting the study adheres to the following safeguarding procedures:

The safeguarding protections offered to all participants are included in the invitations and introduction to the survey instruments after review by the appropriate institutional human protections’ office (for the CEC, the UCLA Institutional Review Board; for other institutions, as required). Respondents are informed that their participation is voluntary and that no consequences are associated with either responding or not responding. For example, the online surveys have the following statements: “Your participation in the evaluation of the NIH Diversity program is voluntary. You are free to withdraw your consent and discontinue participation without penalty at any time. You are not waiving any legal claims, rights or remedies because of participating in the NIH Diversity program” and “All information you provide for this program is private. You will be assigned a code number, and all surveys will use this number. All the information you provide will remain locked in a filing cabinet or protected on a secure server to prevent disclosure. Information provided by you during participation in the surveys for the evaluation of the NIH Diversity program will only be disclosed to the Coordinating Center staff, the NIH Diversity program staff, and the Diversity Program staff at your institution”.
Data are stored in a manner such that restricted information (e.g., name, address, contact information) is stored in a different system from study data such as survey responses. The restricted information is stored in a system behind the CEC firewall and operates on a private IP range. Only local users can access these IP addresses and the number of authorized users is strictly limited. The study data is maintained in a separate system requiring authorized users using encryption.
Any paper files used for data collection (such as handwritten interview notes) are stored in locked cabinets with access limited and controlled as with electronic data.
Publications only report the data in aggregate and will not contain any identifying information.
The UCLA CEC follows the methods detailed in the NIH Privacy Act Systems of Record Notice 09-25-0156 which provides authority for the NIH to conduct and fund research and to provide training assistance, and its general authority to maintain records in connection with these and its other functions (42 U.S.C. 203, 241, 289l-1 and 44 U.S.C. 3101), and Section 301 and 493 of the Public Health Service Act, as it relates to records of participants in programs and respondents in surveys used to evaluate programs of the Public Health Service.
The data collected from the surveys and the interviews are stored in the UCLA CEC secure server. A password-protected directory allows only authorized users access to the data. All computer-based systems comply with the Privacy Act.

A.11 Justification for Sensitive Questions

The proposed data collection involves few, if any, sensitive questions in the surveys, site visits, and case studies. The student and faculty surveys contain questions regarding respondents’ race, ethnicity, gender and income. It is important to collect income data (along with other personal characteristics) in order to document the diversity of the participants in this initiative which is specifically designed to enhance the diversity of the NIH-funded workforce and to understand the contextual factors and variables that contribute to success in any intervention.

Note that items regarding race included in the CIRP HERI surveys are similar but not identical to the definitions for racial categories in OMB Directive 15. The CIRP HERI surveys also ask only that single item and do not include a separate item on ethnicity as outlined in OMB Directive 15. The CEC is unable to modify the items in the CIRP HERI surveys as HERI seeks to maintain comparability of their data over time, having administered those same items over the past 50 years. However, for federal reporting the CEC collapses those responses into the 4 racial categories (American Indian or Alaska Native, Asian or Pacific Islander, Black, White) and 2 ethnic categories (Hispanic origin, Not of Hispanic origin) as defined in the OMB Directive 15. In addition, in the annual follow-up surveys for both students and faculty, the CEC asks the standard ethnicity item (Hispanic/not Hispanic) and a separate race item (e.g., see item 43 in Attachment 13). The race item is more detailed than those listed in OMB Directive 15 because several of the participating institutions desire more detailed race information. However, the resulting data allows us to classify participants according to the race/ethnic classifications in OMB Directive 15 for reporting purposes. Similarly, for sexual orientation/gender identity, since CIRP HERI is a historical survey, the CEC cannot change the wording (see items 1-3 in Attachment 12); however, when reporting in these categories, the CEC will employ current practices at the Department of Health and Human Services.

The exit and follow-up surveys for BUILD students and faculty and NRMN mentees and mentors contain questions regarding work-related information (type of employer organization and career field), career status (student/faculty level – graduate, post-graduate, Assistant/Associate/Full professor), number of NIH applications, number of publications. This information allows NIH to analyze the survey data by subgroups and support NIH’s long-standing efforts to enhance the diversity of the biomedical research workforce.

Survey participants may skip any or all of the Personally Identifiable Information (PII) questions that they do not wish to answer. To avoid fear of disclosure of sensitive information, participants are assured that their responses will be kept private and reported in terms of de-identified, aggregate numbers or summary statistics.

Institutional level data does not contain any highly sensitive questions.

A.12.1 Estimates of Hour Burden Including Annualized Hourly Costs

A.12.1 Estimates of Hours Burden

Every effort is made to minimize the burden on the respondents. All surveys are given online (with paper version available on request). Estimates of hourly burden are provided only for the surveys that will be administered during the requested approval period.

BUILD Students – The entrance survey for the BUILD undergraduate students take on average 45 minutes to complete. Similarly, the surveys at the end of the senior years also take approximately 45 minutes each. The annual follow-up survey takes on average 25 minutes. If this current request for clearance is approved, the same surveys will be used to gather data from applicants for the duration of the clearance.

BUILD Faculty – The initial survey for BUILD faculty takes on average 45 minutes to complete. The annual follow-up survey takes on average 15 minutes to complete.

BUILD Site Visits – For Phase II, a maximum of 1 site visit per BUILD site will be conducted. A team of 3-5 CEC team members will participate in the site visits. On average, 12-15 individuals from a given BUILD site participate in the site visit. The time estimates are 24 hours per person based on 8 hours preparation, 16 hours for a 2-day site visit, including the group interviews.

BUILD Case Studies – On average, 40 hours (5 days * 8hr/day =40 hours) will be needed from someone at each of the case studies sites to assist in pulling together needed materials and assisting in scheduling the requested individual and group interviews. Semi-structured interviews are conducted at each BUILD site, including: Individual interviews (1.5 hours each) with the Principal Investigator(s), Program manager(s), and Partner Institution Directors (up to a maximum of 4 such individuals); and Group interviews (1.5 hours each) with BUILD Faculty Leads (5 participants), Faculty participants (including mentors) in site BUILD activities (5 participants), undergraduate students (2 groups of 6 active BUILD student participants; one group of 4 BUILD drop-outs) and one group of 6 graduate/post-doctoral students. Two additional case studies will be done at 2 matched non-BUILD institutions; these will include interviews (1.5 hours each) with one Institutional Director/Program Manager and group interviews with 5 faculty, 5 undergraduate students, 5 graduate/post-doctoral students.

NRMN – Completing the annual follow-up survey is estimated to take an average of 25 minutes, with variability depending on the academic productivity of the respondent. The biannual program-specific components, which are all fixed choice questions, add about five minutes per program component.

NRMN Case Studies Visits – The case studies have been discontinued because of the restructuring in Phase II.

Table A.12.1 displays the annualized estimate of hour burden. For each survey/data collection instrument included in Table A.12.1, the relevant Attachment illustrating that instrument is also listed. The expected burden level per year for this study is 55,132 hours. This is less than the first OMB application (61,950 hours).

Table 12-1 Estimated Annualized Burden Hours

Form Name	Type of Respondents	Number of Respondents	Number of Responses per Respondent	Average Time per Response (in hours)	Total Annual Burden Hours
2019 CIRP HERI Freshman Survey- (Attachment 12)	BUILD and Non- BUILD Student	15,000	1	45/60	11,250
Student Annual Follow-up survey (Attachment 13)	BUILD and Non- BUILD Student	15,000	1	45/60	11,250
2019 College Senior Survey (Attachment 14)	BUILD and Non- BUILD Student	15,000	1	45/60	11,250
Student Annual Follow-up Survey (Attachment 13)	2020 Student Cohort	5,000	3	25/60	6,250
Student Annual Follow-up Survey (Attachment 13)	2021 Student Cohort	5,000	2	25/60	4,167
Student Annual Follow-up Survey (Attachment 13)	2022 Student Cohort	5,000	1	25/60	2,083
2019-20 HERI Faculty Survey Core National Instrument (Attachment 15)	BUILD and Non- BUILD Faculty Survey	500	1	25/60	208
Faculty Annual Follow-up survey (Attachment 16)	BUILD Faculty Annual Follow-up survey	500	2	25/60	417
BUILD Institutional Record & Program Data Requests (Attachment 19)	Personnel and Administrators at BUILD Institutions	10	3	16	480
BUILD Site Visits (Attachment 18)	BUILD Students, Faculty, and Institution	120	1	24	2,880
BUILD Case Studies Preparation (Attachment 18)	BUILD Students, Faculty, and Institutions	24	1	40	960
BUILD Case Study Interviews (Attachment 18)	Undergraduate BUILD Students	170	1	90/60	255
BUILD Case Study Interviews (Attachment 18)	Graduate/post-doctoral BUILD students	70	1	90/60	105
BUILD Case Study Interviews (Attachment 18)	BUILD PI’s, Program Managers/Directors, & Faculty	162	1	90/60	243
NRMN Annual Follow-up Surveys (Attachment 17)	NRMN 2020 mentee cohort	500	3	25/60	625
NRMN Annual Follow-up Surveys (Attachment 17)	NRMN 2021 mentee cohort	500	3	25/60	625
NRMN Annual Follow-up Surveys (Attachment 17)	NRMN 2022 mentee cohort	500	2	25/60	417
NRMN Annual Follow-up Surveys (Attachment 17)	NRMN 2020 mentor cohort	500	3	25/60	625
NRMN Annual Follow-up Surveys (Attachment 17)	NRMN 2021 mentor cohort	500	3	25/60	625
NRMN Annual Follow-up Surveys (Attachment 17)	NRMN 2022 mentor cohort	500	2	25/60	417
	Total	64,556	85,076		55,132

A.12.2 Annualized Cost to Respondents

An hourly earning rate for undergraduate students was estimated by average of the state minimum wage for the locations of the BUILD institutions ($8.73). The average hourly earnings for graduate students are $17.50 and $31.15 for post-doctoral students. An hourly earning rate for faculty was estimated using the American Association of University Professors’ Annual Report on the Economic Status of the Profession (https://www.aaup.org/report/annual-report-economic-status-profession-2018-19). The amount of average hourly earnings rate was based on the salary of full professors with doctorates. The estimated average hourly earnings for the faculty was calculated to be $64.34. The average hourly earnings rate for Institutional Research staff is estimated as the same as faculty.

Table A.12.2 shows that the estimated annual cost for all respondents to participate in the Diversity Program Consortium evaluation would equal approximately $863,395. This value is less than that provided in the first OMB application ($1,304, 459).

Table 12-2 Annualized Cost to Respondents

Type of Respondents	Total Annual Burden Hours	Hourly Respondent Wage Rate*	Respondent Cost
BUILD and Non- BUILD Student - (Attachment 12)	11,250	$8.73	$98,213
BUILD and Non-Build Student – (Attachment 13)	11,250	$8.73	$98,213
BUILD and Non- BUILD Student – (Attachment 14)	11,250	$8.73	$98,213
2020 Student Cohort (Attachment 13)	6,250	$8.73	$54,563
2021 Student Cohort (Attachment 13)	4,167	$8.73	$36,375
2022 Student Cohort (Attachment 13)	2,083	$8.73	$18,188
BUILD and Non-BUILD Faculty (Attachment 15)	208	$64.34	$13,404
BUILD and Non-BUILD Faculty (Attachment 16)	417	$64.34	$26,808
Personnel and Administrators at BUILD Institutions (Attachment 19)	480	$64.34	$30,883
BUILD Students, Faculty, and Institution (Attachment 18)	2,880	$64.34	$185,299
BUILD Students, Faculty, and Institution (Attachment 18)	960	$64.34	$61,766
Undergraduate BUILD Students (Attachment 18)	255	$8.73	$2,226
Graduate/post-doctoral BUILD students (Attachment 18)	105	$17.40	$1,827
BUILD PI’s, Program Managers/Directors, & Faculty (Attachment 18)	243	$64.34	$15,635
NRMN 2020 mentee cohort (Attachment 17)	625	$8.73	$5,456
NRMN 2021 mentee cohort (Attachment 17)	625	$8.73	$5,456
NRMN 2022 mentee cohort (Attachment 17)	417	$8.73	$3,638
NRMN 2020 mentor cohort (Attachment 17)	625	$64.34	$40,213
NRMN 2021 mentor cohort (Attachment 17)	625	$64.34	$40,213
NRMN 2022 mentor cohort (Attachment 17)	417	$64.34	$26,808
Total	55,132		$863,395

*Bureau of Labor Statistics: The General Public rate was obtained from the http://www.bls.gov/oes/2018/may/oes_nat.htm#00-0000

The Health Professionals wage rate was obtained from http://www.bls.gov/oes/2018/may/oes290000.htm

Occupation title “Healthcare Practitioners and Technical Occupations”, occupation code 29-0000;

A.13 Estimate of Other Total Annual Cost Burden to Respondents or Record Keepers

There are no capital, maintenance or operating costs to respondents.

A.14 Annualized Cost to the Federal Government

The annualized costs to the federal government for the administration of the entire Diversity Program Consortium is approximately $867,966.95. The funds include the salaries of the Program Staff, Grants Management, Contract and Support Staff, and travel costs to the Annual Meeting and Site Visits.

Cost Descriptions	Grade/Step	Salary*	% of Effort	Fringe (if applicable)	Total Cost to Gov’t
Federal Oversight
Project Lead 1	Title 42	$184,234.00	0.4		$73,693.60
Grants Management 1	GS-13/10	$129,906.00	0.4		$51,962.40
Grants Management 2	GS-13/10	$126,062.00	0.4		$50,424.80
Grants Management 3	GS-13/8	$119,597.00	0.2		$23,919.40
Grants Management 4	GS-13/7	$116,365.00	0.3		$34,909.50
Grants Management 5	GS-13/6	$113,132.00	0.3		$33,939.60
Grants Management 6	GS-13/4	$106,668.00	0.2		$21,333.60
Program Officer BUILD 1	GS-14/3	$122,230.00	0.6		$73,338.00
Program Officer BUILD 2	GS-14/1	$114,590.00	0.25		$28,647.50
Program Officer BUILD 3	GS-14/1	$114,590.00	0.1		$11,459.00
Program Officer BUILD 4	GS-13/2	$100,203.00	0.35		$35,071.05
Program Officer CEC 1	GS-15/10	$164,200.00	0.1		$16,420.00
Program Officer NRMN	GS-14/9	$145,148.00	0.2		$29,029.60
Project Scientist BUILD 1	GS-15/10	$164,200.00	0.2		$32,840.00
Project Scientist BUILD 2	GS-15/10	$164,200.00	0.1		$16,420.00
Project Scientist BUILD 3	GS-15/10	$164,200.00	0.2		$32,840.00
Project Scientist BUILD 4	GS-14/10	$148,967.00	0.2		$29,793.40
Project Scientist BUILD 5	GS-14/7	$137,508.00	0.2		$27,501.60
Project Scientist BUILD 6	GS-14/4	$126,049.00	0.1		$12,604.90
Project Scientist CEC	GS-14/1	$114,590.00	0.1		$11,459.00
Project Scientist NRMN	GS-15/10	$164,200.00	0.1		$16,420.00
Contractor Costs
Communications Specialist		$110,406.40	1	$9,443.20	$119,849.60
Analyst		$138,340.80	1	$9,443.20	$147,784.00
Travel					$10,000.00
Total					$867,966.95

*the Salary in table above is cited from https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/18Tables/html/DCB.aspx

A.15 Explanation for Program Changes or Adjustments

Although the consortium-wide evaluation plan is highly similar to the first submission, some adjustments have been made. They are described below.

Inclusion of the latest funding announcements

New funding announcements (RFAs) were issued to solicit applications for the program’s second 5-year phase (see Attachments 2 and 5). The funding announcements from the program’s first 5-year phase that were part of the original submission are also included (see Attachments 1, 3, and 4).

Eliminating or consolidating certain surveys

Four surveys are being eliminated or consolidated from the program. In the original submission, transfer students took the CIRP HERI Transfer Student Survey (Attachment 9 in the original submission). In this submission, the transfer student survey has been eliminated because transfer students now take the CIRP HERI Freshman Survey (see Attachment 12). In the original submission, two versions of the CIRP HERI Freshman Survey were submitted: CIRP HERI Freshman Survey On-line Version and the CIRP HERI Freshman Survey Paper Version (Attachment 8a and 8b in the original submission). In this submission, because the online and paper versions of this survey are the same, and to reduce redundancy, the CIRP HERI Freshmen Survey On-line Version is not included. In the original submission, the HERI Your First College Year survey was administered (Attachment 10 in the original submission). In this submission, pertinent questions from the HERI Your First College Year survey were added to the existing Student Annual Follow-up Survey (Attachment 13) and the Your First College Year survey is discontinued. In the original submission, separate surveys were administered to NRMN faculty mentors and student mentees (Attachments 15, 17, 18, 19, 20, and 21 from the original submission). In this submission, items from those surveys have been consolidated into a single survey: the NRMN Follow-up Survey (see Attachment 17).

Improving the documentation of the evaluation components

Steps were taken to improve the documentation of the evaluation components. The Hallmarks of Success (outcomes referenced in Section A.1) represent key factors at each stage of a biomedical research career that contribute to development of a successful career in the biomedical research workforce. Attachment 6 is an updated document of the Hallmarks of Success. Attachment 7 is a new document that maps the Hallmarks of Success to the administered surveys. Attachment 8 is a new document that outlines high-priority evaluation products (e.g., data briefs and journal articles) that correspond to each of the Hallmarks of Success. Attachment 9 is an updated document that provides a high-level view of the main interventions being implemented by each of the 10 BUILD institutions. Attachment 10 is an updated document of the BUILD logic models pertaining to student-, faculty-, and institutional-level interventions. Attachment 11 is an updated document of the NRMN logic models pertaining to their key interventions and outcomes for mentors and mentees. Attachment 18 is an updated document of the BUILD site visit and case studies protocol. Attachment 25 is a new document that is the Data Sharing Agreement applicable to the second phase of this program. The Data Sharing Agreement outlines the policies governing access to the different types of data collected as part of the evaluation. Attachment 26 is a new document that provides guidance on statistical methods for analyzing survey data.

Discontinuing new participants for NRMN (eliminating the baseline surveys)

Based on the program’s second phase restructuring of the NRMN initiative, participants will no longer be asked to complete surveys that were required during the first phase. This eliminates the need for the following surveys from the original submission:

Attachment 15-Mentee Mentor Assessment
Attachment 16-NRMN Data Warehouse Baseline Data
Attachment 18-NRMN Student/Mentee Core Follow-up Survey
Attachment 19-NRMN Mentor Skills Module
Attachment 20-NRMN Research & Grant Writing Module
Attachment 21-NRMN Coaching Training Module
Attachment 22-NRMN Institutional Context Module

Discontinuing NRMN Site Visits and Case Studies

Based on the program’s second phase restructuring of the NRMN initiative, NRMN site visits and case studies will no longer be included in the evaluation components. This eliminates the need for the NRMN Site Visit & Case Studies Protocol, which was included in the original submission (Attachment 24 in the original submission).

Phasing out reports no longer needed in Phase II

To reduce administrative burden on program participants, the BUILD Implementation Reports will no longer be required in the second phase. Originally, each BUILD program was asked to complete an implementation report annually, providing information on BUILD activities that were implemented and numbers of participants (Attachment 26 in the original submission).

Updating the external advisors

The Advisory Committee to the Director (ACD) Working Group on Diversity (WGD) was formed in response to the ACD Working Group on Diversity in the Biomedical Research Workforce (WGDBRW) recommendations. The WGD is a permanent working group of the ACD and has been charged with providing regular advice to the to the ACD and National Institutes of Health Director on effective strategies to increase the representation of diverse individuals underrepresented nationally in biomedical research and to reduce disparities in research awards from diverse applicants underrepresented nationally in biomedical research. The roster of current ACDWGD members is included in Attachment 22, and the roster of advisors from the original ACD WGDBRW is included in Attachment 21. The Diversity Program Consortium subcommittee of the ACDWGD was formed to provide external input specific to this program. The roster of current Diversity Program Consortium subcommittee members is included in Attachment 23.

Adjustments of payment or gift to respondents

In the original submission, participants who completed surveys were offered gift card incentives through a raffle. The incentives included 15 $100 gift cards raffled to each set of 50 participating faculty at each of the 10 BUILD institutions; 75 $20 gift cards raffled to each set of 500 participating students at each BUILD institution, and 200 $100 gift cards raffled to the overall pool of NRMN participants each year. The use of a raffle at the time of the original submission was based on budget constraints and available funds. Since that time, the budget has changed to allow for survey participants to receive $25 gift cards for each completed survey.

A.16 Plans for Tabulation and Publication and Project Time Schedule

The data collection and analysis are conducted by the UCLA CEC with oversight from the NIH Project Lead, Program Officer and Project Scientist. The Project Scientist and the Principal Investigators at the CEC meet weekly by phone to ensure progress. The CEC and NIH officials meet monthly by conference call to discuss participation rates, strategies for increasing participation, and progress. During the survey administration period, all members of the consortium are notified of participation rates. Documents, including a detailed implementation calendar, memos, surveys, and response rates are posted on the Diversity Program Consortium intranet.

Activity	Time Schedule
BUILD
CIRP HERI The Freshman Survey (TFS)	Fall, Annually
College Senior Survey (CSS)	Spring/Summer, Annually
Student Annual Follow-up Survey (SAFS)	Spring/Summer, Annually
HERI Faculty Survey	Spring, Annually
Faculty Annual Follow-up Survey (FAFS)	Winter, Annually
BUILD Institutional Records Data	Fall, Annually
Participation Data (Tracker Rosters)	Spring, Annually
BUILD Site visits	Spring (if needed)
BUILD Case studies	Spring (one in Phase II)
NRMN
NRMN Annual Follow-up Survey	Winter, Annually

A.17 Reason(s) Display of OMB Expiration Date is Inappropriate

No exceptions are sought.

A.18 Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are sought.

1 Wilder, EL, Tabak, LE, Pettigrew, RI, Collins, FS. Biomedical Research: Strength from Diversity. Science, 2013; 342, p. 798.

2 Ginther DK, Schaffer WT, Schnell J, Masimore B, Liu F, Haak LL, Kington R. Race, ethnicity, and NIH research awards. Science, 2011 Aug 19; 333(6045):1015-9. doi: 10.1126/science.1196783. PubMed PMID: 21852498; PubMed Central PMCID: PMC3412416.

3 Ginther DK, Schaffer WT, Schnell J, Masimore B, Liu F, Haak LL, Kington R. Race, ethnicity, and NIH research awards. Science, 2011 Aug 19; 333(6045):1015-9. doi: 10.1126/science.1196783. PubMed PMID: 21852498; PubMed Central PMCID: PMC3412416.

4 NIH Advisory Committee to the Director, Diversity in the Biomedical Workforce Working Group Report. NIH, Bethesda, MD. 2012.

5 Porter, SR, Whitcomb ME. Non-Response in Students Surveys: the Role of Demographics, Engagement and

Personality. Research in Higher Education, 2005, 46:127-152.

Sharkness, J. Why Don’t They Respond? An Investigation of Longitudinal Survey nonresponse Among College Students

Attending Four-Year Institutions. Electronic Thesis and Dissertations, UCLA, 2012.

6 Singer E & Ye C. The use and effects of incentives in surveys. Ann Am Acad Polit Soc Sci, 2013; 645: 112-141.

7 Estrada M, Woodcock A, Schultz PW. Tailored Panel Management: A Theory-Based Approach to Building and

Maintaining Participant Commitment to a Longitudinal Study. Evaluation Review, 2014; 38: pp 3-28.

LaRose R & Tsai HS. Completion rates and non-response error in online surveys: Comparing sweepstakes and pre-paid

cash incentives in studies of online behavior. 2014. Comp Hum Beh, 34: 110-119.

Singer E & Ye C. The use and effects of incentives in surveys. Ann Am Acad Polit Soc Sci, 2013; 645: 112-141.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Subject	Supporting Statement A
Author	Lopez, Maria (NIH/NICHD) [E]
File Modified	0000-00-00
File Created	2021-01-14