Form 19 NRMN Site visit and Case Studies Protocol

OMB #0925-XXXX
EXP. XX/XXXX

Attachment 24:  
NRMN Site Visit & Case Studies Protocol 

1

OMB #0925-XXXX
EXP. XX/XXXX
Note: The first pages of this document describe the protocols for the proposed activity.  The required burden statements are 
included in Appendix A ‐ the observation protocol for the site visits (page 4) and Appendix B ‐ the interview protocols for the case 
studies to be found with each of the specific data collection activities described subsequently (pages 6, 8, and 10).

NRMN Qualitative Evaluation Narrative
The qualitative component of the NRMN evaluation will largely focus on processes and procedures that build
capacity and infrastructure for NRMN to work as a cooperative to increase the representation and success of underrepresented people in biomedical research. The evaluation will consist of both site visits and a case study. Overall,
this qualitative inquiry will capture what cannot be measured elsewhere or has been otherwise overlooked, and aids
in the triangulation of findings.

NRMN SITE VISIT PLAN
Site Visit Focus
Site Visits will largely focus on describing the activities NRMN implementing and promoting to support URG biomedical research training. Site visits will offer the CEC the opportunity to provide a narrative description of the
relationships amongst NRMN’s core program areas and a characterization of the overarching NRMN collective.
Site visits are an occasion for NRMN to showcase the defining features of their programs as well as to discuss any
challenges related to program implementation and evaluation. Site visits provide an opportunity for a three-way
exchange of information between NRMN, the CEC and NIH that will allow for critical face-to-face learning to
transpire. They are venue through which to develop trusting relationships that help to promote knowledge exchange
and learning from the evaluation.

Site Visit Guiding Evaluation Questions
The following evaluation questions will be answered through collection of qualitative site visit data.
1. How is NRMN, and its supplement and pilot programs, implementing their vision for advancing URG
bio-medical research training?
Sub- Questions:
a) How do the Mentoring and Networking, Mentor Training, Professional Development and Research
Resources and Outreach cores within NRMN interact with one another?
b) How does NRMN and its cores interact with the supplemental and pilot programs sponsored by NIH?

Site Visit Protocol
Semi-structured observation will be conducted in bi-annual planning meetings held by NRMN. Participants
will include faculty and staff charged with implementing NRMN and its four cores, as well as implementation
partners, such as BUILD sites, supplement and pilot programs. Observation protocol is included in Appendix A.
1

OMB #0925-XXXX
EXP. XX/XXXX

Site Visit Timeline

Planning meetings hosted by NRMN are scheduled two times per grant year. As such, site visits will be
conducted over the course of one-two days by a team of CEC faculty and staff in coordination with NIH project
officers and scientists in grant year 3, 4, and 5. To the extent possible, there will be continuity in team membership
across the site visits. Additional CEC staff may attend the site visit as deemed necessary.

NRMN CASE STUDY PLAN
Case Study Focus
The primary focus of the NRMN case study it to describe how the NRMN program is enhancing
stakeholder capacity to attract, serve, and promote the success of URGs in biomedical research. The NRMN case
studies will focus on the processes and procedures that build capacity and infrastructure to advance bio-medical
research training. This work requires an understanding and analysis of how to build and successfully implement the
“systems structures” at consortium- and program-levels to achieve this goal. The primary theoretical/conceptual
framework that will guide the analysis of case study is the consortium level logic model, developed by the CEC.

Case Study Guiding Evaluation Questions
The overarching evaluation questions are:
1) How is NRMN working to increase the representation and success of under-represented people in biomedical
research through training and career development of individuals of diverse backgrounds, communities and cultures?
2) To what extent is NRMN integrated within the programs and activities of BUILD and other partner institutions?
Additional evaluation questions include:
 How and to what extent do Mentoring and Networking, Mentor Training, Professional Development and
Research Resources and Outreach cores – as well as the NRMN supplement and pilot programs - work in
collaboration to achieve Diversity Consortium goals?
 How are the strategies that are being implemented by NRMN and its cores enhancing student, faculty, and
institutional participation and engagement in bio-medical research training for URGs?
 How and to what extent do these cores collaborate and interact with BUILD sites?

NRMN Case Study Participants and Timeline
Table 1 details the participants, data collection methods, and timeline proposed for the NRMN evaluation.
NRMN case study interview participants will include the principal investigator (PI), co-principal investigators (CoPI’s) and management staff from each core, supplement and pilot program as well as participant mentors/coaches
and mentees. Interview protocols are provided in Appendix B.
Mentor and coach sample: Mentor and coach interviewees will be purposefully selected from a population of
mentors who agree to be contacted for further/future interviews after completing the CEC NRMN tracking survey.
We will select a sample of 10 mentors each year to participate in a semi-structured interview. Interviewees will be
selected so that our sample includes mentors and coaches from across the NRMN core programs, level of NRMN
2

OMB #0925-XXXX
EXP. XX/XXXX

mentor training (i.e., “certified” mentors), as well as career stage (early (0-8 years), mid (9-20 years), and later (21+
years)).

Mentee and coached sample: Mentee and coached person interviewees will be purposefully selected from those who
agree to be contacted for further/future interviews after completing the CEC NRMN follow-up survey. We will
select a sample of 10 mentees each year to participate in a semi-structured interview. Interviewees will be selected
so that our sample includes mentees from across the NRMN cooperative programs as well as career stage
(undergraduate student, graduate student, post-doctoral fellow, early-career researcher (0-8 years).
Table 1. NRMN case study participants, methods, and timeline
Individual
Title
Interviews
NRMN Principal Investigator
x
Co-Principal Investigator(s)
x
Program Manager(s)
x
RROC
Co-Principal Investigator(s)
x
Program Manager(s)
x
MTC
Co-Principal Investigator(s)
x
Program Manager(s)
x
RMNC
Co-Principal Investigator(s)
x
Program Manager(s)
x
PDC
Co-Principal Investigator(s)
x
Program Manager(s)
x
Supplement Projects
Co-Principal Investigator(s)
x
Program Manager(s)
x
Pilot Projects
Project Lead
x
Mentoring/Coaching Participants
Faculty Mentors/Coaches x 10/year
x
Student/Junior Faculty Mentees/Coachees
x

3

Estimated Timeline
Annual
Annual
Annual
Annual
Annual
Annual
Annual
Annual
Annual
Annual
Annual
Annual
Annual
Annual
Annual
Annual

OMB #0925-XXXX
EXP. XX/XXXX

APPENDIX A:
NRMN SITE VISIT SEMI-STRUCTURED OBSERVATION PROTOCOL
FOR NRMN STAKEHOLDERS

Public reporting burden for this collection of information is estimated to average 16 hours per response, including
the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and
completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person
is not required to respond to, a collection of information unless it displays a currently valid OMB control
number. Send comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC
7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-xxxx*). Do not return the completed form to this address.
The purpose of observation in this context is to gain a contextualized understanding of the ways in which different
groups of NRMN stakeholders (i.e., program leadership, program implementers, and partners) interact and
collaboratively strategize to advance NRMN’s overarching objective of increasing URG participation and
engagement in bio-medical research. While this can often be a dynamic, fluid process wherein observable details
may frequently change, or which may be spontaneously revealed, there are several key areas of interest that are
considered most relevant to this research inquiry. These are as follows:
DOMAIN

KEY AREAS OF INTEREST

Context

Persons/roles present
Stated purpose/objectives
Meeting agenda (planned and as it is actually implemented)
Questions, answers, comments raised (and by who)

Capacity of core, institution,
or organization to successfully
implement NRMN activities

Successes and challenges in implementation

Goal-setting and progress
monitoring

Planning implementation timelines

Negotiation of human, financial, time resource allocations
Achievement of, and challenges in achieving scheduled milestones
Strategic planning to achieve NRMN program objectives
Perceived stakeholder accountability to implementation partners (e.g., NRMNcore, core-core, NRMN-pilot, NRMN-supplement, NRMN-partner institution)
Perceived stakeholder accountability to evaluation theory of success (e.g.,
output, intermediate outcomes, and long-term outcomes)
Alignment of goals and objectives

Identifying and cultivating
potential synergies across
program activities

Alignment of implementation timelines
Alignment of program resources (human and financial)

Identifying and resolving
barriers to effective
collaboration across program
activities

Benefits and drawbacks of partner collaboration in implementation

Investments in establishing/sustaining partnerships
Identification of systemic, structural, procedural weaknesses in links between
NRMN stakeholders
Actionable next steps developed to resolve identified issues

4

OMB #0925-XXXX
EXP. XX/XXXX

Evidence of follow-up and implementation of action items
Addressed/unaddressed,
prioritized/un-prioritized
programmatic needs
Systems, structures, and
processes required to promote
NRMN partnership
sustainability
Data-informed decisionmaking

Resource and support needs identified by various NRMN stakeholders
Response to, and consideration of expressed needs by implementing partners
Perspective and opinions that take precedence in determining implications for
action
Identification of factors promoting long-term NRMN partnership sustainability
Resource allocation (human, financial, time) in support of sustainability needs
Data sources presented
Perceived reliability/quality of data presented
How discussion and dialogue is facilitated (and by who) around data
interpretation
What and how data are used to inform next steps
Perspectives and opinions that take precedence in determining implications for
action
Expressed data needs

5

OMB #0925-XXXX
EXP. XX/XXXX
APPENDIX B: NRMN CASE STUDY SEMI-STRUCTURED INTERVIEW PROTOCOL
Principal Investigator/Co-Principal Investigators/Core Management Staff
In-person Interview (Approximately 1 hour)

This semi-structured interview protocol contains a list of prompts to be drawn from in interviews with individuals participating in the
NRMN Case Study. As a flexible framework, questions may be added or omitted from the interview in response to participant
feedback. Interview questions, however, are expected to stay within the content areas detailed below. (Interviews will each be preceded by
approved processes of participant consent.)
Public reporting burden for this collection of information is estimated to average 1 hour per response, including the
time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and
completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person
is not required to respond to, a collection of information unless it displays a currently valid OMB control
number. Send comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC
7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-xxxx*). Do not return the completed form to this address.
Understanding Overall NRMN Implementation
1) How would you characterize NRMN’s overall mission/vision?
a) Please provide an overall summary of the roles and responsibilities of [each of the NRMN cores/your core]
in advancing NRMN’s mission/vision.
2) From your perspective, what about your current programming elicits faculty engagement and participation in
[NRMN/core name] programming?
a) What impediments might exist to greater [mentor/coach] engagement and participation?
3) [if applicable] From your perspective, what about your current programming elicits student engagement and
participation in [NRMN/core name] programming?
a) What impediments might exist to greater student engagement and participation?
Understanding collaboration amongst NRMN cores
4) How would you describe collaboration amongst the NRMN cores? [Probe for specific examples]
a) What are some of the systems, structures, and processes that support collaboration amongst the cores?
b) What would an ideal level of collaboration look like?
c) What are some of the challenges in establishing an ideal level of collaboration between the cores?
d) How would you describe knowledge sharing and learning amongst the cores? (including data) [Probe for
specific examples]
Understanding NRMN participants
5) How would you characterize NRMN’s mentors and coaches (ex. generally, where are they drawn from?) and
their needs as mentors or coaches?
a) In what ways do you feel that [NRMN/core name] addresses and/or supports those needs?
6) How would you characterize NRMN’s mentees (ex. generally, where are they drawn from?) and their needs as
mentees?
a) In what ways do you feel that [NRMN/core name] addresses and/or supports those needs?
Understanding BUILD partnerships
7) In what ways has [NRMN/core name] collaborated with BUILD institutions?
a) How would you characterize NRMN’s engagement with BUILD institutions?
8) What are some of the challenges in establishing effective partnerships with BUILD institutions?
a) In your estimation, what elements are essential to sustaining collaboration with BUILD institutions?
6

OMB #0925-XXXX
EXP. XX/XXXX

Understanding work with pilot and supplement programs
9) How would you characterize NRMN’s engagement with the pilot and supplement programs?
a) What are some of the challenges in establishing effective partnerships with the pilot and supplement
programs?
b) In your estimation, what elements are essential to sustaining collaboration with pilot and supplement
programs?

Sustainability
10) Assuming that NIH funding ends after another 5-10 years, how could your core continue its work?
a) What indicators are there (if any) of institutional commitments to sustaining your efforts? (where, elaborate)
11) Can you think of any practices or policies that have changed as a result of NRMN that might continue even if
the program does not?
Reflection
10) Overall, how would your characterize NRMN’s work to date?
a) Do you feel that additional supports, systems, or structures could enhance NRMN’s ability to successfully
implement this programming? If so, what?
b) Are there any questions I haven't asked, or are there additional points you would like to raise that we have
not yet discussed?

7

OMB #0925-XXXX
EXP. XX/XXXX

NRMN CASE STUDY SEMI-STRUCTURED INTERVIEW PROTOCOL
Mentor/Coach Participants – Phone-Based Interview (Approximately 1 hour)

This semi-structured interview protocol contains a list of possible prompts to be drawn from in interviews with individuals participating in
the NRMN Case Study. As a flexible framework, questions may be added or omitted from the interview in response to participant
feedback. Interview questions, however, are expected to stay within the content areas detailed below. (Interviews will each be preceded by
approved processes of participant consent.)
Public reporting burden for this collection of information is estimated to average 1 hour per response, including the
time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and
completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person
is not required to respond to, a collection of information unless it displays a currently valid OMB control
number. Send comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC
7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-xxxx*). Do not return the completed form to this address.
Understanding NRMN participation and engagement
1) Please tell me your name, your title, your department, and your current area(s) of research.
2) How did you find about NRMN programs?
3) Why did you decide to participate in NRMN programs?
4) To what extent are in you involved in NRMN programs? (What activities do you participate in, and how
involved in those activities are you?)
Participant feedback on overall NRMN programming
5) In your opinion, has participation in these activities been beneficial to you as a (mentor or coach) to URG’s?
a) What (if any) opportunities or experiences has NRMN participation exposed you to that you might not have
otherwise experienced?
b) What might improve your experiences as a NRMN participant?
i) Are there systems, structures, or processes that could further improve your participation in NRMN
activities as a (mentor or coach) to URG’s?
ii) Are there systems, structures, or processes that could further improve how useful or meaningful your
engagement has been in NRMN activities as a (mentor or coach) to URG’s?
iii) Are there any particular mentoring skill areas you feel could be further addressed by NRMN
programming?
iv) How, if it all, did NRMN training help you address diversity issues in your mentoring/coaching of
URG’s?
(1) Can you give me an example of when you used that information/skill?
Participant feedback on mentoring/coaching
6) How many mentees do you currently (advise or coach) as a result of your participation in NRMN?
a) How did you initially connect with them?
7) What do you discuss in mentoring sessions with your mentee?
a) How often would you say you meet with your mentee(s), and for how long?
8) In what ways do you feel that being a faculty (mentor or coach) has been beneficial to URG’s in bio-medical
research?
9) In what ways, if any, do you feel that your NRMN training experiences could be improved to better support you
as a (mentor or coach) to URG’s in bio-medical research?
Reflection
10) Would you recommend NRMN participation to other faculty? Why or why not?
8

OMB #0925-XXXX
EXP. XX/XXXX

a) What, in your opinion, would further incentivize additional faculty to participate in BUILD programming?
b) What barriers, if any, exist to greater participation in NRMN programming overall? What do you think
would be the best ways to reduce those barriers?
c) Do you have anything else you would like to say about the NRMN programs?

9

OMB #0925-XXXX
EXP. XX/XXXX

NRMN CASE STUDY SEMI-STRUCTURED INTERVIEW PROTOCOL
Student & Faculty Mentee/Coachee Participants – Phone-based Interview (Approximately 1 hour)

This semi-structured interview protocol contains a list of prompts to be drawn from in interviews with individuals participating in the
NRMN Case Study. As a flexible framework, questions may be added or omitted from the interview in response to participant
feedback. Interview questions, however, are expected to stay within the content areas detailed below. (Interviews will each be preceded by
approved processes of participant consent.)
Public reporting burden for this collection of information is estimated to average 1 hour per response, including the
time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and
completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person
is not required to respond to, a collection of information unless it displays a currently valid OMB control
number. Send comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC
7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-xxxx*). Do not return the completed form to this address.
Understanding NRMN participation and engagement
1) Please tell me your name, your current area of study, and what inspired your interest in pursuing bio-medical
research? Are you an undergraduate student, a graduate student, a post-doctoral student, or a junior faculty
member?
2) How did you find out about NRMN programming?
3) Why did you decide to participate in NRMN programming?
4) To what extent are in you involved in NRMN programming? (What activities do you participate in, and how
involved in those activities are you?)
Participant feedback on overall NRMN programming
5) In your opinion, has participation in these activities been beneficial to you as [title – student, post-doc, junior faculty]?
If so, how? If not, why not?
a) In particular, what (if any) opportunities or experiences has NRMN participation exposed you to that you
might not otherwise have experienced?
b) What might improve your experiences as a NRMN participant?
i) In particular, are there systems, structures, or processes that could further improve your participation in
NRMN activities?
ii) Are there particular skill areas you feel could be further addressed by NRMN programming?
Participant feedback on mentoring
6) Tell me about your NRMN mentor, including how long you have been working with them.
a) How did you initially connect with them?
7) What do you discuss in mentoring sessions with your faculty mentor?
a) How often would you say you meet with your mentor and for how long?
8) In what ways do you feel that having a faculty mentor has been beneficial to you as a [title] in bio-medical
research?
9) In what ways, if any, do you feel that your mentoring experience could be improved to better support you as an
URG bio-medical researcher?
10) In what ways do you feel your NRMN mentor has, or has not acknowledged and respected your cultural, racial
and/or gender background?
11) In what ways, if any, do you feel that your faculty mentor could be better prepared to support you as a URG
[title] in bio-medical research?
Reflection
10

OMB #0925-XXXX
EXP. XX/XXXX

12) Do you plan to pursue a career in bio-medical research?
a) What are the motivating factors behind this decision?
13) Would you recommend NRMN participation to other URG [title]? Why or why not?
a) What, in your opinion, would further incentivize additional [title] to participate in NRMN programming?
b) What, if any, barriers to participation exist amongst URG [title]?

11