Priority Consortium Evaluation Products

OMB #0925-0747
EXP. 11/2019

Attachment 8:
Priority
Consortium
Evaluation
Products

Priority CEC CWEP Dissemination Products - 1

On September 25, 2015, the Consortium-Wide Evaluation Plan (CWEP) was presented to and
approved by the Diversity Program Consortium Executive Steering Committee (ESC). The
evaluation topics were presented in broad terms (see 2015 CWEP Plan Final) with the
understanding that the data would be used to compare outcomes for BUILD vs. non-BUILD
participants at BUILD institutions and comparator institutions (with additional analysis focusing
comparing outcomes for students from historically underrepresented and well-represented
groups in the biomedical sciences), and assess the impact of BUILD on grantee institutions.
This document describes the CEC CWEP Dissemination Products in more detail, tying the
evaluation questions and potential positive findings to the associated Hallmark of Success
(2019 ESC approved version). For reference, the Evaluation Questions from the 2015 CWEP
presentation are listed in the first column and mapped to the updated 2019 Hallmarks to
demonstrate the continuity in the CWEP and reflect that although the Hallmarks have been
updated, the intent and themes of the consortium-wide evaluation have remained consistent
throughout the process. Tables are presented for the BUILD Student, Faculty, and Institutionlevel products.
Topics listed in these tables are evaluation products and will be produced by the CEC,
providing a framework for hypothesis-driven research by any member of the consortium. After
submitting a proposal to the Publications and Presentations Subcommittee (PPsC; see P&P
Guidelines), consortium members may develop hypothesis-based research to look more
closely at these evaluation topics. For additional consortium and research topics open for
consideration by consortium writing groups, see the PPsC Master list of Research Topics
spreadsheet.
Because of the significant updates that have been made to the NRMN structure, logic model
and evaluation plan, evaluation products for NRMN will be outlined and presented at a later
time.

BUILD Student
2015 CWEP Evaluation
Question

CWE Hallmark (2019)

Updated Evaluation
Questions
For BUILD and nonBUILD students at
BUILD and comparator
institutions:

1) Are BUILD underrepresented group
(URG) students more
likely to show increased
early predictors
(hallmarks) of success in
pursuing a biomedical
science career when
compared to non-BUILD
students at the same
institution (URG and

High academic selfefficacy
(STU-1)

What is the change in
academic self-efficacy
(SE) over time?

High self-efficacy as a
researcher
(STU-2)

What is the change in selfefficacy as a researcher
over time?

Potential Positive Findings*
BUILD students, compared to
non-BUILD students at BUILD and
comparator institutions, and for
underrepresented versus wellrepresented students across
those categories:
Have a greater increase in academic
self-efficacy over time; note,
academic SE commonly declines in
the first year – we will see if BUILD
students have less decline and later
have overall higher SE
Have a greater increase in research
self-efficacy over time.

Priority CEC CWEP Dissemination Products - 2
2015 CWEP Evaluation
Question

CWE Hallmark (2019)

Updated Evaluation
Questions
For BUILD and nonBUILD students at
BUILD and comparator
institutions:

non-URG) and nonBUILD students at other
similar institutions (URG
and non-URG)?

Potential Positive Findings*
BUILD students, compared to
non-BUILD students at BUILD and
comparator institutions, and for
underrepresented versus wellrepresented students across
those categories:
Have a greater increase in science
identify over time.

High science identity
(STU-3)

What is the change in
science identity over time?

Satisfaction with
quality of mentorship
(STU-4)

How does satisfaction with
faculty mentorship change
over time?

Are more satisfied with faculty
mentorship.

Perceived sense of
belonging within the
university
(STU-5)

What is the change in
perceived sense of
belonging within the
university?

Have a greater increase in perceived
sense of belonging within the
university over time.

Perceived sense of
belonging within the
research community
(STU-6)

What is the change in
perceived sense of
belonging within the
research community?

Have a greater increase in perceived
sense of belonging within the
research community over time.

Intent to pursue a
career in biomedical
research
(STU-7)

How does the intent to
pursue a career in
biomedical research
change over time?

Are more likely to express an intent
to pursue a biomedical research
career over time.

2) Are BUILD URG
students compared to
non-BUILD students
(URG and non-URG)
and students at nonBUILD institutions (URG
and non-URG) more
likely to show increased:
- Completion of
undergraduate degree
in biomedical sciences
- Intent to apply to
graduate program in
biomedical sciences
- Application,
acceptance, &
enrollment in a
graduate program in a
biomedical sciences

Entry into an
undergraduate
biomedical degree
program
(STU-8)

What is the likelihood that
intent to pursue a
biomedical major results in
entering a biomedical
major?

Are more likely to enter a biomedical
major after reporting their intent to
pursue a biomedical major.

Persistence in
biomedical degree or
other formal research
training program
(STU-9)

What is the persistence in
biomedical science
disciplines over time?

Are more likely to persist in a
biomedical science discipline over
time.

3) What is the student
experience of BUILD

Frequent receipt of
mentoring to enhance

How frequently do trainees
receive mentoring in areas

Are more likely to receive frequent
mentoring in areas that are related to

Priority CEC CWEP Dissemination Products - 3
2015 CWEP Evaluation
Question

CWE Hallmark (2019)

Updated Evaluation
Questions
For BUILD and nonBUILD students at
BUILD and comparator
institutions:

activities and how does
that impact program
goals?

2, part 2) Are BUILD
URG students compared
to non-BUILD students
(URG and non-URG)
and students at nonBUILD institutions (URG
and non-URG) more
likely to show increased:
- Completion of

Potential Positive Findings*
BUILD students, compared to
non-BUILD students at BUILD and
comparator institutions, and for
underrepresented versus wellrepresented students across
those categories:
the biomedical pathway (e.g.
research and career mentoring).

success in the
biomedical pathway
(STU-10)

that are related to the
biomedical pathway, and
has the frequency
changed over time?

Participation in
mentored or
supervised biomedical
research
(STU-11)

What are the rates of
participation in mentored
or supervised biomedical
research, and do they
change over time?

Are more likely to participate in
mentored or supervised biomedical
research.

Evidence of
competitiveness for
transitioning into the
next phase in the
biomedical career
pathway
(STU-12)

What is the change in
indicators of
competitiveness for
admission to graduate
school over time?

Are likely to be more competitive for
admission to graduate school (e.g.
GPA and research experience).

Participation in
academic or
professional
organizations related
to biomedical
disciplines
(STU-13)

How does participation in
academic & professional
organizations change over
time?

Are participating more in academic &
professional student organizations.

Evidence of excelling
in biomedical research
and scholarship
(STU-14)

How does evidence of
excelling in biomedical
research and scholarship
change over time?

Strong academic and
professional networks
(STU-15)

What is the change in
academic and professional
networks over time?

Completion of
biomedical degree or
other formal training
program
(STU-16)

Is there a difference in
completion of
undergraduate degree in
biomedical sciences?
(medium-term outcome)

Are more likely to demonstrate
evidence of excelling in biomedical
research and scholarship change
over time, e.g. conference
presentations and biomedical related
awards/honors).
Are more likely to have strong
academic and professional networks
(e.g. number of different coauthors
on publications, number of research
and career mentors).
Are more likely to complete a
biomedical science degree over
time.

Application and
acceptance to a
subsequent research

Is there a difference in
application and
acceptance in a

Are more likely to apply and to be
accepted in a research-oriented
biomedical post-bac or graduate

Priority CEC CWEP Dissemination Products - 4
2015 CWEP Evaluation
Question

CWE Hallmark (2019)

Updated Evaluation
Questions
For BUILD and nonBUILD students at
BUILD and comparator
institutions:

undergraduate degree
in biomedical sciences
- Intent to apply to
graduate program in
biomedical sciences
- Application,
acceptance, &
enrollment in a
graduate program in a
biomedical sciences

training program in a
biomedical discipline
(STU-17)

subsequent research
training program in a
biomedical science
program? (medium-term
outcome)

Entrance into a
subsequent research
training program in a
biomedical discipline
(STU-18)

Is there a difference in
entrance (matriculation)
into a subsequent
research training program
in a biomedical discipline?
(medium-term outcome)

Potential Positive Findings*
BUILD students, compared to
non-BUILD students at BUILD and
comparator institutions, and for
underrepresented versus wellrepresented students across
those categories:
program over time.

Are more likely to enter (matriculate)
in a research-oriented biomedical
post-bac or graduate program over
time.

Priority CEC CWEP Dissemination Products - 5

BUILD Faculty
2015 CWEP
Evaluation Question

CWE Hallmark (2019)

3) What is the faculty
experience of BUILD
activities and how
does that impact
program goals?

1) Are BUILD faculty
compared to nonBUILD faculty and
faculty in non-BUILD
institutions more
likely to show
increased mentor
self-efficacy,
mentoring, and
quality of mentoring?

2) Are BUILD faculty
compared to nonBUILD faculty and
faculty at non-BUILD
institutions more
likely to show
increased research
self-efficacy,
research, and
scholarly
productivity?

Evaluation Questions

Potential Positive Findings

For BUILD and nonBUILD faculty at BUILD
and comparator
institutions:

BUILD faculty, compared to nonBUILD faculty at BUILD and
comparator institutions:

High self-efficacy as an
instructor in a biomedical
field (FAC-1)

What is the level of
instructor self-efficacy in
a biomedical field, and
how does it change over
time?

Are more likely to show high, or
increased, instructor self-efficacy
over time.

High self-efficacy as an
instructor to a diverse
group of biomedical
students
(FAC-2)

What is the level of
instructor self-efficacy to
a diverse group of
biomedical students, and
how does it change over
time?

Are more likely to show high, or
increased, instructor self-efficacy in
teaching to a diverse group of
biomedical students over time.

High self-efficacy as a
mentor to biomedical
research trainees
(FAC-3)

What is the level of selfefficacy as a mentor to
biomedical research
trainees, and how does it
change over time?

Are more likely to show high, or
increased, mentor self-efficacy over
time.

High self-efficacy as a
mentor to a diverse group
of biomedical research
trainees
(FAC-4)

What is the level of
mentor self-efficacy in
working with diverse
groups of biomedical
research trainees, and
how does it change over
time?

Are more likely to show high, or
increased, mentor self-efficacy to a
diverse group of biomedical students
over time.

Frequently mentors
students, post-docs,
and/or more junior faculty
on biomedical-related
issues
(FAC-5)

What is the frequency of
mentoring, and does it
change over time?

Are more likely to engage in
mentoring, and to report increases
over time.

High self-efficacy as an
independent biomedical
researcher
(FAC-6)

What is the level of selfefficacy as an
independent biomedical
researcher, and how
does it change over time?

Are more likely to show high, or
increased, self-efficacy as
independent biomedical researchers
over time.

High self-efficacy in the
ability to secure external
funding
(FAC-7)

What is the level of selfefficacy in securing
external funding, and how
does it over time?

Are more likely to show high, or
increased, self-efficacy in securing
external funding over time.

Engaged in activities to

What is the level of

Are more likely to show high, or

Priority CEC CWEP Dissemination Products - 6
2015 CWEP
Evaluation Question

CWE Hallmark (2019)

Evaluation Questions

Potential Positive Findings

For BUILD and nonBUILD faculty at BUILD
and comparator
institutions:

BUILD faculty, compared to nonBUILD faculty at BUILD and
comparator institutions:

secure research or
research training funding
(FAC-8)

activity to secure
research or research
training funding, and how
does it change over time?

increased, activity to secure
research or research training funding
over time.

Securing research or
research training funding
(FAC-9)

What is the level of
success in securing
external funding, and how
does it change over time?

Are more likely to show high, or
increased, success in securing
external funding.

Evidence of scholarly
productivity
(FAC-10)

What is the level of
scholarly productivity, and
how does it change over
time?

Are more likely to show increased
scholarly productivity over time.

Evidence of professional
recognition and service
(FAC-11)

What is the level of
professional recognition
and service, and how
does it change over time?
How robust are their
professional and
academic networks, and
how does it change over
time?

Are more likely to show increased
professional recognition and/or
service over time.

3, part 2) What is the
faculty experience of
BUILD activities and
how does that impact
program goals?

Strong academic and
professional networks
(FAC-12)

2, part 2) Are BUILD
faculty compared to
non-BUILD faculty
and faculty at nonBUILD institutions
more likely to show
increased research
self-efficacy,
research, and
scholarly
productivity?

Advancement to next
career stage
(FAC-13)

What promotions and/or
career advancements
have taken place over
time?

3, part 3) What is the
faculty experience of
BUILD activities and
how does that impact
program goals?

Advancement to
leadership positions in
biomedical research and
research training
(FAC-14)

What administrative or
leadership roles have
they held over time?

Are more likely to have held
administrative and/or leadership
roles over time.

Evidence of receiving
training in areas to foster
inclusive research training

What trainings to foster
inclusive research
training environments

Are more likely to have received
trainings to facilitate inclusive
research training environments over

Are more likely to have expanded
their professional and academic
networks over time (e.g. numbers of
coauthors on publications,
presenting at scientific conferences,
holding office in professional
organizations, providing service to
federal or other agencies).
Are more likely to have advanced in
their career (e.g., promotions,
tenure)

Priority CEC CWEP Dissemination Products - 7
2015 CWEP
Evaluation Question

CWE Hallmark (2019)

environments (FAC-15)
Strong self-efficacy to act
as a change agent to
enhance diversity in
biomedical research and
research training
environments
(FAC-16)
1, part 2) Are BUILD
faculty compared to
non-BUILD faculty
and faculty in nonBUILD institutions
more likely to show
increased mentor
self-efficacy,
mentoring, and
quality of mentoring?

Uses evidence-based
practices in teaching and
mentoring
(FAC-17)

Evaluation Questions

Potential Positive Findings

For BUILD and nonBUILD faculty at BUILD
and comparator
institutions:

BUILD faculty, compared to nonBUILD faculty at BUILD and
comparator institutions:

have individuals received
over time?
What is the level of selfefficacy to act as a
change agent to enhance
diversity in biomedical
research and training
environments, and how
does this change over
time?

time

How frequently are
evidence-based practices
used in teaching and
mentoring, and how does
that change over time?

Are more likely to show high, or
increased, activity self-efficacy to act
as a change agent to enhance
diversity over time.

Are more likely to use evidencebased practices in teaching and
mentoring, and to report increases
over time.

Priority CEC CWEP Dissemination Products - 8

BUILD Institutional
2015 CWEP
Evaluation
Question

CWE Hallmark (2019)

Evaluation Questions

2) How have BUILD
institutions
embraced
organizational
changes that
promote institutional
commitment to
diversity?

Commitment to efforts that
create, enhance and/or
maintain diversity and
inclusion at all levels of the
institution
(INST-1)

What efforts have been
made to create, enhance
and/or maintain diversity
and inclusion at all levels
of the institution changed
over time?

Demonstrate increased
commitments to institutional
efforts to create and
enhance diversity and
inclusion at all levels of the
institutions. If institutions
were initially at a high level,
they will have maintained
these efforts.

1) How have BUILD
and partner
institutions
developed the
capacity for
biomedical science
research training and
mentoring and in
what ways is this
sustainable?

Evidence of creating,
enhancing, and/or
maintaining diverse,
inclusive and culturally
appropriate research and
research training
environments
(INST-2)

What evidence is there of
creating, enhancing
and/or maintaining
diverse, inclusive and
culturally appropriate
research training
environments over time?

Show evidence that they
have created and enhanced
diverse, inclusive and
culturally appropriate
research training
environments. If institutions
were initially at a high level,
they will show evidence of
maintaining these
environments.

2, part 2) How have
BUILD institutions
embraced
organizational
changes that
promote institutional
commitment to
diversity?

Demonstrated institutional
commitment to creating,
enhancing and/or
maintaining the diversity of
the biomedical faculty on
campus by recruiting a
diverse pool of potential
applicants
(INST-3)

Has the institution taken
steps to recruit a diverse
pool of potential
applicants for biomedical
faculty?

Have taken steps to recruit
a diverse pool of potential
applicants for biomedical
faculty.

Implementation of
sustainable, institutionally
supported intra-institutional
activities to achieve
positive outcomes related
to biomedical research
capacity building and
faculty development
(INST-4)

What sustainable, intrainstitutional activities have
been implemented to
achieve positive
outcomes related to
biomedical research
capacity building and
faculty development over
time?

Have implemented
sustainable, intrainstitutional activities and to
demonstrate progress
toward achieving positive
outcomes related to
biomedical research
capacity building and faculty
development.

Enhanced inter-institutional
collaborations to achieve
positive outcomes related
to biomedical research,

How have new interinstitutional collaborations
been developed to
achieve positive

Have developed and/or
enhanced inter-institutional
collaborations to
demonstrate progress

In comparison to preBUILD baseline:

Potential Positive
Findings
When compared to the
pre-BUILD baseline,
BUILD institutions are
currently more likely to:

Priority CEC CWEP Dissemination Products - 9
2015 CWEP
Evaluation
Question

3) Does the number
and/or diversity of
students graduating
in biomedical
sciences in BUILD
institutions increase
over time?

CWE Hallmark (2019)

Evaluation Questions
In comparison to preBUILD baseline:

Potential Positive
Findings
When compared to the
pre-BUILD baseline,
BUILD institutions are
currently more likely to:

research training, and
faculty development
(INST-5)

outcomes related to
biomedical research,
research training, and
faculty development? If
applicable, how have preexisting inter-institutional
collaborations been
leveraged to achieve
positive outcomes related
to biomedical research,
research training, and
faculty development?

toward achieving positive
outcomes related to
biomedical research,
research training, and
faculty development.

Implementation of
sustainable, institutionally
supported activities to
achieve positive outcomes
related to biomedical
research training
(INST-6)

What sustainable,
institutionally supported
activities to achieve
positive outcomes related
to biomedical research
training have been
implemented over time?

Have implemented
sustainable, institutionally
supported activities and
made progress toward
achieving positive outcomes
related to biomedical
research training.

Enhancing or maintaining
the diversity of students,
e.g., those from nationally
underrepresented groups,
who pursue degrees in
biomedical fields
(INST-7)

What changes are there
in the numbers and
demographics of students
enrolled in biomedical
science majors over time?

Demonstrated institutional
commitment to efforts that
sustain the interest of
trainees from all
backgrounds pursuing
degrees in biomedical
fields that increase
persistence
(INST-8)

How have the numbers
and demographics of
students enrolled in
majors/ minors in
biomedical sciences
changed over time?

Have enhanced
demographic diversity
among students pursuing
degrees in biomedical fields
(or maintained if initially at
high level). These changes
will be greater than changes
seen at comparator
institutions.
Have enhanced the number
and diversity of students
graduating in biomedical
sciences (or maintained if
initially at a high level).

Employs evidence-based
approaches to establish
and attain goals for
graduation rates, time-todegree, and the ability to
transition to biomedical
graduate and professional
degree programs for

What evidence-based
approaches have been
employed to attain goals
for graduation rates, timeto-degree, and the ability
to transition to biomedical
graduate and professional
degree programs for

Will have employed
evidence-based approaches
to attain institutional goals
for graduation rates and
time-to-degree for students
from all backgrounds. In
addition, students from all
backgrounds who graduate

Priority CEC CWEP Dissemination Products - 10
2015 CWEP
Evaluation
Question

CWE Hallmark (2019)

Evaluation Questions
In comparison to preBUILD baseline:

Potential Positive
Findings
When compared to the
pre-BUILD baseline,
BUILD institutions are
currently more likely to:

students from all
backgrounds
(INST-9)

students from all
backgrounds?

will have improved abilities
to transition to biomedical
graduate and professional
degree programs.

1, part 2) How have
BUILD and partner
institutions
developed the
capacity for
biomedical science
research training and
mentoring and in
what ways is this
sustainable?

Demonstrated institutional
commitment to
implementing and
sustaining mentoring
practices that promote the
development of researchoriented students from all
backgrounds
(INST-10)

What commitments to
implement and sustain
mentoring practices that
promote the development
of research-oriented
students from all
backgrounds have been
implemented?

Demonstrate increased
commitment to implement
and sustain mentoring
practices that promote the
development of researchoriented students from all
backgrounds.

3, part 2) Does the
number and/or
diversity of students
graduating in
biomedical sciences
in BUILD institutions
increase over time?

Institutional infrastructure to
track regular reporting of
student demographics and
outcomes with respect to
biomedical fields
(INST-11)

How has the institutional
infrastructure to track
regular reporting of
student demographics
and outcomes with
respect to biomedical
fields been improved over
time?

Demonstrate a structured
way to track and regularly
report on student
demographics and
outcomes with respect to
biomedical fields (e.g., wellstaffed Institutional Records
office)

Note: Case studies will be the primary method for evaluating the BUILD institutional-level hallmarks. Data from
Institutional Records requests and the National Center for Science and Engineering Statistics (NCSES) can be
used to supplement these data. The CEC will use the case studies of BUILD institutions to identify areas of
institutional change that contributed to positive outcomes. The case studies will identify the challenges in
achieving institutional change as well as the conditions that made change possible. BUILD Case Study Data
cannot be anonymized, it will only be available for use by the CEC; it is not be accessible to consortium members
through data requests.