Attach 5_Reflections and Directions Interviews Focus Groups_CARRA

Reflections and
Directions

Interviews and Focus Groups with
CARRA Members and NCI Staff

September 2007

Prepared for:

Prepared by:

Office of Liaison Activities
National Cancer Institute

WESTAT
1650 Research Boulevard
Rockville, Maryland, 20850

Reflections and
Directions
Interviews and Focus Groups with
CARRA Members and NCI Staff

Authors:
Keith MacAllum
Izabella Zandberg
Barbara Queen
Colleen Ryan

September 2007

Prepared for:

Prepared by:

Office of Liaison Activities
National Cancer Institute

WESTAT
1650 Research Boulevard
Rockville, Maryland 20850

TABLE OF CONTENTS

Page
Executive Summary............................................................................................................

1

Background...................................................................................................................
CARRA Evaluation Methods .......................................................................................

1
1

Interviews With CARRA Members ....................................................................
Interviews With NCI Staff ..................................................................................
Focus Groups ......................................................................................................

2
2
3

Key Findings.................................................................................................................

3

Programmatic Structure ......................................................................................

3

Recruitment and Vetting ...........................................................................
Managing Expectations .............................................................................
Matching Skills to Activities.....................................................................
Personal Contact........................................................................................
Training ...................................................................................................

3
4
4
4
4

Marketing Messages and Mechanisms................................................................

4

Increasing Awareness................................................................................
Recognizing Value ....................................................................................
Increasing Usage .......................................................................................
Marketing Mechanisms .............................................................................

4
5
5
5

Reflections and Directions: Analysis of CARRA Member and NCI Staff
Interviews and NCI Staff Focus Groups...........................................................................

7

Background...................................................................................................................
CARRA Evaluation Goals and Methods ......................................................................

7
8

The 2007 CARRA Member Interviews and NCI Staff Interviews and Focus
Groups .......................................................................................................
Instruments..........................................................................................................
Samples and Recruitments ..................................................................................
Organization of Analytical Sections ...................................................................

9
9
10
11

Data Presentation and Findings ........................................................................................

13

Section I. Findings From CARRA Member Interviews ..............................................

13

Rationale for Joining CARRA and Initial Expectations .....................................

13

iii

TABLE OF CONTENTS—CONTINUED

Page
Four Key Rationales for Joining CARRA.................................................
Rationale 1: To Better Convey the Patient Perspective to NCI,
Researchers, Scientists, and Doctors .....................................
Rationale 2: To Interact and Have the Patient Perspective Heard
at a National Level.................................................................
Rationale 3: To Expand Their Own Knowledge Base ....................
Rationale 4: To Better Help Cancer Survivors and Their
Families in Their Communities .............................................

14

14
15
15
16

Expected Level of Activity........................................................................
Members’ Availability to Participate ........................................................

17
17

CARRA Experience, Involvement, and Managing Expectations .......................

18

The Subjective CARRA Experience and Perceived Level of
Involvement ...............................................................................
Managing Expectations .............................................................................

18
19

CARRA Members are Enthusiastic..................................................
CARRA Members May Have Unrealistic Expectations From the
Start........................................................................................
CARRA Members Are Presented With a Wide Range of CARRA
Activities................................................................................
CARRA Members Recall Event-Related Activities More Readily
Than Process-Type Activities................................................
CARRA Members May View CARRA as Generating Work for
Members ................................................................................

20

CARRA Members’ Communication With Constituency Groups .............
Providing Support to CARRA Members...................................................

23
24

CARRA Members Wish for Personal Interactions With OLA ........
Network Building and Member Profiles Could Help Personalize
CARRA..................................................................................

24

Building Awareness and Understanding of CARRA Within NCI ............

25

Screening, Skills, and Training ...........................................................................

26

Screening of New Members ......................................................................

27

CARRA Members Support Increased Screening .............................
Recognize Personal Strengths and Characteristics and Match
Them to Tasks........................................................................

27

iv

20
21
21
22

25

28

TABLE OF CONTENTS—CONTINUED

Page
Identifying Critical Skills ..........................................................................

29

Science Backgrounds Are Advantageous.........................................
There Is Consensus on the Need for Strong Interpersonal Skills.....
Members Bring Preexisting Professional Skills in Addition to
Advocacy ...............................................................................
Match Member Skills and Interests to CARRA Activities ..............

29
30
31
31

Training Needs and Options......................................................................

32

Customized Training ........................................................................
Mentoring Is Valued ........................................................................

34
35

Section II. Findings From NCI Staff Interviews..........................................................

35

Characteristics of the NCI Respondents..............................................................
Summary of Interview Results............................................................................

36
36

Value of Consumer Advocates..................................................................

36

Contributions of Advocates..............................................................
Concerns of Advocates ....................................................................

37
40

Awareness of the CARRA Program..........................................................

40

Experience With CARRA and Consumer Advocates .........................................
Qualities of CARRA Members That Are Most Valued ......................................

41
42

CARRA Members Are Vetted ..................................................................
CARRA Members Are Trained.................................................................
CARRA Members Have Superior Communication Skills ........................
CARRA Members Are Dedicated .............................................................

42
43
43
43

Skills and Knowledge of CARRA Members ......................................................

44

Excellent Communication Skills ...............................................................
Personal Experience With Cancer .............................................................
Understanding of Science..........................................................................

44
45
45

Terminology, Concepts ....................................................................
How Science is Done (IRB, Function of NCI).................................

45
45

v

TABLE OF CONTENTS—CONTINUED

Page
Training for CARRA Members ..........................................................................

46

Basic Research Processes and General Science Training .........................
Specialized Training..................................................................................
Train the Staff That Use CARRA Members .............................................
Suggestions for Communication Skills .....................................................

46
47
48
48

Barriers and Facilitators ......................................................................................

48

Barriers ...................................................................................................
Facilitators.................................................................................................

48
51

Marketing Mechanisms.......................................................................................

51

Respondent Recommendations for Marketing Strategies .........................
Reactions to OLA List of Marketing Strategies ........................................
Recommendations for CARRA.................................................................

52
54
55

Section III. Findings From NCI Staff Focus Group.....................................................

55

Marketing Messages............................................................................................

56

Participant Views on the Value CARRA Advocates Offer.......................
Additional Views on the Value CARRA Advocates Offer .......................
Challenges to the Perceived Value of CARRA Members.........................

56
60
62

Marketing Mechanisms.......................................................................................

64

Participant Views on Communication Approaches...................................
Additional Views Related to Communication Approaches.......................
Views Related to Communication Content and Resource Material ..........
Other Relevant Topics...............................................................................

64
67
68
69

Familiarity With CARRA ................................................................
Use of the Term “Consumer Advocate” ..........................................

70
71

Conclusions, Implications, and Recommendations ..............................................................

73

Program.........................................................................................................................

73

Recruitment.........................................................................................................
Vetting and Selectivity........................................................................................
Levels of Member Activity and Patterns of Program Use ..................................

74
74
76

vi

TABLE OF CONTENTS—CONTINUED

Page
Clarifying Roles and Activities During the Application Process ..............
Better Clarification on How Members Are Invited to Join Activities.......
More Explicit Discussion About Activity Levels During Training ..........
Ongoing Communication ..........................................................................

76
77
77
77

Matching Member Skills to Activities ................................................................
Personal Contact .................................................................................................
Training...............................................................................................................

78
79
79

Interpersonal Communication ...................................................................
A Basic Understanding of Science and Scientific Methods ......................
Understanding the NCI Structure and Culture ..........................................

80
81
81

Recommendations on Training ...........................................................................

82

Marketing Within NCI..................................................................................................

83

Increasing Awareness..........................................................................................
Recognizing Value ..............................................................................................

83
84

The Perceived Value of Consumer Advocates..........................................

84

Consumer Advocates Convey the Patient Perspective.....................
Advocates Are Attuned to Human Subjects Protection ...................
The Value of Advocates Extends Beyond Their Status as Cancer
Survivors................................................................................
Advocates Provide Value by Communicating Accurate and ProNCI Information to Stakeholders...........................................
Advocates Can Take on Roles the Government Cannot (Such as
Lobbying and Fund Rising) ...................................................

85
85

86

The Special Value of CARRA Members ..................................................

86

Increasing Usage .................................................................................................
Marketing Mechanisms.......................................................................................

87
89

Personal Outreach .....................................................................................
Website ...................................................................................................
Other Marketing Mechanisms ...................................................................

89
90
90

Recommendations for Marketing Mechanisms...................................................
Recommendations for Messages and Communication .......................................

91
92

vii

86
86

TABLE OF CONTENTS—CONTINUED

Page
LIST OF TABLES

Table
1

Study workplan for recruitment .......................................................................

10

2

Characteristics of NCI staff respondents, by CARRA user status ...................

36

LIST OF EXHIBITS
Exhibit
1

Potential values CARRA advocates bring to the work of NCI........................

57

2

Ways to communicate about the CARRA program.........................................

64

LIST OF APPENDICES
Appendix
A.

Letter of Invitation Sent From NCI Office of the Director..............................

A-1

B.

CARRA Member In-depth Interview Protocol................................................

B-1

C.

Informed Consent Form...................................................................................

C-1

D.

NCI Staff Member In-depth Interview Protocol (CARRA Member Users) ...

D-1

E.

NCI Staff Member In-depth Interview Protocol (CARRA Member Non-Users)

E-1

F.

Leadership Focus Group Moderator Guide .....................................................

F-1

G.

NCI Staff Focus Group Moderator Guide........................................................

G-1

H.

Focus Group Handouts ....................................................................................

H-1

I.

Thank You Letter Sent From Project Director.................................................

I-1

viii

EXECUTIVE SUMMARY
Background
The National Cancer Institute (NCI) has long recognized the importance of involving consumer
advocates, both individuals affected by cancer and those who care for them, in working with NCI staff on
program development and grant review activities. The decision to intentionally build relationships with
consumer and voluntary advocacy groups can be traced back to a 1993 National Cancer Advisory Board
survey on the needs of these groups. Respondents indicated overall satisfaction with NCI but
recommended that NCI strengthen its collaboration with consumer advocates to improve communication
and interaction between the Institute and these outside groups.
NCI’s Office of Liaison Activities (OLA) was established in 1996 to help strengthen NCI’s
communications and relationships with national advocacy and voluntary organizations that work with
consumer advocates, and scientific and professional societies concerned about cancer. OLA administers a
variety of formal and informal programs that ensure the voices of those affected by cancer are included in
NCI’s priorities, programs, and policies. OLA took the lead in designing and implementing the
Consumer Advocates in Research and Related Activities (CARRA) program.
The CARRA program provides a vehicle for consumer groups to be a part of the research and
decision-making process by exchanging their views and perspectives with NCI. The initiation of CARRA
represents the culmination of planning that began in early 2000. The first cohort of CARRA members
began their official term in September 2001. The program represents an important step in expanding and
systematizing access to skilled advocates who can contribute to the NCI research and communication
agenda. Building on existing, individual efforts to involve consumer advocates, CARRA is intended to
provide a timely and user-friendly process for involving advocates in the work of NCI.
In 2002, NCI contracted with Westat, a social science firm in Rockville, Maryland, to evaluate the
CARRA program. As part of its evaluation support, in 2006 Westat conducted surveys with CARRA
members and NCI staff designed to capture their perspectives on the experience of participating in and the
perceived value of the program. Analyses of the 2006 survey data provided valuable insights into the
workings of CARRA and the opinions of the two critical stakeholder groups. They also raised additional
questions for study and exposed opportunities for program improvement. The OLA tasked Westat with a
subsequent phase of research designed to gather rich qualitative data from members and staff, using
interviews and focus groups to help illuminate specific areas of interest and invite grounded feedback for
program improvement. This report describes the recent evaluation efforts.

CARRA Evaluation Methods
In the summer of 2007, Westat conducted a series of interviews with CARRA members and NCI
staff and a series of focus groups with NCI staff. Data were collected in a two-step process. Interviews,
comprising individual feedback to open-ended questions, were completed first. Systematic thematic
analyses were used to develop preliminary findings of individual responses and recommendations. These

1

findings were sorted and consolidated, and, in turn, served as the core dimensions to be explored in the
focus groups. Thus, feedback from the interviews served to construct the dimensions of the focus group
moderator guides. For example, suggestions from individual interviews on marketing strategies were
compiled into a set of potential strategies whose appeal and feasibility were then “tested” in focus group
settings.

Interviews With CARRA Members
Twelve interviews with CARRA members were conducted by telephone with a diverse group of
CARRA members located within eight states. Half were identified as High Active and half as Low
Active. Seven were male and five were female. While all were actively involved as consumer advocates,
most were retired from their professional careers or were working part-time. To gain the benefit of
perspective, the respondent group was skewed toward longer term members, with all but three enrolling
in CARRA in 2001. Data emerging from the CARRA member interviews have been organized along the
following three themes:
•

Rationale for joining CARRA and initial expectations;

•

CARRA experience, involvement, and managing expectations; and

•

Screening, skills, and training.

Interviews With NCI Staff
Twenty-four in-person interviews were conducted with NCI staff members. The 24 NCI
respondents were not quite evenly split between CARRA users (n=13) and CARRA non-users (n=11).
There were eight male and sixteen female respondents. Nineteen respondents were involved in scientific
work, primarily the administration of extramural research. Five were involved in programmatic work.
The interviews focused on the following topics:
•

Value of CARRA and consumer advocacy to NCI;

•

Awareness of the CARRA program and how respondents learned about it;

•

Experience with CARRA and consumer advocates;

•

Qualities of CARRA members valued the most;

•

Skills, knowledge, and/or experience that make CARRA members desired members of project
teams in respondents’ areas;

•

Type of training that should be provided to CARRA members to make them more valuable;

•

Barriers and facilitators to using CARRA; and

•

Ways to increase awareness and use of CARRA within NCI.

2

Focus Groups
Following the in-depth interviews with NCI staff, three focus groups were convened to explore in
more detail some of the interview findings. Participants in the focus group were:
•

NCI staff members who have used the CARRA program;

•

NCI staff members who have not used the program; and

•

Senior-level leadership staff at NCI.

Eight participants were recruited for each group with the goal of having six to eight participants in
each session. All but one of the recruits participated in their respective sessions.
The focus group discussions addressed two key topics – marketing messages and marketing
techniques. Marketing messages were defined as value statements describing the use and utility of
consumer advocates in the work of NCI. Marketing techniques were defined as communication
mechanisms for reaching NCI staff with messages concerning advocates in general and CARRA in
particular.
In addition to the two primary topics, participants were asked questions to assess their familiarity
with CARRA and were provided an opportunity to describe CARRA in their own words. Members of
NCI leadership group were asked to share their thoughts on the structure and culture of NCI and how
these might promote or hinder staff members’ use of CARRA advocates. The focus groups concluded
with a discussion of the term “consumer advocate.”

Key Findings
The summary of key findings is divided into two parts: Programmatic Structure and Marketing
Messages and Mechanisms. The former describes steps that NCI and CARRA could take to improve the
functioning of the program and its usefulness to NCI. The latter describes ways to improve awareness of
the CARRA program within NCI.

Programmatic Structure
•

CARRA members, NCI leadership, and NCI staff alike share a general belief in the value of
garnering advocates’ insights for the work of NCI.

•

Members described their participation in CARRA with enthusiasm. Even among Low Active
members, interest in continuing the program and dedication to it was high.

Recruitment and Vetting
•

CARRA members and NCI staff agree that selection and vetting of CARRA members must be
made more rigorous to maintain the perceived advantage of the CARRA program.

3

•

Respondents were in favor of a rigorous vetting process designed to screen potential members,
more for certain personal characteristics than for technical expertise.

Managing Expectations
•

Four suggestions for better managing member expectations were offered:
-

Better clarifying roles and activities during the application process;

-

More clearly describing how members are invited to join activities;

-

Initiating more explicit discussion about activity levels during training; and

-

Maintaining ongoing communication with CARRA members.

Matching Skills to Activities
•

CARRA members and NCI staff would welcome increased efforts to match individual
member’s personal qualities and professional skills to particular tasks.

•

Despite the wide range of activities in which CARRA members can and do participate, peer
reviews have assumed a powerful position of prominence, such that participation in other
activities is relegated to a lower status or otherwise discounted by many.

Personal Contact
•

The importance of developing and nurturing personal relationships was a continuing theme for
building rapport between OLA and NCI divisions, as a springboard to generate invitations for
members to participate in NCI activity, and as a way of managing member expectations.

Training
•

Both members and staff perceive current training as strong. Among those who suggested that
training enhancements were of value, three topics pervaded the discussion:
-

Interpersonal communication skills for members, such as negotiation;

-

Basic understanding of science and scientific methods; and

-

Understanding of NCI structure and culture.

Marketing Messages and Mechanisms
Increasing Awareness
•

Despite basic awareness among the majority of NCI respondents, a lack of clarity remains
concerning the CARRA program, its mission, its value, and its members.

4

Recognizing Value
•

Staff perceived value in a range of ways that consumer advocates can contribute to the work of
NCI. The most prominent of these was the advocates’ ability to convey the patient perspective.

•

Three dimensions of CARRA members distinguish them from other advocates: they are vetted,
trained, and possess superior communication skills.

•

NCI staff place a premium on the vetting and training of CARRA members, although some
indicated they had experience in which the quality of a CARRA member did not meet their
expectations.

Increasing Usage
•

Barriers to increased use tend not to be structural but, rather, programmatic and related to
inadequate clarity and understanding of CARRA that could be addressed through stronger
outreach and marketing.

•

Concrete examples modeling how others have effectively used well-trained CARRA members
with measurable results would facilitate increased use.

Marketing Mechanisms
•

A clear preference for in-person presentations and personal outreach emerged. Direct
communication by CARRA representatives at regularly scheduled NCI meetings is identified
as the most efficacious mechanism to market the presence and value of CARRA within NCI.

•

The use of traditional marketing products, such as brochures, posters, and branded giveaways,
was discouraged. Prepared materials are valued only as backup, supporting documentation
available via the website.

•

The benefits of a well-designed website as a communication mechanism were alluded to
repeatedly by respondents. The website is the first place most NCI and CARRA respondents
said that they turn for information about CARRA.

•

Access to practical, concrete resources that would help staff identify and recruit the appropriate
CARRA member was requested repeatedly.

5

6

REFLECTIONS AND DIRECTIONS
Analysis of CARRA Member
and NCI Staff Interviews and
NCI Staff Focus Groups
Background
The National Cancer Institute (NCI) has long recognized the importance of involving consumer
advocates, both individuals affected by cancer and those who care for them, in working with NCI staff on
program development and grant review activities. The decision to intentionally build relationships with
consumer and voluntary advocacy groups can be traced back to a 1993 National Cancer Advisory Board
survey on the needs of these groups. Respondents indicated overall satisfaction with NCI but
recommended that NCI strengthen its collaboration with consumer advocates to improve communication
and interaction between the Institute and these outside groups. Since that time, research and experience
has confirmed that the public viewpoint is an essential component in setting policies and priorities.
NCI’s Office of Liaison Activities (OLA) was established in 1996 to help strengthen NCI’s
communications and relationships with national advocacy and voluntary organizations that work with
consumer advocates, and scientific and professional societies concerned about cancer. OLA administers a
variety of formal and informal programs that ensure the voices of those affected by cancer are included in
NCI’s priorities, programs, and policies. OLA took the lead in designing and implementing the
Consumer Advocates in Research and Related Activities (CARRA) program.
The CARRA program provides a vehicle for consumer groups to be a part of the research and
decision-making process by exchanging their views and perspectives with NCI. The initiation of CARRA
represents the culmination of planning that began in early 2000.
OLA recognizes that consumer advocates have the capability and desire to disseminate health
messages and programs with credibility and clout to their members across the country. Hence, promoting
interaction to build and accomplish common agendas is essential. OLA serves as a catalyst to link these
groups with NCI programs and to ensure that their viewpoints are heard. This position is reflected in the
Consumer Advocates in Research and Related Activities: Program Overview (February 2001, p. 3):
Consumer advocates offer unique and important perspectives that enhance the NCI’s
efforts to reduce the Nation’s cancer burden. As lay representatives, consumer advocates
represent the concerns of those affected by cancer and bring a sense of urgency to the
research agenda. They put a human face on the Nation’s cancer burden and their presence

7

in the scientific dialogue about cancer stimulates a straightforward presentation of ideas
that often enhances scientific inquiry.
The first cohort of CARRA members began their official term in September 2001. The program
represents an important step in expanding and systematizing access to skilled advocates who can
contribute to the NCI research and communication agenda. Building on existing, individual efforts to
involve consumer advocates, CARRA is intended to provide a timely and user-friendly process for
involving advocates in the work of NCI. Specifically, the CARRA program offers the following:
•

A system for selecting and supporting a diverse pool of consumer advocates who can
contribute to NCI activities;

•

A straightforward process for matching NCI staff needs with appropriately skilled CARRA
members; and

•

A system for assuring that the needs of NCI staff and CARRA members are met.

Two broad goals have been articulated for the CARRA program by consensus of advocates and
NCI staff:
•

Increase opportunities for CARRA members to provide input to NCI’s research activities
involving scientific research and communication of scientific research; and

•

Help foster an organizational atmosphere that values the contributions of consumer advocates
through the involvement of CARRA members in NCI activities.

CARRA Evaluation Goals and Methods
In 2002, NCI contracted with Westat, a social science firm in Rockville, Maryland, to evaluate the
CARRA program. As part of its evaluation support, in 2006 Westat conducted surveys with CARRA
members and NCI staff designed to capture their perspectives on the experience of participating in and the
perceived value of the program. This initial phase of work included the CARRA Member Survey, the NCI
1
staff Survey and Post-Activity Surveys administered to both groups.

1

Additional activities include the development of a logic model and evaluation design for examining the program, post-activity surveys of NCI
staff and CARRA members who have participated in specific activities involving consumers, and an NCI staff attitudes and beliefs survey, the
results of which appear in the Westat report Assessment of NCI Staff Member Attitudes and Behaviors with Regard to the CARRA Program
(2006).

8

The 2007 CARRA Member Interviews and NCI Staff
Interviews and Focus Groups
Analyses of the survey data, documented in a series of analytical reports,2 provided valuable
insights into the workings of CARRA and the opinions of the two critical stakeholder groups. They also
raised additional questions for study and exposed opportunities for program improvement. The OLA
tasked Westat with a subsequent phase of research designed to gather rich qualitative data from members
and staff, using interviews and focus groups to help illuminate specific areas of interest and invite
grounded feedback for program improvement.
With an eye toward generating practical
recommendations, the lines of questioning focused on 1) the development and efficacy of marketing
mechanisms and materials that promote visibility of and participation in CARRA, 2) the value statement
messages OLA needs to convey to NCI staff, 3) support mechanisms and materials that will help manage
member expectations, and 4) identifying specific training needs of members that will enhance their
productivity and interaction with NCI staff.
This phase of work operated on a fast-track basis. The initial workplan was completed by Westat
and submitted to OLA on June 1, 2007. The workplan and draft instruments were reviewed and approved
by the NCI’s Institutional Review Board (IRB) on June 12. The final version of the workplan was
approved June 18. Data collection took place between June 27 and August 9. Data were analyzed on a
rolling basis, beginning July 9, and the findings were written up between August 10 and August 31. The
opportunity to probe deeply on information gathered through survey research represents the culminating
phase of CARRA data collection.
Data were collected in a two-step process. Interviews, comprising individual feedback to openended questions, were completed first. Systematic thematic analyses were used to develop preliminary
findings of individual responses and recommendations. These findings were sorted and consolidated,
and, in turn, served as the core dimensions to be explored in the focus groups. Thus, feedback from the
interviews served to construct the dimensions of the focus group moderator guides. For example,
suggestions from individual interviews on marketing strategies were compiled into a set of potential
strategies whose appeal and feasibility were then “tested” in focus group settings.
Due to wide dispersion in geographical location of respondents, the CARRA member interviews
were conducted via telephone. However, since NCI is located just a short distance from Westat
headquarters, the NCI staff interviews were conducted in person.

Instruments
Interview protocols were based on the research questions developed by OLA as part of the Office
of Science Planning and Assessment (OSPA) evaluation matrix and were augmented by questions that
emerged from Westat’s analysis of existing survey data. Copies of all instruments can be found in
Appendices B through G.

2

See the Initial Report on the Results of Post-Activity Surveys of CARRA Members and NCI Staff (August 2006); Second Report on the Results of
Post-Activity Surveys of CARRA Members and NCI Staff (September 2006); Assessment of NCI Staff Member Attitudes and Behaviors with
Regard to the CARRA Program (November 2006); Third Report on the Results of Post-Activity Surveys of CARRA Members and NCI Staff
(January 2007); and Assessment of CARRA Member Attitudes and Behaviors with Regard to the CARRA Program (March 2007).

9

Data from interviews were compiled and used to inform construction of moderator guides and
focus group materials. Specifically, two handouts were constructed that summarized findings from
interviews with NCI staff on perceived value of advocates and efficacy of suggested marketing strategies.
The moderator used these tools to gauge participant reaction.
Focus group discussions focused primarily on two topics:
1. Value statements, or ways of describing the potential value and benefits of including CARRA
members in one’s NCI work. These statements represent potential communication messages
intended to make a persuasive case to NCI audiences for CARRA members.
2. Communication approaches or, more specifically, the best ways to go about ensuring that NCI
staff members are aware of CARRA and have an accurate understanding of how the program
can contribute to their work.
In addition to discussing those two core topics, participants were asked questions to gauge their
familiarity with CARRA and were given an opportunity to describe CARRA in their own words.
Members of the NCI leadership group were asked to share their thoughts on the culture of NCI and how it
may promote or inhibit staff members’ use of CARRA advocates. The discussions concluded with
questions on the connotations of the term “consumer advocate,” which is the shorthand used for CARRA
advocates. The moderator guides for staff members (both CARRA users and non-users) and the guide for
managers (leadership) are included in Appendices F and G.

Samples and Recruitment
The study workplan sought to recruit 12 CARRA members and 18 NCI staff for interviews and 24
NCI staff for focus groups, as depicted in Table 1. Survey findings suggested that the level of activity for
CARRA members had a strong effect on member experience, so a mix of members with varying levels of
activity was sought. With respect to NCI staff, the survey work had not collected data from non-users of
the CARRA program; therefore, non-users were deliberately sought. It was also determined that
perspectives of both NCI leadership and staff were valuable when exploring strategies for marketing
within NCI, so a mix of interviewees along this dimension was sought as well.
Table 1.—Study workplan for recruitment

Interviews

Focus groups

CARRA members
High Active
Low Active
6 via telephone
6 via telephone

NCI staff
Participating
9 in-person
7 at scientist level
2 at line manager level
2 groups of 6 to 8 (onsite)
1 scientist at level
1 manager at level

Nonparticipating
9 in-person
7 at scientist level
2 at line manager level
1 group of 6 to 8 (onsite)
1 at scientist level

For each target group of interview respondents and focus group participants, a pre-identified pool
of eligibles was constructed by the OLA and submitted to Westat. Westat recruited from these pools of
eligibles rather than from the universe of CARRA members or NCI staff. As a result, the potential of
selection bias cannot be ignored. OLA provided Westat with contact information for these eligible
interviewees and focus group participants.

10

To expedite recruitment and manage interview scheduling logistics, all eligible NCI staff
interviewees and focus group participants received an e-mail invitation announcing their eligibility to
participate (see Appendix A). Text for the e-mail invitation was prepared in cooperation with OLA and
was sent to selected NCI staff under the auspices of the NCI Office of the Director. A Westat staff
member contacted eligible candidates via telephone for specific recruitment to either interview or focus
group.
Twelve CARRA members were recruited and interviewed from a pool of 24 eligibles identified by
OLA. For CARRA members, OLA divided the pool into those classified as High Active and Low Active
based on OLA records of recent CARRA activity. Six of the CARRA member interviewees were
recruited from each of these categories.
Twenty-four NCI staff were recruited and interviewed from a pool of 38 eligibles identified by
OLA. For NCI staff, OLA divided the pool into those classified as users and non-users of the CARRA
program based on recent program records. Thirteen NCI staff identified as CARRA users and 11
identified as non-users were recruited. While the workplan called for interviewing 18 NCI staff, an
additional six were recruited in anticipation of attrition; however, all 24 staff were interviewed.
Out of 41 NCI staff deemed eligible by OLA to participate in the three focus groups, 24 were
recruited. Of these, eight had been identified as CARRA users, eight as non-users, and eight as holding
leadership/managerial positions within NCI.
To ensure adequate participation in each focus group (n=6–8 each) and to handle scheduling
logistics, a Westat staff member followed up on the e-mail invitation by phoning potential participants to
recruit them into the appropriate group. Notably, participants spontaneously mentioned that the initial
letter provoked a heightened level of interest in the project, but the follow-up phone call was instrumental
in garnering their participation. Eight participants were recruited for each group.
Interviews and focus groups were audiotaped in support of analyses. Confidentiality was promised
to all participants, so titles and affiliations of respondents are not provided in this report. Participants did
vary along dimensions of age, gender, educational attainment, professional background, and
organizational affiliation. Within NCI specifically, a wide variety of programs and roles were
represented. Among the users and non-users were individuals working in communications, therapeutics
development, clinical investigations, writing and editing, scientific review, and health education.
Likewise, among the NCI leadership group, there was variety in the roles and NCI organizations
represented. Participant affiliations included the Office of Communications and Education, the Division
of Cancer Control and Population Sciences, the Division of Cancer Prevention, the Division of Cancer
Biology, and the Office of the Director.

Organization of Analytical Sections
The remainder of this report is divided into two analytical chapters. In the first, Data Presentation
and Findings, the specific findings that emerged from the interviews and focus groups are organized into
three corresponding sections, one each for CARRA interviews, NCI staff interviews, and NCI focus
groups. These sections provide OLA with summarized responses to questions in the core instruments
organized by major themes. Direct quotations from respondents are included in the write-up to provide
some of the flavor of their responses. As stated above, findings from the interviews served as the
foundational material for the final focus group data collection.

11

The concluding chapter offers a synthesis of key findings, cross-cutting themes, and
recommendations. These are presented to provide the OLA with interpretive analyses on which to base
programmatic decisions and development of marketing strategies.
For convenience, the key findings have been extracted and presented in a concise fashion as a
forward to this report. These have been classified as relating to program or marketing.

12

DATA PRESENTATION
AND FINDINGS
Section I.
Findings From CARRA Member Interviews
Twelve interviews with CARRA members were conducted by telephone with a diverse group of
CARRA members located within eight states. Half were identified as High Active and half as Low
Active. Seven were male and five were female. While all were actively involved as consumer advocates,
most were retired from their professional careers or were working part-time. To gain the benefit of
perspective, the respondent group was skewed toward longer term members, with all but three enrolling
in CARRA in 2001. Data emerging from the CARRA member interviews have been organized along the
following three themes:
•

Rationale for joining CARRA and initial expectations;

•

CARRA experience, involvement, and managing expectations; and

•

Screening, skills, and training.

Rationale for Joining CARRA and Initial Expectations
As OLA considers new strategies with respect to recruiting and informing new members about the
CARRA program, it is important to better understand how members first found out about the program.
Interestingly, of all the questions in the interview protocol, respondents had the hardest time answering
this one. The most common reply was, “That’s a really good question. I’m not really sure.” Only two of
the 12 respondents could confidently state how they learned about CARRA. The remaining stated they
could not recall or were not sure, or they listed a number of possibilities.
I don’t recall. I’m thinking it was over the internet but I'm really not sure.
I think I received a mailing. I think they found me?
It might have been an e-mail, or maybe through a colleague.
While this finding might simply reflect a lack of recall, respondents’ ability to easily retrieve other
information suggests an alternative interpretation. The number of respondents who offered multiple
avenues suggests that the CARRA program may enjoy good penetration into the consumer advocacy
communities, permeating various networks and organizations. As discussed below, respondents question
whether CARRA is as well known within NCI as it is within the advocacy community.

13

Four Key Rationales for Joining CARRA
With respect to members’ original interest in joining CARRA, responses converged around four
themes: conveying the patient perspective to the research community, interacting at the national level,
expanding their own knowledge base, and directly helping others. Because all of the respondents were
actively involved as advocates in their local and regional communities, the interview protocol sought to
tease out the value-added aspects of CARRA that differentiate the program from other advocacy
opportunities. Many respondents commented that while they had similar opportunities through other
agencies and organizations, the distinguishing characteristics of CARRA were its well-thought-out and
intentionally planned structure along with its direct affiliation with NCI. Moreover, the opportunity to be
trained on and participate in peer reviews made the CARRA program stand out as a unique opportunity.

Rationale 1: To Better Convey the Patient Perspective to NCI, Researchers, Scientists, and
Doctors
The concept of conveying the patient perspective pervaded the language of CARRA members.
However, the audience to whom the patient perspective is conveyed is multidimensional. It ranges from
scientists involved in the highest levels of research down to local medical doctors. The unifying theme
was that members want to put a face on cancer and ensure the “voice of the consumer” is heard in settings
that might be somewhat removed from the very human aspect of this disease.
I wanted to remind researchers that patients are not subjects, but people.
I wanted to get involved in anything that would do a better job of getting the patient’s
perspective.
More importantly, I joined so that patients would have a voice.
Because of interest in patient protection. CARRA seemed to offer an opportunity to get involved
in patient protection. Patients are truly outside of the loop in navigating the health care system.
One of the things I liked when I first found out about the CARRA was I would get a chance to be a
participant, as a consumer advocate, to discuss with researchers how the patients are looked at
and how the studies will impact the patients. One thing I’ve found working in health care for the
past 20 years is that when you work in the health care circle you tend to live in a bubble, and you
have a different view on how health care affects people. You have a sense of grandiosity or
arrogance that we’re on the top of the hill and everyone’s going to come up and see us and we’ll
fix everything. In the CARRA program, I’ve been able, again, to see that not everyone
understands what the health care system is all about. They don’t know how the research, or
sometimes the health care organization itself is unaware how the system itself affects the
population it’s supposed to treat. CARRA gave me the opportunity to give a voice to say to these
organizations, to say, “Hey, look, we’re here to serve the consumer, they’re not here to serve us.”
More specifically, CARRA appeared to offer opportunities for advocates to provide input into the
peer review process.
It was innovative, no one had done it up to that point. I wasn’t aware of any consumer advocate or
research related activity or training going on to use the consumers and people who were cancer
survivors on panels, so to be able to represent other cancer patients was appealing to me.

14

The thing that really hooked me was the fact that I would have a chance to interact with the
scientific community, and it would be a planned interaction, and the fact that CARRA was willing
to train me to go out there.
What I really liked was the opportunity to be involved in the peer review process. To sit down with
world renowned scientists, it was an honor to be in the same room. You’re a true part of the team.
In some earlier groups that involved advocates, which were disingenuous, it was not to invite their
opinions but to mold them. I was skeptical going in [to CARRA]. My expectations weren’t high,
but I was looking for the opportunity to be in the review process. That was very exciting to me.

Rationale 2: To Interact and Have the Patient Perspective Heard at a National Level
Virtually all CARRA members are active in local and regional organizations, but few had the
opportunity to participate at the national level. Members valued this access and welcomed the unique
opportunity to help influence planning and policy.
I joined CARRA to be active at the national level. To be active at the national level is even more
important that to be active at the local level. It’s where everything happens, ’cause it’s a trickle
down effect. Although being active at the local level gets me in touch with people directly, being
involved with CARRA has gotten me involved in politics and policy, which I had never been
involved with before. (This respondent now teaches a college-level course on the Politics of
Cancer.)
I thought it was cool to be on a national level, looking at stuff for scientific cancer research.
CARRA enables me to be involved at both ends where things happen. I’m active in grassroots
activity at the local level where we get patients involved and national policy and politics. If I
hadn’t have had the CARRA involvement I wouldn’t have known what it takes at the national level
and the politics and policy level to be an effective communicator, for example with the MDs. The
biggest thing for me is to be able to go in and talk to the oncologists, and get to know them, and
explain to them why it’s important for them to be able to communicate better with their patients,
and offer more programs, have a cancer navigator in their hospitals. And that all came through
the CARRA program.
I’m actively involved in educational programs and coalitions within my state and community. I
joined CARRA because I saw how this outreach could be brought in at the national level.
It is vitally important to provide awareness to Congress to guide resource allocations.
Actually, mostly advocating for more dollars for research and getting treatments to patients more
quickly and advocating for that.

Rationale 3: To Expand Their Own Knowledge Base
Virtually all respondents were able to provide a long list of examples documenting their level of
engagement as advocates. Some are in essentially full-time positions within advocacy organizations,
while others dedicate the vast proportion of their time on a voluntary basis to advocacy work. Members
saw CARRA as an opportunity to strengthen their abilities in these related capacities. The knowledge

15

they have gained was said to have strengthened their ability to communicate with their own
constituencies. Relevant topics and activities ranged from simple dissemination of resources to learning
about the funding process to helping patients understand the value and goals of clinical trials.
Number one is to learn, become aware, and educate myself about the vast arena of cancer and
seeing where I could fit in and help promote that education.
To be honest, to just stay informed on what’s going on within the cancer community so that I could
put information on my newsletter.
What attracted me was the opportunity to learn about grant funding. I wanted to learn more about
how things got funded, how the money was spent.
I hope to learn more about the science process and the funding process. To learn more about
cutting edge research—especially with respect to leukemia because of my own personal
involvement.
“Rubbing elbows” with scientists and clinicians, who are really at the cutting edge of research in
this disease, is another huge interest of mine.
I’ve had an opportunity to do site visits, I’ve had an opportunity... to take lead training, what I call
“Cancer Science 101,” had one training on quality of care, one on clinical trials, and that gave me
some medical background and vocabulary that helped to make my job a little bit easier.
NCI does things like clinical trials and I help promote those things in my newsletter. I also get
information from some of the NCI people that I actually publish in the newsletter.
So I can help people understand what a trial is, why it’s important...it helps to make the clinical
trial more relevant to people, and this way we’re dispelling the myth that with clinical trials,
you’re not going to be a guinea pig.

Rationale 4: To Better Help Cancer Survivors and Their Families in Their Communities
Advocates with personal experience with cancer conveyed their urgent need to find information
and a support system to help them and their families find some semblance of clarity and sense of
direction. They described how they learned to cope through acquiring information and support and how
surprisingly difficult it was at times to obtain reliable, practical information. One respondent described
how her interest in helping others emerged from trying to help her daughter who was diagnosed with
brain cancer. Their cancer experiences sensitized them to the fact that others dealing with cancer likely
have similar needs. CARRA, they said, provides an avenue through which the members can help meet
others’ needs in this regard.
Here we were, very well educated, you know, our daughter was a doctor. We had lots of contacts,
and yet we were completely lost when our daughter was diagnosed with a brain tumor. So we
understood seeing it from that side and I wanted to help others. I felt desperate for hope and, I
thought, “Well, at least I can help others.”
I wanted to just advocate for people who had very little voice. I think I did it out of desperation to
help the next person and to just maybe ease the way for them.

16

I’m interested in helping guys work through the initial shock of becoming diagnosed and then
helping them think through what they’re dealing with and kind of “navigate them” through these
decision things. You’ve really felt like you’ve helped somebody out.
It looked like an opportunity to serve and to perhaps make a difference in some small way. The
CARRA program provided opportunity to provide input, consult, and to stay abreast of what
exactly is going on, so that when you give people information, you’re not misinforming.
Many of the people I work with, and I'm an old reading teacher, either were illiterate or had poor
low reading skills, and I wanted to [share] materials but without having to do a reading lesson to
get people to refer back to them, and see, there’s a lot of literature with loads of information that I
can present to a group... but if they can’t understand the print that’s there, it doesn’t mean
anything.
Some joined because they saw a wide range of opportunities in which they could become involved.
I thought that I would be getting more literature for the patient to read, so the patient would
understand. I was thinking that the NCI would put out information, and I thought my role would be
to take the information and put it in layman’s terms so the average person would be able to
understand it, and I also thought I’d be on a peer review grant panels as a patient advocate and in
the time since I’ve joined I’ve only been called to do one, and that was only recently.

Expected Level of Activity
As reflected in CARRA member surveys, these interview respondents indicated that they expected
to participate in two to four activities per year. “Two to three” was the most common response. The
highest number of anticipated CARRA activities per year was “four or five.” Survey responses
uncovered some concerns that actual levels of activity did not match expected levels.
Maybe 4 or 5 times a year. Particularly after the Intensive Training… It would have been good to
have served immediately after that.
While some members expected to be involved in a range of activities and others were not quite sure
what to expect, responses nevertheless emphasized opportunities to participate in peer reviews.
Scientific reviews of applications for cancer grants. I wanted to make sure that the people applying
were doing the right thing and protecting the patient in their research.
Peer reviews of grants and a fair amount of a chance to stay engaged with things. Some of it was a
bit amorphous from the beginning in terms of the potential of consulting of NCI whenever they
needed patient advocate input.

Members’ Availability to Participate
Most respondents were able to recall a time when they were unable to participate in a CARRA
activity. Some respondents cited more than one of these situations. But in every case discussed, the
member had a preexisting commitment or scheduling conflict. These findings are also echoed in CARRA
member survey findings.

17

Interviews, however, revealed some members’ expressed fear that declining an invitation to
participate (even if due to scheduling conflict or a health issue) might be perceived by NCI staff as a lack
of interest on the member’s part. They worried that declining an invitation might result in their being
blackballed and, therefore, not called again. While not an issue that is deeply or widely believed, this
concern is one that OLA may want to address.
Said one respondent in a questioning tone, “I doubt that this is really the case,” half expecting the
interviewer to relieve him of this concern completely. Acknowledging that fellow CARRA members
might suspect it to be the case, another respondent stated, “I do know for a fact that if you’re not available
and you turn a PI down, you’re not put on a blacklist; you know, that doesn’t keep others from calling
you.”

CARRA Experience, Involvement, and Managing
Expectations
The Subjective CARRA Experience and Perceived Level of Involvement
As they had in surveys, virtually all respondents used the interviews to reconfirm that they rate
their CARRA experience very highly.
To obtain perspectives of both Low Active and High Active members, a stratified sample of
respondents was deliberately chosen. Indeed, their responses reflected a range of CARRA activity levels.
Yet, analyses revealed no discernible difference between High Active and Low Active respondents in
terms of how they felt about their CARRA experiences.
As survey findings had also indicated, there was disappointment among members around the issue
of level of involvement. Nonetheless, dissatisfaction with low involvement did not appear to translate into
hostility or resentment with the program; instead, it appears to simply reflect a remarkably high interest
level. In general, the members we interviewed were not overly dissatisfied or upset about their level of
involvement. They appeared to understand the constraints of the program.
I’m satisfied. But I could be more satisfied.
I could be more satisfied and would enjoy doing it more, but I’m not sure that’s CARRA’s fault
‘cause I know they have to ask everyone be involved.
I’d like to be a little more involved. I could be more satisfied. But I understand the nature of it. I
value the participation greatly and enjoy it, but my level of involvement has not been a concern.
Not at all.
Both Low Active members and High Active members alike commented on their desire to be more
involved. This desire for more involvement demonstrates the extraordinarily high levels of enthusiasm
and interest CARRA members bring to the program. Interest and dedication to CARRA is high among all
members, as the comments of one Low Active member demonstrate:
Honestly, I think this is a great program. Its biggest value is opening each others’ eyes about how
the process works. You get to see the other side of the street. I was so naive about how research
worked, the funding process and now I can explain that better to others. We need to figure out how
to facilitate that interchange on a regular basis. Some scientists never leave the lab and CARRA

18

members can’t get into the lab. If there was more… scientist meetings, so we simply meet, so there
is more interaction.
At least two Low Active members reported that they had been more active when first on board, and
commented that their level of activity fluctuated over time. Others considered themselves active on a
daily basis, as one respondent indicated, “I’m a CARRA advocate every day.” And all considered
themselves to be fully engaged CARRA members. Therefore, this report does not distinguish
respondents by level of activity.

Managing Expectations
One of the key findings from the CARRA member surveys was the noticeable disconnect between
member expectations with respect to involvement compared to their actual involvement. That is, a
substantial proportion of members expected to be more involved in CARRA activities than they were. A
major goal of the interviews was to determine what accounted for this disconnect, and specifically to
explore whether the OLA might be contributing to unrealistic expectations in the way they communicate
with members.
It is noted that nine of the 12 respondents joined CARRA in 2001 when it was first launched. They
recognized that the program was new and, in turn, that perhaps even the program designers were not sure
what to expect early on.
I’m sure that when CARRA put this [program] out, they might not have had a really good idea
because they were developing it on both ends.
CARRA was built “on the run.” I think it deserves a real hard look, with a hand-picked consumer
advocate component, to make it better.
Other responses reflect a lack of clarity with what to expect as a new CARRA member. It is here
that the prominence of peer review as the quintessential CARRA activity first became apparent during
interviews.
Because I hadn’t had a lot of experience with how government programs are set up, I didn’t know
what to expect, this isn’t my normal thing.
I think I could be used more, I'm just not sure how to answer that because I don’t know what the
level of involvement is for other people, so I don’t know if they can use me more.
Somewhat satisfied. I first understood I’d be doing peer reviews, site visits. My original
expectations were high.

Several respondents raised the concern that the selection process itself raises expectations.
No, my level of participation has not met expectations. When they make a big deal about an
application process, and when you’re picked, you think, “Oh, well then they’re probably going to
use me,” but I get the sense that there is probably some people who are being used and the greater
majority of us are not being used.

19

Some respondents recalled that during their training, there were a few opportunities to discuss
expectations, but afterwards there were none. Some of these respondents stated that members themselves
share some responsibility for failing to raise questions about what to expect from the program.
Reflecting on the interviews as a whole, several complex factors appear to be at play. While only
some of them are within the complete control of the OLA, recognizing the range of forces at play may
help OLA manage member expectations in the future. These forces include the fact that members:
•

Are enthusiastic;

•

Acknowledge that their own expectations may be unrealistic;

•

Are presented with a wide range of potential CARRA activities;

•

Tend to recall event-related activities, such as site visits, more readily than process-type
activities (such as reviewing communication materials); and

•

May forget that NCI, not CARRA, generates assignments.

CARRA Members Are Enthusiastic
Members appear to have an unlimited amount of enthusiasm. A striking dimension of the CARRA
member interviews was the level of commitment and interest each respondent had in the face of
sometimes overwhelming personal and medical challenges.
I'm one of those people who give everything, if I'm working on it, I give over 100%.
When you tell these volunteers that they’re going to be able to do something, boy do they want to
know when! That’s the whole problem with advocacy—they do, do, DO want to be used.
[I’m] not satisfied. I always tried to see what more I could do and at the same time be effective and
have impact.
I don’t think it was ever communicated to me how often we would serve by the office, it was just
that I was eager to do whatever I could.
I expected more than I’ve been given the opportunity to do, and at that point in my life, I could
have certainly devoted as much time as they wanted, so it was kind of disappointing to not be
called more often.
When I got my first packet [to do a peer review], I wanted to do as many as I could without getting
fired from my full-time job. I wanted to learn as much as I could. I was fascinated by it all.

CARRA Members May Have Unrealistic Expectations From the Start
As active, passionate, and successful consumer advocates in their respective local communities
who are fueled with the sense of being selected to join an elite group, CARRA members confess to
possibly suffering from an inflated sense of self-importance, leading to unrealistic expectations. These
expectations were said to be further increased with the special training offered by CARRA.

20

I went into it thinking, “I’m the most important person that ever lived. They cannot have a
program without ME!” Then I got there, and I realized, “Oh, yes they can.” This respondent goes
on to explain how group training tempered his expectations. [Seeing] all the other capable
members, I’m not the only person with experience.
The training is great. You learn how important you can be to the process. But that feeds the
monster. You feel you need to be involved in everything.
Part of the problem is that you finish this intensive training that was extremely helpful and, you get
this passion from being at the training and, of course, all the knowledge you acquire.
I felt I became immersed in this and that, I could provide insights.
These first two factors—enthusiasm and inflated expectations—appear to be functions of the
CARRA members themselves and apparently are characteristics of the type of person who is drawn to a
program like CARRA in the first place.

CARRA Members Are Presented With a Wide Range of CARRA Activities
In an effort to project a wide range of potential opportunities for involvement, OLA shares a
laundry list of activities (see the attachment to Appendix B) comprising of at least a dozen ways in which
members may be engaged. Presented with a long list of ways they can be involved, these members
indicated that they perceive CARRA as being in the middle of some very important national activities.
Many conclude they will be involved in all or at least a large number of them. More often, for obvious
reasons, members fixate on the most glamorous ones (i.e., peer reviews and site visits). Members reason
that since there is so much going on, they will be called quite often.
You see this long list of different kinds of things that one could be drawn into and you consider
your own resume and figure, “Well, wow, I could probably play a role here or there.”
From my initial call, to the materials I received, the impression I got was there was this huge
need.
I think that things were said that led me to think that way [that there would be a lot of activity],
otherwise I wouldn’t have thought that up in my mind.
I think it [communication about activities] was very broad in nature and what would’ve been
helpful was to … well, they did have a menu, but the menu was very broad, and I think what it
comes down to is specificity and relevance to the individual who wants to be there, and I think
most of this was done by email. There really wasn’t a dialogue like this for individuals to have
that personal connection.
Need to explain more to people up front that this isn’t going to be a full time job.

CARRA Members Recall Event-Related Activities More Readily Than Process-Type
Activities
Members appear to have a harder time recalling process activities in contrast to event activities.
Thus, when asked to cite the activities in which they have participated, members tend to comment on peer

21

review panels and site visits. They tend not to comment on reviewing educational materials or
disseminating information from NCI to their respective constituency groups. Despite their active
involvement in the latter type of activities, they tend not to recount them when asked to reflect on their
involvement and may be underreporting (to OLA and themselves) this kind of work.
I was anticipating more scientific research, peer review.
Some people think CARRA is about going on site visits and reviewing protocols. And to me,
CARRA is a lot more. It’s ensuring that people are aware of resources. To me, a CARRA
member is a liaison between the researcher and the community, because we become more aware
of different resources. We become more aware of how this process works. For me, I came away
from the training realizing that there are a lot of things CARRA members are involved in.
I think the interpretations or the expectations of other CARRA members, this is what people get a
little confused about.
At [one meeting] I picked up a minor focus among a minority of people as to they were almost
insulted why they haven’t been invited to a Peer Review, and they’d been there a while, and I was
kind of like, “get over it,” we’re not there to be invited to Peer Review; it isn’t about us, it’s
about them [NCI] needing us.

CARRA Members May View CARRA as Generating Work for Members
Some respondents suggested that not all CARRA members fully appreciate that CARRA
assignments are generated within NCI and not from within the OLA office. Accordingly, members
sometimes believe they should be used to the level of their availability, overlooking that they can only be
used in relation to actual need. While no respondents said that they personally have this lack of
understanding, some members raised the possibility that others might.
I understand this is on an as-needed basis. I don’t see what they can do unless there are more
activities. What is the point to reach out [to us] if there is no activity? It’s as needed. They need
to communicate that clearly. This is a privilege to participate, not a right or an expectation.
I don’t see that as a big deal because from the beginning I understood it would be “As required”
or “As needed” and then if the need wasn’t there, then okey dokie, I'm not going to take that
personally.
While some of the member comments above suggest that expectations were brought into alignment
as a result of the training, at least one member pointed out how the training itself can serve to elevate
member expectations. At the same time, some respondents indicated that they did not go through training
until well after they joined, allowing preconceived notions to solidify. Taken together, these observations
underscore the need to have clear honest communication with members during the recruitment process,
during training, and during their tenure as members.
The website provides basic information. It’s helpful. But it doesn’t explain the process. Only
training did that. I don’t think there is anything that can prepare people or explain expectations
better than real time, actual training.
Training was fantastic. I had been a member a couple of years before I had my training.

22

CARRA Members’ Communication With Constituency Groups
As anticipated, the interviews helped to draw out an important distinction with respect to how
CARRA activities are conceptualized. Most respondents tended to conceive of “activities” as a high
profile events, namely a peer review or site visit. While some respondents listed “development and
review of educational materials” or “communication and dissemination” as a rationale for joining, these
are ongoing processes, sometimes only tangentially related to CARRA. As mentioned above, when
members are asked to recount their activities, they tend to recall events more readily than processes.
Yet, members’ communicating information gained through CARRA with their respective
constituency groups is considered by OLA to be a major component of the CARRA program. That
members generally shared this goal was confirmed through the CARRA member surveys. To learn how
members interpreted and enacted their role as conduits of information, the interview protocol explored
respondents’ understandings of expectations and actual activity. Responses indicate that, indeed, all
respondents are well aware of expectation to share information with constituencies. Consistent with
survey findings, this expectation appears to have been communicated clearly.
When asked how often they have shared information with their constituencies, most have done so
frequently.
Constantly. Twice a week at meetings. I’m involved in two community programs a week. It’s
continuous.
I’m forwarding CARRA updates to others, discussing with patients, through word of mouth,
presentations, and I’ve included CARRA info in all of these practices.
CARRA provides the information; they tell me what’s going on and I’m just involved. I disseminate
as much information as possible.
I have the potential to share information every month, but probably share every other month. We
have a larger newsletter, so that’s when I would put information in, and if there’s something that
comes up in between I could say it.
Very much so. I know that when I first was participating with CARRA, I always shared with the
local affiliate of the Komen Foundation, these are all people I'm involved with, and so every time I
was doing something with CARRA, these people were aware.
It is intriguing that while many respondents reported a desire to improve communication with
constituencies as a motivating factor for joining CARRA, respondents tended not to conceive of
communicating information gained through CARRA to constituencies as a CARRA activity.
Respondents rarely mentioned communication or dissemination when asked to describe or list their
CARRA activities. And when directly asked if they think the communication and dissemination efforts
catalogued in the quotes above constitute a CARRA activity, few clearly saw it that way. Prefacing their
responses with “I guess,” or “I think,” conveys further that this expectation, while acted upon with
regularity, is not a conscious or concrete component of their CARRA identity.
I guess I never really thought of it as a formal part of the program. I want other people to have
this information, but I never thought of it as a CARRA activity.
I think it’s not viewed as a CARRA activity. It’s something they [other members] do through their
other organizations.

23

I guess, I think I consider that I’m jointly working with CARRA.
Members don’t understand this aspect of CARRA. They know to do it, but don’t connect it, [with
CARRA].
One member suggested that, “A monthly checklist of [pre-assigned CARRA] activities would help
people reflect on all they have done. It would remind them that simple communication is really
valuable.”

Providing Support to CARRA Members
In addition to specifically asking how OLA can better manage member expectations, the interview
protocol invited feedback on other ways OLA can assist and support its membership. A common thread
emerged from respondents: personal contact.

CARRA Members Wish for Personal Interactions With OLA
Although respondents thought the “website is fine” for reference materials and appreciated the
regular electronic messages that offered updates and resources, they clearly crave personal connections,
such as those created during the training workshops and activities. The call for direct contact with OLA
staff and other CARRA members was a common theme.
Interestingly, suggestions concerning ongoing support of members often had direct implications for
helping to manage members’ expectations. The notion of informal, but periodic, personal contacts and
two-way updates that invited a response was a frequent theme. Members also called for meetings or other
events that would allow them to interact directly with other members and tighten the CARRA network.
Check in, informally, but personally, not through a survey. Just a quick call. “What’s going on?
What have you been involved in?” Track the activity level of members to see what types of
members get called, like once a year.
I guess a personal e-mail would be good, just to say “we realize you haven’t done anything and
there really isn’t anything for you to do, thank you and goodbye” or “there wasn’t the right
opportunity to involve you, but we still want you involved.”
I don’t remember receiving any personal e-mails in a little while. I’m not offended by lack of
contact, but....
Convene annual meetings or regional meetings. I know they can’t fly everyone into Washington all
the time. A day-long seminar to learn what is going on. I know there are webcasts or
teleconferences, but to be honest, it’s easier for me to take a couple of days off to attend a meeting
than steal a couple of hours here or there. You lose the personal touch too.
The training was fantastic. It was a great opportunity to interact with lots of people.
One member who has been active in CARRA since its inception shared the opinion that the quality
of communication from OLA has improved since the member first joined in 2001. But the same
respondent concurred that “more personalized e-mails and updates that are specific,” would help manage
expectations on an ongoing basis, not just at the outset. In this case, the respondent was using the word

24

“specific” to describe communication that is exclusive to the particular member, preferably personalized,
but at the very least, specifically forwarded to select members sharing similar interests.

Network Building and Member Profiles Could Help Personalize CARRA
When asked what other kinds of help and assistance they would like to receive, responses again
tended to reference personal calls, personalized e-mails, regional meetings, and personal dialogue—
basically anything that increases interaction among members and between members and OLA. (It should
be noted that a number of respondents commented on how much they value the interview, interpreting it
as an example of personal outreach from OLA). Several respondents specifically mentioned wanting to
build a stronger CARRA network.
CARRA updates are great, but interviews like this are better, because it’s a personal point of
contact. The opportunity to talk out loud, as advocates we all like to hear ourselves talk, is good.
Getting us more connected in regional networks. Help us become more involved with local
designated cancer centers.
I went to the training, and then there wasn’t anything that made me feel hooked up with the
people who were there, so that you could network, because oftentimes networking often helps
with what else you’re doing in your life.
There really wasn’t a dialogue like this for individuals to have that personal connection.
I did appreciate the time I had [to attend meeting in DC] to have that personal interaction.
One significant area was communication; I did not see or hear or read much from NCI over a
period of time so that would be one thing.
While calling for group meetings, respondents did recognize the costs involved. More than one
respondent indicated that they did not expect these to be “lavish affairs.”
They could be low budget. Just provide travel reimbursement.
An even lower cost example of personalization that was mentioned during the course of interviews
was “developing and sharing profiles of specific CARRA members, like spotlights.” These could be
disseminated in monthly newsletters or e-mail blasts, giving all members a chance to better know their
colleagues. (Like CARRA members, NCI staff specifically requested profiles of members to help staff to
better know CARRA members and select members who are a good fit for particular projects.) While such
profiles would not address the desire for face-to-face interaction, they would advance peer networking.

Building Awareness and Understanding of CARRA Within NCI
Members would appreciate efforts that better help them integrate into NCI activities. This desired
activity was most frequently described as “educating NCI staff on the CARRA role” to inform and better
prepare the NCI staff with whom CARRA members interact.

25

We’re well trained to look out for human subjects and I think that it just has to be made clear that
that’s what we’re supposed to be doing.
Every advocate that comes into a meeting for peer review has some high esteem for researchers,
they are learned people. But there’s a lot of researchers and clinicians that are unsure about us,
the advocate.
Basic scientists are not people people. Some might not see that the patient advocate view is
important. They might sense that they don’t have the necessary expertise. But maybe they [the
scientists] lack knowledge about CARRA.
Respondents indicated a desire that OLA heighten awareness of CARRA within NCI, unaware that
a parallel component of this phase of CARRA research was to identify efficacious ways that this can be
done.
Building awareness and interest is more essential within NCI: there is no shortage of interested
advocates.
There are more groups within NCI who could use CARRA members. OLA needs to expand options.
While the action-oriented activities that characterize CARRA are highly valued, the mere act of
increasing communication and interaction between advocates and scientists is equally valued. One
respondent, commending OLA for accomplishing this and urging them to explore new opportunities to
facilitate this interaction, said it as follows:
The biggest value of CARRA is opening each others’ eyes about how the process works on the
other side of the street. I was so naive [about NCI] and now I can explain it better to others. We
need to figure out how to facilitate that interchange on a regular basis.... Some scientists never
leave the lab, and advocates can’t get into the lab, and so never the two shall meet.
Additional suggestions included providing training to NCI staff, having the chairs of review panels
do a better job of introducing CARRA members and soliciting their input, and providing CARRA
members with a clearer understanding of how NCI works so they are better able to operate appropriately
within the NCI context.

Screening, Skills, and Training
In this section, respondent comments concerning the screening of new members, the skills needed
to serve as a CARRA member, and the types of training they perceive as most important are discussed.
As the interviews transpired, it became clear that these topics were very closely related. While each is
examined in turn, the interconnections among screening, skills, and training become obvious, and their
distinctions become blurred. An important theme of identifying and matching member strengths (i.e.,
skills) to their CARRA activities is evident throughout. Another theme underscored in conversations with
members was the prominence of peer review. Its significance was reflected in discussions of skills that
members need, how they should be trained, and how they should be screened. The opportunity to
participate on a peer review panel appears to be the dimension of CARRA that truly distinguishes it from
other advocacy experiences in the minds of the respondents.

26

Screening of New Members
CARRA Members Support Increased Screening
Given the members’ otherwise inclusive orientation, it was surprising to find the interviews
yielding many comments on the need for CARRA to be very careful in its recruitment. Indeed, nearly all
of the respondents called for more stringent screening. There was a general agreement that to maintain
CARRA’s reputation as having recruited the “cream of the crop,” the program should remain selective
and perhaps become “more selective in recruiting.” To distinguish CARRA members from the large
contingent of consumer advocates, especially in the minds of the NCI staff who they imagined may
harbor stereotypical impressions concerning advocates, respondents were in favor of a rigorous vetting
process designed to screen potential members on certain personal characteristics rather than technical
expertise.
CARRA needs to be more selective in recruiting, especially when working with researchers.
They need to screen CARRA members more carefully, some are too emotionally involved. Need to
separate them out.
You’re better off having a small, focused group of people; I mean not everybody can be a CARRA
member, not everyone understands it, and nobody has a “right” to be a CARRA member.
You also need to screen people, because I do get concerned from interacting with some of the
CARRA members that they may be good advocates but not good CARRA members, because they
are too emotionally involved in their cancer, they can’t separate themselves.
I worry about that, because I’ve heard comments from the scientists that they were grateful for my
participation, because they’ve had bad experiences with people who can’t understand, who can’t
approach this logically, and they approach it too emotionally. And it gets to be a contentious
thing. I understand why they are that way, but I don’t think that’s appropriate for the job they are
supposed to be doing.
As these comments indicate, CARRA members want to screen out potential members considered
“too emotionally” or too personally involved. Their concern is that advocates cannot fulfill their
responsibilities as CARRA members if they cannot keep their emotions in check. Reflecting on their
personal experience interfacing with NCI staff, members have identified this ability as prominent and
essential. While passion and emotion may be crucial elements of advocacy, CARRA members believe it
is their ability to function within NCI on behalf of the consumer—to promote and protect the best
interests of the patient—while keeping those emotions in check. As one respondent stated, those who are
unable to do so “may be good advocates but not good CARRA members.”
I observed it during my training. I could tell this person was not ready. For them, everything was
too personalized. They couldn’t take the global view.
Some people are just confrontational. Somehow they got it in their mind that that’s their mission.
Your mission is not to go in there and stir up the pot.
When I'm in a group of people, and we’re looking at a particular research proposal, the clinicians
and scientists are looking at this from a different perspective than the consumer advocate. Now,
granted, we’re getting full voting rights, and we can ask questions, but it’s not our job to trip these

27

guys up, to second guess someone. The CARRA member is not really competing with those
clinicians; we’re there to provide another perspective. So if you’re talking about how to work and
play well with people, then yeah, everybody’s going to need those things [negotiation skills /
interpersonal skills] or you aren’t going to do well. We’re there because we have a connection to
this disease, because it puts a face on that cancer, and it gives them some insights.
I don’t think I'm a physician and I don’t think I'm a researcher. I know that I am an advocate, and
there is a difference. I don’t think a lot of people have accepted that fact. There are many
advocates who... when they go into meetings, they are trying to be someone they’re not.
Recognize Personal Strengths and Characteristics and Match Them to Tasks
Members again raised the issue of screening when discussing training and assignment to
appropriate CARRA activities. At the core, these factors appear to revolve around the notion of matching
CARRA members’ strengths, either identified through screening or developed through training, to the
activities in which they take part. As discussed below, members felt that training ought to be matched to
incoming member needs, presumably identified through the screening process.
Respondent: There are some CARRA members that might have an A after their name, a B after
their name, a C after their name.
Interviewer: What do you mean by “A, B, C?”
Respondent: It’s a combination of their interests, what they do, their background and experience,
and how much time they have for this. You’re looking to see whether you got an A
team member or a B team, and you can use a B team member for this kind of stuff,
and maybe there’s fewer of them than in the A team, etc. Cause, you know, a happy
volunteer will speak very well for your organization.
Given this theme of “matching,” it is noted that respondents’ calls for more rigorous and careful
screening do not necessarily equate to being more elitist in who can and cannot join the ranks of CARRA,
but instead relate more to determining future members’ strengths and weaknesses so that they can be
deployed most effectively. Respondents indicated that they do not necessarily believe potential members
should be screened out due to a lack of technical expertise in science or medicine or the practical aspects
of peer review, which can be trained.
Again, the prominence of peer review is clear in respondent comments. For example, one member
brought up the topic of screening by stating, “CARRA needs more screening, more up-front effort on
selecting the right member.” When probed on what “right member” meant, this respondent said the
phrase meant identifying members who are right for peer review:
For those who aren’t ready for peer review, spread other work around.
Thus, members envisioned screening as serving primarily as a safeguard, that is, to protect the
integrity and reputation of CARRA within the peer review setting. Those “who aren’t ready” can be
assigned other CARRA tasks.
You know, some of these [CARRA member] people can’t do peer review and if they don’t know a
little bit about anything, they’re going to be overwhelmed. It’s always risky business when you just
put a CARRA member out there. Picking and choosing needs to be very careful. So in the recruiting
process, you kind of need to get into their head. These [CARRA] people are usually cancer

28

survivors, they want to do something, they want to help, but the real trick is to match their abilities
to their expectations, and it’s always been that problem. So I would spend more time having a
dialogue and I think it would be worth the time to have an interview.
As this report reflects, the theme of matching strengths—defined as skills and abilities, but more
often as personal characteristics—to appropriate CARRA activities permeated the interviews.

Identifying Critical Skills
While some generic skills were identified, members tended to discuss strengths and weaknesses
more in terms of matching them to specific tasks. There was fairly strong consensus on the following as
standard skills all CARRA members should have:
•

A basic understanding of science;

•

The ability to critically analyze;

•

A working knowledge of how NCI research and funding takes place;

•

Self-confidence and strong communication skills; and

•

The ability to interact with NCI staff on the common ground of reason, logic, and the scientific
method.

The most prominent task correlated in members’ minds with CARRA is peer review. As a result,
much of the discussion during the interviews focused on skills related to this strand of activity. For
example, even questions about general skill requirements yielded much discussion of peer review panel
skill requirements.
Respondents often recognized that CARRA members arrive with a set of professional experiences
and skills that ought to be taken into consideration. These comments pointed out that advocates are not
simply people with or affected by cancer. They are also teachers and business owners and officers from
the military and executives from the private sector. Respondents noted that while it is important to think
about what skills CARRA advocates need, it is equally important to recognize and utilize the skills they
bring to CARRA.
Consequently, respondents discussed skills in relationship to tasks or activities. Their comments
suggested that matching one’s natural characteristics and preexisting professional skills benefits both the
program and the individual. While training can hone and augment these skills, the natural match was
favored over the shaped match.

Science Backgrounds Are Advantageous
Technical skills, like knowledge of basic science concepts and communication skills can be
enhanced, but respondents tended to prefer that these should be in strong evidence upon arrival, forming a
platform on which to build. A few comments reflected the premium placed on members with a training
or background in science. A skill often related to science, methodical analyses, was also referenced.
Still, none of the respondents suggested that members need to be scientists. As one respondent stated

29

earlier, “I don’t think I’m a physician and I don’t think I’m a researcher. I know that I am an advocate,
and there is a difference.” What was valued was a basic understanding of science concepts, methods, and
terminology.
I think it would be very wise to identify future CARRA members who have stronger science
backgrounds.
You have to arrive at CARRA with some sort of skill level in being able to analyze documents. You
have to have that understanding before you ever get there. Then, CARRA’s got the hands-on
workshop training on how to do certain things, like how to go through a grant. I’ve participated in
the training program on how to analyze grant materials; now I knew how to analyze other kinds of
materials, but this was just focused on analyzing R0-3 grants and I thought the training was very
good.
If you have a science background, you have an easier time understanding what the doctors talk
about during grant reviews, but you don’t need to understand everything on a graph to understand
what affects the patient.
There Is Consensus on the Need for Strong Interpersonal Skills
When concerns are voiced about skills, they appear to be less about technical knowledge or know
how and more about process and interaction. Technical aspects such as terminology, reading scientific
papers, and understanding the scientific process were mentioned as important by several respondents. But
every respondent had strong feelings about the need for communication, negotiation, and interpersonal
skills.
[You] have to start off with having excellent communication skills, two-way communication skills
where you learn how to talk but you also learn how to listen.
Need to be willing to ask questions. Not afraid to reach out to scientists. Ask a question if they
don’t understand something.
Being bold and assertive. I'm not shy. I also think people need to be retuned in to how to write.
I think I am the type of person who doesn’t have a problem speaking up if I have a question, so you
have to also be a person who isn’t afraid to look dumb so that you can clarify information.
It’s important to speak out rather than hold your tongue. Even if they prove you wrong, you’ll find
out why you were mistaken. You’ll learn. Through experience, you get more comfortable.
Need to keep an open-mind, take emotion out of it. Approach things with a logical view.
You must be able to effectively argue, in a logical way, not in a critical or judgmental way. You
have to explain your viewpoint and why you feel that way. Argue fairly.
Need to overcome sense of intimidation. They may feel intimidated at first, but the majority of
scientists have been very welcoming.
I brought the ability to separate the emotion out of it. I think that’s something that is hard to teach.
Active listening. You don’t what to jump in right away. You have to avoid knee-jerk reactions.

30

Members Bring Preexisting Professional Skills in Addition to Advocacy
During the interviews, the first question asked of respondents concerned their current employment.
This launched a discussion of what they are doing now, both within and beyond the advocacy community,
as well as their professional backgrounds. While each of the respondents was able to articulate a long list
of advocacy efforts, all of them also invoked their careers and positions prior to becoming engaged as
CARRA members or as consumer advocates.
At various points throughout the interviews, respondents harked back to prior experiences or
current professional commitments as dimensions of their lives that shaped them as advocates or informed
their work as CARRA members. These comments underscored the obvious fact that advocates are not
simply people who are personally familiar with a disease. They have professional careers and rich life
experiences.
Acknowledging that what holds true for them holds true for other CARRA members, respondents
noted that OLA ought to take into account their preexisting professional experiences and skills. Their
comments demonstrate that it is critically important to recognize and utilize the skills members bring to
CARRA when planning training or assigning tasks. Moreover, CARRA members who are able to apply
their professional skills seem happier.
See, with something like that, I can be very engaged—because I can utilize and leverage my skills
that I have from my profession where I have about a dozen years of strategic planning, business
planning and product planning—which kind of put those elements or skill sets to work. That really
does motivate me and I’d really make time for that.
Being able to read technical documents, being able to analyze applications, that’s due to my
background in working with computers.
One particularly active member was asked why she thought she was so frequently sought out as a
CARRA member. The respondent replied that it was her professional background, her outgoing
personality, her experience in community outreach, and her responsiveness to CARRA when invited.

Match Member Skills and Interests to CARRA Activities
Given the diverse professional backgrounds of members, it is not surprising to find they have
diverse interests. Although the activity of peer review overshadowed the interviews, some respondents
clearly want to be involved in activities other than peer reviews, preferably assignments that take
advantage of their preexisting skills. Being “sorted” into different CARRA segments (e.g., peer review,
communication, advisory, product development, etc.) might actually be welcomed. Some members sense
that the major focus of CARRA is on peer reviews because the training seems to emphasize that
dimension over others in which they may have a stronger personal interest.
I mean, I’d like to do something else. I’m happy to help with those [Grant Approval processes] but
certainly, I’d like to do something else that benefits from my expertise, whatever that is.
Yeah, because my expectation was to get some information that might be complicated for the
average reader to read and then make it simpler for them, send it back, that’s what I thought I’d be
doing. But I got trained on being a Peer Grant reviewer.

31

I think the fact that I don’t have a science background, and I don’t have a typical cancer group,
has limited me from being selected, and that’s totally understandable.
Like the screening discussion, these comments underscore members’ desire to match their skills
and interests with NCI needs. At the same time, they are also consistent with earlier comments reflecting
the desire to ensure that only a select subset of CARRA members sit on peer review panels and interface
with NCI staff at that level.
These respondents promote a more intentionally structured and defined approach not only in the
implementation of the CARRA program itself, but also in the selection of its new members, so as to
successfully match their interests and background with CARRA’s needs and specific activities.

Training Needs and Options
During the interviews, each respondent was presented with a list of seven training topics identified
by OLA:
•

Peer review training;

•

Human subjects protection courses;

•

Technology-specific workshops;

•

How to serve on an IRB;

•

Computer training;

•

How to conduct Internet research; and

•

Negotiation skills, business etiquette, interpersonal communication.

The goal of the question was to determine which types of training OLA should invest in or
augment in the future. However, it is noted that respondents answered the question in somewhat different
ways. Because many received CARRA training in each of these areas, some interpreted the question to
mean which was most important, or which ones they valued most personally. Attempts to clarify did not
always succeed, so interpretation is somewhat compromised.
During one such effort to clarify, a respondent indicated that he/she was unaware that CARRA
training had been provided. “You know, I didn’t know CARRA offered training, so that’s news to me.”
Therefore, this respondent answered the question as if OLA was planning to develop these workshops
from scratch.
Nevertheless, a relatively clear pattern emerged, consistent with findings presented above. Most
respondents favored training related to peer review and human subjects’ protection. Even those who
advocated for training on how to serve on an IRB tended to subsume that topic under human subjects’
protection. Only a few saw value in training on technology, computers, or Internet research. And, by far,
the category receiving the most interest was “negotiation skills and business etiquette.” This latter
category tended to be interpreted as how to interact directly and effectively with NCI staff, and more
specifically in a peer review context.

32

As discussed in the previous section, peer review activities have become nearly equated with
CARRA activity in the minds of these respondents. As a result, feedback on training is closely, almost
exclusively, associated with peer review activity. Feedback emphasizes the need to provide a solid
grounding in the peer review process to ensure CARRA members are fully prepared for this demanding
experience.
I wouldn’t have been prepared at all if I hadn’t gone through the training and still, again, I feel in
a “one down position” when I’m at those peer review meetings because I don’t have enough
knowledge of the science, no matter how many times I read the grant requests, I still don’t
understand a lot of the science.
I’ve done RO-3 Grant Reviews a number of times: once this year, twice last year, and two twice the
year before that. I’ve also participated in the CARRA training for grant reviews where I learned
how to do it and I immediately went on to some more grant reviews which was very taxing, I didn’t
know what to expect on that one but it was a biggie and once I figured out how to do them the
smaller grant reviews have been easy.
Well, just to become a CARRA advocate, I did not get the training I needed from CARRA; I got it
from NBCC—the scientific materials, the vocabulary, the use of computers, going to different sites.
At the same time, members did not want to be placed in a situation in which they feel inadequately
prepared. Relying on their preexisting professional skills provides them with a certain comfort zone,
while adequate training can overcome a lack of experience or other uncertainties.
I felt prepared for the assignments that were linked to my work experience, but not for peer review,
because peer review training was not available [at that time]. I felt overwhelmed. It was not clear
to me what I was supposed to do. I was not prepared, nor was I ready to call the lead scientist and
ask. I was intimidated both by the size of the task and my obligations. I didn’t want to look stupid.
It’s clear to me how others might feel intimated.
Again, I would like to be more involved and, these grant approval processes of course, you know,
it’s very intimidating. The doctors involved and the scientists involved and here’s just a “layperson” doing the best they can. I’ve never felt on the same level as they are.
During the course of the interviews, CARRA members put forth the notion that certain essential
skills or characteristics cannot be trained. For example, several respondents questioned whether the
ability to bring an objective, global view rather than a deeply personal view, can be taught or trained.
You’re so close to the patient experience.
It’s a characteristic that you either have or you don’t.
Assuming future members are brought on board whose interpersonal skills and abilities in this area
are not fully formed, training or coaching was advised. Such professional development may be able to
help overly passionate advocates control and channel their emotions or build self-confidence in those who
may lack it.
It’s great to have a passion to whatever degree but it’s got to be channeled in a very strategic
manner. You want the opportunity to have impact, not drive people away with your passion.

33

You have to be assured of yourself in order to even participate in any activity like CARRA to start
with, but it would probably be a good training program for some people.
When discussing training, one respondent downplayed the value of training on computer use or
Internet searching, while mentioning the value of training on technology from a very scientific
perspective:
If you’re talking about searching on the internet, then I don’t think any of that is needed but if you
were talking about technology from the perspective about the newest molecular stuff, then that
would be different; but knowing how to make attachments and email, no I don’t think so.
Making a related point, some respondents mentioned the value of learning basic science and
medical terminology, having a refresher on the basic scientific method, learning how and why clinical
trials are undertaken and, gaining techniques on how to approach reading scientific papers. Other
respondents mentioned that training should include an overview on how NCI is structured, the vast array
of work conducted there, its culture and way of doing business. These comments reflect an understanding
that CARRA advocates can be more effective if they understand the culture and structure of the
organization they are about to enter.

Customized Training
In keeping with themes mentioned previously, several respondents proposed that a more
discriminating vetting process be used to sort new members into different segments based on
characteristics or skills. This notion of segmentation was returned to again when discussing training; the
obvious implication being that standardized training would no longer be necessary if CARRA members
were arrayed along a predetermined division of labor. Instead, only those who were prescreened to serve
on peer review panels would receive the peer review training, presumably the most complicated form of
training offered. Others would receive training as needed, in other areas such as communication, human
subjects protection, or serving on an IRB.
Comments projected a scenario where CARRA training was customized to the individual, based on
a combination of skills identified through the screening process, the needs of NCI at the moment, and the
interests expressed by the members themselves. Such a modularized approach to training was considered
to be more cost-effective. Some went farther and questioned whether all members needed to be trained in
person, proposing teleconferences as an alternative.
I think they could probably do some online things over the web; you could do training over the
web.
However, this approach seems to contradict the thematic value placed on in-person contact above
and beyond the purpose of the meetings. And in practice, it may be more expensive to organize and
deliver customized, modular training rather than a standardized series of workshops to all members.
However, earlier comments did indicate that expectations to complete peer reviews and other activities
did increase following the training, so it is clear that the structure of training does contribute to shaping
members’ expectations.

34

Mentoring Is Valued
Several respondents mentioned the value of mentoring, often in the context of wishing they had a
mentor themselves when coming on board. Others suggested that it was their less experienced colleagues
who would benefit from coaching or mentoring. Members suggested that mentors could help future
CARRA members become more productive contributors to CARRA and NCI, help them gain a better
idea of what to expect from the program, identify the essential skills necessary for each particular activity,
help build those skills on a one-to-one basis, and encourage members to obtain addition help if needed.
Mentoring would also provide another form and level of activity for current members while
simultaneously helping to address the desire for social networking.
Respondent: I would have liked a mentor.
Interviewer: You sound like you could be a mentor for a future CARRA member.
Respondent: Oh, I would love to be!
Interviewer: Should CARRA consider using its current and experienced to serve in that role?
Respondent: It would be wonderful, but CARRA should also have some standards so that all
mentors are mentoring the same way.
In a slightly different context, one advocate described, in colorful terms, how a more mature,
experienced advocate could provide guidance to a less experienced one, in this case, one who is taking
too emotional a stance within a peer review panel.
When you do a Peer Review panel, there’s usually more than one advocate on the panel, so maybe
you can identify the senior advocate who can “take Earl behind the woodshed” and have a “come
to Jesus meeting” at least for the sake of that particular scientific session.

Section II.
Findings From NCI Staff Interviews
As part of the effort to gather qualitative data from CARRA members and NCI staff, Westat
interviewed 24 NCI staff members between June 27 and August 9, 2007. The interviews focused on the
following topics:
•

Value of CARRA and consumer advocacy to NCI;

•

Awareness of the CARRA program and how respondents learned about it;

•

Experience with CARRA and consumer advocates;

•

Qualities of CARRA members valued the most;

•

Skills, knowledge, and/or experience that make CARRA members desired members of project
teams in respondents’ areas;

35

•

Type of training that should be provided to CARRA members to make them more valuable;

•

Barriers and facilitators to using CARRA; and

•

Ways to increase awareness and use of CARRA within NCI.

The results of the interviews with NCI staff members are presented in this chapter. Before
presenting the results, a brief description of the characteristics of the NCI respondents is provided.

Characteristics of the NCI Respondents
NCI staff were oversampled in anticipation of attrition. However, every scheduled interview took
place resulting in six more interviews than called for in the workplan. As shown in Table 2, the 24 NCI
respondents were not quite evenly split between CARRA users (n=13) and CARRA non-users (n=11).
Respondents were typically female (16 females, 8 males). Most were involved in scientific work (n=19),
primarily the administration of extramural research (n=16, not shown). Respondent characteristics by
type of work performed and CARRA user status are associated with direct quotations.
Table 2.—Characteristics of NCI staff respondents, by CARRA user status
Description
Total..................................................................................................
Gender..............................................................................................
Female .........................................................................................
Male .............................................................................................
Type of work performed
Scientific......................................................................................
Programmatic...............................................................................

CARRA
user
13

CARRA nonuser
11

Total
24

9
4

7
4

16
8

10
3

9
2

19
5

Summary of Interview Results
Respondents were first asked whether they were familiar with the term “consumer advocate.” With
the exception of one staff member, a CARRA non-user who was nonetheless familiar with the CARRA
program, all were familiar with that term.
Several respondents expressed concern that the word “consumer” was inappropriate in the context
of the mission and objectives of the CARRA program, and most of the interviewees used the terms
“patient advocate” or “community advocate” instead. One NCI staff member also indicated that there is
some negative baggage associated with the word “advocate,” that might be associated with a
confrontational or overly individually oriented attitude. However, no substitutes were offered.

Value of Consumer Advocates
Both users and non-users of the program were asked about the value that consumer advocates bring
to projects at NCI. A key contribution that emerged was that advocates convey the patients’ perspective.
Further probing yielded other contributions that consumer advocates make to the research process.

36

Contributions of Advocates
Advocates convey the patient perspective. An overwhelming majority of respondents stated that
the value of consumer advocates is the patient perspective they bring to the research process. Advocates
provide information that scientists might not know. They provide an insight into what it is like living with
a condition under study. Respondents said that advocates provide their own point of view and objective
information, bringing a “humanizing factor” into the research process.
Obtaining the consumer perspective is very important…. Consumers are on one end of the
spectrum, NCI on the other end, but NCI is there to serve the public, therefore, obtaining the
consumer’s perspective is valuable. (Science administrator, user)
They tell us what impact it will have on them, what ever it is we are thinking of doing, and how we
can do it better to meet the needs of out target audience. (Programmatic, user)
They look at projects with more of a “what does this mean to someone like me” approach.
(Science administrator, user)
Input from a patient advocate is a good way to screen for potential patient issues and I personally
think their input or presence at meetings is important. It is highly recommended, if not required, of
us, as a part of Research Programs and Review Grants (RPRB). (Science administrator, user)
While there was a general appreciative attitude about consumer advocates, non-users were more
skeptical than users about the overall quality and type of input provided by advocates.
They are cancer survivors, and I would imagine that the input we get from them gets us a better
picture of the reality of what the patient goes through. But, when it comes to research, I don’t think
they can provide the scientific information we need. (Programmatic, non-user)
It also becomes a politically correct thing— you have to have one no matter if it makes sense or
not, or if they make any contribution. (Science administrator, non-user)
Advocates provide grounded “reality checks” in research planning and development. As one
respondent noted, each proposal sent to NIH or NCI should have some sort of public health relevance.
Consumer advocates help to ensure that even the most technical and basic science proposals maintain
such relevance.
[Advocates serve as a] reality check that helps keep researchers in the world of cancer connected
to what the disease is really like and how it affects people. (Programmatic, user)
[Advocates] keep the disease real in the minds of the investigators so they don’t get too
intellectual. (Programmatic, user)
[Using consumer advocates at meetings] keeps attendees grounded in the real disease.
(Programmatic, user)

37

One CARRA user working in the area of basic research told the following anecdote about why he
wanted a consumer advocate on his project:
In this division, we spend about $900M per year on basic science research. One doesn’t do basic
research strictly for the sake of doing basic science. We try to discover things about human cancer
that will enable us to translate knowledge to human clinical studies or human population studies.
When we began this project with the goal to see if we could mimic human cancer in mice, I made it
known, to the amusement of my colleagues, that I wanted a consumer advocate, a patient advocate,
on our project because I wanted to be sure that scientists were constantly reminded that the goal of
this was not to cure cancer mouse cancer, but to cure human cancer. (Science administrator, user)
Advocates are attuned to human subjects protection. A third contribution that emerged was that
consumer advocates are concerned with human subjects issues, including how subjects are treated during
a research project. Advocates are also often concerned with how well proposed research addresses the
broader population of cancer survivors and healthy people. NCI staff members involved in administration
of research grants, in particular, mentioned the value of advocates in making sure that issues related to
informed consent and patient protection are addressed in the grant review process:
Advocates are especially valuable on issues related to informed consent and on treatment from the
patient’s perspective, for example if a procedure should be done in a hospital or on an out-patient
basis, or other quality of life issues. Input on informed consent is very important; readability, on
what they think is important, the appearance of an informed consent form, as well as the text.
Communicating its purpose to others. (Science administrator, user)
They help focus on the human issues in the science projects. (Science administrator, user)
Advocates help with development of communication materials. Another contribution of
consumer advocates that emerged with further probing is that they can help translate research materials
such as recruitment materials, study descriptions, and findings into a form that is understood by lay
people. CARRA users involved in health communication, in particular, appreciated the potential role of
advocates in the development of products and materials. One respondent mentioned that the advocates
provided information on the management of side effects for a decision aid booklet for prostate cancer
patients.
Things that may not seem from a patient’s standpoint to be something they want to participate in
and we get feedback as to why. (Science administrator, user)
I feel that they are usually cancer survivors or represent/work closely with that constituency. They
have a lot of insight and perspective. It may not change the scientific way that we’re saying
something, but I think a CARRA advocate can help us put our information in an understandable
way. (Programmatic, user)
[CARRA member involvement] makes for a better product. [Using them] makes a product go from
mediocre to resonating with people in a better way. (Programmatic, user)
Advocate involvement makes for a better product (video, print, something in conceptual stage). It
doesn’t mean we have to take everything they recommend. Early on, people didn’t really know
what to do, so we get a lot of opinions. Each group has their own agenda. By involving advocates,

38

expectations are managed better over the course of the reviews—for the greater good and the spirit
of improvement. (Programmatic, user)
Advocates are poised to promote dissemination of information. Several respondents observed
that advocates bring information and inform research questions posed by scientists, which contributes to
how a study approaches its subjects, but at the same time, they can communicate back with their
advocacy groups and inform them about the quality of the peer review process. Thus, they make this
process more transparent. By inviting advocates to attend grant reviews, NCI is gaining “ambassadors” to
go back to the community.
When asked to speak at a local function, then they can relay what the government dollars are doing
for cancer patients. (Programmatic, user)

Advocates bring valuable professional skills beyond personal survivorship perspective.
Several respondents recognized that advocates can bring valuable insight from their professional
experience:
I think that something you always have to remember about potential advocates is that they have a
life beyond having survived cancer. They have skills that are extremely valuable to us. The first
time around, I guess I hadn’t thought about the patient advocate who had other skills. I was simply
looking for someone who could ask us, hey guys, what are we learning about human cancer. But I
found someone who gave me a lot more than that. Our engagement with patient advocates has been
much richer than a simple reminder that there are human patients who need our help. (Science
administrator, user)
Advocates can assist with recruitment to clinical trials. Clinical trials only exist if people
participate, explained one of the CARRA users. Through their communications with advocacy
communities, advocates can encourage other patients to participate, and if they are involved personally,
they can contribute effectively to the clinical trial.
Advocates can take on roles the government can not, like lobbying and fund raising. Finally, a
couple of respondents indicated that advocates can take on roles such as lobbying. As one person
observed:
One of the great benefits I saw was the recommendations coming from the PRGs [Progress Review
Groups]. The advocates used them to start their own programs, their own organizations, to
educate Capitol Hill, to get additional dollars for particular disease site. They were responsible in
large part for increases in funding. Prostate cancer in particular was one that was very successful.
They doubled or tripled that funding. I think it was very successful. (Science administrator, user)
Advocates gain perspective and important insights into science. Respondents also pointed out
that by bringing consumer advocates into the research process at NCI, they gain a perspective and insight
into cancer science and return to their advocacy communities with hands-on knowledge of how scientific
research is done. As one CARRA user commented,
On the flip side, what I’ve seen is that there is great understanding that happens with the advocate
when they are exposed to the science and the programs at NCI. They get a better understanding of
what it takes. (Science administrator, user)

39

In this context, one of the NCI staff members, a science administrator, mentioned that the feedback
they were getting from advocates in Progress Review Groups “is one of the best things that NCI has ever
done. The community as a whole felt very involved in doing priority setting,” Another person, a CARRA
non-user working in the area of nutrition, pointed to the existence of various conspiracy theories prevalent
among consumer advocates, particularly in the area of nutrition, claiming that there are some miraculous
supplements that someone does not want to approve or study. Involving advocates in the research process,
this person contended, might counteract such theories.

Concerns of Advocates
When discussing the general value of cancer advocates’ involvement in NCI programs, respondents
also raised several concerns. Among them, the lack of scientific background of consumer advocates and
issues related to the political correctness and obligation to involve advocates were most profound.
Lack of scientific background. Several respondents pointed out that advocates are not scientists.
Some of the advocates cannot understand the documents they are supposed to review. Concerns were
raised that CARRA can facilitate their involvement in the activities at NCI if consumers are scientists (or
are scientifically inclined/versed), but it can be less easy if they are not. However, one respondent
explained that CARRA members can still comment on related scientific tasks, such as how to enroll
minorities and children in clinical trials, data sharing among the public, outreach activities, consent
information dissemination, and so on.
Obligation to involve or bow to political correctness. Several respondents observed that the
requirement to use advocates in some units might be seen as a bow to political correctness rather than a
true recognition of a value that consumer advocates might bring. As one CARRA user commented:
I don’t think very many other programs, workshops, meetings utilize advocates. I feel it is a bit of
a struggle. Some scientists feel they must monitor what they say when there is an advocate in the
room, and don’t want to have an advocate. BUT those who have been most negative at first, have
been positive after having advocate present.
I think at NCI it’s an underutilized treasure.
(Programmatic, user)

Awareness of the CARRA Program
Most staff, including non-users, were aware of the CARRA program. Only four of the 11 non-users
had not heard about CARRA until they received the e-mail invitation to participate in this study. Three of
those four were biostatisticians involved in intramural research, and another administered nutrition
prevention programs. Most staff members who knew about CARRA heard about it when it began or soon
after they were hired by NCI (if that was later than 2001).
Non-users learned about the program in a variety of ways, including word of mouth, attendance at
a board or other meeting where CARRA was presented, e-mail alerts about CARRA activities, and a
CARRA presentation to senior leadership; one person had an office that was physically close to the office
where the CARRA program was housed.
Most of the CARRA users are mandated to use the program and learned about it as a part of their
job duties description. One respondent, a science administrator, user, was sent to observe the CARRA

40

training for new members immediately after he was hired. This, he observed, “gave a whole picture of
what was the purpose of this organization, and what were the opportunities and possibilities to interact
with it.” Among NCI staff that are not mandated to use consumer advocates, one person was introduced
to CARRA by her co-worker at a time when she was looking for advocates. Another CARRA user first
learned about the program through a colleague who went to a CARRA meeting and reported back about
it. Yet another respondent remembered having CARRA staff come to her branch to talk about the
program at a staff meeting. Staff members involved directly in communication activities pointed to their
intimate knowledge of OLA and CARRA: “[Our] office embraces it! It is not a tough sell. We work down
the hall from OLA- we all came from there. We all know about it,” remarked a programmatic user.

Experience With CARRA and Consumer Advocates
As described in other sections of this report, the types of activities in which CARRA members are
involved varied depending on the type of units represented by the NCI respondents. The experience from
CARRA members’ involvement in respondents’ projects varied as well. The majority of CARRA users
were generally satisfied with the performance of CARRA advocates. Respondents involved in
programmatic activities were more likely to be satisfied than those involved in science-related activities.
They were very informed in a broad sense. [They] may not have known specific mouse genetics, but
[were] impressive in background and understanding. Contributed when appropriate… (Program
administrator, user)
I had trouble picking people because everyone seemed great, qualified, great choice, especially in
major cities where there were a lot of people to choose from. (Program administrator, user)
A very good element to include in our process. (Programmatic, user)
We are getting to these CARRA members to reach underserved populations to take our cancer
information to the underserved populations. (Programmatic, user)
Any time I do something, I always build CARRA review into it. It makes for a better product (video,
print, something in conceptual stage). It doesn’t mean we have to take everything they recommend.
(Programmatic, user)
[Advocates] are basically required in our environment, so it’s not really optional, and aside from
that, when they’re good, they’re really helpful. (Science administrator, user)
They turned out to be useful for an enormous number of translational things, so having an
advocate now makes sense. (Science administrator, user)
I’ve been involved with two CARRA members so far, and I’ve been very impressed with their
dedication and willingness to sit down with an application like this one and tease out the pieces
that apply to them. However, what we get from them doesn’t look like a scientific critique…. They
definitely have helped but that doesn’t mean I don’t have to go back through it and critically
evaluate what they have done, as they don’t have the critical savvy of scientists and miss some
things. But it is good to have them there. They do add a unique perspective. (Science administrator,
use)

41

One respondent, experienced working with approximately 20 CARRA advocates, estimated that in
70–80 percent of cases he/she had good interaction with the advocate. At the same time, two CARRA
users expressed deep disappointment with the contribution provided by CARRA members on review
panels. Among them, one scientific review administrator, experienced working with approximately 10–20
CARRA members, pointed to the low quality of contribution provided by advocates on the review panels.
In this respondent’s opinion, the problem results from asking advocates who cannot intelligently talk
about science to do a peer review. This staff member also explained that CARRA members usually
review the human subjects section of the grant proposals, is a small section that could be reviewed by
mail, thus saving NCI the cost of flying the advocates from all over the country and paying for their hotel
accommodations.
They are not scientists and never will be respected by scientists as equal partners.” (Science
administrator, user)
They don’t accomplish anything in terms of the needs of the consumer community. (Science
administrator, user)
Another respondent in this group raised a different source of disappointment with CARRA
advocates. This person was not mandated to use consumer advocates on her review panels, but she had
reached out to the CARRA program to have the patient voice represented in her reviews. This science
administrator, user observed that the CARRA volunteers seemed to want to discuss scientific issues, and
did not seem to appreciate that the other reviewers were on the panel for that purpose, and that volunteers’
focus should be on “how would I feel if I was a patient and they were going to tell me that they were
going to do all those things to me.” In other words, CARRA members tried to bring in scientific
discussion and be equal partners in the process by providing scientific input (which this respondent found
to be inappropriate and not what the respondent would expect from them.) As a result, she observed, the
scientific reviewers had a tendency to dismiss the remarks that the advocates made.

Qualities of CARRA Members That Are Most Valued
NCI staff that have used the program pointed to several qualities of the CARRA members that
result from their involvement in CARRA program activities. The most frequently mentioned qualities
were that CARRA members are vetted, trained, have superior communication skills, and are dedicated.

CARRA Members Are Vetted
The notion of vetting was mentioned often by respondents. Potential CARRA members are
screened and go through an application process. One CARRA user noted that knowing that members are
vetted gives potential users confidence in involving CARRA members in their project work.
We used CARRA extensively to get names of advocates. CARRA solved an important problem
because the advocates in their membership are vetted. (Science administrator, user)
For some project work, I inherited the name of an individual and this person, although they had
done a lot of advocate service, was kind of a loose cannon, and was not a member of the CARRA
program. Some problems started coming up, so I wanted to sever ties with this person. I wanted to
replace her. So another NCI staff member said to call the CARRA program to identify a member

42

as a replacement. I did not know CARRA existed, even though I worked nearby. Fortunately, I met
someone who was dynamic and had graduated from the CARRA program. I asked her to sit in on
meetings in different cities. We also wanted to find a local person on the CARRA registry to come
to meetings when they were held in their locale. So we identified meeting locales and got a list of
CARRA folks in the area. I contacted everyone, and then talked to a couple of people and got
members to attend a meeting. (Programmatic, user)

CARRA Members Are Trained
Several program users pointed out the importance of the training provided to CARRA members at
the beginning of their involvement with NCI.
They prepare the advocates, they give them an orientation to clinical trials, their design and
function. (Science administrator, user)
We’ve used the CARRA program to identify consumer advocates to be board members on the
Central Institutional Review Board. We limit the participation on the CIRB to those advocates who
have been through the CARRA program because they come to us with a frame of reference of
clinical trials, of cooperative groups, of how they can make a difference with their input. They
have an orientation already built in when they come, so we only use CARRA trained patient
advocates on the Board. (Science administrator, user)

CARRA Members Have Superior Communication Skills
CARRA users generally find that the members they work with are willing to speak up and are
respected by the scientific reviewer. They are sophisticated, articulate, and posses a broad understanding
of the issue.
I think that they contribute way more by what they say during the review than by what they write in
the final review. (Science administrator, user)
It’s the level of sophistication that you get—you know, they’re not just off the street. They’re
usually knowledgeable and articulate. (Science administrator, user)
CARRA members seem to be an articulate bunch of people, who express their ideas well. Someone
with real world experience, who’s plugged into a cancer advocacy group, so their perspective is
larger than their own personal one—this is one of the criteria for CARRA members—it’s not just
personal agenda, they have to represent a constituency, a larger group of people in an organized
way. (Programmatic, user)

CARRA Members Are Dedicated
Users appreciate the dedication of CARRA members. One user pointed out that their dedication is
perhaps prompted by either being a survivor or having a family member who is a survivor.

43

They’re all very reasonable people, and they give a lot of time. I’m always amazed to look at the
members and their personal challenges. They’re in treatment and still willing to be part of
CARRA. The process of screening is so good, so they do seem capable. (Programmatic, user)
If CARRA did not exist or was taken away or scaled back, it would be a real loss. We would lose
the perspective from survivors that we don’t have internally. They bring a plain language aspect of
it, bringing our work to a place that resonates with other people with cancer. That’s what CARRA
members do. I don’t think we’re going to get specific content. We need them to review and to
identify the reality of treatments. We gain good knowledge from their personal experience. Their
contributions make our products go from mediocre to ones that resonate with people in a better
way. (Programmatic, user)

Skills and Knowledge of CARRA Members
Once respondents established if and how they used CARRA members, they were asked what skills,
knowledge, and/or experiences make a CARRA member a desired member of the project team in the
types of activities that they conduct at NCI.
Users reported that they know that they are going to get quality feedback if they use a CARRA
member. Whatever skills they list on the request for a CARRA member, they are confident that CARRA
members assigned are going to have the desired skills. Users are confident that they will get a level of
sophistication from CARRA members that includes being knowledgeable and articulate.
In describing the skills of CARRA members, several noted that it was difficult to specify the skills
because the skills need to match the activity or study. For example, if you are formulating policy, then
you must understand data; if you are an advocate, then you need to know the patient perspective; if you
are part of a research team, then you need a different set of skills. One respondent’s comment
summarizes a general sentiment:
There is no template to describe the ideal member because they have different tasks.
(Programmatic, user)
With this caveat in mind, respondents mentioned a number of skills that were important for
CARRA members to have. These are described below.

Excellent Communication Skills
Almost half of the respondents, users and non-users alike, mentioned some kind of communication
as the most important skill to have. Users mentioned good communication skills in general and, more
specifically, being articulate, confident, and able to express themselves. Respondents see the need for
great communicators who will persevere, engage people, and educate people around them. Consumer
advocates should ultimately be able to listen, educate, and communicate.
Knowing the audience and community they are speaking to and the organization they represent is
another facet of communication.
Valued attributes included involvement in committees and

44

understanding their dynamics, being plugged into an advocacy group, and understanding minority issues
and having cultural sensitivity.

Personal Experience With Cancer
About half of the respondents, both users and non-users, also mentioned the importance of having a
personal experience as a cancer survivor or having a close experience with cancer.
Overall, patient advocates have to have an intimate experience with cancer, be articulate and able
to synthesize information. That runs across whatever they do. (Scientific, non-user)
[CARRA members] should be survivors, or have been touched very closely by cancer. And they
have to be good communicators, because they’re going to be in a review and if they’re afraid to
open their mouth about something that’s going to impede their being there. (Scientific, user)
Other respondents mentioned the need for a good, basic understanding of cancer. There needs to
be an appreciation of cancer in general or of their specific area of interest/expertise. Overall, the advocate
should have an intuitive sense of what is important and think critically rather than just advocate for
research. Several respondents pointed out that in spite of their own very specific experience with cancer,
advocates must be able to separate themselves from the individual experience and care and speak about
broader cancer issues.
Some respondents felt there were not enough advocates involved directly in prevention. There
needs to be an interest and willingness to advocate prevention.

Understanding of Science
Understanding science was the third most mentioned skill or kind of knowledge desired in a
CARRA member. This complex issue elicited mixed comments. The desired levels of and importance of
understanding science varied. Respondents focused on 1) just knowing science, terminology, basic
concepts, and 2) knowing how science is done—the research process and how NCI functions.
Institutional Review Boards were mentioned often, as at least three respondents were required to use
consumer advocates in peer review.
Terminology, Concepts
Some respondents, mostly non-users, indicated that basic scientific concepts, terminology, subject
matter knowledge, or scientific method being employed are important skills to have. One respondent said
that regardless of being involved in science, per se, the understanding of science would be good
knowledge. Non-users more commonly suggested the need to have some knowledge of scientific terms,
subject matter, and method.

How Science is Done (IRB, Function of NCI)
Given that many of the respondents are involved in scientific research, the idea of “how science is
done” and the research process itself surfaced often as a suggestion. Advocates should know the function

45

of the local IRB and have prior IRB experience as well as the knowledge and appreciation of
confidentiality. They might also have an understanding of the concept of prevention and participating in
a prevention trial. It was also suggested they know how clinical trials are designed and coordinated and
the stages of a clinical trial.
[Advocates] must have an understanding of science, have a feel for [issues such as] dose response.
Most of us think that “more is always better.” (Programmatic non-user, administration of nutrition
prevention programs)
Increase in didactic learning of a discussion on how clinical trials are structured, so that
advocates would understand, for example, why do controls. … controls are very important because
placebos can have powerful effects. (Science administrator, user)
A few respondents expressed the need for advocates to understand the function of NCI itself,
bringing in these bits of knowledge: the distinction between intramural vs. extramural; how we do what
we do; how funding, grants work; the competitive process; what do investigators do. One respondent also
thought work experience in science or a healthcare profession would be helpful.
In general, most respondents stressed overall knowledge and experience so that the advocates can
fit more roles.

Training for CARRA Members
Respondents were asked about the types of training they would recommend that OLA provide to
CARRA members to help members meet NCI staff expectations for advocates’ skills and expertise. Both
non-users and users of the program responded to this question. Respondents were also asked to comment
on a list of specific types of training prepared by OLA. This section describes respondents’
recommendations for training, as well as their reactions to the specific types of OLA trainings.
Respondents’ recommendations focused on the following training areas:
•

Basic research processes and general science training;

•

Specialized training;

•

Training NCI staff who use CARRA members; and

•

Suggestions for communication skills.

Basic Research Processes and General Science Training
Users and non-users indicated that it would be useful to train CARRA members about how the
research process works, underlying concepts of doing conventional trials, and interpreting evidence. In
addition, it would be useful to provide CARRA members with an overview of what the peer review
process entails. Respondents also suggested seminars on scientific lingo, jargon, the scientific process
and training in scientific and quantitative methods.

46

Overall, respondents said that some generic training was worthwhile (e.g., science terminology).
However, while science background is useful, respondents did not recommend training CARRA members
to be scientists.
Instead, they recommended training advocates to understand the science
process/scientific method—more specifically, how science works NCI or how things “get done here at
NCI” (procedures, policy). Armed with that, advocates can integrate and work from within.

Specialized Training
When respondents were asked to mention specialized training that might be useful, they
commented that different tasks require different skills. “It depends” was the most often mentioned
qualifier. For example, respondents asked why an advocate should train on review processes if he/she is
unlikely to do reviews. Rather, it might be advisable to have a diverse group of advocates, “a stable of
experts,” and then lay out what is going to happen, prepare members for upcoming assignments, and train
members for specific study sections as they are assigned.
The list of OLA trainings that NCI staff were asked to respond to included more peer review
training; human subject protection courses; how to serve on an IRB; negotiation skills, business etiquette,
etc.; technology-specific workshops; computer training; and how to conduct internet research. Peer
reviewing training and human subject protection courses received the most support.
More peer review training. Users and non-users liked this training idea the most, though more
users agreed. Still, even among these respondents there was emphasis that the members already get this
training and do not need too much more.
Human subject protection courses. Human subject protection courses and how to serve on an
IRB were frequently supported, and were also considered hand-in-hand skills. Some respondents
mentioned these two types of training without prompting because these were the areas in which they used
CARRA members exclusively. They expect that members have such training, but recommend it if they
do not.
Negotiation skills. About one-quarter of respondents thought this would be a good training point,
but they did not agree that it was always pertinent. A few respondents said that members already had this
skill.
Technology. Any suggestion related to technology and computer training was least important to
NCI staff. While these were good skills to have, training in this area was not a priority. However, such
training was suggested for those members who are not computer savvy especially because of electronic
submissions and other electronic-related project work. More users than non-users agreed with this
suggestion.
One programmatic non-user offered these concluding, general remarks:
The gestalt of it all, gestalt of everything that is being done—it involves training, leadership,
clarification of the role, getting the word out, engaging people so that people will start thinking
about them (the members).“ (Program administrator, non-user)

47

Train the Staff That Use CARRA Members
Some suggested that it would be worthwhile to train NCI staff how to effectively use CARRA
members. That is, educate NCI staff about how CARRA members can be used by increasing awareness
of the CARRA program, the roles that CARRA members can play, and the training that members receive.
Moreover, educate NCI staff about the expectation of the CARRA members and teach NCI staff to treat
advocates as members of the team.

Suggestions for Communication Skills
The following training ideas were offered: provide training in health literacy and numeracy to
increase level of sophistication; provide training on how to provide evaluative input, not just didactic
learning; use case scenarios and mock study sessions (sit on a panel, listen, and then provide an opinion;
know when it is appropriate to speak) to train; and provide training in cooperative groups and outreach
activities.
Most respondents recognized the value of advocates in communication but not all saw their value
in research projects, especially basic science. Several scientific users expressed frustration that CARRA
members did not have an understanding of science and that rather than representing the patient side, they
were trying to take part in scientific discussions. Among scientific non-users, there was mixed reaction to
the skills and training CARRA members might receive. As with users, those who were more negative
about CARRA did not want members involved in the research process. Those who were more positive,
however, recognized that members needed an understanding of science, a background in cancer, and
familiarity with NIH and NCI, and felt that members could obtain knowledge through training.
A few respondents (users) indicated that the current training was adequate and they would not
change a thing. One respondent mentioned seeing CARRA members as having these skills through
training more than did non-CARRA-trained advocates. Some respondents, however, did not know what
training CARRA members received, wanted to be informed of that training, and even expressed interest
in being involved in the training.

Barriers and Facilitators
Lack of specific skills and training may already be barriers to including an advocate in staff work at
NCI. Respondents did not always interpret barriers to mean structural, organizational barriers, but more
literally as impediments that would stop a consumer advocate from participating.
The findings reflect lack of skills and more generally deeply rooted feelings in NCI about involving
advocates. Several clear themes emerged and are described below.

Barriers
Lack of awareness of the CARRA program. Respondents indicated that there was insufficient
knowledge about who exactly CARRA members are, where they are from, how they are identified, and
what their value to a project would be. To overcome this barrier, educating potential users and marketing
the program were suggested.

48

Educate them [investigator] as to the value of having more involvement of consumer advocates.
Bringing a greater awareness to the value and importance of their involvement. It is important to
the institute, and to the individual projects. (Scientific, non-user)
We all have intuitive sense that it would be good to have a consumer perspective, but the devil is in
detail. (Scientific, non-user)
Lack of clarity on the program’s mission. Respondents felt that there was a lack of clarity about
the CARRA mission. The program appears to have a dual purpose: (1) creating good will with advocacy
organizations, and (2) reaching out to grass roots. The stated goal does not match the reality. This lack of
clarity might be solved by training and communication.
Lack of understanding about how advocates can “help me accomplish my work.” This barrier
goes hand in hand with program definition. If one does not try using an advocate, one may not realize
how much he/she can contribute. It was suggested that this barrier might be overcome by opening up
other applications to review by CARRA members since everything that is done by NCI is an outcome for
public health. Also, educate the investigators as to the value of having more involvement of consumer
advocates.
If they had concrete specific ideas for how to engage them, how they can add value to what we are
doing, I think people would be attentive and appreciative. (Programmatic, non-user)
Lack of understanding how NCI can support the work of advocates. One respondent felt that it
was not office leadership and culture discouraging using CARRA members, but rather that “We don’t
have the mechanism to bring advocates to the office,” (Programmatic, user). Another staff member
indicated that CARRA members would have to be invited by the Center for Scientific Review to
participate in peer review. This person wasn’t clear on the legality of it, and assumed there would be legal
barriers to such involvement.
Lack of time. Lack of time is a problem for potential users as well as for CARRA members.
According to one programmatic user, it is time-intensive to sift through potential members, so “if there is
a time crunch, it slows you down.” It takes a lot of time to process comments [from OLA on each
advocate]. From the advocates’ perspective, many have full-time jobs, so they do not have a lot of time.
An incentive for advocates to spend more time might be to provide them with compensation such as
honoraria.
Lack of funding. It would be helpful if the Institute provided travel funds. One respondent felt it
was not cost-effective to have a CARRA reviewer travel, stay in hotels, and have input from him/her on
cancer therapy trials only.
[A barrier is] putting CARRA members on the review panels. But, CARRA members don’t add
much to the peer review. The cost of flying those people, paying for their training, etc. is high, and
it is not economical because they don’t add too much value to the reviews. So, is NCI getting a
bang for its buck? (Science administrator, user)
Lack of scientific background. This loudly echoes one of the major concerns of the value of using
advocates. “Advocates are not scientists…” One respondent, a science administrator, user, felt there may
be “fear of having to stop and explain things to advocates; elitist attitude. They don’t know the science

49

that well, so they make comments extreme in either direction.” Again, this barrier might be overcome by
educating the advocates in scientific language and terminology and training them about their appropriate
role in the research process.
There are often controversial trials being conducted at NCI and so involving CARRA members,
who find the disease to be extremely personal to them and emotions may run high, want treatments
approved before they are proven 100% effective. (Scientific, non-user)
It helps to have the consumer advocate coming from the science backgrounds, or at least personal
interest in some of the medical background. However, when you choose this kind of people for
advocates, they want to talk about science. So it’s a Catch 22. If you bring people from those
fields, they feel compelled to talk about things that they are not really asked to speak about.
(Scientific, user)
[It’s] not a good idea [to involve consumer advocates in research process at NCI.] Research has to
be directed primarily by scientists. In any of the reviews that they have advocates, they would
comment on inane issues. Involvement of advocates may make things too touchy-feely, therefore
the selection process is extremely important to get a technically competent person. Those seem to
work out best, and then people steal them. (Science administrator, non-user)
Lack of focus on what advocates have been tasked to do. Some respondents were skeptical as to
advocates’ goals in participating in activities. They have found that consumer advocates are advocating
for funding instead. Advocates give unrealistic scores in reviews and have shown bias, which loses
credibility for consumer advocates altogether. One respondent noted that each advocacy group pushes for
its own special cancer, which results in them competing against each other for limited resources. It would
be helpful if various advocacy groups could coalesce around the broader topic of cancer.
Sometimes, the idea of a consumer advocate could be a negative from the scientist’s perspective
because the advocate may come with their own agenda, their own preconceived notions. Just as
they could bring positive press, they could bring negative press. (Science administrator, non-user)
Open communication in reviews. Some scientists feel they must monitor what they say when
there is an advocate in the room; consequently, they do not want one there. In turn, an advocate might be
unwilling to speak up frankly in a meeting.
They have been vetted, they have the skills we need, but will they really speak up? (Programmatic,
non-user)
Other barriers mentioned include health of the advocate, willingness of advocate to get involved in
prevention, need for a bigger pool of advocates, and use of the term “advocate.” Regardless of the
barriers, many respondents recognized that advocates should be involved in the scientific review process.
I think that we realize that we are supported by taxpayer dollars, and the consumer has the right,
that we have accountability. I don’t feel that advocates shouldn’t be involved. (Scientific, user)
This is about the most high powered science review that there is, costing some of the biggest bucks,
and we have consumer advocates with us every step of the way. (Scientific, user)

50

Facilitators
While barriers seem quite prevalent, respondents listed some ideas that would facilitate the use of
consumer advocates in activities at the NCI.
Top-down support. If someone in authority supports having advocates, then it filters down. It is
important to get upper management to support advocates. As one programmatic user put it, “If the
director tells you to do something, you do it. Convince upper management.” An example of where it
would be useful to have top-down support is as follows: Applications being reviewed in the Center for
Scientific Review are discouraging the use of advocates—CSR feels that people who are not scientists
should not be involved. A solution to this might be facilitated by applying subtle pressure to CSR from
NIH management. Top-down might be facilitated by writing a memo to staff about when is it appropriate
to use CARRA members or issuing a policy requiring CARRA members on each committee for the
extramural program at NCI, including everything (technical, small business, etc.). Such a policy would
protect NCI from the complaints coming from taxpayers.
Examples of how others have used advocates. The OLA needs concrete, specific ideas for how to
engage advocates and how to add value to what NCI staff is doing.
Marketing. When prompted about making presentations, a few respondents specified that OLA
should make a presentation in front of leadership every few years. Within branches, divisions, and other
entities of NCI, the senior staff also needs to disseminate that information to junior staff and PIs.
Good experiences with former/existing advocates. NCI staff ought to convey to scientists that
consumer advocates add value at every step of the process, during research process and communication
process, as well as during grant reviews.
Funding. It would be helpful if NCI provided travel funds for CARRA members.
Education. Members might come from science backgrounds or have interest in medical
background, but they should know when it is appropriate to talk. One respondent stated that once the
advocate goes through the process of reviewing, they get used to talking to scientists.
It is apparent that use of advocates would be facilitated by making sure that advocates see the big
picture, which can be addressed in training. Further, it is essential that CARRA members are looking out
for the best interests of the entire cancer population. One respondent said ideally, NCI should choose
people outside the system whose involvement is totally altruistic.

Marketing Mechanisms
Respondents were asked about potential marketing strategies that OLA might employ to encourage
NCI staff to learn about and ultimately use the CARRA program. Respondents were first asked to offer
ideas for potential marketing and/or communication mechanisms that “would work for them” if the
CARRA program were to conduct an outreach to the NCI staff. After recording their spontaneous
responses, staff members were asked to react to a list of seven marketing strategies prepared in advance
by OLA: posters or standing displays about CARRA, items that would increase name recognition such as
pens/pads of paper with the CARRA logo, presentations from OLA staff, articles in NCI Bulletin about
CARRA, reading/hearing success stories of advocate involvement from other NCI staff, recognition from
the NCI Director for the use of advocates, and encouragement/support from leadership.

51

The answers to the open-ended request for marketing and communication recommendations
suggested that OLA’s marketing efforts should be designed with three goals in mind:
•

Raise awareness of the CARRA program by making NCI staff aware that the program exists.
Several respondents—both users and non-users of the CARRA program—indicated that they
had not seen the program advertised. According to one programmatic user, “[The] CARRA
program is very unknown. Their function inside NCI is unknown.”

•

Clarify and provide concrete examples of how the advocates can be used in the work of
particular units, how they fill a specific need, and what are the positive impacts that they can
bring into unit-specific programs.

•

Encourage and help NCI staff (especially those that need advocates) to reach out to the
CARRA office, and engage CARRA members.

The desirable intensity of a marketing campaign was also commented upon. As one programmatic
user said, “I don’t think success should be defined by blanketing everybody using a shotgun approach. If
you have key offices in NCI that use CARRA effectively—this is a sign of success. If CARRA has
influenced the content areas, that’s enough!”

Respondent Recommendations for Marketing Strategies
The following suggestions were made about how to market the CARRA program and achieve the
three goals listed above.
Presentations by OLA staff. Face-to-face dissemination methods were identified by the
respondents as the most desirable marketing tool. Said one programmatic user, “CARRA is too
multifaceted of a program to just say, ‘here’s CARRA, did you know?’ Better by personal connection to
understand what it is. ” NCI staff members considered in-person presentations to be the most valued not
only because such methods of communication have the strongest impact on the audience, but also because
they would allow the CARRA message to be tailored and customized to specific audiences.
Such customization of the messages was seen as one of the crucial benefits of the in-person
CARRA outreach, since the contributions of consumer advocates can be diverse and wide-ranging, with
some dimensions more appropriate to particular audiences. Moreover, it was stressed that NCI staff
members do not respond well to generic, one-size-fits-all pronouncements. In-person presentations allow
for suggestions on how to engage advocates in one’s own work, and can be very specific to the area of
interest of the targeted unit. Consequently, the CARRA program needs to be aware of the specific needs,
processes, and projects conducted in various units of NCI.
One suggestion for OLA is to have strategic meetings with those who have used CARRA or who
think that they might use CARRA, and be more strategic in who they target, and come back later
and have reinforcement conversations with people developing content, or reviewing protocols for
studies. Meet with them and say “we’re still here”; ask what else they can do, and stay focused on
specific needs instead of going broad. (Programmatic, user)
Respondents suggested that specifically targeting division directors and upper management might
be beneficial, since “if you convince them, then they will tell their staff to use CARRA,” according to one

52

programmatic user. Additionally, one of the respondents pointed out, different messages may be needed
for the extramural versus intramural, scientific versus programmatic users. Overall, presentations provide
opportunities to answer the most important question, “What’s in it for me?” After conducting such inperson presentations, OLA staff would leave the participants with concise take-away materials about the
program.
Respondents shared several ideas for the most appropriate settings for such in-person presentations.
Among them, presentations at regularly scheduled meetings, such as division meetings, all-hands
meetings, program meetings and brown bags were mentioned most often. Only one person indicated that
such regular meetings might be too busy, and that having a separate meeting devoted to CARRA might be
a more productive approach. Additional ideas of receptions sponsored by CARRA in collaboration with
group or branch leadership were offered. Several respondents also suggested that CARRA should be
mentioned in Director’s Lectures, and other educational seminars where different cancer topics were
discussed.
Written communications. Written materials were considered effective as reminders, but not as the
main communication vehicle. Generally, there was more support for articles in the NCI Bulletin than
other vehicles. Respondents suggested introducing a regular column to the NCI Bulletin authored by
advocates. However, one respondent commented that each new issue of the NCI Bulletin used to be
introduced to the staff via an e-mailed reminder, which was very helpful in keeping everyone up to date
with this publication; however, those reminders are not sent out anymore, and it is easy to forget to go to
the NCI website and thus to read this publication. For this reason, reinstating electronic remainders for the
Bulletin might be advisable. Other recommended written vehicles included one-page hard copy bulletins
about the CARRA program to be delivered to in-boxes, a weekly debrief from the Director’s office, and
articles in the NIH Record. Additionally, several respondents mentioned that e-mail messages could be
sent periodically as reminders.
In general, all written materials should convey to the NCI staff illustrative, real-life examples
(“case studies”) of successful inclusion of consumer advocates in individual projects to “promote
awareness of all the positive reasons to have an advocate.” Those articles need to be hands-on, and should
present success stories where involving consumer advocates really made an impact. They should also
highlight stories that would let the NCI staff know that they do not work with advocates in a vacuum.
Several ideas were provided for such materials, among them the following:
•

Lists of concrete examples of effective consumer advocates showing relevance of their
involvement in the particular projects of individual units at NCI. These materials should
demonstrate how advocates helped solve particular problems that faced the NCI staff.

•

Lists of concrete examples of effectively integrating consumer advocates into the research
process.

•

Lists of critical substantive skills that CARRA members have, outlining how those skills are
relevant to tasks in which they might be involved.

•

Portfolio of CARRA members with biographical sketches and personal/professional skills
outlined, which would serve as a reminder that advocates are more than survivors. Many of
them hold professional positions and have backgrounds that could potentially contribute to the
NCI projects (examples were provided of advocates who were also experienced lawyers, IT
professionals, and sociologists, who were able to contribute their professional skills to the
projects.)

53

Reactions to OLA List of Marketing Strategies
As noted above, respondents were asked to react to a list of seven potential marketing strategies.
Respondents had raised some of these strategies themselves while making marketing suggestions. Their
reactions to the OLA list are summarized below.
Posters and standing displays and brochures. The majority of respondents agreed that posters
and standing displays would not be a good marketing tool, particularly on the NCI campus, where two of
the three buildings are leased and where there are severe restrictions on the types of materials that can be
displayed on the bulletin boards. Some respondents candidly confessed to ignoring posters and banners
already on display.
Branded giveaways, such as pens and pads of paper with CARRA logo, to increase name
recognition. In general, respondents did not like the idea of branded giveaways. As one programmatic
user phrased it, “I don’t feel we need to turn into a salesman type of thing.”
Presentations by OLA staff. As described in the previous section, presentations by OLA staff,
particularly in-person presentations that are tailored to a specific audience, were widely endorsed.
Articles in the NCI Bulletin about CARRA. Respondents generally liked the idea of articles in
the NCI Bulletin, but noted that because the Bulletin is only available electronically and reminders about
its availability are no longer sent out, that it is not the best vehicle for marketing.
Reading/hearing success stories of advocate involvement from other NCI staff. Respondents
strongly endorsed sharing concrete examples of advocate involvement as a way to market the CARRA
program and educate NCI staff about how CARRA members can be used in their projects. One
respondent suggested reaching out to the NCI community to collect directly from NCI staff real-life
examples of the effective use of advocates.
Recognition from the NCI Director for the use of advocates. Respondents recognized that
recognition from the NCI director would help increase awareness of the CARRA program and encourage
use of CARRA members on projects.
If CARRA has an in road with the Director, get his blessing, and make sure there is an advocate on
all high-level advisory boards. Would be nice if he would send out a memo saying, “from this day
forth, you should have 1–2 advocates present at each one of your meetings or workshops”—– like
an honorary advisory member to each one of these specialized programs; that would be the best of
all worlds. If that doesn’t happen, the next level down is the Executive Committee, made up of all
the division directors—do a presentation and “wow” them/convert them to the importance of
having an advocate serve. (Programmatic, user)
Encouragement and support from the leadership. Most respondents, particularly those who
have used the CARRA program, indicated that the use of advocates is encouraged from top down.
Respondents also recognized that more top-down support would promote a wider use of consumer
advocates in the NCI projects.

54

Recommendations for CARRA
Strengthen CARRA website. Several respondents noted issues with the quality of the CARRA
website. It might be useful for CARRA to carefully review its website to ensure that it accurately
represents CARRA’s mission, describes the benefits of using CARRA members, and provides some of
the information that respondents are requesting, including more information on the background,
recruitment process, and training of CARRA members. The CARRA website might be one place where
success stories of using advocates on projects could be highlighted.
Engage NCI Staff to help CARRA develop materials. One respondent pointed out that CARRA
could ask NCI staff members to help develop materials for advocates and the CARRA office or to provide
help with training. Calling on the skills and expertise of NCI staff members would introduce them to the
CARRA program, establish a personal connection to it, and ensure that promotional materials are targeted
to the needs of NCI staff and that advocates are trained in a manner that is useful to NCI staff. One of the
users of the CARRA program volunteered to observe the training sessions and provide CARRA with
feedback on what additional training might be necessary.
One staff member suggested that OLA have strategic meetings with those who have used CARRA
or who think that they might use the program, so that the office can become more strategic in whom they
target. Later, return and have reinforcement conversations with people developing content or reviewing
protocols for studies.
Market CARRA beyond NCI. Two respondents suggested that the CARRA program should be
marketed to the extramural community beyond NCI. First, respondents advised CARRA to bring a booth
or exhibit to some major national cancer meetings, such as the annual meetings of the American
Association for Cancer Research, American Society of Clinical Oncology, and American Association for
Clinical Research. One person suggested making presentations at universities. Respondents also offered a
suggestion to incorporate learning about CARRA program to training for the extramural staff.

Section III.
Findings From NCI Staff Focus Groups
Following the in-depth interviews with NCI staff, three focus groups were convened to explore in
more detail some of the interview findings. Participants in the focus group were:
•

NCI staff members who have used the CARRA program;

•

NCI staff members who have not used the program; and

•

Senior-level leadership staff at NCI.

Eight participants were recruited for each group with the goal of having six to eight participants in
each session. All but one of the recruits participated in their respective sessions. Direct quotes are
associated with the focus group session in which they were made.

55

The focus group discussions addressed two key topics. Marketing messages were defined as value
statements describing the use and utility of consumer advocates in the work of NCI. Marketing
techniques were defined as communication mechanisms for reaching NCI staff with messages concerning
advocates in general and CARRA in particular. Two handouts that summarized findings from the NCI
staff interviews on the topics of the perceived value of advocates and the efficacy of suggested marketing
strategies were used to help structure the discussions and gauge participant reactions (see Appendix H).
In addition to the two primary topics, participants were asked questions to assess their familiarity
with CARRA and were provided an opportunity to describe CARRA in their own words. Members of
NCI leadership group were asked to share their thoughts on the structure and culture of NCI and how
these might promote or hinder staff members’ use of CARRA advocates. The focus groups concluded
with a discussion of the term “consumer advocate.”
This section summarizes participants’ views on the value that CARRA advocates offer (i.e.,
marketing messages) and on the best ways to communicate with NCI staff about CARRA (i.e., marketing
mechanisms). The former can help in the development of effective promotional material about consumer
advocates and the CARRA program. The latter can inform the design and implementation of effective
mechanisms for communicating messages about CARRA within NCI. Both the marketing messages and
the marketing mechanisms subsections conclude with a summary of additional information of relevance
to CARRA revealed by the focus group discussions.

Marketing Messages
Participant Views on the Value CARRA Advocates Offer
As noted above, participants in each group were presented with a document listing the potential
value that CARRA advocates bring to the work of NCI (Exhibit 1). The moderator used this list to guide
the discussion and generate feedback that can be used to craft marketing messages that will resonate with
NCI staff. Participants were invited to write notes on this handout and these were collected at the close
of each focus group session.
Several participants in all three focus groups made clear that NCI as an Institute values the
viewpoints of people external to NCI who are affected by cancer. All three focus groups took pride in
that fact. Discussions with NCI leadership, in particular, highlighted this pride. In fact, one manager
described NCI as a leader among the NIH Institutes in terms of seeking and considering the viewpoints of
patients and advocates. The CARRA program is one of the ways that NCI demonstrates a commitment to
including the views of people external to NCI affected by cancer and is valued for that reason.
I think most folks here at NCI accept the underlying general principle of the CARRA program… the
initial fundamental issues about whether involving advocates is a useful activity have been
resolved. Now it’s more about implementation and making people more aware of things like the
selection process and the briefing process for advocates. (Leadership Group)
We have offered some criticisms, but I want to say that CARRA is a good and useful program. We
are ahead of the other Institutes. (Leadership Group)
I wanted to say that this is an important program. These people have passion. They bring
freshness to a project. These are our jobs every day and sometimes we get stuck in the mundane.
(CARRA Users Group)

56

Look for ways to invest in this resource [CARRA]. It is better to have them than not to! That
would be chaotic. Any resource needs investment. (CARRA Users Group)
Exhibit 1.—Potential values CARRA advocates bring to the work of NCI
•

CARRA advocates convey the patient perspective.

•

CARRA advocates provide powerful reminders to scientists about why scientists do what they do—
especially in cases where basic scientific research activities may feel far removed from healing
human cancer.

•

CARRA advocates provide grounded “reality checks” about how patients might respond in
research and product development.

•

CARRA advocates can actively help identify issues of human subjects protection because they are
particularly attuned to such issues.

•

CARRA advocates bring valuable professional skills beyond their personal survivorship
perspectives.

•

CARRA advocates are poised to disseminate quality information in their communities based on
what they learn from their experiences at NCI.

•

CARRA advocates can take on roles that the government cannot, like using what they learn in
lobbying and fundraising efforts.

•

CARRA advocates come to better understand NCI and the scientific process and are better
equipped to then accurately discuss those topics with others in the community.

Participants highlighted both directly and indirectly that the core value that CARRA
advocates offer is that they bring the patients’ perspective to the table. If participants prioritized one
value statement above the others, it was this one. This value was said to underlie all of the others
presented on the list that participants reviewed. They did not limit themselves to thinking of this
perspective as only a “patient” perspective; rather, throughout the discussions they also used varied terms
such as “non-scientific” perspective, “external” or “outside NCI” perspective, and “the public’s”
perspective.
I marked “patient perspective” as a strong positive for me. That is CARRA members’ main role.
It’s what we’re going to ask them to do. (CARRA Users Group)
CARRA is a resource of interested and informed individuals who can advise from a unique
perspective, a patient perspective. (Leadership Group)
Moderator:

How would you describe the CARRA program?

Participant 1:

It’s an NCI program that finds advocates to help you with projects.

57

Participant 2:

Not just help you with projects, also to broaden your perspective and get input
from people who are not just the scientific community. (Non-users Group)

We have health professional summaries and patient summaries. We want the CARRA member to
look at the patient summaries, because they’re written by us, and answer from a patient
perspective, “Does it make sense?” “Does it have what you need?” “Is it too technical?” “Does
it answer your questions?” (CARRA Users Group)
Participants had generally positive reactions to the statement that CARRA members can
provide “reality checks.” Most viewed the statement as just another way of describing the advocates’
ability to provide the patient perspective. Some participants used the term “reality check” unprompted.
For example, one participant said that CARRA members provide a reality check for his team when the
advocates review brochures. Others called the reality check function critical. At least one vocal
participant did not think advocates could deliver reality checks in a scientific context.
This point on reality checks is critical. On grant reviews I’ve managed, I’ve seen that after reading
consent documents and risks, CARRA members do verbalize whether the consent documents are
appropriate. (CARRA Users Group)
This list [of potential value] captures the original intent of CARRA, especially the reality check.
(Non-users Group)
This sentence needs a “reality check” because many scientific activities are beyond a CARRA
member’s skills. (CARRA Users Group)
Participants had mixed responses to the value statement, “CARRA advocates provide
powerful reminders to scientists about why scientists do what they do—especially in cases where
basic scientific research activities may feel far removed from healing human cancer.” For example,
one non-user was particularly drawn to this statement. She told the story of a scientist who was wishing
aloud at a professional meeting for larger samples of brain tissue to examine. A cancer survivor at the
meeting reminded the scientist that he was talking about people’s brains, and should be striving to leave
as much intact as possible. Conversely, a basic scientist in the focus group saw no value in such
reminders for scientists. Echoing this sentiment, members of the leadership group said that such
reminders are not particularly helpful. Rather, they said, scientists should be doing good science. The
jotted notes of the CARRA users group also reflected both of positive and negative opinions. The notes
included negative comments like “not really” written beside the “reminders” sentence and positive
comments like “great!” and, “I’ve heard this is true.”
Sometimes in practicing science, people forget. It helps the scientists remember that we have to be
respectful of human beings. (Non-users Group)
I don’t know if they really need a reminder. I think it’s already there. (Leadership Group)
On a certain level [such reminders] are irrelevant because what you need in the very basic
research is people who are committed to finding out the “why.” And, somebody else is really
going to take that forward and incorporate it…On a certain level [reminders can be a] distraction
because what you really want from the basic scientist is that they keep being interested and driven
by science and that they do good science. I think it’s only when you get sort of into the
translational level where the input of the patient and how this impacts the patient is more relevant.
(Leadership Group)

58

Participants agreed that CARRA members deliver value because they are particularly
attuned to human subjects protection issues. While participants agreed with this statement and
specified that CARRA members do a better job than many in reviewing human subjects protections, none
of the participants believed that that was the most important value offered by CARRA members.
In my experience, which has been in study sections, we’ve had pilot projects that CARRA members
have ended up reviewing. We’ve had very positive experience. One thing that they’ve been
assigned is not the technical information, but notions related to human subjects protection. They
end up doing that whole section on human subjects…they have a great expertise with regard to that
and most [other] people kind of blow that off. They’ve sort of cultivated that as their expertise. I
think it ends up serving a very valuable niche. My experience has been that they do that very well.
(Leadership Group)
On issues of human subjects protections, [CARRA members] really do take the time and review
quite carefully, as opposed to physicians on the review panel who may just look at it peripherally
because they’re more interested in the science. (CARRA Users Group)

Although participants agreed that CARRA members can bring valuable professional skills to
NCI work, they remained strongly focused on their questions about how they can know more about
the CARRA members offered to them and determine the best fit for a project. As described in the
subsection of this report on communication, this theme of how to know more about CARRA members in
advance and how to effectively match projects and the advocates best suited for them emerged in many
ways throughout the discussions.
I don’t know what valuable professional skills they can offer, especially if I don’t know someone.
(Non-users Group)
I have a new member who is looking at summaries for me. And, because he has a history as a
writer and an editor, he is picking up some good things in his reviews. This is a skill that is not
written in his CARRA bio, but he brought this up. (CARRA Users Group)
They are bringing writing skills and speaking skills and other skills they have developed in life.
That is valuable to the process. (CARRA Users Group)
Participants viewed the three messages that advocates 1) are poised to disseminate quality
cancer information in their communities, 2) can take on roles such as lobbying and fundraising that
government cannot, and 3) come to better understand NCI and the scientific process and talk about
those topics in their communities as being outside of their control. Although some participants
thought that these three messages were important, not everyone believed that these external activities
were happening often enough to make a difference. Overall, these benefits of CARRA were less
motivating for CARRA users, non-users, and NCI leaders because CARRA members cannot specifically
be asked or told to do these activities, rather they happen on their own naturally.
On this list these [three messages] are things that are byproducts, but we can’t ask them to do
these. I know [my group] would love to have people lobbying on our behalf, but that’s not
something we can dial up the CARRA list and say. (Non-users Group)

59

I marked “poised to disseminate quality information” as a positive statement. Maybe it’s wishful
thinking that CARRA members do this. But, it’s a gap we need to fill. I’m not sure how much of
this they do, but I think we can use a lot more of it. (CARRA Users Group)
In terms of taking on roles that government cannot, I’m not sure that CARRA members are really
doing that. It’s not a negative. It’s a role I wasn’t familiar with. (CARRA Users Group)
The last three [messages on the list], I see those as communication with the outside world and I
think that’s great. (Leadership Group)

Additional Views on the Value CARRA Advocates Offer
Participants prized the fact that CARRA members are vetted and trained. In fact, many
spontaneously mentioned the vetting process early in each group during open-ended questioning on how
they would describe CARRA (in CARRA participant and nonparticipant focus groups) and questioning
on the fit between CARRA and the culture at NCI (in NCI leadership focus group). However, many
participants asked questions indicating that they know little of how CARRA advocates are actually vetted
and selected in practice. A handful indicated that more extensive vetting and training would be
beneficial.
There is a vetting process. So, not just anyone can self-declare [themselves to be a CARRA
advocate]. (Non-users Group)
I’ve used CARRA members since the early days of the program. I’ve always appreciated [the
program’s] ability to identify good people [to be members]. (CARRA Users Group)
Moderator:

One of the things I’m hearing [in participants’ top-of-mind descriptions of
CARRA] is that CARRA members are “prevetted” and “informed.” You put that
right up front in your descriptions.

Participant 1:

Well, that is the goal, but that has not been the experience. My understanding
when CARRA was set up was that there would be a group of individuals who would
understand the expectations and be knowledgeable about how NCI or NIH
functions…that they would have a good idea of what the role of the advocate would
be…that they would be vetted to the point that they would be able to represent the
patient perspective and not their own individual perspective. (Leadership Group)

Definitely continue the training programs. CARRA had a mentorship program where a former
advocate would work with a new advocate. I’ve heard that was a useful tool. (CARRA Users
Group)
In a related theme, the “goodness of the fit” between a CARRA member and the job he or she
is given was of paramount importance. This issue was raised spontaneously by participants in every
group. The managers group went so far as to recommend that each CARRA member develop a few niche
skills (e.g., grant review, clinical trials work). Similarly, the CARRA users group highlighted the
importance of being able to work again and again with the same member to help that person develop a
particular skill set and knowledge base.
I want to follow up on something [Name] said on the fact that some of the advocates are really
good at what they do and others aren’t. I wondered what kind of a role there is for OLA to do

60

additional screening or guidance in terms of which [CARRA members] would be good for which
assignments. (Leadership Group)
It’s unrealistic to assume that all CARRA members are going to learn everything about all of NCI.
I think part of it is the matching process where lay members kind of develop a niche or a focus
area. It might be ethics. It might be review in a particular scientific domain. It might be a larger
advisory role in a particular program. It might be on a particular project. (Leadership Group)
You may want people who are clearly in advocacy positions. Or, you may want people who just
represent the patient perspective. Or, you may want people who just represent a different scientific
perspective…If [CARRA members] know that they are volunteering or participating in one piece,
then it’s also easier to attract a broader group of people because the expectation for them is not
this overarching thing. Given how overwhelmed everybody is, having just one small piece is a
whole lot easier than this broad amorphous thing. (Leadership Group)
The training gives them the mechanics of [peer] review. But, getting into it, it would be so nice to
use the same person over and over so that you can really train them. It takes a person more than
just one time through to be able to really comment. I think you can ask for the same person, but I
think CARRA should make it a point. (CARRA Users Group)
When I look at a list of CARRA members to review a brochure, I look to see if they have reviewed
in the past. (CARRA Users Group)
Participants in all three groups said that they would appreciate having more detailed
information about each CARRA member including information about personal and professional
background, key skills, and other work the person has done with NCI. Not only would these
participants like to be able to read this information in some form such as a detailed biosketch, but they
would also like to be able to contact other NCI staff who have worked with a particular member to learn
more about the member. Current participants in the CARRA program acknowledged having received
biographical information, but said that they wanted more detail. All of the groups acknowledged that
privacy matters may limit the ability of OLA to provide such full detail, or at least would keep OLA from
being able to post such detailed biosketches online publicly.
If you recommend an advocate to me, can you tell me about work they’ve done in the past so that
I can get some background? (Non-users Group)
I think there are levels of privacy where you can’t just search around online to get the CARRA
list. If you could profile people in some way so you would know what you’re getting… but you
can’t do that. There are too many privacy issues. (Non-users Group)
Moderator:

In these CARRA bios, is there enough information now?

Participant:

No, no. I think they’re somewhat skimpy. (CARRA Users Group)

Although advocates’ presence and advice overall is valued, these discussions highlighted the fact
that CARRA members’ contributions are most prized when they are able to help get something
done that needs to be done (e.g., reviewing protocols for human subjects matters) or help make
someone’s job easier (e.g., reviewing materials written for the general public, helping to anticipate public
reaction to a press release).

61

We develop materials and we’re going from a science background…To have a brochure and be
able to ask [a CARRA member] whether people will be receptive to it before we get to the pilot test
phase, that’s a very important step and value to help make sure we’re in touch with reality.
(CARRA Users Group)
There was a press release on a… profiling tool and we thought there would a lot of questions
afterward. We wanted to make sure we got the patient perspective so we worked with CARRA to
set up a teleconference. (Non-Users Group)3
I didn’t want to say this in the group because it doesn’t sound nice. But, if you can demonstrate
that CARRA members are like help or a service that you can get for little or no cost to your project,
that would be appealing. (Leadership Group)

Although almost all participants agreed that a patient/consumer perspective matters, several
participants offered examples of instances in which CARRA members should not be included.
Examples of these are situations in which confidentiality is an issue and situations of “intense scientific
activity.”
There are some activities that are intensely scientific. For instance, we have some working groups
that are plowing through the literature on some marker. It’s only when we begin talking about
how we’re going to move that marker into a clinical trial or into the community where then you
definitely want to hear from the advocates…But, the intense scientific part of it doesn’t really
benefit from having advocates’ input. (Leadership Group)
With what I do, I do not see where [including an advocate] comes into play. Now, I’m open to
negotiation on that. But, I am more at the “discovery” end of the drug development system. (Nonusers Group)
We deal with industry, academia, big biotech, small biotech, nonprofits, and governments. It’s
confidential information. If I open my mouth I totally destroy the program. Confidentiality is a
major issue. If you’re funding [the participating organizations], you’ve got them reasonably welltrained on confidentiality. An advocate has no strings. So, you may have a ticking time bomb.
(Non-users Group)

Challenges to the Perceived Value of CARRA Members
A top concern among participants was whether they might obtain the services of a member and
then discover that the person is disruptive to the process or difficult to work with because he or she comes
to the table with an agenda and focuses solely on that. This concept of one’s “own agenda” was
spontaneously raised in each of the three groups, with three participants indicating that they had
personal experience with this type of situation arising in their past work with a CARRA member.

3

Participant 1:

It is a total crapshoot as to the quality of the person, their agenda…

Participant 2:

You can have one person that just doesn’t get it who can completely tank any
opportunity that you have to fund anything. (Non-users Group)

Note that members of the non-users group may have used CARRA in the distance past, but not within the past year.

62

Participant 1:

In the process of reviewing grants, there is an issue when you have a CARRA
member as a full member because they get an equal vote to everyone else. There
are instances I have run into recently of the CARRA member voting with their
heart and not with their brain. While this doesn’t skew the vote a lot, it does
skew the vote.

Participant 2:

It is rare, but it does happen.

Participant 3:

To CARRA’s credit, they do ask after [a CARRA member participates on a grant
review panel] about the ratings…

Moderator:

About the performance of their members?

Participant 3:

Yes. At least, I know they used to do that.

Participant 4:

They still do.

Participant 5:

We got a little of that, but we can take care of it with education [of the CARRA
member]. (CARRA Users Group)

Some CARRA non-users said that they have established their own network of cancer
advocates whom they know and trust to provide the same kinds of input into communications that
a CARRA member would. These staff members do not feel a need to access CARRA members except
in rare cases when their own network is not filling the role they need. Conversely, several CARRA users
saw it as redundant to find their own advocate resources when they could use the CARRA program.
I think one of the reasons we don’t use CARRA is that our organization focuses a lot on developing
partnerships. We have partnerships with advocacy groups, so it doesn’t make sense to pull in
someone when we’ve got access to advocates as part of our partnerships. (Non-users Group)
Yes, I have a better relationship with the advocates in various groups. So, if I need someone I call
them and not CARRA. I don’t know anyone at CARRA…it seems duplicative when I’ve got
someone I already know. (Non-users Group)
When I think about what we do with large prevention trials… We don’t go to CARRA at all. We
look inside the trial for participants who are on the trial to serve on what we call Participant
Advisory Boards. We’re tapping into the voice of the participant on that trial. (Leadership Group)
NCI has invested a lot into the CARRA program. And, if we don’t use it, the alternative is to go out
and find your own advocates. And, that’s redundancy. (CARRA Users Group)
Where would other people find a good source of patient advocates? This is a resource that is in
place. (CARRA Users Group)

Two members of the non-users group said that they had difficulty finding CARRA members
with an interest in prevention. In their view, most members have shifted their focus to treatment due to
their own cancer experience. A member of the users group working in prevention said that he did not
have difficulty finding helpful CARRA members, but he also said that those interested in prevention were
in high demand and at times too busy to work with him.

63

We have used it less because the people on the list are, for the most part, cancer survivors, or
people with cancer, or people with an agenda about cancer, and not people at risk for cancer and
not helpful at advising us about prevention. The people on grant reviews have been destructive.
They have not been at all helpful because they have an agenda that is not consistent with
prevention. (Non-users Group)
We can’t get much interest from people on the list for prevention because they already have the
disease. They’re past that point. (Non-users Group)
We have a hard time finding CARRA members that aren’t already overbooked who have an interest
in prevention. (CARRA Users Group)

Marketing Mechanisms
Participant Views on Communication Approaches
As noted earlier, focus group participants were asked to respond to a list of suggestions that NCI
staff interviewees made about the best ways to communicate within NCI about the CARRA program and
its members (Exhibit 2). The focus group participants offered insight into the best communication
approaches or mechanisms and into content that would be helpful for them to receive from CARRA.
The moderator used the list of ways to communicate about CARRA to guide the discussion and
generate feedback that can be used to identify the most effective mechanisms for communicating
messages about CARRA within NCI. A summary of this feedback follows.
Exhibit 2.—Ways to communicate about the CARRA program
•

Face-to-face interactions were advocated more strongly than written or electronic approaches.

•

Written communication can be effective sometimes, but mostly as reminders.

•

Static strategies such as posters, displays, and brochures were not viewed as offering much value,
but could possibly be useful as reminders.

•

Branded giveaways such as pens or pads of paper were perceived as potentially lowering
credibility, although some said such material may support name recognition.

•

Support/encouragement from top leadership—Director, Executive Committee—in the form of
announcements or messages was said to be important, but not sufficient to move people to access
the CARRA program more frequently.

•

Hard copy and/or web-based support materials could help to encourage use of CARRA.
Suggestions included concrete examples showing how CARRA members have been effectively
integrated into NCI work and a portfolio providing biosketch details on each member’s background
and skills.

•

Allocating budget to engage consumer advocates would send a strong signal and increase use of
consumer advocates in NCI activities.

64

The theme that face-to-face communication is valued, which emerged in the NCI staff
interviews, continued strongly in the focus groups. In particular, non-users of the CARRA program
and NCI leadership emphasized the importance of face-to-face communication. They believed this
approach would work best if CARRA representatives were to join standing group meetings (e.g., division
or branch staff meetings), rather than booking other special CARRA-specific meetings. More
specifically, they thought that featuring CARRA as an agenda item at such meetings about once each year
would be sufficient. Non-users, especially, indicated that the presence of CARRA representatives would
serve not only to inform them about CARRA, but also to give them a sense of connectedness to the
program and the feeling that they could comfortably call a single, specific person for access and help.
CARRA users did not feel as strong a need for face-to-face contact. Rather, as individuals already
involved in the program, they seemed to see such contact as unnecessary for them personally. However,
one CARRA user emphasized that face-to-face communication was critical for him in that he got
involved in working with CARRA because of a face-to-face encounter.
We have invited CARRA staff to come and present at our Division’s All Staff meeting a couple of
times and that was just a real efficient, helpful way of getting all the staff familiar with what
CARRA does and what they have to offer. And, that was a result of CARRA presenting to the EC
and saying, “We’re happy to come to any Division or Office at any time and present what we can
do and explain what our function is.” (Leadership Group)
There is no point person. We don’t know what they do. So, if they came to a Branch meeting…it
would be a small enough setting that we’d get to know who the person was. (Non-user Group)
Participant 1:

Face-to-face is not necessary for me.

Moderator:

Because you’ve already made the connection to CARRA?

Participant 1:

Yes.
(CARRA Users Group)

I don’t think I would’ve gotten involved with CARRA had it not been for face-to-face. And, I’m a
strong advocate for face-to-face because I think that we do so much electronically. I met a person
[knowledgeable about CARRA] at an NCI exhibit at a national meeting. Then, I went through the
process of learning about CARRA. (CARRA Users Group)
Participants were not particularly receptive to the idea of brochures or other written
materials. There existed a general feeling that these materials would end up among a stack of other
written materials and that NCI staff would simply go to the website to find information they needed,
rather than unearth a brochure.
Written materials? I think there’s too much. We’re inundated. Twenty years ago you could have
written a memo and we would have read it. (Non-users Group)

Focus groups echoed the interviewees’ sense that support from NCI leadership for CARRA is
important, but not sufficient. Participants had already concluded that NCI leaders believe that externalto-NCI viewpoints provide an important balance and help guard against the Institute becoming too
insular. When participants were given an opportunity to comment on any of the statements on the page,
this item stirred little interest. Just one participant noted a desire for demonstrable support for CARRA
among the leadership, rather than merely statements of support or mandates.

65

Getting e-mail directives is one thing. But, if you see someone at a high level actually using the
program, pulling in a CARRA member, then it makes more sense to you. (Non-users Group)
The CARRA website was said to be an important stop for NCI staff wanting to know more
about the program. Participants in every group said that they had or would visit the CARRA website to
become involved with the program or to obtain answers to their questions. Participants had specific
questions and suggestions related to the site. For example, they wished that the site used examples or
suggested ways in which CARRA might be useful to them. In addition, they asked a variety of questions
throughout the groups (e.g., How are CARRA members selected?) that could be considered in future
website reviews undertaken by CARRA. (All of their questions are summarized in the upcoming
subsection Views Related to Communication Content.)
When I wanted to learn more, I just went on the web and typed in CARRA. (CARRA Users Group)
I think training of NCI staff is also important. We have so many new people coming in all the time.
I think information on [the CARRA] website—a template and a list of examples on how people at
NCI have successfully used CARRA members in various roles and lessons learned. (Leadership
Group)
Before the [focus] group, I visited the [CARRA] website. And, I still don’t know how it applies to
me. I comprehended it. But, the link to me was not clear. (Non-users Group)
I was told that we needed to do more outreach and involve advocates. I looked on the [CARRA]
website and I still didn’t understand how to really get someone involved. It was through personal
interaction [with CARRA] that I was really able to make the pieces fit. (CARRA Users Group)
How about website? How much information do they have on that? I haven’t been to it in about a
year. (Leadership Group)
Not only would allocating budget to engage consumer advocates signal NCI leaders’ support
of the program, but it would remove a barrier to using CARRA. Not all participants were receptive
to this idea, but most indicated that they would be more likely to involve CARRA members if costs such
as funding travel were not an issue.
If programs knew that there would be money so that, if you have to bring people to meetings and so
on, that there was an additional source for those funds from CARRA. Then, that would be a carrot.
(Leadership Group)
I could see where budget might be effective, especially in a time where budgets are so tight. So, if
I’m trying to get advocate input and I need to fly that person in. That’s a budget item that I would
like for the CARRA program to pick up, just because I’m trying to make the dollars in my program
go as far as possible. (Non-users Group)
I don’t think throwing more money is an answer. (Non-users Group)

66

Additional Views Related to Communication Approaches
Overall, participants strongly indicated that communication from CARRA should be regular,
but not frequent. This theme applies to both in-person and electronic/written forms of outreach.
Participants suggested visiting staff meetings annually and conducting reminder outreach annually or
biannually. It was suggested that reminder outreach may be better received if there is a reason for it (e.g.,
“visit the updated CARRA website,” “CARRA now provides more background information on
members,” “staff members’ top questions about CARRA answered,” “we’ve updated our member list”).
Current CARRA participants did not seem to feel as strong a desire for contact from CARRA. They felt
they were already involved and understood how to use the program for their tasks. Yet, most of them
acknowledged early in the discussion that they are not familiar with all the work CARRA does.
I suppose [outreach] could be done in a varied way—so that every half-year you just send around
an e-mail with a link…and, then, at alternate times send outreach in a different format asking, “Do
you need ____?” or “Have you thought about____?” (Leadership Group)
Moderator:

How often would you think a CARRA program person should visit a Division or
Office All Staff meeting?

Participants:

Annually. Every year. Every year or two years. (Leadership Group)

Yes, [reach out to say] “We’ve updated our website. We’ve changed. We’ve added.” Whatever it
is. (Leadership Group)
Many focus group participants raised issues of time. For example, one interviewee suggested
that CARRA not only provide NCI staff with information, but also continue to actively seek NCI staff
input and specific contributions (e.g., training CARRA staff) as a means of improving CARRA’s outreach
to staff. Although focus group participants responded well to this notion in principle, many were very
concerned about any activity that might take up time in their busy schedules.
The moment you say “training,” [as a way of giving input into CARRA], my blood pressure goes
up. We’re inundated with training. It’s so time consuming. (Non-users Group)
Moderator:

What about the idea of a small reception?

Participant 1:

I would never show because it’s outside of the time that I have.

Participant 2:

I’m agreeing [with Participant 1].

Participant 3:

I’m agreeing, too. A meet and greet, I can’t do to that because I’m trying to handle
the business at hand.
(Non-users Group)

I bet many people would say, “I’m too busy [for face-to-face communication].” (CARRA Users
Group)

67

Views Related to Communication Content and Resource Material
Participants said that real examples of the kinds of work CARRA members have done would
be helpful to them in envisioning ways that they could work with CARRA members. As reiterated
throughout this report, there was a strong demand for concrete examples as a way of both demonstrating
the value of CARRA and, most importantly, helping potential CARRA users to see how they could use
advocates’ services in their own work.
Instead of just having a presentation of what it [CARRA] offers, I think that knowing what the
outcomes [of collaborating with a CARRA member] could be would be helpful. (Non-users Group)
Participant 1:

If you know where [CARRA members] have been used…examples.

Participant 2:

Or, even testimonials.

Participant 1:

Exactly. On this [printout of CARRA website] there are no examples of how
CARRA members have been used. What have they done? Where have they been
successful?
(Non-users Group)

I think the value of the advocates really needs to be demonstrated for buy-in from the NCI
community. (Non-users Group)
Also, positive outcomes of some of CARRA’s activities as they relate to people in this quad [EPS,
EPN] and the Clinical Center are important to know. (CARRA Users Group)

Participants had a number of questions (e.g., “How are CARRA members trained?”) and
specific informational requests (e.g., more detailed biosketches). They suggested that such information
posted online would be helpful. They acknowledged that some of these answers may be provided
already, but it is helpful to know what questions the audience for CARRA messages has in mind.
Specific questions and topics participants mentioned wanting to know or to see on the website included
the following:
•

Concrete examples of how CARRA members have been tapped to help with NCI work and
succeeded (as a way of helping other staff members imagine ways CARRA members might be
able to help their work);

•

Outcome data or other information demonstrating the program’s effectiveness;

•

Details on how the CARRA membership list is created/how members are chosen, emphasizing
criteria for selectivity and vetting;

•

Details on how members are trained;

•

Information on how the membership list is updated and how often;

•

Procedures for procuring the help of a member;

68

•

Lessons learned or tips for the NCI staff member to use (e.g., what the CARRA member needs
to know from NCI about a task, key questions to ask CARRA members when contacting them
for NCI work); and

•

Procedures to follow if something goes wrong (e.g., member is not completing assigned work,
member takes a stand on own agenda to the detriment of the project overall, member has
reoccurrence of cancer during work).

Participant 1:

How is the member pool developed?

Participant 2:

There is an application process, a vetting process. They were being vetted through
DEA. That was about 5 years ago. I don’t know if that’s still being done.
(Leadership Group)

I would want more information on the advocates and how they were selected. Before I bring
someone into my meeting, I would want to have done my due diligence. (CARRA Users Group)
More information on the CARRA people would help, biosketches. It would more sensitive, but it
would be good to be able to get information on how individuals performed [on past tasks/roles at
NCI]. (Leadership Group)
One thing I’d like to see is outcome data. Some of the things the program can do seem nebulous.
What is the bang for the buck for me if I’m going to pull someone in? (Non-users Group)
One question I’ve been getting recently is, “What is the difference between DCLG [Director’s
Consumer Liaison Group] members and CARRA members?” I think there is confusion about that.
DCLG is a formal, advisory body of lay members some of whom have been in CARRA. That group
has become more active in the last year. (Leadership Group)

Other Relevant Topics
NCI staff who used CARRA members to review communication materials seemed to have
had more consistently good experiences than those who used CARRA members for activities that
required greater scientific knowledge and knowledge of NCI processes. In other words, participants
who were the most positive about CARRA had generally used the advocates’ services to review materials,
such as brochures, annual reports, and summaries about treatments, written for the lay public. Those who
used CARRA members’ services in peer review situations had more concerns about their scientific
knowledge, understanding of their own roles and NCI processes, and the potential that members would
serve their own agenda without regard for other factors.
We’ve run into some of the same difficulties that others have in terms of lack of scientific
knowledge [needed for tasks like] going through the literature. So largely, we use [CARRA
members] to review patient-oriented versions of summaries or publications. (Leadership Group)
In our office, we’ve used CARRA a lot, mainly for brochures and material review. We really rely
on the feedback from CARRA members. (CARRA Users Group)

69

Familiarity With CARRA
While focus group participants in all three groups understood CARRA and valued it, they did not
seem to have an awareness of CARRA as a dynamic, ever-improving, entity. For example, each group
raised many questions about how the program works, how effective it is, and the extent to which it is
monitored and strengthened with feedback. In addition, some focus group participants expressed a desire
for access to the research conducted on CARRA, especially outcome data about the work of CARRA
advocates. This desire was particularly strong for non-users of the program. Yet, the topic was raised by
users and leadership as well.
When CARRA was started, there was a great effort to inform the community about its existence and
what it could do for you. But since then…I’m not sure I’ve heard anything from CARRA over the
last 5 years. I have had essentially no communication, and therefore, I tend not to think of CARRA
when I’m trying to identify advocates. (Leadership Group)
The [CARRA] program started out. But, since then, nobody has heard whether or not it has been
updated, what the utility is, whether or not there has been feedback. Well, I know there has been
feedback from program staff, but I don’t know whether that has gone into a black hole. (Leadership
Group)
I don’t know whether the CARRA list is ever culled through. Do they revise the list? Do they
update it? Do they get feedback? Do the advocates get feedback from program staff in a way that
allows NCI to modify the membership of CARRA? We do this with peer review all the time. [We
say], “Oh, this person is really not an appropriate reviewer.” (Leadership Group)
CARRA is just out there. If I don’t think of them, they don’t get thought of. They’re not necessarily
pushing [information about CARRA] out there. (Non-users Group)
In terms of familiarity with CARRA, non-users varied widely. Current CARRA users
described themselves as fairly familiar with CARRA. Each of the two groups of NCI non-leadership
staff were asked to describe their level of familiarity with CARRA. They were asked to use a scale of 1
to 10, where 10 was “very familiar” and 1 was “not at all familiar.” In the group of non-users, two people
had never used the services of a CARRA member and assigned themselves a 1 or 2. Others who said they
were more familiar indicated that they had used a CARRA member for past work, but were not currently
familiar with the program. Among participants in the discussion group of current CARRA users, there
was greater familiarity with the program. All respondents said that they were a 5 or above, with the 5s
indicating that they felt familiar with CARRA but believed they did not know the full spectrum of
CARRA’s work so they did not choose a higher number.
We used CARRA members to review our annual report. But, I don’t really know the full spectrum
of what CARRA can do. (CARRA Users Group)
I’m familiar that basically CARRA is the source of names that I must go to. (CARRA Users Group)
I’ve used CARRA members for reviews since the early days of the process. I appreciate their
ability to identify good people. (CARRA Users Group)
Early on I was involved with the formation of CARRA, but have not directly used it more recently.
What I understood was to have a pool of advocates, or interested people, who could serve on
various forums where we needed the outside public, the educated public. (Non-users Group)

70

Use of the Term “Consumer Advocate”
In light of feedback from interviewees that the term “consumer advocate” may be troublesome in
some ways, focus group participants were asked to comment on it. Their responses were mixed, and no
one objected so vigorously to the term that they would not use or misunderstand the CARRA program.
Like the interviewees, several focus group participants preferred the term “patient advocate,” calling it
more precise. Yet, one believes that survivors, at some point, can and should stop thinking of themselves
as patients. While many agreed with the interviewees that the term “advocate” connotes someone
confrontational and focused on his or her own issues, they did not come up with a substitute word they
preferred. One manager suggested the term “representative,” as in “community representative,” “patient
representative,” or “public representative.”
I hate the term “consumer.” Consumer? In response to what we do at NCI? I just think it doesn’t
fit. I like the idea of “public representative” because it is bigger. (Leadership Group)
Whenever I write, I call them patient advocates. I don’t like the word consumer. I guess if it’s not
just patients you’re dealing with, that’s where the word consumer comes in. (CARRA Users
Group)
You could say patient representative or community representative. But, I think there are some
settings where using the term advocate can get in the way. But, in general, we live in a society
where there are advocates for all kinds of things. So having an advocate for the patient
perspective is not unreasonable. (Leadership Group)
I think NIH did it right when they developed COPR (Council of Public Representatives) when they
chose to say “public representative.” (Leadership Group)
Moderator:

The term advocate had some baggage in the interviews.

Participants:

Own agenda. Own agenda.
(Non-users Group)

I personally like the word advocate. I think of an advocate as a positive. I must be hand-picking
my advocates because I’m the only one who doesn’t have that negative feeling. I “match” the
advocates. That’s a better way to say it [than handpicking]. I once had an advocate come into a
meeting—a really contentious one where experts would disagree. I had no idea what her position
was on the issue. But, I knew that she would try to represent the community. She’s just a good
advocate. (Non-users Group)
You can put a negative connotation on just about any word. Somebody is always going to do that.
(CARRA Users Group)

71

72

CONCLUSIONS, IMPLICATIONS,
AND RECOMMENDATIONS
Through intensive interviews and focus groups, this phase of the CARRA program evaluation
yielded a considerable number of insights into the views about the program held by CARRA members as
well as NCI leadership and staff. While the cancer community and the research scientists aiming to
ameliorate cancer are the ultimate beneficiaries of the CARRA program, the NCI staff and the members
themselves are the direct customers of the CARRA program.
The perspectives presented in this report provide insights into how the program is experienced by
its customers, the value it represents to them, the benefits it offers, and the barriers to maximizing its
potential. Taken together, these findings and insights have numerous implications for continual
improvement of the CARRA program. A summary of conclusions, implications, and recommendations
follows, with special consideration given to identifying the most promising mechanisms for
communicating the value of CARRA advocates in the work of NCI.
Perhaps the most important overall message shared by CARRA members, NCI leadership,
and NCI staff alike is a general belief in the value of garnering the patient perspective in the work
of NCI. A minority of NCI respondents maintained that the purpose of engaging advocates in the
conduct of basic science was not as obvious to them as their role in translational research, clinical
research, communication, dissemination, and product development. Yet, the overriding message,
particularly from within NCI leadership, was a strong belief in the need for and benefits of viewpoints
that come from “outside NCI” that represent the “patient perspective.”
The responses shared by members, leadership, and staff shed light on many facets of the CARRA
program. These findings are summarized around two fundamental dimensions:
•

Ways that the CARRA program itself can be strengthened, and

•

How best to market the program, that is, how to communicate its current and evolving
strengths within NCI.

Program
Responses in interviews and focus groups reflect a generally positive orientation toward CARRA.
Members described their participation in CARRA with enthusiasm. Even among Low Active members,
interest in continuing and dedication to the program was high. Likewise, among NCI staff, most CARRA
users were pleased with their experience. For example, of the 13 CARRA users who responded to the
question, 10 said they accomplished what they had hoped to do by engaging a CARRA member.

73

Curiously, 10 out of 12 members could not recall or were not sure how they first learned about the
program. However, their responses, which typically listed multiple possibilities, suggest that the CARRA
program is well penetrated into the consumer advocacy communities through its various networks and
organizations.

Recruitment
Current members cited fairly consistent rationale for their interest in joining CARRA. Responses
converged around four themes:
•

Conveying the patient perspective to the research community, ensuring the “voice of the
consumer” is heard in a wide variety of settings;

•

Interacting with researchers, scientists, physicians, and administrators at the national level;

•

Expanding their own knowledge base by strengthening their abilities in communicating with
constituencies; and

•

Directly helping other cancer patients by providing support systems and access to reliable,
practical information.

Many respondents commented that while they had similar opportunities through other advocacy
agencies and organizations, the distinguishing characteristics of CARRA were its well-thought out and
intentionally planned structure and direct affiliation with NCI. Moreover, the opportunity to be trained on
and participate in peer review made the program unique.

Vetting and Selectivity
A unique feature of CARRA structure is its selectivity accomplished through an application and
vetting process, and members and NCI staff alike cited CARRA’s selectivity as an important component
of the program’s success. When commenting on the value of using the CARRA program, staff mentioned
the vetting process, and members promoted an increased, more stringent screening process.
There was a general agreement that to maintain CARRA’s reputation as having recruited the
“cream of the crop,” the program should remain selective and perhaps become “more selective in
recruiting.” To distinguish CARRA members from the large contingent of consumer advocates,
especially among the NCI staff, respondents favored a rigorous vetting process designed to screen
potential members more for certain personal characteristics than for technical expertise.
Over the course of interviews and focus groups, a composite view of the idealized CARRA
advocate emerged. While some technical knowledge of science and technology were mentioned, it is
clear that the idealized characteristics are more a function of personality, approach, and demeanor. These
characteristics were typically presented as generalized preferences along a continuum and were consistent
among members and staff alike.

74

Qualities of an Ideal Advocate
Analytic
Community orientation
Thoughtful
Critical thinking
Articulate
Cooperative
Confident
Some science background
More science background
Computer savvy

vs
vs
vs
vs
vs.
vs
vs
vs
vs
vs

emotional
personal orientation
reactionary
preconceived notions
outspoken
adversarial
intimidated
none
some
novice

The chief concern mentioned by NCI staff was that advocates in general, including CARRA
members, may have their own agendas and a tendency to focus on them to the detriment of their NCI
work. The scenario of an advocate pursuing just his or her own concerns, or advocates “thinking with the
heart instead of the head,” was mentioned by interviewees and focus group members.
This issue, while not necessarily CARRA-specific, was expressed by even the staunchest backers
of advocate inclusion. Some staff expressed apprehension about engaging someone who will be
disruptive rather than productive. Put another way, the qualities most valued in advocates were the ability
to take a calm, thoughtful approach, and to demonstrate a sense of responsibility to the process/task at
hand and to the job of representing the community.
Recognizing that it is much more difficult to screen for personal characteristics than for technical
skills or experience, the interviews probed for suggestions. Five of the 12 CARRA members had the
same recommendation, that is, use of a mock review.
Yeah, you need to screen people out, it’s a very hard thing to do. You could use a mock review,
almost grade people on that. There must be standardized screening tools on anxiety or depression.
The mock review should be on their cancer. See how they react in that situation. Use it to incite
some emotional response, to see if they look logically or if they respond emotionally.
I think something else that would help is to be able to have sessions, some way where you read a
scientific piece and be asked to critique it and then get constructive criticism. Like in a mock
session.
Along these lines, a member of the leadership focus group proposed that NCI staff suggest traits
OLA should be aware of during such mock reviews with CARRA members, for example, whether the
person asks questions and what kinds of questions they are. Such rigorous screening need not result in
denial of membership in CARRA. Instead, respondents suggested this step be used as an assessment of
what activities each member is best suited for or should be assigned to. Here again, the prominence of
peer review is inferred, with the goal of this screening process to be sorting out those not suited for peer
review activities. Such a vetting process is not considered as important as training for other member
activities.
Every CARRA member has some type of position, is a member of another organization. I believe
the value of CARRA is to help members be more effective members of their [home] organization.

75

NCI staff also ranked the vetting of CARRA members as one of the most valuable components of
the program. They perceived the vetting process as an important step in assuring the advocates they
requested are well aware of how NCI operates and are personally capable of performing their duties in a
professional and productive manner.
Recommendation: Explore strategies to project selectivity in the application process and
strengthen the vetting process. CARRA members themselves support a rigorous vetting process to
maintain the program’s reputation among NCI staff, to reduce apprehension among NCI staff about
advocates, and to better determine the skills and attributes new CARRA members bring to the program.
The latter could possibly be achieved by employing a mock peer review session designed to draw out how
potential CARRA members would interact with NCI staff under actual conditions.

Levels of Member Activity and Patterns of Program Use
In keeping with the sampling strategy, levels of reported activity varied across respondents. More
interesting was how the level of activity varied over time for individual members, with members typically
more involved at the outset. Despite classification as High and Low Active, most members expressed
interest in being more active, and all considered themselves fully engaged as CARRA members. On
average, respondents indicated that they participated in two to three activities per year.
When first joining CARRA, members expected to be involved in a range of activities, but their
responses to interviews and focus group questions emphasized a strong interest and desire to participate in
peer reviews. Others reported a lack of clarity on exactly what to expect. The theme of uncertain
expectations was common, even among those who had specific interests.
Due to initial and continuing levels of commitment and interest, many respondents indicated
disappointment with the number of activities in which they were involved. While members wished to be
more frequently involved, they generally understood the constraints of the program (i.e., that they are
placed in activities on an as-needed basis). While their disappointment has not escalated to overt anger or
frustration, there is a perceptible disconnect between member expectations and actual involvement.
The interview protocol invited suggestions on how to better manage member expectations
and a handful of recommendations were offered. Foremost among these were:
•

Doing a better job of clarifying roles and activities during the application process;

•

Providing better clarification on how members are invited to join activities;

•

Initiating more explicit discussion about activity levels during training; and

•

Maintaining ongoing communication.

Clarifying Roles and Activities During the Application Process
When asked what would help manage member expectations with respect to level of involvement,
respondents asked for three clarifying items:

76

•

A clear statement of CARRA’s mission and goals;

•

Specific examples of how individual members can accomplish those goals; and

•

A better sense of how an appropriate role can be integrated for them.

In short, it is necessary that NCI communicate to potential new members that CARRA offers a
wide range of opportunities to become involved, but each specific member’s experience will vary and be
determined by his/her specific background and skill set. Potential members need to temper expectations
by understanding the highly variable nature of participation from the outset.

Better Clarification on How Members Are Invited to Join Activities
The respondents themselves generally appeared to understand that CARRA operates on an asneeded basis, responding to needs generated by NCI. However, they indicated that other members may
mistakenly believe OLA creates the opportunities for participation and thereby directly determines the
level of member involvement. Greater clarification on this indirect process would help lower
expectations by making it clear that the skills and enthusiasm of members will likely outpace the need for
their services as generated by NCI. At the same time, it will buffer OLA from this misplaced
responsibility.

More Explicit Discussion About Activity Levels During Training
Respondents had many favorable things to say about training, but they commented that there were
few opportunities to discuss expectations. Training provides a highly valuable opportunity for OLA to
explicitly reiterate how the process of being invited to participate actually works and how member
experiences are variable, thereby making it clear that there is no standardized CARRA experience.
Training should also emphasize the full range of CARRA activity. Responses indicate that most
members equate peer review as the quintessential CARRA activity. Training topics that focus on peer
review, human subjects protection, and participation on IRBs tends to reinforce such conceptions.
Training that moved beyond peer review as the central focus and elevated the value of other activities,
such as dissemination of information and communication activities, would help avoid members’
misperception that they are not active unless they are involved in peer reviews.
While respondents were well aware of the expectation that they share information with
constituents, they do not interpret such activities as being a “CARRA” activity but, rather, as their own
personal activity for their constituency/advocacy group(s). Training would help rectify this situation.

Ongoing Communication
Through ongoing communication, the messages and understandings identified above can be
reiterated to members over time, reinforcing their understandings. Members also suggested that simple
but regular check-ins by phone or e-mail would help them manage expectations by providing
opportunities to exchange information and reconcile member interest with current NCI need.

77

Recommendation: Clarify expectations through improved lines of communication with
potential and current members. OLA can bring member expectations into alignment with the likely
level and type of CARRA activity through improved communication with potential new members and
current members. Specifically required are clarifying roles and activities during the application process,
providing better clarification on how members are invited to join activities, initiating more explicit
discussion about activity levels during training, and offering ongoing communication.

Matching Member Skills to Activities
Leadership and staff emphasized that when there is alignment of a CARRA member’s interest,
aptitude, and skills with the task at hand, great things happen. The theme of better matching the member
to the activity was expressed by members and NCI staff alike. Suggestions for strengthening this matchmaking process extend from the selection and vetting phase to post-activity follow-up with the relevant
members and staff.
During the vetting phase, a more rigorous screening process (described above) would help identify
the particular strengths of members that would later be used in determining degree of fit. Such an
approach would have the secondary value of reducing misconceptions that all members are eligible (or
appropriate) for all activities.
Another suggestion was to expand on the information made available to NCI staff when
considering engaging a CARRA member. This information would include details on members’
professional experience, personal background, and prior work with NCI as well as their advocacy
credentials. One participant in the leadership focus group suggested collaborating on developing a set of
key questions that would be especially relevant to CARRA users (e.g., about the person’s background,
interests, and current workload).
NCI staff expressed a willingness to collaborate with OLA to make better matches. For example,
NCI staff invited suggestions on how they might inform the decision-making of CARRA members to
participate in particular projects, perhaps by better specifying the professional skills and personal
characteristics expected. They invited feedback from OLA as to what they should be providing CARRA
members to help guide members’ involvement and increase their productivity.
Lastly, maintaining feedback loops between members and OLA and between NCI staff and OLA
will provide valuable lessons for establishing better matches in the future. Westat gathered useful general
privacy information in three rounds of post-activity surveys, but concerns prevented OLA from
connecting experiences to particular members. Future feedback loops would ideally allow OLA to
triangulate data from all parties—the member, the NCI staff, and OLA records.
Recommendation: Redouble current efforts to ensure positive matches between members
and activities. CARRA should be more strategic in leveraging specific skills to specific tasks by
identifying measures to ensure a good fit between CARRA members and the tasks they are invited to do.
Recommendation: Maintain a systematic feedback loop following activities. Post-activity
feedback provides NCI staff and CARRA members with the opportunity to share opinions and
impressions of their experiences with OLA. Gaining permission to specify members and staff by name
will provide OLA with the information it needs to make better matches in the future. Simple forms built
into the process of CARRA involvement should make obtaining feedback standard operating procedure.

78

Personal Contact
With respect to supporting the membership, personal contact between members and the OLA was a
common request that emerged from respondents. Almost without exception, personal contact was noted
as the best way OLA can assist and support its membership.
Respondents thought the “website is fine” for reference materials and appreciated the regular
electronic messages that offered updates and resources, but members clearly crave personal connections
such as those created during the training workshops and activities. The call for direct contact with OLA
staff and other CARRA members was a common theme.
Members suggested that informal, but periodic, personal contacts and two-way updates that invited
a response would have dual benefit of information exchange and keeping member expectations realistic.
While acknowledging that there are associated costs, members promoted the value of meetings and other
events to facilitate tightening of the CARRA network.
Recommendation: Convey a sense of personal contact and belonging to a community. The
camaraderie generated during the training was highly valued. Assuming the costs of additional physical
meetings are prohibitive, OLA should consider alternative means for creating a sense of connectedness
between and among its members. “Member of the month” profiles and personalized e-mail are two
suggestions. Reinstituting mentoring and coaching roles would create additional opportunities for
connections among members.

Training
Impressions and feedback on the training needs of CARRA members were explored in both the
member and staff interviews. Both interview protocols allowed for open-ended responses and also
invited specific feedback on the following predefined set of training topics.
•

Peer review training

•

Human subjects protection courses

•

Technology-specific workshops

•

How to serve on an IRB

•

Computer training

•

How to conduct Internet research

•

Negotiation skills, business etiquette, communication

The congruence between members and staff with respect to perceived training needs was
substantial. In general, both groups perceived that current CARRA training is strong, covering most of
the essential dimensions. A few CARRA users explicitly stated that the current training was adequate and
they would not modify anything. Thus, respondents tended to suggest that OLA should simply continue

79

providing these forms of training rather than allocate resources to enhance them. The exceptions are
discussed in detail below.
Among staff who have used CARRA members, awareness of and understanding of human subjects
protection issues among members was virtually taken for granted. Only non-users, unfamiliar with the
skill set of CARRA members, mentioned this area as worth expanding. In contrast, CARRA members
themselves favored additional training on peer review and human subjects protection, reporting that they
would like to obtain a solid grounding in the peer review process to ensure they are fully prepared for this
demanding experience. However, much of their concern was related to appropriate communication and
interaction with NCI staff. Other members sometimes gave the impression that peer review was the
dominant focus of CARRA training. If anything, these members felt greater attention should be given to
other activities, for example, dissemination and refinement of media products, to help place peer review
in a more balanced perspective.
The topics of technology, computers, and Internet research did not generate much discussion or
reaction other than being advantageous for those who did not already have those skills, assumed to be a
very small fraction of advocates.
Among those who suggested that training enhancements were of value, three topics pervaded the
discussion:
•

Interpersonal communication;

•

Basic understanding of science and scientific methods, and

•

Understanding of NCI structure and culture.

It is noted that the latter two, although not specifically listed in the protocol as an option,
nevertheless emerged as important training components to add.

Interpersonal Communication
NCI staff tended to identify excellent communication skills as a key skill demanded of CARRA
members. They expected members to be articulate, confident, and able to express themselves clearly.
The dual abilities to engage in active listening as well as educational communication were considered
equally import. Staff tended to assume that CARRA members—as experienced advocates—know their
external audience, the community to which they are speaking, and the organizations they represent. They
believed members could benefit from additional training about the internal dynamics of research
committees, peer review panels, and NCI as an organization.
In this regard, of the topics listed in the interview protocol, “negotiation skills/business etiquette/
communication” was a theme emphasized by both members and staff. Communication appeared to have
two dimensions: within NCI and beyond NCI. The former focused on communication between CARRA
members and NCI scientists within the context of a peer review. The latter focused on members’ ability
to share current, relevant, and accurate information with constituencies and other stakeholders.
With respect to communication within NCI, a premium was placed on an advocate’s ability to
manage his/her emotions and interact with scientists in an intellectual, analytical fashion. This theme

80

echoes CARRA members’ perceptions that the passion advocates typically rely on must be modulated
when working within NCI.
With respect to communication beyond NCI, a premium was placed on advocates’ ability to judge
what is important to communicate to the public and to think critically rather than simply advocate for
research of any kind. Staff stated that strong advocates must be able to speak about broader cancer issues,
unfettered by personal experience and emotions linking them to specific cancer type. They also
mentioned an advocate’s interest and willingness to advocate for prevention as well as treatment as
important.

A Basic Understanding of Science and Scientific Methods
Discussions on the amount of scientific knowledge and understanding advocates need to have or
gain through training ranged widely. For a small minority of NCI staff who tended to be non-users, the
lack of technical knowledge concerning cancer, medicine, and science was identified as a major
shortcoming among advocates and CARRA members. For the vast majority, however, the concerns dealt
more with basic understanding of the scientific process and how cancer research unfolds within NCI.
Reactions among members were mixed as well, with several members claiming that science
backgrounds provide obvious advantages for members. However, the general consensus was that training
should not attempt to turn advocates into scientists. Instead, members and staff seemed to agree that basic
levels of scientific knowledge were adequate for advocates to interact with their NCI-based colleagues.
Many mentioned learning terminology, basic concepts, and the principles of the scientific method as the
core dimensions required.
Many respondents included knowledge of how the research process unfolds within NCI as a related
core dimension. How clinical trials are structured, how grants are evaluated, and how to approach
reading and critiquing scientific papers were often mentioned as specific examples. Members desired
additional training on these topics and staff indicated such training would be useful.

Understanding the NCI Structure and Culture
To maximize their effectiveness, members and staff suggested the opportunity for CARRA
members to become more familiar with the culture of NCI and how NCI is structured. This goes beyond
the fundamentals of science and refers more to organizational culture, policy, and protocol. In particular,
staff mentioned understanding the importance of interpersonal communication (discussed separately,
above), knowing the distinction between intramural vs. extramural research, knowing how funding
streams flow, understanding how grants are designed, and understanding how the competitive process of
grant reviews is managed. They also mentioned understanding the organizational structure of the NCI,
the distinctions between divisions, and the roles individuals, especially principal investigators, play.
For their part, members concurred with the value of such knowledge. Their comments often hinted
at the potential clash of two cultures, one coming from an advocacy perspective and the other from a
scientific perspective. Recognizing that the science community often harbors assumptions concerning
advocates, the responding CARRA members expressed willingness on their part to learn the language and
customs of the science community in hopes of meeting them more than halfway.

81

Yet, it is worth noting that some members urged OLA to do a better job of communicating with
NCI about CARRA, its goals, and its members. And at the same time, some NCI staff acknowledged that
they did not know much about CARRA or the training CARRA members receive, so the staff felt they
needed to be educated as well. Some expressed interest in becoming involved in the design and/or
delivery of the training itself. To some degree, each side recognized the need to become more familiar
with the other.

Recommendations on Training
Align training with broader set of presumed skills. Those skills include a basic understanding
of science, the ability to critically analyze, a working knowledge of how NCI research and funding takes
place, self-confidence and strong interpersonal communication skills, and the ability to interact with NCI
staff on the common ground of reason, logic, and basic knowledge of the scientific method. Some
generic training on basic science concepts also is advisable.
Consider specialized training opportunities keyed to member screening. Reconsider the utility
of standardized training. Training may need to be individualized (segmented) because members come
with so many different backgrounds and expertise. Some suggested, for example, that only those who
were prescreened to serve on peer review panels would receive the peer review training.
Reinstitute the mentoring component. Several respondents indicated that they would either like
to have a mentor or to serve as a mentor to new advocates. NCI staff who use CARRA made similar
suggestions. The value of mentors, or coaches, as a supplement to training were discussed in terms of
helping new advocates:
•

Become more productive contributors to CARRA and NCI;

•

Gain a better idea of what to expect from the program;

•

Identify the essential skills necessary for each activity;

•

Obtain additional help, if needed; and

•

Increase members’ comfort and understanding of their role, particularly when they take on new
roles related to peer review.

The reestablishment of a mentoring component or coaching role would also provide an additional
form and level of activity for current members while simultaneously helping to address the overall desire
for social networking. By pairing members and building member skills on a one-to-one basis, the
CARRA network will be strengthened. A similar suggestion made by NCI staff to help transition a new
CARRA member to a new role would be to have a veteran member introduce the task to the new member
or invite the new member to shadow an existing CARRA member. Such a coaching process would help
with the new member’s understanding and comfort level.

82

Marketing Within NCI
Most NCI respondents, even the non-users, reported some awareness of the CARRA program.
Still, a few confessed that the first time they learned of CARRA was when they were invited to participate
in this study. Most of those who had been at NCI prior to 2001 first learned of CARRA around the time
the program was launched. More recent staff learned of it soon after beginning their employment at NCI,
suggesting a fairly established presence within NCI. Some CARRA users reported learning of the
program in conjunction with learning of the organizational requirement to engage consumer advocates in
certain NCI activities.
Staff reported a variety of ways they became informed of CARRA. Among the most common
were word of mouth and presentations at meetings. Others recalled receiving e-mails alerting them of
CARRA activities or corridor conversations due to physical proximity to the OLA office.
OLA’s interest in pursuing a more aggressive internal marketing campaign was validated through
the interviews and focus group discussions. The essential goals of marketing the CARRA program are
threefold:
•

Increase awareness, that is, bring CARRA to the attention of a broader cross-section of NCI
staff, raise its profile among those already familiar with the program, and distinguish CARRA
from other consumer advocate initiatives by clarifying objectives and goals.

•

Recognize value, that is, communicate how CARRA members can contribute to the work of
NCI, how they fill a specific need, and the positive impacts they can bring to unit-specific
programs.

•

Increase usage, that is, encourage and help NCI staff (especially those that need advocates) to
reach out to the CARRA office, and engage CARRA members in the process.

Increasing Awareness
Users of the CARRA program are, by definition, aware of the program, and most non-users were
also aware of the program’s existence. Despite basic awareness among the majority of participating NCI
staff, lack of clarity remains regarding the CARRA program, its mission, its value, and its members. Not
surprisingly, when NCI focus group members were asked to rate level of familiarity with CARRA on a
scale of 1 to 10 (with 1 being “not at all familiar” and 10 being “very familiar”), non-users reported levels
of 1 or 2. Of some concern, however, was that several current CARRA users reported levels as low as 5.
Perhaps more importantly, there was little knowledge of CARRA’s commitment to quality control.
This blind spot became evident within the focus groups, where lack of awareness of CARRA as a
dynamic, ever-improving entity surfaced, especially among non-users. Participants confessed to not
knowing how the program works, how effective it is, or the extent to which it is monitored and
strengthened with feedback.
Many staff reported successful use of CARRA advocates in one activity (e.g., communication or
peer review) but were unable to imagine their involvement in other types of activities (e.g., peer review or
communication). Several staff who engage consumer advocates reported using other networks to identify

83

advocates. Some expressed confusion as to how CARRA was related to or different from other NCI
initiatives, such as the Director’s Consumer Liaison Group. Others, expressing support for the generic
value of engaging advocates in the work of NCI, applauded CARRA for assuming this responsibility—so
“other divisions didn’t have to”—clearly reflecting a lack of understanding. While generally sympathetic
to the idea of engaging consumer advocates, staff are not entirely clear how CARRA figures into this
process. The research findings revealed resonating themes around which useful messages can be
formulated and also identified promising marketing mechanisms for delivering those messages within
NCI.

Recognizing Value
While building awareness of CARRA among NCI staff, it is critical to help them recognize the
value of the program and the members themselves. As the designers and managers of CARRA, OLA
already possesses the knowledge to construct effective messages that will ensure clarity of CARRA’s
structure and mission. Therefore, this report focuses its attention on information that could be used to
construct messages concerning the perceived value of the program. The interviews and focus groups
specifically sought to elicit feedback from NCI staff around the value of CARRA members and consumer
advocates for this purpose.
A few preliminary remarks about the value of CARRA and advocates may be useful. First,
numerous comments conveyed a sincere belief that NCI as an Institute values the viewpoints of people
external to NCI who are affected by cancer. This belief was particularly emphasized by the members of
NCI leadership. As mentioned earlier, staff appear genuinely sympathetic to the idea of engaging
consumer advocates in the work of NCI.
Second, most were familiar with the term “consumer advocate,” and many had some personal,
though not always direct, experience working with advocates in some capacity. But there was significant
variation among staff as to the full range of activities in which consumer advocates should be involved.
For example, most staff appreciated the value of an advocate in communication, but not all saw their
value in research projects, especially basic science projects.
Hence, several staff specifically referenced situations where advocates are not appropriate. These
tended to be situations characterized by intense, highly specialized scientific activity or in which
confidentiality is an issue. Importantly, the issue of confidentiality in question referred less to patient
confidentiality than product or process confidentiality within the private sector or government involving
the development of proprietary medicines or techniques.

The Perceived Value of Consumer Advocates
Staff perceived value in a range of ways that consumer advocates can contribute to the work of
NCI. The most prominent of these was the advocates’ ability to convey the patient perspective. A
subcategory of this value included grounded reality checks in research and product development. A
second subcategory of the patient perspective is the advocates’ close attention to human subjects
protection issues.
Importantly, many respondents commented that the value advocates bring is often associated with
the valuable professional skills they possess beyond their own personal survivorship perspective.

84

Respondents also saw the value advocates bring by taking on roles the government can not, such as
lobbying and fund raising.
Lastly, a set of comments reflected the value of involving advocates in the work of NCI so that
advocates are poised to promote dissemination of new and accurate information. Given the influence of
advocates and the range of their activities in the field, the value of intelligently engaging advocates in the
work of NCI is realized through a better informed and well-educated cadre of CARRA advocates who
have firsthand insights into the work of NCI.

Consumer Advocates Convey the Patient Perspective
The most commonly cited value associated with consumer advocates was their ability to bring the
patient perspective into the research and communication development process. For many this was
specifically referred to as the “non-scientific perspective.”
Staff valued the consumer advocates’ ability to provide insight into the “uniquely personal”
experience of actually living with the multitude of scientific, medical, and clinical conditions that staff
study. Consumer advocates brought to the surface valuable, but sometimes subtle or hidden information
of which scientists are not always cognizant.
An additional element of the patient perspective was the notion of “grounded reality checks.” In
this capacity, advocates provide scientists with realistic reflections on how the community will respond to
research planning and development. Beyond helping to “keep researchers in the world of cancer
connected to what the disease is really like and how it affects people,” grounded reality checks provide
insight into how patients interpret and react to research opportunities such as clinical trials, thereby
providing researchers with practical information on how to improve recruitment and participation.
It is noted that there were mixed, but mostly negative reactions to statements suggesting that
scientists need to be reminded about the human relevance of their work. Although at least one science
administrator remarked that advocates “provide powerful reminders to scientists about why scientists do
what they do—especially in cases where basic scientific research activities may feel far removed from
healing human cancer,” the general consensus was that the scientific community does not require or
respond well to suggestions that they are unaware or detached from the personal trauma associated with
cancer.

Advocates Are Attuned to Human Subjects Protection
Respondents acknowledged that consumer advocates are naturally concerned with human subjects
issues, including the degree to which proposed research addresses the broader population of cancer
survivors and how subjects are protected and treated during the actual research project. NCI staff
members involved in the administration of research grants in particular mentioned the value of advocates
in making sure that issues related to informed consent and patient protection are addressed in the grant
review process. Concentrating attention on human subjects protection protocols, anticipating concerns,
and offering potential solutions based on a grounded understanding of the consumer community are
examples of the value NCI staff perceive.

85

The Value of Advocates Extends Beyond Their Status as Cancer Survivors
Several respondents who have worked with consumer advocates commented that the value
advocates provide specific projects can often extend well beyond representing the patient perspective.
Most CARRA members interviewed had a wealth of education, training, experience, and expertise that
they brought with them as advocates. The advocacy community represents a cross-section of the
population and comprises people from all walks of life and professions. Thoughtful NCI staff find ways
to take advantage of the valuable professional skills advocates bring to the table beyond their personal
survivorship perspective and commitment.

Advocates Provide Value by Communicating Accurate and Pro-NCI Information to
Stakeholders
Well-informed advocates bring value to NCI and the research process by communicating up-todate and accurate information within their advocacy circles. Several staff members commented on the
need to bring greater transparency to the peer review process and clinical trials. By inviting advocates to
attend grant reviews, for example, NCI gains “ambassadors” to bring positive experiences and
information back to the community.
Programs such as CARRA are valued because they give NCI staff the opportunity to teach
advocates about progress being made at NCI, the steps that are taken to combat cancer, and important
insights into science. Such exchanges provide advocates with the ability to return to their advocacy
communities with hands-on knowledge of how scientific research is performed. Non-users tended to find
this component particularly valuable given the prevalence of misperceptions and “conspiracy theories”
they perceive are held within the advocacy community. Thus, in addition to traditional dissemination of
findings and information, a well-informed advocate can build bridges between the consumer and research
communities, help eradicate misconceptions, share understandings into the purpose and importance of
critical research activities such as clinical trials, and encourage other patients to participate.

Advocates Can Take on Roles the Government Cannot (Such as Lobbying and Fund Rising)
As federal employees, NCI staff must abide by well-established strictures concerning the use and
dissemination of scientific information to avoid impropriety and conflicts of interest. Advocates,
however, tend not to be as restricted. Indeed, one of the many goals an advocate may have is to influence
decision-makers, especially with respect to funding and resources. While being careful to specify that
NCI would never explicitly ask an advocate to take on these roles, staff nevertheless recognized the value
advocates bring when they do.

The Special Value of CARRA Members
Having established the perceived value of consumer advocates in general, the protocols collected
feedback from NCI staff on the specific value of CARRA. In addition to being recognized as
extraordinarily dedicated, CARRA members were distinguished from other advocates along three
dimensions: they are vetted, trained, and possess superior communication skills.

86

•

CARRA members are vetted. Because CARRA members go through an application process
and are individually screened, potential staff users have increased confidence in involving
CARRA members in their project work.

•

CARRA members are trained. Although the details of training are not widely understood,
staff are aware that CARRA members receive training, making the CARRA member a more
valuable member of the team.

•

CARRA members have superior communication skills. Experience has indicated to staff
that CARRA advocates can both clearly represent their constituency during NCI activities and
communicate effectively to their respective communities.

Staff appreciated the passion and ability of CARRA advocates to constantly strive to connect the
appropriate information in a clear way with a wide audience, increasing the influence of NCI’s research
findings. Staff members have come to expect that CARRA members will have the desired and necessary
skills needed to perform well. Moreover, users expressed belief that they will obtain quality input and
service if they use a CARRA member.
Yet, some respondents indicted that they were unsure exactly how advocates were vetted, selected,
and trained, suggesting that more information about this process would lead to greater confidence and use
of CARRA members. A few concerns were reported by staff respondents that represent challenges to
perceived value:
•

Non-users were more skeptical about the overall quality and type of input provided by
advocates.

•

Several respondents believed that advocates bring a personal agenda, which may cause
difficulty and conflicts of interest when working with NCI staff.

•

Non-users and some users cited lack of scientific background as a threat to value.

•

The requirement to use advocates is perceived by some staff to be based on an obligation to
remain politically correct rather than a sincere recognition of value added.

•

Disappointment was aired that not enough advocates appear interested in prevention work but
are instead focused solely on treatment of their particular cancer type.

Increasing Usage
With respect to increasing use of CARRA members, a number of barriers and facilitators were
identified. Although the goal of the research question was to identify structural factors that promote or
inhibit use (i.e., those resulting from the way NCI is organized) the vast majority referred either to
programmatic issues (i.e., those resulting from the way CARRA is organized), such as inadequate clarity
surrounding CARRA or shortcomings identified with individual CARRA members themselves. Sorted in
this manner, the list of cited barriers included the following:
Structural (NCI based)
•

Lack of funding specifically to cover costs of consumer advocate involvement (e.g. travel).

87

•

Access to alternative networks of cancer advocates trusted to provide the same kinds of input
as CARRA members. (However, some staff felt that they had faster, more immediate access to
CARRA members than to other whom they would have to contact and recruit.)

•

The staff member’s type of work colors their perceptions. In this study the major distinction
was between communication and science activities. Respondents involved in communication
activities were more likely to be satisfied than those involved in science-related activities,
perhaps because they were better able to observe the contribution of the member (e.g., help
translate material into layman’s terms).

Programmatic (OLA based)
•

Lack of NCI staff awareness of the CARRA program.

•

Lack of clarity about CARRA program’s mission.

•

Lack of understanding of the value of consumer advocates.

•

Lack of concrete, specific ideas of how to engage advocates. Relatedly, there is the “goodness
of fit” concern; that is, uncertainty regarding how to determine best fit between CARRA
member skills and project needs.

•

Lack of understanding of NCI’s role in helping advocates become involved in activities.

•

Lack of understanding of the process through which NCI staff can recruit CARRA members
onto their studies.

Individual (member based)
•

Lack of specific skills and training relevant to task at hand (i.e., peer review).

•

Lack of scientific background (e.g., staff reported that they had to rewrite analyses by
advocates because the documents were not in appropriate “scientific critique” form).

•

Lack of confidence that advocates can communicate in the necessary scientific manner
regarding a peer review.

•

Lack of a coalesced focus by advocates around the broader topic of cancer (i.e., counterproductive emphasis on one’s own special cancer type, which results in an “inability to see the
bigger picture” and causes advocates to compete against each other for limited resources).

The cited facilitators, or those that could lead to increased usage, were as follows:
•

Provision of travel funds to CARRA members.

•

A cadre of appropriately trained CARRA members.

88

•

Good experiences with former/existing advocates resulting in repeat use and positive word of
mouth (communicate with staff member who has worked with a particular advocate to learn
more about member and fit with activity).

•

Concrete examples of how others have effectively used advocates and the subsequent results
(create summaries of personal and professional background, key skills, and other work the
person has done with NCI).

•

Top-down support (e.g., statements, behavior or other signals from the Office of the Director,
Executive Committee, division directors, etc., promoting the use of advocates).

•

Internal marketing.

Marketing Mechanisms
Feedback on how to best communicate with NCI staff about CARRA was collected during the
interviews. The protocols focused on obtaining respondents’ reactions to a preidentified set of
communication strategies. These included:
•

Tangible products (such as brochures, posters, standing displays, or branded items);

•

Presentations by OLA staff; and

•

Endorsements by NCI leadership (including Office of the Director, Executive Committee,
division directors, etc.).

These data were coded, tallied, summarized, and then presented to the three focus groups to gauge
reactions from users, non-users, and leadership. Across the interviews and focus groups, a clear
preference for in-person presentations and personal outreach emerged. In contrast to static products, there
was general consensus on face-to-face communication and as-needed access to web-based resources. In
both rounds of data collection, the use of traditional marketing products, such as brochures, posters, and
branded giveaways, was discouraged.
The findings indicate that formal and informal contact with NCI staff is the most effective
mechanism for conveying information about CARRA within the NCI structure. While informal
communications can take a variety of forms, CARRA staff should make arrangements to formally present
information about CARRA at obligatory staff meetings about once or twice a year. Staff explicitly stated
that infrequent contact would be both effective and preferred. The key is to present at well-attended
meetings (i.e., those that require staff attendance). CARRA-sponsored receptions or meetings especially
convened for CARRA were not identified as promising.

Personal Outreach
Customization of marketing messages was seen as one of the essential benefits of in-person
CARRA outreach. Personalized outreach was stressed given that contributions of consumer advocates
can be diverse and wide-ranging, with some dimensions and activities suited to different audiences.
Moreover, it was stated that NCI staff members do not respond well to generic, one-size-fits-all

89

pronouncements. In-person presentations allow for suggestions on “how to engage advocates in the work
that we do,” and would allow presentations to be specific to the areas of interest of the targeted audience
(e.g., health disparities, nutrition, prevention, grant review, basic research, etc.). Involving staff members
who have used CARRA as co-presenters was a suggestion for achieving even greater specificity.
Respondents suggested that relationship building with division directors and upper management
would be beneficial, arguing that “if you convince them, then they will tell their staff to use CARRA.”
After conducting such in-person presentations, OLA staff should leave the potential users with concise
take-away materials about the program, such as simple one-page summaries of the CARRA program (i.e.,
main goals and sample activities) with contact information and website address.

Website
The benefits of a well-designed website as a communication mechanism were alluded to repeatedly
by respondents. The website is the first place most NCI and CARRA respondents said that they turn for
information about CARRA. It can passively deliver key messages about CARRA to a large audience
(i.e., all of NCI and beyond), and it can provide ready access to a set of resources on an as-needed basis to
potential users.
Thus, access to informational documents, such as CARRA program descriptions, profiles of
CARRA members, and guidelines for accessing CARRA members, are seen as valuable products that
need to be made available to NCI staff, although they need not be disseminated as tangible products.
Instead, respondents indicated a preference for being able to download such documents from the CARRA
website on an as-needed basis.
The current status of the CARRA website was referenced by several NCI staff members. Access to
resources that would help staff identify and recruit the appropriate CARRA member was often requested.
In general, staff favored web-based materials that convey illustrative, real-life examples (case studies) of
successful inclusion of consumer advocates in specific but common projects. Such resources need to be
concrete and should present success stories where involving consumer advocates really made an impact.
They should highlight stories that would let the NCI staff know that they do not work with advocates in a
vacuum. As one of the respondents put it, “the story should be ‘here is how consumer advocates made an
impact in actual conditions and here is how they can help you.’”
In addition to its utility for communicating within NCI, the website can be a useful mechanism for
communicating with CARRA members and for external communication as well. For example, should
OLA seek to be more deliberate in its efforts to position CARRA as the leading effort to engage
advocates in federal health and science efforts, the website would be a powerful vehicle for marketing the
prominence of the program both within and beyond NIH. The website is also the most likely source of
information for potential members and, therefore, an important tool for managing expectations. Current
members will likely use the website as their main resource for CARRA-related information.

Other Marketing Mechanisms
Other suggestions for marketing and communication included promotion of CARRA by NCI
leadership and periodic e-mail reminders. Obtaining and broadcasting endorsements by NCI leadership
were mentioned as a valuable, parallel strategy, although not as adamantly as the research team might
have expected. In fact, a small number of respondents, particularly those closest to basic research,

90

indicated that their activities and decisions were based on the specific exigencies of the work at hand
rather than top-down norm setting. Nevertheless, support from NCI leadership (e.g., division directors up
to the Office of the Director) was generally seen as an important buttress to OLA outreach.
Most respondents, particularly those who have used the CARRA program, indicated that the use of
advocates is currently encouraged from top down. Respondents also recognized that greater top-down
support would promote wider use of consumer advocates in NCI projects.
Lastly, e-mail messages and written materials were considered effective as reminders but not as the
main communication vehicle. Written materials that convey illustrative, real-life examples of successful
inclusion of consumer advocates in specific projects to the NCI staff were favored over general
informational messages. Practical, concrete examples that present success stories where involving
consumer advocates really made an impact were often cited as powerful.
With respect to publications, there was more support for articles in the NCI Bulletin than other
dissemination vehicles. Respondents offered an idea of introducing in this newsletter a reappearing
column, an “Advocate Corner,” authored by advocates themselves. Other potential products included a
one-page paper about the CARRA program and articles in the NIH Record.
Several respondents mentioned that e-mail messages could be sent periodically as reminders, but
they should focus on specific themes rather than general communication. Thus, these “reminders” are
viewed more as news (e.g., “CARRA Now Provides More Background Information On Members,” “Staff
Members’ Top Questions About CARRA Answered,” “Updated CARRA Member List Now Available”).
Disseminating electronic reminders when information on CARRA appears in the latest issue of the NCI
Bulletin was also advised.

Recommendations for Marketing Mechanisms
Seek opportunities to conduct in-person outreach at standing meetings such as division or
branch meetings. The idea of communicating via face-to-face outreach was generated by the interviews
and was very well received in focus groups. An annual visit to each NCI group—as opposed to more, or
fewer, visits—was said to be appropriate. The CARRA speaker’s purpose at each meeting should be
threefold: 1) to inform NCI staff about CARRA including ease-of-use and benefits to the type of work
those NCI staff members do, 2) to demonstrate CARRA’s responsiveness to NCI staff’s needs, and 3) to
establish interpersonal connections to help staff members feel comfortable calling CARRA staff directly.
In determining allocation of communication funds, dedicate relatively less (if any) to static
strategies (e.g., posters, brochures) and branded giveaways (e.g., pens, pads). These kinds of ideas
tended to fall flat among NCI participants. Most felt ambivalent about them or believed mildly that such
materials could be helpful reminders. A few thought that branded giveaways could detract from the
credibility of CARRA.
Supplement in-person outreach with e-mail outreach. Ensure that e-mail content contains new,
relevant, interesting material. For example, e-mails could notify NCI staff that the CARRA website has
been updated in response to staff feedback or alert staff to other changes made over time. Such an
approach would subtly keep CARRA in view of staff, while demonstrating the responsiveness of the
CARRA program to effectively serve the needs of NCI staff.

91

Communicate widely at NCI, but not frequently. According to interviewees and focus group
participants, a barrage of outreach from CARRA would not be necessary and may even make them feel
less, rather than more, interested in contacting CARRA. Instead, key face-to-face encounters, meaningful
electronic outreach, and a well-designed/maintained website are the most promising approaches.
Position the CARRA website as a cornerstone of CARRA communication efforts. Revisions
to the current website should be made in light of the findings on users’ and potential users’ information
needs and their specific desire to better understand how CARRA advocates are relevant to their work.
Many interviewees and focus group participants said that they turned to the site for information about
CARRA, indicating that it was a powerful communication mechanism. Respondent feedback revealed
key questions and requirements of NCI leadership and staff, including the following:
•

Concrete examples of how CARRA members have been tapped to help with NCI work and
succeeded (as a way of helping other staff members imagine ways CARRA members might be
able to help their work);

•

Outcome data or other information demonstrating the program’s effectiveness;

•

Information on how the CARRA membership list is created/how members are chosen;

•

Information on how members are trained;

•

Information on how the membership list is updated and how often;

•

Procedures for procuring the help of a member;

•

Lessons learned or tips for the NCI staff member to use (e.g., what the CARRA member needs
to know from NCI about a task, key questions to ask CARRA members when contacting them
for NCI work); and

•

Procedures to follow if something goes wrong (e.g., member is not completing assigned work,
member takes a stand on his/her own agenda to the detriment of the project overall, member’s
health prevents participation).

Recommendations for Messages and Communication
Include concrete examples of varied and successful work of CARRA members in
communications, both to explain what members can do and to persuade audiences of their
usefulness. Interviewees and focus group participants expressed a desire for examples to help them
understand CARRA, see its value, and grasp how a member might be able to help them specifically.
Prioritize examples that demonstrate how CARRA members’ contributions serve NCI’s goal of including
external or patient perspectives and help to get work done.
Highlight that CARRA members bring a patient perspective, that is, a “non-NCI
perspective.” The belief that such a perspective is important was strong among participants. And, this
value—of a different and fresh perspective—was the one that resonated best with staff and leadership. To
them, it is the central benefit that CARRA members bring. Other values discussed in the focus groups
were said to simply be manifestations of advocates’ “perspective,” such as being more attuned to human
subjects protections and being able to provide reality checks.

92

In addition to spotlighting the patient perspective, highlight how CARRA advocates help
teams to accomplish tasks or achieve their goals more easily. While it is true that merely getting the
patient perspective was valued, focus group participants spoke of CARRA members’ contributions as
being the most valued when they actually made a difference in day-to-day work. For example, grant
reviewers were pleased to know that a CARRA member was looking carefully at human subjects issues
that might otherwise be overlooked, and NCI staff producing brochures were grateful for CARRA
members’ help in avoiding pitfalls and communicating clearly.
Focus on selling points other than the possibility that advocates offer great value to basic
scientists working on early development processes.
For example, avoid messages that may be
construed as condescending or assert that CARRA members “can remind scientists why they do what
they do.” This was a potential message identified in interviews that was poorly received when
subsequently tested in focus groups. For many, it did not make sense to involve advocates in early
development processes; for others, the idea actually rankled.
Explore ways to capitalize on the fact that NCI appears to be seen as a leader for patient and
advocate inclusion among the other NIH Institutes. Some NCI staff and leadership expressed the view
that NCI, as an Institute, is a leader on patient and advocate inclusion. Capitalizing on and enhancing this
image can help to promote the CARRA program as well as the continued inclusion of external viewpoints
overall. Consider opportunities ranging from delivering presentations that target other Institutes to
publicizing effective strategies and research data.
Take steps that demonstrate to NCI staff that CARRA is a dynamic and active program that
includes mechanisms and activities for ongoing feedback and strengthening. OLA should consider
sharing information about CARRA’s evaluation activities and proactive approaches to addressing
concerns, such as staff’s desire for a better fit between members and their tasks. Demonstrating that user
input has been sought and is being addressed would boost confidence. Lastly, discussions with leadership
and staff revealed that some think of CARRA has having been launched with fanfare, but quieting down
over time.
I think one recommendation is learning what impact CARRA has had and what it has
accomplished. I don’t think they’ve done that and it’d be helpful to understand in what
categories. It could be as simple as a pie chart, a bar chart, a percentage of certain activities
and how that compares to what the targets were, and how can we improve upon that for the
following years.

93

94

APPENDIX A

LETTER OF INVITATION SENT FROM NCI OFFICE OF THE DIRECTOR

A-1

A-2

Dear Colleague,
I am writing to you to ask you to participate in an evaluation of an important program here at
NCI – the Office of Liaison Activities’ Cancer Advocates in Research and Related Activities
(CARRA) Program. The CARRA program was created to encourage increased involvement of
consumer advocates in the activities you manage and many of you have taken advantage of this
valuable opportunity. Thank you.
Ongoing evaluation data indicate that the CARRA program is achieving its goals and holds
promise for continued enhancement of our work. OLA has contracted with Westat, an
independent research firm, to complete additional research.
Over the next three weeks, a representative from Westat will contact you by phone to determine
your availability to participate in a brief interview or a focus group. Both activities are designed
to be minimally intrusive on your work day and maximally response to your schedule.
I encourage you to share your candid impressions of CARRA and its potential, even if you have
yet to participate in the program itself. Participation will provide you with an excellent
opportunity to share your ideas how we can most appropriately involve cancer advocates in our
work.
Should you have questions about this project, please contact Elizabeth Neilson, CARRA
Program Manager, OLA, or NCI.
Sincerely,
John

A-3

A-4

APPENDIX B

CARRA MEMBER IN-DEPTH INTERVIEW PROTOCOL

B-1

B-2

CARRA Member In-depth Interview Protocol

CARRA Member Name: ___________________________________________

Interviewer: _____________________________

Date: ___________________________________

Hello, this is Keith MacAllum calling from Westat. Thank you for agreeing to participate in a
interview for the Evaluation of the CARRA program.
Is this a convenient time for us to talk?

Our company, Westat, has been contracted by the NCI to evaluate CARRA and come up with
ideas for making it more efficient and effective.
Your name has been selected from a list of all CARRA members. We would like to ask you
several open-ended questions and give you an opportunity to freely express your views and
opinions about the program. Your participation is very important, since we are only able to talk
to a limited group of the CARRA members. The interview will take approximately 30 minutes.
Before we get started there are a few things that I’d like to mention to ensure we have informed
consent.
First, because it is extremely important that you feel free to express your views openly, please be
assured that your responses will be kept confidential. For analysis and reporting purposes, the
collected data will be aggregated, and no names of our respondents will be provided to NCI, so
that your name will not be associated with anything you say during our discussion.
Also, to ensure that I have an accurate record from which to analyze these responses, I would
like to tape-record our discussion today. Is that OK with you? As I explained before, only our
research team will have access to these recordings.
Do you have any questions before we begin?

OK, let’s begin.

B-3

1. Let’s first start with a few background questions, confirming your status.
Are you currently _____

2. Would you describe yourself as active in the cancer advocacy community? How so?

3. And you joined the CARRA program in __________________?

4. Why did you become a member of the CARRA program? What made you consider
joining the CARRA program?

a. How did you find out about the program?

b. What were the types of activities that you were most interested in when you
joined the program? [SEE APPENDIX FOR THE LIST OF CARRA ACTIVITIES]

c. Why did these particular types of activities interest you most?

5. At the time you joined the program, what were your expectations?
a. What kinds of activities did you expect or hope to be involved in?
b. About how many activities per year did you expect to participate in?
PROBE: About how much time did you expect to spend on those activities?

B-4

c. If you can, reflect a bit on how you developed this expectation.

PROBES: Can you help me understand what this expectation was based on?

Was it based on:

your own assessment?
communication with the NCI’s Office of Liaison
Activities?
communication with someone else?

6. Have you participated in any activities since becoming a CARRA member?
a. If yes, What types of activities? [SEE APPENDIX FOR THE LIST OF CARRA
ACTIVITIES]
b. If yes, About how many?
c. How satisfied are you with your level of involvement in the program activities?
Why?

For those who felt their involvement was too low, explore the barriers. For
example, was the participant’s interest/availability a barrier? Was a perceived or
real lack of invitations from NCI a barrier?

d. Have you ever been unable to participate in a CARRA activity?
(IF YES: what were the circumstances?)

B-5

7. Has your level of involvement in activities matched your expectations?

Has this been a concern for you?
a. (IF YES…) How could the Office of Liaison Activities have better communicated
with you about what to expect?
b. (IF NO…) Tell me what the Office of Liaison Activities did right in
communicating with you about expectations.
c. How could the Office of Liaison Activities better manage the expectations of
CARRA members in the future?
d. Based on your experience, what questions should OLA anticipate that new
members may have about level of involvement?

8. There is an expectation on the part of the CARRA program that CARRA members will
share information about NCI with their constituencies. Are you aware that such
information sharing with constituencies is expected of CARRA members?
a. (IF YES…) How was this communicated to you?
b. Have you shared information about NCI with your constituencies?
(IF NOT, PROBE: Why not? SKIP TO Q.9)
(IF YES, PROBE: What kinds of information have you shared and how?)
c. Why have you been involved in these particular types of information sharing
activities in contrast to other types?
d. How often have you been involved in information sharing activities with your
constituencies?
e. What accounts for this frequency of information sharing?

B-6

9. a. (If member has participated in an activity) Based on your experience in working on
CARRA-related activities, have you felt prepared to effectively participate?
(If yes: PROBE: What skills were required? How did you gain these skills?)
(If no: PROBE: What skills were required? What type of training would you need
to feel prepared to participate next time?

b. We’re interested in determining what other skills CARRA members might need to
effectively participate in NCI activities? What skills would you highlight?

c. Specifically, what type of training could CARRA leadership offer that would help
members develop these skills?
For example:
more peer review training
human-subjects protection courses
technology-specific workshops
how to serve on an IRB
computer training
how to conduct internet research
negotiation skills, business etiquette, communication

10. What other kinds of help and assistance would you like to receive from the CARRA
program?

11.

Do you have any recommendations on how to improve the CARRA program?

B-7

Attachment to Appendix B
List of CARRA program activities (protocol appendix)

Peer review of grant applications (in-person or by phone)
One-time discussion and learning workshop
One-time meeting
On-site visit to a potential grantee
Progress Review Group to assess science and make recommendations
Review of informed consent documents
Standing Committee (a committee that meets more than once)
Educational materials development (print, electronic, or other types of materials)
Contract review (review of responses to a request for research contract proposals).
Research tool development (review of surveys, interview guides, or other research tools)
Editorial boards for NCI publications
Web site usability testing
Other

B-8

APPENDIX C

INFORMED CONSENT FORM

C-1

C-2

Informed Consent Form
CARRA Evaluation
Identification of
Project

Telephone and in-person interviews and focus groups to obtain personal and
professional opinions on the program and services of the National Cancer
Institute’s (NCI) Consumer Advocates in Research and Related Activities
(CARRA) Program, managed by the Office of Liaison Activities.

Purpose

The purpose of this research is to gather qualitative information on the experiences
and opinions that NCI staff and CARRA members concerning the CARRA
program. Findings will be used to communicate more effectively about the
program and help to strengthen its services.

Procedures

Individuals who participate will be asked to discuss their opinions about CARRArelated topics in an interview, either by phone or in person as is convenient for
them, or participate in a focus group of 6 to 8 individuals. The total time for the
interviews will be no more than 40 minutes. Focus groups are scheduled for 1
hour.

Confidentiality

All information collected in this study is confidential. I understand that the data I
provide will be grouped with data others provide for the purpose of reporting and
presentation and that my name will not be used. I understand that the interview or
focus group will be audio-taped but my voice will not be played for others beyond
the independent research team without my written permission.

Risks

I understand that the risks of my participation are expected to be minimal in nature.

Benefits, Freedom to
Withdraw, & Ability
to Ask Questions

I understand that this study is not designed to help me personally but that the
investigators hope to explore effective ways to communicate about the CARRA
program, identify strategies for overcoming any barriers to using the program by
NCI staff, and strengthen the role of consumer advocates in the work of NCI. I
understand that I am free to ask questions or withdraw from participation at any
time and without penalty.

Statement of Age of
Subject

I state that I am at least 18 years of age, in good physical health, and wish to
participate in this program of research being conducted by Westat for the Office of
Liaison Activities of the National Cancer Institute, Bethesda, MD.

Contact Information
for Principal
Investigator

Keith MacAllum, PhD
Director, CARRA Evaluation, Westat, Rockville, MD
301-610-8854
keithmacallum@westat.com

Printed Name of Research Participant __________________________________
Signature of Research Participant _____________________________________
Date___________________________

C-3

C-4

APPENDIX D

NCI STAFF MEMBER IN-DEPTH INTERVIEW PROTOCOL
(CARRA MEMBER USERS)

D-1

D-2

NCI Staff Member In-depth Interview Protocol
CARRA Program Users

1. Our questions today revolve around the use of consumer advocates in the work of NCI.
In your opinion, what is the value of inviting a consumer advocate to participate in
projects at NCI?

PROBE:

Does type of project matter? What about science-related projects?
What about other types of projects?

2. Have you worked with a consumer advocate in NCI work?

a. What did you want to accomplish by inviting a consumer advocate?
b. Did you accomplish that?

3. We understand that you have involved CARRA members in the work of your office.
a. IF NOT OBVIOUS, in the example you gave above, were you referencing a
CARRA Member?

b. Are you currently working with a CARRA member?

c. How many CARRA members have you involved so far?

4. What made you decide to use the CARRA program and invite a consumer advocate(s) to
work with you?

D-3

5. When did you first find out about the CARRA program?

6. Do you recall how you find out about the CARRA program?

a. How well-known would you say the CARRA program is among your colleagues?

b. Have you mentioned it to others? (PROBE: Have you recommended it to
anyone?)

c. Have you ever been asked about it?

7. What skills, knowledge and/or experience make a CARRA member a desired member of
the project team in the type of activities at NCI that you conduct?
[Note reaction to term “member of project team.” Explore level of involvement.]

a. Do you think that CARRA members in general have those skills and knowledge?
b. Need to probe on personal characteristics (confidence, communication, etc.)

8. NCI would like to see more consumer advocates provide input into NCI’s research and
communication activities. What are the barriers to having CARRA members involved in this
work at NCI?

a. How can these barriers be overcome?
b. Do you feel your office leadership and culture encourage using CARRA members?
c. In your experience, what are some persuasive reasons to use the CARRA program?
- What other incentives can be used?

D-4

9. Did you get the support that you needed from the CARRA program to better incorporate
CARRA members into your project?

10. What can the Office of Liaison Activities do to encourage NCI staff in general to learn
about the CARRA program?
PROBE ON POTENTIAL MARKETING / COMMUNICATION MECHANISMS

†

Posters or Standing Displays about CARRA

†
Items that would increase name recognition such as pens/pads of paper with the
CARRA logo
†

Presentations from OLA staff

†

Articles in NCI Bulletin about CARRA

†

Reading/hearing success stories of advocate involvement from other NCI staff

†

Recognition from the NCI Director for the utilization of advocates, etc.

†

Encouragement / Support from leadership

D-5

11. What other things can the Office of Liaison Activities do to encourage NCI staff in
general to use the CARRA program?

a. Based on your experience at NCI, which of the mechanisms and materials we have
discussed do you see as the best way for the Office of Liaison Activities to increase
awareness of the CARRA program among the NCI staff?

NOW LET’S TAKE A LOOK INTO THE FUTURE
12. In thinking about the initiatives and/or projects you anticipate in the next 18 months, do
you see opportunities to include CARRA members in any of those initiatives?

a. Do you plan on seeking out CARRA members? What about your colleagues?

13. What “value” would you anticipate getting from CARRA member involvement in future
activities?

14. We talked earlier about the skills you believe CARRA members need. Should you
decide to invite a consumer advocate to participate in those upcoming initiatives / projects,
does the CARRA program currently offer an adequate pool of candidates representing the
skills and expertise that you might need?

IF ANSWERS “NO,” ASK:
a. What types of skills or experience are the CARRA members missing, that would be
of interest to you?

D-6

15. Which of the following types of training would you recommend that the Office of
Liaison Activities provide to the CARRA members to help them meet those expectations?
[Allow for open-ended response. If appropriate, probe on following...]

†
†
†
†
†
†
†

more peer review training
human-subject protection courses,
how to serve on an IRB,
negotiation skills, business etiquette, etc.
technology-specific workshops,
computer training,
how to conduct internet research,

16. Aside from providing the above training, what can the Office of Liaison Activities do to
improve the quality of participation of the CARRA members in your initiatives and
programs?

17. Do you have any other suggestions for improving the CARRA program?

D-7

ONLY IF TIME ALLOWS:

FINALLY, LET’S TALK BRIEFLY ABOUT THE REQUEST PROCESS THAT YOU
WENT THROUGH IN ORDER TO INVITE A CARRA MEMBER TO PARTICIPATE IN
YOUR PROJECT.

18. How did you request the CARRA member (s) with whom you cooperated/cooperate?

(PROBE: Are you the only person in your office in charge of requesting a CARRA member,
or are there more staff members who can do so?)
a. Did you find this process easy?
b. What did you like about this process?
c. What did you dislike about this process?
d. How could the Office of Liaison Activities make the request process better?

D-8

APPENDIX E

NCI STAFF MEMBER IN-DEPTH INTERVIEW PROTOCOL
(CARRA MEMBER NON-USERS)

E-1

E-2

NCI Staff Member In-depth Interview Protocol
CARRA Member Non-users

1.a Our questions today revolve around the use of consumer advocates in the work of NCI.
Are you familiar with the designation “consumer advocate”?

If not, Consumer advocates are members of the public (lay persons) who have been affected
by cancer, often personally, and who volunteer in a wide range of activities in order to
represent the collective viewpoint of the cancer community.

1.b. In your opinion, what is the value of inviting a consumer advocate to participate in
projects at NCI?

PROBE:

What about science-related projects?
What about other types of projects?

2. Have you ever involved consumer advocates in your programs at NCI?

IF YES,
a. In what capacity? What types of projects were they involved in?

b. How did you get in touch with those advocates?

E-3

3. Have you heard about the CARRA program? (Consumer Advocates in Research and
Related Activities.)
IF NOT, SKIP TO Q. 6
IF YES, PROBE:
a. Approximately when did you hear about it?

b. Do you recall how you learned about this program?

c. What have you heard about it? From whom?

d. How much awareness about the program is there among your colleagues, as far as you
can tell?

e. Have you mentioned it to others?

f. Have you ever been asked about it?

4. What is your opinion about the CARRA program?

a. How would you describe the potential value, if any, of CARRA members’ involvement
in research and related activity projects at NCI?

E-4

5. Have you ever involved CARRA members in your work?

IF PARTICIPATED IN THE PAST, SWITCH TO THE CARRA USERS PROTOCOL,
start with Q3
IF NEVER PARTICIPATED, CONTINUE...

6. NCI would like to see more consumer advocates provide input into NCI’s research and
communication activities. IN your opinion, what are the barriers to having consumer
advocates involved in the research process at NCI?

a. Thinking about your work environment, such as office leadership and culture, would
you say you feel encouraged to use consumer advocates in your work?

b. Do you have any suggestions for how to overcome barriers to involving consumer
advocates?

7. What is the primary reason that you have not sought out the CARRA program?

(PROBE: Is it the lack of skills, administrative issues, lack of time, other reasons?)

E-5

LET’S TALK A LITTLE BIT MORE ABOUT INVOLVEMENT OF CONSUMER
ADVOCATES IN RESEARCH AND OTHER ACTIVITIES AT NCI.

8. What skills, knowledge, and/or experience would make consumer advocates in general,
and CARRA members in particular, desired team members at NCI?

a. Probe on personal characteristics (Confidence, communication, etc.)

9. Do you believe that consumer advocates have those necessary skills, knowledge, and/or
experience? (PROBE: What are those skills?)

9.a. Are you aware that CARRA members receive training on peer review process, human
subjects protection, and other cancer research related topics?

10. What types of training or activities can the Office of Liaison Activities (OLA) undertake
in order to improve the skills, knowledge, and/or experience of consumer advocates?

E-6

11. Which of the following types of training would you recommend that the Office of Liaison
Activities provide to the CARRA members to help them meet those expectations?

†
†
†
†
†
†

more peer review training
human-subject protection courses
understanding the research process and medical statistics
how to serve on an IRB
negotiation skills, business etiquette, etc.
technology-specific workshops
computer training
how to conduct internet research

12. What can the Office of Liaison Activities do to encourage NCI staff in general to learn
about the CARRA program? [Allow for open ended response ON POTENTIAL MARKETING
/ COMMUNICATION MECHANISMS then probe on...}

† Posters or Standing Displays about CARRA
† Items that would increase name recognition such as pens/pads of paper with the
CARRA logo
† Presentations from OLA staff
† Articles in NCI Bulletin about CARRA
† Reading/hearing success stories of advocate involvement from other NCI staff,
† Recognition from the NCI Director for the utilization of advocates, etc.
† Encouragement / Support from leadership

13. What would convince you and more of your colleagues to use the CARRA program?

E-7

NOW LET’S TAKE A LOOK INTO THE FUTURE

14. In thinking about the initiatives and/or projects you anticipate in the next 18 months, are
there opportunities for consumer advocates to contribute?
a. In what way?
b. How many advocates?

15. We have discussed already some ideas for training to be offered to consumer advocates.
Aside from providing the appropriate training, what else can the Office of Liaison Activities
do to improve the quality of participation of consumer advocates/the CARRA members so
that they would meet your needs and expectations?

16. Do you have any additional suggestions for improving the CARRA program or the use of
consumer advocates in general?

E-8

APPENDIX F

LEADERSHIP FOCUS GROUP MODERATOR GUIDE

F-1

F-2

NCI LEADERSHIP - DISCUSSION GUIDE
FOCUS GROUP RESEARCH ON COMMUNICATING EFFECTIVELY ABOUT CARRA

National Cancer Institute
Consumer Advocates in Research and Related Activities (CARRA) Program
August, 2007
1. Welcome and Informed Consent (5 minutes)

2. Introductions (5 minutes)
a Interviewer and notetaker introductions
b Describe purpose of the group/What to expect
As you know from your invitation, our purpose today is to talk about the CARRA
program—specifically about how best to ensure that NCI staff are familiar with the
program, and have an accurate sense of its purpose and potential usefulness. This
discussion group is part of a larger project that has included interviewing CARRA
members and NCI staff, both those who have and have not utilized the CARRA program.
One thing we will do today is talk about some of what we have heard and see what you
think.
c Process items
− Open, honest opinions
− Will conclude on time
− Audiotaping, as noted in informed consent and as you can see by the tape
recorder on the table
− All input will be kept confidential, with findings reported in aggregate and not
attributed to any one person.
d Participant introductions
− Your name and what you do at NCI

3. Initial Understanding of CARRA (10 minutes)
We recognize that NCI is very large Institute with a huge number of programs and activities
underway simultaneously and CARRA is only one of them.
a How would you describe the purpose of the CARRA program?
b Every organization has its own culture. How does the CARRA program fit into the
culture of this Institute as you see it? [Probe for both positive/supportive aspects of
the context at NCI and anything seen as a barrier to/not supportive of participating
with CARRA.]
c We’ve gotten feedback from other NCI staff that there are instances in which it is a
“perfect fit” to involve a CARRA member and instances in which it is not ideal. Can
you give me some examples that we have not mentioned yet of each type of situation?

F-3

[Probe to differentiate between situations that very clearly should not involve
consumer advocates and those that are less clear (e.g., involvement would cost only
time or effort).]
4. Value Statements about Advocates (15 minutes)
a

Through research among NCI staff and CARRA members, we have identified some
specific potential benefits of working with CARRA that might be valuable to NCI
staff. [Give each participant a handout showing the bolded items below.]
•
•
•
•
•
•
•
•

CARRA advocates convey the patient perspective.
CARRA advocates provide powerful reminders to scientists why they do
what they do—especially in cases where basic scientific research activities
may feel far removed from healing human cancer.
CARRA advocates provide grounded “reality checks” about how patients
might respond in research and product development.
CARRA advocates can actively help identify issues of human subjects
protection because they are particularly attuned to such issues.
CARRA advocates bring valuable professional skills beyond their personal
survivorship perspective.
CARRA advocates are poised disseminate quality information in their
communities based on what they learn from their experiences at NCI.
CARRA advocates can take on roles that the government cannot, like using
what they learn in lobbying and fundraising efforts.
CARRA advocates come to better understand NCI and the scientific process,
and are better equipped to then accurately discuss those topics with others in
the community.
− When you read this list, which bullet item strikes you most in a positive way,
and why?
− Do any bullet items strike you negatively or raise questions for you?
− Is there anything you would add?

5. Communication Mechanisms (20 minutes)
a Now let’s move from the subject of what CARRA advocates might have to offer to
talking about how best to get information about CARRA to you and others at NCI.
The research team has spent several weeks talking with NCI staff in an effort to
understand the best ways to communicate about the CARRA program and CARRA
members. I’d like to get your reactions and talk more fully about what we’re hearing.
[Give each participant a handout showing the bolded items below.]
Take a moment to look over these items and jot down some of your thoughts,
reactions, or questions. [BRIEF PAUSE] I’m curious about what you’re thinking
and what you wrote… [OPEN-ENDED DISCUSSION. THEN, COVER ANY
QUESTIONS BELOW NOT COVERED IN OPEN DISCUSSION.]

F-4

•

•

•
•
•

•

•

b

Face to face interactions were advocated more strongly than written or
electronic approaches.
− What kinds of interactions might be most beneficial for you?
− Specifically, what pros and cons exist to reaching out via Division/All Hands
meetings? Summit meetings? Sponsored receptions?
− What kinds of “leave behind” materials would be useful to you?
Written communication can be effective sometimes, but mostly as reminders.
− What similar examples have you seen of effective written outreach?
− How often would be comfortable for you to be contacted with information
related to CARRA?
− What types of information would you like to see in these reminder-type
outreach efforts?
Static strategies such as posters, displays, and brochures were not viewed as
offering much value, but could possibly be useful as reminders.
Branded giveaways such as pens or pads of paper were perceived as
potentially lowering credibility, although some said such material may
support name recognition.
Support/Encouragement from top leadership—Director, Executive
Committee—in the form of announcements or messages was said to be
important, but not sufficient to move people to access the CARRA program
more frequently.
− Thinking about the reminders we spoke of, what would your thoughts be on
such reminders coming from leadership sources as compared to coming from
OLA?
Hard copy and/or web-based support materials could help to encourage use
of CARRA. Suggestions included concrete examples showing how CARRA
members have been effectively integrated into NCI work and a portfolio
providing biosketch details on members’ background and skills.
− What information would you be hoping to get from the CARRA website?
− What other support materials could simplify the process of accessing CARRA
members?
Allocating budget to engage consumer advocates would send a strong signal
and increase use of consumer advocates in NCI activities.
− Are there potential unintended consequences that come to mind?

One suggestion that was made during the interviews was this. Rather than only
telling NCI staff about CARRA and involving advocates, consider turning that
communication flow around as a way of enabling people to learn about CARRA and
consider its value. For example, staff members could be asked to share what they
know about cancer research that advocates should know or they could provide
training or programmatic input.
− What value, if any, do you see in the suggestion to use this means of
communication in service of increasing use of CARRA?

F-5

− How would you predict that this kind of approach be received by you and
those in your work environment?
− In the context of the kinds of communication outreach we’ve been discussing,
this is a somewhat novel idea. Are there other approaches to internal
communication at NCI, beyond typical communication mechanisms, that you
have in mind or have seen work?
c

Members of CARRA are currently called “consumer advocates.” We have heard
some hints that each term has some particular connotations. First, many interviewees
used the terms “patient advocate” or “community advocate.” Second, the term
“advocate” connoted for some an individual who is confrontational or focused only
on their own concerns.
− Can you speak to those concerns?
− Are there any ideas—even just brainstorming—for alternate terms?

[PERMISSION TO LEAVE FOR ANYONE WHO NEEDS TO.]

6. Wrap Up (5 minutes)
a One last question—Within the CARRA program, as part of their training, there is
some effort to bolster members’ scientific background. And, from earlier research,
we understand that CARRA members with scientific backgrounds may be more
valued and called upon more than others. Yet, we have also heard that NCI staff
value the unique, consumer perspective, rather than the scientific perspective, that
these consumer advocates offer. Please tell me your thoughts about reconciling this
tension.
b Before we close, is there anything that any one in the group would like to say about
our topic but has not had the opportunity?
c Thank participants for their time and helpful input.
d Ensure all who wish to have copy of informed consent form have one.

F-6

APPENDIX G

NCI STAFF FOCUS GROUP MODERATOR GUIDE

G-1

G-2

NCI STAFF - DISCUSSION GUIDE
FOCUS GROUP RESEARCH ON COMMUNICATING EFFECTIVELY ABOUT CARRA

National Cancer Institute
Consumer Advocates in Research and Related Activities (CARRA) Program
August, 2007
1. Welcome and Informed Consent (5 minutes)
2. Introductions (5 minutes)
a Interviewer and note taker introductions
b Describe purpose of the group/What to expect
As you know from your invitation, our purpose today is to talk about the CARRA
program—specifically about how best to ensure that NCI staff are familiar with the
program, and have an accurate sense of its purpose and potential usefulness. This
discussion group is part of a larger project that has included interviewing CARRA
members and NCI staff, both those who have and have not utilized the CARRA program.
One thing we will do today is talk about some of what we have heard and see what you
think.
c Process items
− Open, honest opinions
− Will conclude on time
− Audio taping, as noted in informed consent and as you can see by the tape
recorder on the table
− All input will be kept confidential, with findings reported in aggregate and not
attributed to any one person.
d Participant introductions
− Your name and what you do at NCI
3. Warm Up (10 minutes)
a NCI is very large Institute with a huge number of programs and activities underway
simultaneously. Using a scale of 1-10, how familiar would you say you are with the
work of the CARRA program? [Probe reasons.]
4. Initial Understanding of CARRA (10 minutes)
a How would you describe the CARRA program, say to a new colleague?
b

[FOR NON-CARRA-PARTICIPANTS] To make sure we hear a range of opinions,
we specifically invited people who have not participated in the CARRA program to
be at the table today. So, as the program looks to effectively define what it has to
offer, one obvious to question to ask you is, “What are the reasons that some people
do not participate?” Can you shed light on that for me?

G-3

c

[FOR CARRA PARTICIPANTS ONLY] Everyone at the table today has
participated with the CARRA program in some way. Tell me what prompted you to
request a CARRA member.
d. We’ve gotten feedback from other NCI staff that there are instances in which it is
a “perfect fit” to involve a CARRA member and instances in which it is not ideal.
Can you give me some examples that we have not mentioned yet of each type of
situation? [Probe to differentiate between situations that very clearly should not
involve consumer advocates and those that are less clear (e.g., involvement would
cost only time or effort).]

5. Value Statements about Advocates (10 minutes)
a

Through research among NCI staff and CARRA members, we have identified some
specific potential benefits of working with CARRA that might be valuable to NCI
staff. [Give each participant a handout showing the bolded items below.]
•
•
•
•
•
•
•
•

CARRA advocates convey the patient perspective.
CARRA advocates provide powerful reminders to scientists about why
scientists do what they do—especially in cases where basic scientific research
activities may feel far removed from healing human cancer.
CARRA advocates provide grounded “reality checks” about how patients
might respond in research and product development.
CARRA advocates can actively help identify issues of human subjects
protection because they are particularly attuned to such issues.
CARRA advocates bring valuable professional skills beyond their personal
survivorship perspectives.
CARRA advocates are poised to disseminate quality information in their
communities based on what they learn from their experiences at NCI.
CARRA advocates can take on roles that the government cannot, like using
what they learn in lobbying and fundraising efforts.
CARRA advocates come to better understand NCI and the scientific process,
and are better equipped to then accurately discuss those topics with others in
the community.
− When you read this list, which bullet item strikes you most in a positive way,
and why?
− Do any bullet items strike you negatively or raise questions for you?
− Is there anything you would add?

G-4

6. Communication Mechanisms (20 minutes)
a Now let’s move from the subject of what CARRA advocates might have to offer to
talk about how best to get information about CARRA to you and others at NCI. The
research team has spent several weeks talking with NCI staff in an effort to
understand the best ways to communicate about the CARRA program and CARRA
members. I’d like to get your reactions and talk more fully about what we’re hearing.
[Give each participant a handout showing the bolded items below.]
Take a moment to look over these items and jot down some of your thoughts,
reactions, or questions. [BRIEF PAUSE] I’m curious about what you’re thinking
and what you wrote… [OPEN-ENDED DISCUSSION. THEN, COVER ANY
QUESTIONS BELOW NOT COVERED IN OPEN DISCUSSION.]
•

•

•
•
•

•

Face to face interactions were advocated more strongly than written or
electronic approaches.
− What kinds of interactions might be most beneficial for you?
− Specifically, what pros and cons exist to reaching out via Division/All Hands
meetings? Summit meetings? Sponsored receptions?
− What kinds of “leave behind” materials would be useful to you?
Written communication can be effective sometimes, but mostly as reminders.
− What similar examples have you seen of effective written outreach?
− How often would be comfortable for you to be contacted with information
related to CARRA?
− What types of information would you like to see in these reminder-type
outreach efforts?
Static strategies such as posters, displays, and brochures were not viewed as
offering much value, but could possibly be useful as reminders.
Branded giveaways such as pens or pads of paper were perceived as
potentially lowering credibility, although some said such material may
support name recognition.
Support/Encouragement from top leadership—Director, Executive
Committee—in the form of announcements or messages was said to be
important, but not sufficient to move people to access the CARRA program
more frequently.
− Thinking about the reminders we spoke of, what would your thoughts be on
such reminders coming from leadership sources as compared to coming from
OLA?
Hard copy and/or web-based support materials could help to encourage use
of CARRA. Suggestions included concrete examples showing how CARRA
members have been effectively integrated into NCI work and a portfolio
providing biosketch details on members’ background and skills.
− What information would you be hoping to get from the CARRA website?
− What other support materials could simplify the process of accessing CARRA
members?

G-5

•

b

Allocating budget to engage consumer advocates would send a strong signal
and increase use of consumer advocates in NCI activities.
− Are there potential unintended consequences that come to mind?

One suggestion that was made during the interviews was this. Rather than only
telling NCI staff about CARRA and involving advocates, consider turning that
communication flow around as a way of enabling people to learn about CARRA and
consider its value. For example, staff members could be asked to share what they
know about cancer research that advocates should know or they could provide
training or programmatic input.
− What value, if any, do you see in the suggestion to use this means of
communication in service of increasing use of CARRA?
− How would you predict that this kind of approach be received by you and
those in your work environment?
− In the context of the kinds of communication outreach we’ve been discussing,
this is a somewhat novel idea. Are there other approaches to internal
communication at NCI, beyond typical communication mechanisms, that you
have in mind or have seen work?

c

Members of CARRA are currently called “consumer advocates.” We have heard
some hints that each term has some particular connotations. First, many interviewees
used the terms “patient advocate” or “community advocate.” Second, the term
“advocate” connoted for some an individual who is confrontational or focused only
on their own concerns.
− Can you speak to those concerns?
− Are there any ideas—even just brainstorming—for alternate terms?

[PERMISSION TO LEAVE FOR ANYONE WHO NEEDS TO.]
7. Wrap Up (5 minutes)
e One last question—Within the CARRA program, as part of their training, there is
some effort to bolster members’ scientific background. And, from earlier research,
we understand that CARRA members with scientific backgrounds may be more
valued and called upon more than others. Yet, we have also heard that NCI staff
value the unique, consumer perspective, rather than the scientific perspective, that
these consumer advocates offer. Please tell me your thoughts about reconciling this
tension.
f Before we close, is there anything that any one in the group would like to say about
our topic but has not had the opportunity?
g Thank participants for their time and helpful input.
h Ensure all who wish to have copy of informed consent form have one.

G-6

APPENDIX H

FOCUS GROUP HANDOUTS

H-1

H-2

FOCUS GROUP HANDOUT RE: VALUE
National Cancer Institute (NCI)
Consumer Advocates in Research and Related Activities (CARRA) Program

August, 2007
Examples of the Value CARRA Advocates Potentially Offer to NCI
•

CARRA advocates convey the patient perspective.

•

CARRA advocates provide powerful reminders to scientists about
why scientists do what they do—especially in cases where basic
scientific research activities may feel far removed from healing
human cancer.

•

CARRA advocates provide grounded “reality checks” about how
patients might respond in research and product development.

•

CARRA advocates can actively help identify issues of human subjects
protection because they are particularly attuned to such issues.

•

CARRA advocates bring valuable professional skills beyond their
personal survivorship perspectives.

•

CARRA advocates are poised to disseminate quality information in
their communities based on what they learn from their experiences at
NCI.

•

CARRA advocates can take on roles that the government cannot, like
using what they learn in lobbying and fundraising efforts.

•

CARRA advocates come to better understand NCI and the scientific
process, and are better equipped to then accurately discuss those
topics with others in the community.

H-3

GROUP DISCUSSION HANDOUT RE: COMMUNICATION
National Cancer Institute (NCI)
Consumer Advocates in Research and Related Activities (CARRA) Program

August, 2007
NCI Staff Interviewees’ Input on the Best Ways to Communicate About
the CARRA Program and CARRA Members
•

Face to face interactions were advocated more strongly than written
or electronic approaches.

•

Written communication can be effective sometimes, but mostly as
reminders.

•

Static strategies such as posters, displays, and brochures were not
viewed as offering much value, but could possibly be useful as
reminders.

•

Branded giveaways such as pens or pads of paper were perceived as
potentially lowering credibility, although some said such material may
support name recognition.

•

Support/Encouragement from top leadership—Director, Executive
Committee—in the form of announcements or messages was said to be
important, but not sufficient to move people to access the CARRA
program more frequently.

•

Hard copy and/or web-based support materials could help to
encourage use of CARRA. Suggestions included concrete examples
showing how CARRA members have been effectively integrated into
NCI work and a portfolio providing biosketch details on each
member’s background and skills.

•

Allocating budget to engage consumer advocates would send a strong
signal and increase use of consumer advocates in NCI activities.

H-4

APPENDIX I

THANK YOU LETTER SENT FROM PROJECT DIRECTOR

I-1

I-2

Thank You Email Message to Respondents

August 2007
Dear
On behalf of the National Cancer Institute (NCI), NCI’s Office of Liaison Activities, and
the Westat research team, I want to thank you for participating in the most recent round
of data collection on the CARRA Program (Consumer Advocates in Research and
Related Activities). We appreciate the generosity of your time and your thoughtful
reflections.
The information you shared with us will be aggregated and summarized into a report for
the Office of Liaison Activities (OLA) which manages the CARRA program. Your name
and identity will not appear in any report or product delivered to OLA. The findings will
be used by OLA to make modifications to the CARRA program and strengthen the role
of consumer advocates in the work of NCI.
Thank you again for your contributions to this important effort.
Keith MacAllum, PhD
Director, CARRA Evaluation
Westat
1650 Research Blvd.
Rockville, MD 20850

I-3

I-4