Principal Investigators and project directors (In-depth Interviews(PSLA)

ATTACHMENT C

PSLA—IN-DEPTH TELEPHONE INTERVIEWS WITH PIS AND PDS

ARRA COMPARATIVE EFFECTIVENESS RESEARCH EVALUATION (ACERE)

INTERVIEW PROTOCOL FOR PRINCIPAL INVESTIGATORS AND PROJECT DIRECTORS; PROJECT-SPECIFIC LEVEL OF ANALYSIS

Description of the protocol

The purpose of qualitative interviews with principal investigators (PIs) and project directors (PDs) is to learn more about the factors that limit or facilitate success, challenges faced, limitations, and missed opportunities, unanticipated positive benefits, or unintended negative consequences for a sample of ARRA-funded CER projects. We will conduct up to 50 hour-long telephone interviews with PIs or PDs from a mix of different projects (see the draft approach of the project-specific level of analysis for more details).

This protocol consists of five sections. In the General Introduction and Background section, we briefly present the purpose of the project and interview to the interviewee; such information will also be included in a transmittal email message sent to all candidates for interviews. In the Context for Your CER Project section, we discuss the research objectives of the project with the PI/PD, including the short- and long-term goals of the work. Interviewers will tailor this section based on what they already know about the project via either the evaluation document review or, perhaps, the PI-PD survey. In the Conducting Your CER Project section, we ask about the interviewee’s internal and external research environments and begin a discussion of challenges. In the Project Outputs and Outcomes section, we ask interviewees to describe any products they may have already produced during the funding period, how the ARRA CER project has shaped their future research plans, and what challenges, successes, and unintended consequences (favorable and unfavorable) they have encountered to date. Finally, in the Wrap Up section, we provide an opportunity for interviewees to share any final thoughts.

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990- . The time required to complete this information collection is estimated to average 1 hour per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Health & Human Services, OS/OCIO/PRA, 200 Independence Ave., S.W., Suite 336-E, Washington D.C. 20201, Attention: PRA Reports Clearance Officer

The total burden for this protocol is 50 hours.

1. GENERAL INTRODUCTION AND BACKGROUND (5 minutes)

We appreciate you taking the time to speak with us today. Before we begin, let me introduce myself and tell you a little bit about the evaluation that Mathematica Policy Research is conducting for the Assistant Secretary for Planning and Evaluation (ASPE) of the U.S. Department of Health and Human Services.

My name is [name], and I work for Mathematica. (If note taker, introduce him/her as well.)

The evaluation we are conducting for ASPE is a midstream impact assessment of the American Reinvestment and Recovery Act (ARRA) comparative effectiveness research (CER) portfolio. As you probably know, ARRA expanded federal resources devoted to this research by directing $1.1 billion to HHS. As part of our evaluation, we are conducting a number of in-depth interviews with principal investigators and project directors to get their feedback about their ARRA-funded CER project(s).

Our primary goals for today are to discuss your role in working on ARRA-funded CER projects and, more importantly, obtain your feedback on:

• The factors that facilitate and limit success of your project(s);

• Challenges you have faced and what you are doing to overcome them;

• Limitations to your project(s) as well as potential missed opportunities or unintended consequences (both favorable and unfavorable); and

• The successes you and your project(s) have experienced.

The results of our discussion will be synthesized in a final report and only general themes that emerge from our discussions will be reported. We will not attribute specific comments or quotes to named individuals without permission and your individual answers will be kept private to the extent permitted by law. Only the Mathematica evaluation team will have access to individually identifiable information.

We expect this discussion to take about one hour, and your participation is voluntary. Any questions before we begin?

2. CONTEXT FOR YOUR CER PROJECT (5 minutes)

I’d like to start with a broad question about comparative effectiveness research, which is sometimes also called patient-centered outcomes research, before we turn to your specific project(s).

1. With all the attention paid to CER, different definitions are emerging and evolving. In your own words, how would you define CER to a colleague who is unfamiliar with it?

[Note to Interviewer: Revise the exact wording of this next question based on information we have already gathered to sound like you already know a little bit about this project.]

2. Now, I would like to learn about your project. Can you describe at a very high level (within a minute or two) the research objectives of your CER project(s), [shortened version(s) of project name(s)]?

1. What are the short- and long-term goals and outcomes of your CER project(s)? (Probe for project-specific information based on what is known from document review and possibly the survey. Probe for what have been the most successful project components to date.)

2. CONDUCTING YOUR CER PROJECT (20-25 minutes)

Now let’s discuss how your project(s) [is/are going / went]. I’m interested in your experiences to date, including the challenges, successes, and surprises you’ve encountered along the way.

3. What factors have facilitated you and your team’s ability to carry out your CER project(s)?

1. What factors are missing that would have been helpful? (Probe: Intramural or extramural funding availability? Data or patient availability? Researcher/investigator time? Other local expertise in CER? Appropriately trained research team?)

3. You submitted your project(s) for funding under [RFA/RFP funding mechanism(s) identified through the document review]. In what ways do you think that funding mechanism is promoting or hindering your project(s)? [Note to Interviewer: This might have been discussed under #3; if already explicitly addressed, this can be skipped.]

3. In what ways were any stakeholders (meaning individuals who would benefit from information based on your project, such as payers, providers, and other decision makers) involved in developing your proposal(s) or participating in your project(s)?

1. What types of stakeholders have been involved? At what point did you involve them? In what ways did you engage them? (Probe: In-person meetings, phone, etc.)

2. How do you think involving stakeholders has affected your work?

3. Would you do anything differently if you were to work with stakeholders again?

3. Are you working with collaborators on your CER project(s), either at your own institution or elsewhere?

1. Were these collaborations already in place, or are they new?

2. How were the collaborations established?

3. In what ways are you collaborating?

4. What sort of collaborations are you considering for the future? Why?

Qs 7 and 8 are lower priority and should be skipped if time is short.

[Note to Interviewer: There are important differences between academic and non-academic institutions and large or small institutions. Familiarize yourself on where the investigator’s institution falls among these characteristics.]

3. What about [name of institution]’s research supports? Is the research infrastructure at your organization sufficient to conduct CER on your topic? What about other CER topics?

1. What do you need to carry out CER effectively? (Probe: What about supports for research in general, such as an easily accessible mechanism to facilitate patient recruitment or adequate computing, programming, and statistical services? How about institutional supports that are more CER-specific, like expertise in CER methodology? [Note to Interviewer: Tailor these probes based on what is known about the project and the institution.])

3. To what extent does [name of institution] explicitly prioritize CER [grants/contracts]?

1. In what ways does [name of institution]’s leadership implement this priority?

3. PROJECT OUTPUT AND OUTCOMES (20-25 minutes)

[Note to Interviewer: Please adjust the wording in this section depending on whether the project you are talking about has ended or is ongoing.]

Let’s switch gears and discuss what your project [is producing/produced]. I’m interested in what’s happened so far, as well as how you’re thinking about conducting more comparative effectiveness research in the future.

9. Many of the non research projects funded under ARRA are nearing completion, but some ARRA research projects had different schedules. [Some parts of your project(s) are probably not yet complete/I understand your project is now complete.] How do you feel your work [is progressing/progressed] relative to its schedule?

1. [Is/Was] that schedule more or less what you had originally proposed, or did you have to make substantive changes to what you were doing in your project? (Probe for details on how extensive changes were, why they were necessary, role of funding agency/project officer in making changes, when in the project timeline changes occurred, and perceived effect of changes on project outcomes.)

[Note to Interviewer: In what follows, choose questions based on a review of details of the project that we already have collected. For projects that fall into more than one category, ask questions relevant to all project aspects. Questions that we ask will include those that elaborate on details or ask any remaining questions about details. Keep in mind that some of these questions might have been answered earlier.]

(Probe for project-specific details based on document review, survey, and previous interview responses:

If project categorized as Research:

Are you developing novel uses for existing data resources?

What do you hope your project will contribute to the existing knowledge base on CER in your field?

What are your plans for disseminating the findings of your project to the real-world decision makers it was intended to inform?

If project categorized as Human and Scientific Capital:

Tell me more about the training [program/activities] you are [developing/expanding]. What sort of activities are in the works? How many trainees are you overseeing? Will your program continue after the ARRA funding ends?

Can you describe, in layman’s terms, the methods you are developing? How do you see those methods being used or contributing to the CER methods knowledge base? How are you getting the word out?

If project categorized as Data Infrastructure:

For projects that are directly related to data:

I understand your project involves [creating/linking] datasets. What do the datasets include (e.g., # cases, conditions covered)? Do you anticipate those datasets to become publicly available? Will they be self-sustaining, such as through user fees?

Are you using any existing data resources in novel ways?

For projects that are primarily infrastructure:

How has your project improved you or your institution’s ability to conduct CER?

Do you think the activities that have been funded with ARRA monies are sustainable? What factors make them more or less sustainable?

If project categorized as Dissemination and Translation:

What can you tell me about the [workshops/symposia/other convening activity] you are organizing? What are your topics, and who is in your target audience? Are those the people who are actually attending? How many have participated to date? Will your activities continue after ARRA funding ends?

What sorts of new dissemination and translation methods are you developing?

9. Have there been any significant challenges or barriers that you haven’t already mentioned in conducting your project(s)? (Probe: Delays related to IRB, HIPAA, or OMB approval/clearance; data not being readily available; timeline issues.)

1. What have you done, or do you intend to do, to overcome these challenges or barriers?

2. If you were conducting the project again, is there anything you would do differently? (Probe: Were there any missed opportunities?)

9. What have been the impacts on knowledge or practice that have resulted from your ARRA-funded CER project(s) so far?

9. How has your ARRA-funded CER affected your research priorities and career plans? (Probe: Are you considering building on your current CER work or going in a new direction?)

1. If necessary: How likely are you to conduct future CER projects? (Probe if new direction: What new topics are you interested in?)

9. How aware are you of the Patient Centered Outcomes Research Institute (PCORI) that was established in the Patient Protection and Affordable Care Act?

1. If aware: In what ways has the establishment of PCORI affected your future research plans?

9. What recommendations would you make to federal agencies and/or PCORI as they decide how to proceed with funding future CER and CER-related projects? (Probe: Are there any future directions for CER research that you see as being high priority that warrant future federal investment?)

4. WRAP-UP (5 minutes)

We’re almost done. I really appreciate you talking with me today about your project. I have really learned a lot about it and all the information you’ve provided has been invaluable to our evaluation. Before we end the call, I wanted to know...

15. Is there anything else you’d like to add about your project that we have not touched on? One last item that you really think I should know?

Check with note taker to see if anything was missed or if he/she has any follow-up questions.

Thank you for taking the time to speak with us. These insights are very valuable to our evaluation for ASPE. If we notice we missed anything while reviewing our notes, would you mind if we followed up with you?

Thanks again for your time.