Attachment F: Pretesting_Policymaker Script

Script for policymaker interview [pretesting]

Topic of interview: [TOPIC]

Intended audience: [policymaker]

Location: [enter location]

Date of the interview: [enter date]

Testing materials

• Audio recording equipment, speaker phone

• Interviewer clock

• Materials emailed to policymaker 1 day before interview:

  • PDF version of policymaker product

  • Consent form

• Packet of testing materials for interviewer

  • 1 version of policymaker product

  • Informed consent form

Procedures for obtaining informed consent

Policymaker will be sent an informed consent form before the interview. At start of interview, interviewer will ask particpant if he/she has any questions about the consent form and if agrees to be interviewed and audiotaped.

Testing goals

Testing will use draft material, sent to interviewee prior to interview via email. This translation product focuses on [CONDITION/TREATMENT]. Interviewer will navigate policymaker through materials as needed to focus on specific points or sections – namely, the following:

• Usability of information presented: Is information presented in a way that policymakers can understand and use?

• Understanding of information presented: Can users understand and correctly interpret information as shown? What changes to format or layout may be needed? What text changes may be needed?

Policymaker interview (60 minutes total)

Time in minutes for each section	Topic	Elapsed time at end of section
5	Introduction (welcome; background; ground rules; warm-up)	5
10	OVERALL IMPRESSIONS	15
5	Key policy issues	20
5	Background information	25
10	Bottom line	35
5	Outcomes table	40
5	A note regarding this Policymaker Summary	45
3	Gaps in knowledge	48
5	SUMMARY	52
5	FORMATIVE & DISSEMINATION	57
3	Closing	60

Introduction (start at _____ – 5 min – end at _____)

Welcome and Background—explain purpose of the interview

• Thank you for agreeing to do this interview. My name is [NAME] and I’ll be talking with you today. Also on the phone is [NAME], who will be taking notes. I work for a company called [ORGANIZATION NAME].

• We are testing information products developed by the Eisenberg Center at Baylor College of Medicine and sponsored by AHRQ, the Agency for Healthcare Research and Quality. These products are summaries of systematic reviews that are written at various Evidence-based Practice Centers around the country. The Policymaker’s Product on [TOPIC] we have asked you to provide feedback on is an example of the products.

• Based on feedback we get from you and others we will make final recommendations to AHRQ about how to present the information. Once the products are released to the public, they will be available on the AHRQ Web site along with the source document.

• Today, we’re going to be focusing on how the information is expressed and conveyed and less on the conclusions drawn. However, if there is anything that is problematic with the content, we would like to know.

• Did you read the consent form that was sent to you? Do you have any questions?

Go over ground rules.

• Everything you tell us will be confidential. To protect your privacy, we won’t connect your name with anything that you say.

• Please give us your honest feedback. We did not develop the materials and will not be offended by anything you say.

• Here’s how the interview will work: I’ll ask you to examine the materials and tell me what you think about them. As you examine the reports, I need you to think out loud so I can follow and understand your thoughts. We’ll be talking for about an hour.

• Is it OK if I audiotape this interview today?

{Turn on recording equipment.}

Warm-Up

I would like to begin with a couple of general background questions.

• What is your role/position and responsibilities?

• How many years have you been in this job/position?

• How would you describe your organization/healthcare system?

  • Which stakeholders do you represent or interact with most often?

  • What types of decision making problems you most often encounter?

Overall Impressions (10 min – end at ______)

Now, let’s talk about the summary.

{Interviewer should follow the lead of the participant in how information is offered. Spontaneous comments are encouraged. Limited guidance is offered in the early stages of the interview. The interviewer uses the probes/prompts below if the participant does not spontaneously offer feedback on the materials. After all of the material is covered the list of standardized summary questions are asked.}

• Could you tell me if you are looking at an electronic or print copy of the summary?

  • If print: Is it a color or black and white copy? Does the copy print OK? Are you able to read it easily?

• What do you think of the overall layout and design of the summary?

  • Which information would you be likely to read first?

  • Is there anything you would change about the design?

• What do you expect to find inside the summary based on the title?

• Is it clear from the first page who sponsored this brochure?

  • How does that make you feel about the brochure?

  • What sources do you trust the most to provide health information like this? What sources are less trustworthy? (If necessary, probe specifically on government, university, non-profit, pharmaceutical companies, other private or corporate sources)

Key policy issue/s (4 min – end at ______)

Thanks. This is exactly the kind of information we are looking for. Now, I’d like to go section by section and ask specific questions about each. Let’s start on the first page at the key policy issue/s.

• How would you summarize the main points of this section to a colleague?

Background information (5 min – end at ______)

• Was this the kind of information you expected to find under this heading? Is there anything you would add? Anything you would take out?

• How would you summarize the main points of this section to a colleague?

• Is there anything that you think other policymakers would find confusing or have a hard time understanding in this section?

Conclusion (5 min – end at ______)

• What are your reactions to the information in this section? How would you explain it in your own words?

• Is there anything missing here or information you feel like should be taken out?

• Is there anything that you think other policymakers would find confusing or have a hard time understanding in this section?

Bottom Line / Confidence scale (10 min – end at ______)

• Based on the heading, what information did you expect to find?

• What are your overall reactions to this chart?

• What are your thoughts on the confidence scale?

  • Does it help you to evaluate the information in the bottom line?

• What do you think the authors meant by the phrase “level of confidence?”

  • How would you characterize the level of confidence (or evidence) for the information in this chart?

  • How would you explain the relationship between the level of confidence and confidence scale? Is there another way you would like to visualize the level of confidence?

• {Go row by row} What do you think the authors meant by the conclusions in this row? How would you explain it in your own words?

  • [For rows with one bullet] What are your reactions to the low level of confidence in this row? How would you use that information?

• How does the information in this table compare to the information in the conclusions section?

• Is there anything that you think other cpolicymakers would find confusing or have a hard time understanding in this section?

Outcomes table (10 min – end at ______)

• What are your reactions to the information in this table? How would you explain it in your own words?

• How would you use the information in this table? Is there any additional information you would like to see in this table?

• Is there anything that you think other policymakers would find confusing or have a hard time understanding in this section?

A note regarding this Policymaker Summary (3 min – end at ___)

• What do you think of the title to this section?

• What are your reactions to the information in this box? What do you think of the number of studies included in this review? What do you think the authors meant by the term “independent researchers”? How would you define that in your own words?

• What are your reactions to the topic nomination and review process?

• Are you interested in looking at the full report with references, or not? How likely would you be to look at the full report?

• Overall, how does this information affect the credibility of this summary, if at all? [If negative reaction previously] Does this change your opinion or reactions to the information we already talked about?

• Is this information important to you? If so, does it stand out on the page?

• What do you think of the placement of this information on the page? Would you like to see in another section or is it OK where it is?

• Is there anything that you think other policymakers would find confusing or have a hard time understanding in this section?

Gaps in knowledge / Additional issues (4 min – end at ______)

• Based on the heading, what information did you expect to find?

• {Go bullet by bullet} What do you think the authors meant by the information in this bullet? How would you explain it in your own words?

• What are your overall reactions to this section? How would you summarize the main points of this section to a colleague?

Ordering Information / Source (2 min – end at ______)

• What are your reactions to the materials available for ordering? Would you be likely to request any of these materials?

• What do you think about the source material?

  • Did you notice the source information when you first looked at this page?

Summary (5 min – end at ______)

• If you were going to summarize this product for a colleague, what would you say?

• If you were to rate the usefulness of these materials on a scale of 0-10, where 0 = not useful at all and 10 = extremely useful, how would you rate the usefulness of the information in this material for policymakers?

• How do you think you might use this information?

  • Would this information be helpful in decision making? Why or why not?

• Was there any information new to you?

  • Anything surprising?

• What do you think is the most important information in this product?

  • How well does that information stand out?

• Did you see any information that seemed to be inaccurate to you?

• Did you see anything that might be perceived as controversial?

  • Are there any red flags here? Did anything jump out at you that might be a problem for AHRQ?

• How credible or trustworthy do you find this information? Do you believe the findings presented in this product?

• Overall, did you like the way the information was presented? Why or why not?

• What did you think about the order in which the information was presented?

• Do you have any other advice on how to improve the content or format of this booklet?

Formative & Dissemination (4 min – end at ______)

• How do you normally prefer to receive materials like this research summary?

  • Where do you usually get information like this?

  • Are there organizations that you are affiliated with that provide products like this?

• Are you familiar with the term “evidence-based medicine”?

  • (If yes) What does that mean to you?

  • Do you use evidence-based information for policy decision making? If so, how?

  • What information would help you with decision making? (If necessary, give examples such as guidelines, algorithms, outcomes, cost to patients, or something else).

  • How is your decision making for recommending a particular intervention or policy impacted when the evidence is unclear or insufficient?

• Where do you usually get evidence-based information?

  • How would you like to get this type of information (for example, handouts, emails, Internet)?

  • [If typically gets from journal] When do you get a chance to read the journals? Are there other formats that would be useful to you? Would you be interested in reading summaries of the research articles, or not?

• Are you familiar with the term “comparative effectiveness research”?

  • (If yes) What does that mean to you?

  • How is comparative effectiveness research the same or different from evidence-based information?

  • Do you use comparative effectiveness research for decision making? If so, how?

• How do you learn about new ways to treat patients with [CONDITION]?

Closing (1 min – end at ______)

Before we end, I’d like to give you chance to share any additional thoughts or comments about the information we talked about today.

• Is there anything else you would like to add that you didn’t have a chance to say during our discussion today, or something that we didn’t talk about that you wish we had?

Thank you very much for participating in this discussion today. We appreciate your time.

8