Cognitive Testing Guide

Draft FDA Ad Exposure Frequency Cognitive Testing Guide

Round 2 interviews

Survey Item testing

11.17.14

Thank you for agreeing to participate in this study today. During today’s interview, we will show you a TV commercial and ask you to complete a survey afterwards. Your feedback will help us to strengthen the survey.

First, I’d like to provide an informed consent form that describes the study—and your responsibilities—in more detail.

[REVIEW INFORMED CONSENT]

[Note: Participants will be blind to FDA’s sponsorship]

SCREENING / INTRODUCTION / CONSENT

Thank you again for agreeing to participate.

[INSERT OMB LANGAUGE AND NUMBER]

[INSERT CONSENT LANGUAGE-OMB control number to appear on every page]

This survey is being conducted by RTI International (RTI), an independent nonprofit research organization, on behalf of a public health agency.

Now we would like you to view the ad.

[SHOW PARTICIPANT STIMULUS 2 times]

SURVEY INSTRUCTIONS

Now that you have viewed the ad, I would like to ask you some questions about it.

[Recall drug name]

1. One of the ads you saw was for a medication for seasonal allergies. Do you recall the name of that drug? Was it:

• Rheutopia

• Trinase

• Coravaz

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• How did you choose your answer?

• Did you recognize either of the other two drug names?

• How much attention did you pay to the ad?

• What were your general impressions of that ad?

• How did the ad compare to other ads you have seen for prescription medications?

  • Was it similar or different?

  • Was the ad believable?

  • Is there anything about the ad you would change?

Now, I’d like to ask you to answer some survey questions.

Please answer the following questions based on the ad you saw.

As you review the survey, I’d like you to read the instructions and questions aloud and then “think aloud” as you answer each question. This may feel a little unnatural, but it will help us to understand how you think about and answer each question. Here’s an example:

Question: How many times did you go to a doctor’s office for a scheduled appointment in the past three months?

Answer: Well, I see my heart doctor every month, so that’s three visits. I see my primary care doctor twice a year, but I didn’t go in the last three months. I also had to go to urgent care last week for a sinus infection…but that wasn’t a scheduled appointment. I guess my answer is three visits.

After each survey question, I also may ask you some follow-up questions. We are interested in your initial impressions and honest opinions. There are no right or wrong answers, and it is ok to have strong opinions. Please feel free to use the entire range of response options.

Let’s begin…

[Counter arguing (# of negative thoughts)]

2. The name of the seasonal allergy drug you saw advertised was Trinase. Please think of the Trinase ad you watched. List all thoughts (including unrelated thoughts) that came to your mind as you watched it.

[INCLUDE 5 TEXT BOXES WITH CHARACTER COUNTERS]

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• Was this enough space to fill in all of the thoughts you had?

• What did you think of the character counter?

2b. Now that you have listed your thoughts, please rate each one as either positive (+), negative (-) or neutral (0):

[INCLUDE SMALL BOX NEXT TO EACH THOUGHT FOR RATING]

Cognitive Testing Probes

• How easy or difficult was the rating? What would make it easier/more difficult?

[RANDOMIZE ORDER OF QUESTION BLOCK 3-15 and QUESTION 16]

[RANDOMIZE ORDER OF Q3 and Q4]

[Recall-Risks (unaided)]

3. What are the side effects or risks of Trinase? (Please list as many side effects or risks as you can remember.) [open ended]

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• Was this enough space to fill in all of the thoughts you had?

• What do you think of when you hear the word “side effects”? What about “risks”? Do those mean the same thing to you?

• What do you think of when you hear the words “negative effects”? What about “warnings”? Do those mean the same thing to you as “side effects” or “risks”?

[Include same number of text boxes as number of risks in ad-(12)]

[Recall-Benefits (unaided)]

4. What are the benefits of Trinase? (Please list as many benefits as you can remember.) [open ended] [Include same number of text boxes as number of benefits in ad-(5)]

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• Was this enough space to fill in all of the thoughts you had?

• What do you think of when you hear the word “benefits”?

5. What else do you remember from the Trinase ad? [open-ended]

[INCLUDE TEXT BOX WITH CHARACTER COUNTER]

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• Was this enough space to fill in all of the thoughts you had?

• What kinds of things were you thinking of?

[RANDOMIZE ORDER OF Q6 and Q7]

[Recognition-Risks]

6. Please check which of the following statements were mentioned in the ad as side effects or risks of taking Trinase. Select “Mentioned in the Ad” if the side effects or risks are mentioned in the ad, even if the statement does not match word for word what you recall from the ad. Even if you think a statement is true, please select it only if it was mentioned in the ad. Check all that apply.

[RANDOMIZE ORDER]	Mentioned In Ad	Not Mentioned In Ad
a. The most common side effects of Trinase include headache, viral infection, sore throat, coughing and nosebleeds.	X
b.  Trinase can cause nausea in some people.		X
c.  Some people may experience eye problems from Trinase such as glaucoma or cataracts.	X
d.  Trinase may cause slow wound healing.		X
e.  In rare cases, Trinase can cause severe allergic reactions.	X
f.  Trinase can cause extreme dizziness in some people.	X
g.  Contact your doctor if you experience sudden changes in hearing.		X

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• How clear or unclear were the instructions, particularly the 2^nd and 3^rd sentences in the instructions?

• Were any of the statements more difficult than others to choose an answer for? Did you guess on any of the statements? If so, which ones? (i.e. were there any that you were unsure of?)

[INTERVIEWER: NOTE ANY SIGNS OF FATIGUE OR FRUSTRATION.]

[Recognition-Benefits]

7. Please check which of the following statements were mentioned in the ad as benefits of taking Trinase. Select “Mentioned in the Ad” if the benefits are mentioned in the ad, even if the statement does not match word for word what you recall from the ad. Even if you think a statement is true, please select it only if it was mentioned in the ad. Check all that apply.

[RANDOMIZE ORDER]	Mentioned In Ad	Not Mentioned In Ad
a.  Trinase is taken once a day.	X
b.  Trinase is non-drowsy.		X
c.  Trinase is available in pill form.		X
d.  Trinase can treat runny nose.	X
e.  Trinase is non-habit forming.	X
f. Trinase can also be used to treat the common cold.		X
g.  Trinase can treat nasal congestion.	X

[RANDOMIZE ORDER OF QUESTION BLOCK 8-10 and QUESTION BLOCK 11-13]

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• How clear or unclear were the instructions?

• Were any of the statements more difficult than others to choose an answer for?

• Did you guess on any of the statements? If so, which ones? (i.e. were there any that you were unsure of?)

[INTERVIEWER: NOTE ANY SIGNS OF FATIGUE OR FRUSTRATION.]

[Perceived Risk (Likelihood)]

8. In your opinion,if 100 people take Trinase, how many will have any side effects or risks?

Please enter a number between 0 and 100 in the box below.

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• How did you choose your answer? Was it based on any specific words from the ad?

• What would make this question easier to answer?

9. In your opinion, if you were to take Trinase, how likely would you be to have side effects or risks?

• Not at all likely

• Slightly likely

• Moderately likely

• Very likely

• Extremely likely

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• What did you think of the answer options?

• What type of side effects come to mind? (Note whether respondent mentions life-threatening side effects or milder side effects.)

• What did you consider when thinking about your answer? (e.g. current health condition, past experience with other prescription drugs for seasonal allergies)

[Perceived Risk (magnitude)]

10. If you did have side effects or risks, how serious do you expect they would be?

• Not at all serious

• Somewhat serious

• Moderately serious

• Very serious

• Extremely Serious

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• What would make it easier?

• What does the word “serious” mean to you? (i.e. What did you think when you see the word “serious”?)

[Perceived Efficacy (Likelihood)]

RANDOMIZE ORDER

11. In your opinion, if 100 people take Trinase, for how many will the drug work?

Please enter a number between 0 and 100 in the box below.

Cognitive Testing Probes How easy or difficult was this question to answer? What would make it easier? What does “the drug working” mean to you? Do any other words come to mind when you see the words “the drug working”? Which answer would you choose if the answer options were “most,” “some” or “all”?

12. In your opinion, if you were to take Trinase, how likely is it that the drug would work for you?

• Not at all likely

• Slightly likely

• Moderately likely

• Very likely

• Extremely likely

Cognitive Testing Probes

• Which question is easier to answer, number 11 or 12?

• What is the main difference between these two questions?

• What does “the drug working” mean to you? Do any other words come to mind when you see the words “the drug working”?

[Perceived Efficacy (Magnitude)]

13. In your opinion, how effective would Trinase be in helping your seasonal allergies?

• Not at all effective

• Slightly effective

• Moderately effective

• Very effective

• Extremely effective

Cognitive Testing Probes

• How easy or difficult is this question to answer?

• How did you select your answer?

• What is meant by “how effective” would TRINASE be in helping your seasonal allergies?

• What other words come to mind when you see the words “how effective”?

[Risk/benefit balance]

14. Think about the risks and benefits of Trinase. How would you rate the drug overall?

• Risks completely outweigh benefits

• Risks mostly outweigh benefits

• Risks slightly outweigh benefits

• Risks and benefits are equal

• Benefits slightly outweigh risks

• Benefits mostly outweigh risks

• Benefits completely outweigh risks

Cognitive Testing Probes

• How easy or difficult is this question to answer?

• What do you think of the number of response options? (too few, too many, just right?)

[Behavioral Intentions]

15. Based on the ad, how likely are you to do each of the following behaviors?

	Not at all likely	Slightly likely	Moderately likely	Very likely	Extremely likely
Look for more information about Trinase
Talk with a friend or family member about Trinase
Ask your doctor for more information about Trinase
d. Ask your doctor to prescribe Trinase
e. Take Trinase if your doctor prescribed it

Cognitive Testing Probes

• How easy or difficult were these questions to answer?

• How did you choose your answer?

• Were any easier/more difficult to answer than others?

[Attitudes toward using the drug]

16. Based on what you learned in the ad, please tell us how you would feel about taking Trinase:

1 2 3 4 5 6 7

Bad Good

1 2 3 4 5 6 7

Not useful Useful

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• How did you choose your answers?

• What do you think of the response choices for this question – “bad,” “good,” “useful,” “not useful.”

• What is meant by “bad”, “good,” “not useful”, “useful”

• Are there other words you would use instead?

18. How much attention did you pay to the [first] ad you saw about Trinase?

• None

• Very little

• Some

• Quite a bit

• A great deal

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• Was it difficult to remember how much attention you paid to the ad?

• How did you choose your answer?

[Ask 19 only of respondents who viewed the ad more than once]

19. How much attention did you pay to the last ad you saw about Trinase?

• None

• Very little

• Some

• Quite a bit

• A great deal

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• How did you choose your answer?

• What is meant by “a great deal” of attention?

[Perceptions of FDA approval of ads and ad claims]

19. Would you say the following statements are true or false?

	True	False
a. The FDA only approves prescription drugs that have been found to be extremely effective.
b. The FDA only approves prescription drugs that do not have serious side effects.
c. Only prescription drugs that have been found to be extremely effective can be advertised to consumers.
d. Prescription drugs that have serious side effects cannot be advertised to consumers.
e. The U.S. Food and Drug Administration (FDA) approves all prescription drug TV ads before they can be shown to the public.
f. All of the information in prescription drug ads is true and accurate.
g. I believe in all of the information provided in prescription drug TV ads
h. Only the safest prescription drugs are allowed to be advertised to the public in TV ads.

Cognitive Testing Probes

• How easy or difficult were these questions to answer?

• How did you choose your answers?

• Were any statements particularly easy or particularly difficult?

• What do you think is meant by “extremely effective?” What is meant by “serious side effects”?

[Perceptions of ad truthfulness]

21a. How misleading did you find the Trinase ad you saw?

• Not at all misleading

• Slightly misleading

• Moderately misleading

• Very misleading

• Extremely misleading

21b. How misleading do you think the Trinase ad you saw could be to other viewers?

• Not at all misleading

• Slightly misleading

• Moderately misleading

• Very misleading

• Extremely misleading

Cognitive Testing Probes

• How easy or difficult were these two questions to answer?

• How did you choose your answer?

• What is meant by “misleading”?

• Which is easier to answer, the first or second question? Why?

• If answers are not the same:

• What made you choose two different answers to these two questions?

21c. How truthful was the Trinase ad you saw?

• Not at all truthful

• Slightly truthful

• Moderately truthful

• Very truthful

• Extremely truthful

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• How did you choose your answer?

• What does “truthful” mean in this question?

• Which is easier to answer, this question or the two previous questions that used the word “misleading”? Why?

[RANDOMIZE ORDER OF QUESTION 23 and QUESTION 24]

[Attitudes toward amount of risk information in ad]

22. How do you feel about the amount of risk information presented in the Trinase ad?

• The ad did not have enough risk information

• The ad had just the right amount of risk information

• The ad had too much risk information

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• How did you choose your answer?

• What is “the right amount”?

[Attitudes toward amount of benefit information in ad]

23. How do you feel about the amount of benefit information presented in the Trinase ad?

• The ad did not have enough benefit information

• The ad had just the right amount of benefit information

• The ad had too much benefit information

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• How did you choose your answer?

• What is “the right amount?”

[Perceived ease of understanding]

[RANDOMIZE ORDER OF QUESTIONS 24 and 25]

24. Based on the ad you just saw, how easy to understand were the benefits of using Trinase?

• Difficult to understand

• Somewhat difficult to understand

• Neither easy nor difficult to understand

• Somewhat easy to understand

• Easy to understand

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• How did you choose your answer?

• What do you think of the response options in this question?

25. Based on the ad you just saw, how easy to understand were the risks of using Trinase?

• Difficult to understand

• Somewhat difficult to understand

• Neither easy nor difficult to understand

• Somewhat easy to understand

• Easy to understand

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• How did you choose your answer?

[Biased processing/inattention]

26. I would prefer not to think about seasonal allergy treatment at the moment

• Strongly disagree

• Disagree

• Neither agree nor disagree

• Agree

• Strongly agree

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• What do you think this question is trying to say?

[Perceived worry about side effects/drug therapy concerns]

27. If you took Trinase, how concerned would you be about the side effects?

• Not at all concerned

• Slightly concerned

• Moderately concerned

• Very concerned

• Extremely concerned

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• What do you think of when you see the word “concerned”?

• What types of side effects come to mind?

[Need for cognition]

28. How much do you agree or disagree with the following statements?

[RANDOMIZE ORDER]	1 Disagree a lot	Disagree	Uncertain	Agree	5 Agree a lot
a. I like to have the responsibility of handling a situation that requires a lot of thinking.
b. I prefer complex to simple problems
c. I try to anticipate and avoid situations where there is a likely chance I will have to think in depth about something.

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• Were any of statements more difficult than others?

[Health Literacy]

29. How confident are you filling out medical forms by yourself?

• Not at all confident

• Slightly confident

• Moderately confident

• Very confident

• Extremely confident

[TV/Media Use]

30. In the past 7 days, on how many days did you . . .

• Read a newspaper or magazine ____

• Watch television____

• Listen to the radio_____

• Use the Internet for email ____

• Use the Internet, other than for e-mail____

Cognitive Testing Probes

• How easy or difficult was this question to answer? Was it difficult to remember how many days you did each of these things?

• Were any of these items more difficult than others to remember?

[Medication necessity]

30. We would like to ask you about your personal views about medications prescribed for you. Below are statements other people have made about their medications. Please indicate how much you agree or disagree with them by checking the appropriate box. There are no right or wrong answers. We are interested in your personal views.

[RANDOMIZE ORDER]	Strongly Disagree	Disagree	Uncertain	Agree	Strongly Agree
a. My health, at present, depends on my medications
b. My life would be impossible without my medications
c. Without my medications I would be very ill
d. My health in the future will depend on my medications
e. My medications protect me from becoming worse

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• Were any statements more difficult than others to answer?

• How would you have responded to these statements if they were specific to allergy medications?

[Current Prescription Drug Use]

30. Are you currently taking, or have you ever taken, any prescription drugs for seasonal allergies?

• Currently taking

• Have taken in the past but not currently taking

• Have never taken, and not considering taking

• Have never taken, but considering taking

[Ask only of those who answer that they are currently taking a drug to question 33.]

[Satisfaction with current treatment]

30. How satisfied are you with your current seasonal allergy treatment?

• Not at all satisfied

• Slightly satisfied

• Moderately satisfied

• Very satisfied

• Completely satisfied

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• What does “satisfaction with current treatment” mean to you?

[Satisfaction with AVAILABLE treatments]

30. Think about drugs for seasonal allergies that are currently available. How satisfied are you with their ability to control your allergies?

• Not at all satisfied

• Slightly satisfied

• Moderately satisfied

• Very satisfied

• Completely satisfied

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• How did you choose your answer to this question?

• What kinds of drugs were you thinking of? Prescription only? Over the counter?

[History of Side Effects]

30. Have you ever had a serious side effect from a prescription drug?

• Yes

• No

• Don’t Know

30. Have you ever had a serious side effect from a prescription allergy drug?

• Yes

• No

• Don’t Know

Cognitive Testing Probes

• How easy or difficult were these questions to answer?

• What does the word “serious” mean in this question?

• For those who answered “don’t know”:

• How would you have answered if the “don’t know” option was not there?

[Illness Duration]

30. When did a healthcare professional first tell you that you had seasonal allergies?

• Six months ago or less

• More than six months ago but less than a year ago

• More than a year ago but less than 5 years ago

• Five years ago or longer

[Illness severity]

30. In general, how severe are your seasonal allergies? Would you describe them as:

• Very mild

• Mild

• Moderate

• Serious

• Severe

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• What did you base your answer on?

• What does “very mild” mean? What about “severe”?

[Illness knowledge]

39. In general, how much would you say you know about seasonal allergies? Would you say you know:

• Nothing at all

• Only a slight amount

• Some

• More than some but not a lot

• A lot

Cognitive Testing Probes

• How easy or difficult was this question to answer?

• What does “nothing at all” mean? What about “a lot”?

40. Which of the following is a common symptom of seasonal allergies?

• Sneezing

• Chronic pain in the ears and eyes

• Excessive thirst

• Vomiting

• Don’t know

41. Which of the following is another name for seasonal allergies?

• Histamine

• Halitosis

• Hay fever

• Heat rash

• Don’t know

41. Seasonal allergies are often caused by your body’s allergic response to:

• Parasites

• Petals

• Pollen

• Seeds

• Don’t know

41. What typically causes seasonal allergies in the fall?

• Orchard grass

• Bluegrass

• Pollinating trees

• Ragweed

• Don’t know

Cognitive Testing Probes

• How easy or difficult were these four questions to answer?

• Were any more difficult than others?

• Were any particularly easy?

[Age]

41. Please tell us your age

[open ended] (valid age range should be 18-100)

[Gender]

41. What is your sex?

• Male

• Female

[Education]

41. What is the highest level of education you have completed?

• Less than high school

• High school graduate (high school diploma or GED)

• Some college, but no degree

• Associate’s degree (2-year)

• Bachelor’s degree (4-year) (example: BA, BS)

• Advanced or postgraduate degree (example: MA, MD, DDS, JD, PhD, EdD)

[Race]

41. What is your race? (Select all that apply)

• American Indian or Alaska Native

• Asian

• Black or African-American

• Native Hawaiian or Other Pacific Islander

• White

• Some Other Race

[Ethnicity]

41. Are you:

• Hispanic or Latino

• Not Hispanic or Latino

Closing

This concludes the survey. Our goal was to gather patient reactions to important information about prescription drugs. To get your true reaction to this information, we used a fake brand of drug in this project.

Trinase is not a real drug and it is not available for use or sale. Please contact your healthcare provider for any questions about seasonal allergies.

Thank you very much for your time.