Support Statement Part B Physician Interpretation Study 2019

Part B. Statistical Methods (used for collection of information employing statistical methods)

1. Respondent Universe and Sampling Methods

The nonprobability sample of U.S. physicians for the study will be recruited from a panel maintained by SERMO. SERMO has a proprietary database of more than 1.8 million doctors and allied health professionals who have opted in to participate in research studies, including over 800,000 verified physicians. Physicians are added to the database through a variety of means, including by telephone and online. The information in the database is rigorously verified through a three-stage registration process. Physicians first provide basic information, then complete a more detailed profiling survey. Following this, members are contacted at their place of work and asked to provide another person from the workplace that can verify the member’s information. SERMO is ISO 20252 and ISO 26362 certified and uses custom algorithms to detect and prevent duplicate accounts at the point of registration and RelevantID^® digital fingerprinting to prevent fraudulent survey participation. SERMO’s panel tracks closely to the demographics of American physicians, as benchmarked by the AMA. Members have on average 15–20 years of experience in medicine and represent over 90 specialties and subspecialties in all 50 states.

We will recruit participants who are healthcare professionals who practice internal medicine, general medicine, or family medicine. Qualified participants will report spending at least 40% of their office time per week on direct patient care. Soft quotas for age, race/ethnicity, and gender will be used to ensure a diverse sample of participants.

Panel members will be invited to participate by receiving an e-mail invitation (Appendix C) and, if interested, can click on a hyperlink within the e-mail and gain access to the screener (see Appendix A). The sample will not be representative of the healthcare professional population, but soft quotas will be used to ensure recruitment of a demographically diverse set of participants. Final sample sizes for the pretest and main studies are 158 and 566, respectively.

As detailed in the participant screener (see Appendix A), to qualify for the physician study, all participants will meet the following criteria:

• At least 18 years of age

• Live in the United States

• Work as a physician in internal medicine, general medicine, or family medicine

• Spend at least 40% of working week on direct patient care

• Has not participated in market research within the last three months

• Does not work for the Department of Health and Human Services or work in marketing, advertising, or pharmaceutical industries.

• Not accessing the survey using a mobile device

The screener will also ask participants to provide other demographic information. After participants are screened, those who are eligible will be randomly assigned to conditions.

2. Procedures for the Collection of Information

Design Overview

We propose to investigate how physician perception of professional prescription drug communications is influenced by variations in information context, methodologic rigor of the underlying clinical study, and time pressure. We propose to test three different contextual presentations of drug information (medical journal abstract, sales aid without graphic design elements, sales aid with graphic design elements), and two types of study methodological rigor used by Kesselheim et al. (classified as high or low)¹. We have chosen to test a mock sales aid presentation and a medical journal abstract in order to examine the potential differences in perception that may arise by presenting the same information in different vehicles. Mirroring the time constraints of practicing physicians, we will examine the role of time pressure by randomly assigning half of the study participants to a limited amount of available time to read the materials. Figure 1 describes the study design.

Figure 1.--Study Design

			Information Context
			Medical Journal Abstract	Sales Aid without graphic design elements	Sales Aid with graphic design elements2
Limited Time to Read	Methodological Rigor1	High
		Low
Unlimited Time to Read		High
		Low

¹ As defined by Kesselheim et al., 2012

² For example, colors and background images

Procedure

The pretest and main study will be 20 minutes long and conducted using an Internet panel. Participants will be randomly assigned to see one version of the communication and time available to read. After viewing the communication, participants will complete a questionnaire that assesses participants’ interpretation and understanding of the information and their perceptions of the drug (see questionnaire, Appendix B). We will also measure covariates such as demographics and health literacy.

Participants

We will recruit participants who are healthcare professionals who practice internal medicine, general medicine, or family medicine. Qualified participants will report spending at least 40% of their office time per week on direct patient care Soft quotas for age, race/ethnicity, and gender will be used to ensure a diverse sample of participants. We will exclude individuals who work for the Department of Health and Human Services or work in marketing, advertising, or pharmaceutical industries. The studies will be conducted with an Internet panel. Panel members can only participate in one of the studies (pretest or main study).

Research Questions

Our research questions are:

RQ 1: Does the information context in which the information appears affect interpretation of the information?

RQ 2: Does methodological rigor of the study affect interpretation of the information?

RQ2a: Do physicians correctly interpret the methodological rigor of the study?

RQ3: Does the time available to read the information affect interpretation of the information?

RQ4: What are the potential interactions between these factors?

Analysis Plan

We will conduct ANOVAs (for continuous variables) and logistic regressions (for dichotomous variables) with interaction terms and planned comparisons to test the hypotheses outline above.

Power

We conducted a priori power analyses to ensure we obtained a sufficient sample to detect statistically significant differences in the outcome measures of interest across the different experimental conditions. The pretest, assuming the need for power of .80, alpha probability of .05, and a medium effect size (f = .25), will require a sample of 158 participants. The main study, given the experimental design and assuming a power of .90, alpha of .05, and small to medium effect size (f = .15), will require a sample of 566 participants. With these sample sizes, we will have sufficient power to detect small-to-medium sized effects across analyses.

Table 2. – Pretest: A priori power analysis to determine sample size needed in F tests (ANOVA: fixed effects, main effects, and interactions) to achieve power of 0.80 (Faul et al., 2007).2
	Main effect from 3 x 2 x 2 model
Effect size f*	0.15	0.20	0.25
α error probability	0.05	0.05	0.05
Power (1 – β error probability)	0.80	0.80	0.80
Numerator df	24	2	2
Number of groups	125	125	12
Total Sample Size	432	245	158

Table 3. – Main Study: A priori power analysis to determine sample size needed in F tests (ANOVA: fixed effects, main effects, and interactions) to achieve power of 0.90 (Faul et al., 2007).
	Main effect from 3 x 2 x 2 model
Effect size f*	0.15	0.20	0.25
α error probability	0.05	0.05	0.05
Power (1 – β error probability)	0.90	0.90	0.90
Numerator df	2	2	2
Number of groups	12	12	12
Total Sample Size	566	320	206

*An effect size of 0.10 is traditionally considered small, whereas an effect size of 0.25 is considered medium (Cohen, 1988).³ Here we have shown two different effect sizes centering around small to medium effects.

3. Methods to Maximize Response Rates and Deal with Non-response

This experimental study will use an existing research panel to draw a sample.  The panel comprises individuals who have signed up to participate in online studies.  To help ensure that the participation rate is as high as possible, FDA will:

• Design an experimental protocol that minimizes burden (short in length, clearly written, and with appealing graphics);

• Administer the experiment over the Internet, allowing respondents to answer questions at a time and location of their choosing.

4. Test of Procedures or Methods to be Undertaken

We have conducted nine hour-long qualitative interviews to cognitively test the study stimuli and materials and revised the questionnaire accordingly. One pretest will be conducted to test the experimental manipulations and pilot the main study procedures. Finally, we will run the main study as described elsewhere in this document.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The contractor, Fors Marsh Group, will collect the data on behalf of FDA as a task order under Contract HHSF223201510003B. Shane Mannis, 571-444-1109, is the contractor’s Project Director for this project. Data analysis will be overseen by the Research Team, Office of Prescription Drug Promotion (OPDP), Office of Medical Policy, CDER, FDA, and coordinated by Kathryn J. Aikin, Ph.D., 301-796-0569, and Amie C. O’Donoghue, Ph.D., 301-796-0574.

1Kesselheim, Aaron S., Christopher T. Robertson, Jessica A. Myers, et al. (2012). A randomized study of how physicians interpret research funding disclosures. New England Journal of Medicine, 367(12), 1119-1127.

2 Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175-191.

3 Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2^nd Ed). Hillsdale, NJ: Lawrence Erlbaum & Associates, Inc.

5