Supporting Statement Part B Pill Appearance Surveys Final 5-26-15

B. Statistical Methods

1. Respondent Universe and Sampling Methods

Both the pharmacist survey and the first patient survey will be conducted under a collaboration between contractors Harvard Medical School/Brigham and Women’s Hospital and Nielsen, a market information firm that administers online and telephone-based surveys and polls in many industries. In 2014, Nielsen purchased Harris Interactive, the well-known survey research firm that organized polls widely used by academic institutions and government bodies, and have previously been proprietarily conducted by numerous pharmaceutical companies to test patients’ attitudes about medications. Thus, the partners with whom the contractors are working are experienced and have a long reputation for excellence in the survey community. Nielsen also maintains relationships with partners who collect panels of patients in which they can find survey samples with many different chronic conditions, including depression, cancer, high blood pressure, diabetes, high cholesterol, heart disease, HIV, and others. Members of such panels have identified themselves as interested in taking surveys, which can cover any medical or non-medical topic.

Pharmacist survey: The sampling universe comprises United States-licensed pharmacists actively practicing in traditional community pharmacy practice settings (i.e., independent, chain, mass merchandiser, and supermarket pharmacies). The pharmacist sample will be purchased from Medical Marketing Service (MMS), Inc., which maintains a list obtained from the American Pharmacist Association (APhA) of 289,151 licensed pharmacists in the U.S. from every state. The list contains unduplicated licensed individuals and is cleaned and updated whenever a state board of pharmacy provides an updated file.

A proportional allocation sampling plan will be applied to this list to distribute the sample of pharmacists across the nation in relation the number of pharmacists in each state. The purpose of the allocation is to provide sufficient counts of pharmacists in subpopulations defined by groupings of states corresponding to state drug product selection laws. U.S. drug product selection laws can differ among states in several important ways. Some state boards of pharmacy have adopted mandatory generic substitution laws, which require pharmacists to substitute a generic for a brand-name medication if the prescriber does not specify that a brand-name medication should be dispensed. More permissive generic substitution laws enacted in other states give pharmacists discretion by allowing, but not requiring, pharmacists to substitute generic products for brand-name products. Some states require patients to provide consent prior to the substitution of a generic, while other states do not. Finally, some states have specific anti-substitution laws related to certain classes of compounds, such an anti-epileptic drugs.

Patient survey #1: The sampling universe is adults affiliated with Marketing Systems Group (MSG) or Survey Sampling International (SSI), which maintain demographically diverse and nationally representative lists of people interested in taking surveys about a wide range of issues. Patients sampled from these lists will be screened to confirm they are 50 years and older and take one or more generic medications for at least one of six chronic conditions: epilepsy, diabetes, hypertension, hyperlipidemia, depression, and HIV. These chronic conditions involve a range of different morbidities that may differently affect the way patients experience generic drug pill appearance changes. For example, patients with psychiatric disease (depression) may have different perceptions about alterations in their chronic medication regimens than patients with cardiovascular disease. They also invoke a range of different potential concerns about generic medications. For example, there is controversy about the use of generic drugs in treating epilepsy, where some antiepileptic drugs are narrow therapeutic index products. Therefore, we may perceive differences in patient outcomes between users of narrow therapeutic index antiepileptic drugs and wide therapeutic index drugs for diabetes or hyperlipidemia.

Patient survey #2: The sampling universe for the second patient survey is adults who take one of four pre-specified drugs for a chronic condition and who recently experienced a change in the appearance of that medication. Patients will be identified using pharmacy claims from the Optum Research Database.

The Optum Research Database is a proprietary research database of those who receive commercial health insurance from a large commercial insurance company that serves more than 50 million persons through a continuum of health care and specialty services with more than 400,000 physicians and 3,300 hospitals. The affiliated health plans presently reach across the US and include both urban and rural representation. The members of the plans are predominantly employer-based groups but also include individuals from the Medicaid and Medicare populations.

The Optum Research Database contains data from affiliated health care plans and their electronic administrative claims data, reflecting medical management information data for a broad cross-section of the population. The Optum Research Database contains medical and pharmacy claims linked to enrollment information with data from 1993 to current. For 2011, data relating to approximately 12.8 million individuals with both medical and pharmacy benefit coverage are available. Underlying information is geographically diverse across the US and fairly representative of the US population. Claims for pharmacy services are typically submitted electronically by the pharmacy at the time prescriptions are filled. Pharmacy claims data include drug name, dosage form, drug strength, fill date, days of supply, financial information, and de-identified patient and prescriber codes, allowing for longitudinal tracking of medication refill patterns and changes in medications. Pharmacy claims are typically added to the research database within six weeks of dispensing.

The contractors will collaborate with Optum to conduct the second patient survey among participants in the Optum Research Database who are identified as having recently had a generic chronic medication undergo a change in physical appearance for one chronic disease medication, such as quinapril (anti-hypertensive), fluoxetine (antidepressant), lamotrigine (antiepileptic), or simvastatin (antihyperlipidemic). The sub-contractors will be able to identify those patients who had a recent change in pill appearance using the pharmacy claims data, as has been done in prior work.^1,2 The key advantages of this approach are the ability to focus specifically on patients with a chronic condition who have had a recent change in pill appearance and the fact that the research database provides rich clinical and pharmacy information that can be used to stratify patients. For example, the sub-contractors will be able to measure the frequency with which patients have experienced changes in pill appearance in the past. The availability of these data will also reduce the number of questions asked of the patients, which will likely improve the response rate.

Survey response rates

Pharmacist survey: As mentioned above, the pharmacist survey will be mailed to a stratified, random sample of licensed pharmacists in the U.S. based on a master list from the American Pharmacists Association maintained by MMS, Inc. Pharmacists who work in a traditional community practice setting will be identified from the master list prior to mailing out surveys. Only those who work primarily in traditional community pharmacy practice settings will be eligible to participate as the study objectives do not pertain to those who work in hospitals or other settings. This list was used for the 2009 National Pharmacist Workforce Survey, which obtained an 89% deliverable rate and 52% response rate.³ Therefore, based on an 11% undeliverable rate and a 52% response rate, 2,161 questionnaires will be mailed to pharmacists to obtain the 1,000 responses sought.

Patient survey #1: The telephone survey of patients is expected to provide a 20% response rate, based on Nielsen’s prior experience with telephone surveys. According to the Pew Research Center, a nonpartisan fact tank that conducts public opinion polling and other data-driven social science research, the typical response rates for consumer telephone surveys in 2012 was 9%.⁴ This response rate has fallen from 36% in 1997, likely due to the use of call blocking features (such as caller ID), dislike of answering phone calls from unknown numbers, and a general lack of interest in participating in any time of research. Pew Research Center has found that certain techniques can be used to increase the response rate, and Nielsen’s phone center will employ some of these techniques to obtain a response rate as high as possible. These include: releasing the sample in small batches, calling back each number up to eight times, calling non-responders at different times of the day on callbacks, and the use of refusal conversion techniques by phone interviewers. With this significant effort, the survey aims to achieve the 20% response rate, a response rate higher than typical for consumer phone surveys. This response rate is calculated as the proportion of those responding to the survey among those who are called via telephone; it takes into account those who do not answer the telephone, decline to participate, are determined ineligible during screening, and who start but do not complete the survey. Based on the 20% estimated response rate, 5,000 individuals will be called to obtain the 1,000 responses sought. Of these 5,000, we estimate that 3,330 will agree to undergo brief screening questions, which will identify approximately 1,200 eligible participants. Of those eligible, we anticipate that 1,000 will complete the survey.

Despite declining response rates, telephone surveys that include both landlines and cell phones, and are sampled to maintain representativeness, continue to provide accurate and valid data.⁵ In addition, response rate is not the only valid measure for the representativeness of the data.  The sampling approach, use of the phone method, and weighting of the data also will help ensure that the data findings are representative of the population of interest, valid, and reliable.

Patient survey #2: The goal response rate for this mailed survey is estimated based on the contractors’ prior experiences with national surveys. In a 2007 mailed survey of commercially-insured patients on perceptions about generics, the contractors obtained a 48% response rate (1,047 of 2,202).⁶ This mailed survey included a $2 honorarium and up two reminder mailings. Similarly, in a 2012 survey of physicians, the contractors obtained a 53.5% response rate (269 of 503). This was an online survey that included an honorarium of up to $50 and multiple reminder messages.⁷ Other surveys conducted by Optum have achieved response rates closer to 20%, although there are additional components of this survey intended to boost response rate that were not included in Optum’s prior surveys, such as the honorarium and additional reminder packages. Based on a goal response rate of 50%, 2,000 surveys will be mailed to patients in order to obtain the 1,000 responses sought.

2. Procedures for the Collection of Information

Pharmacist Survey: A stratified, random sample of licensed pharmacists will be surveyed about their experiences with dispensing generic drug pills that differ in appearance from previous refills of the same medication and dosage level (e.g., when pharmacies switch generic suppliers). The pharmacists will be asked about the frequency with which patients inquire about the pill appearance changes, the outcome of such inquiries, and strategies used by pharmacists to address the transition to pills that have a different appearance (e.g., alert stickers on pill bottles, verbal warnings).

The pharmacist survey will be conducted using methods the contractors have previously applied to a national survey of health professionals.⁸ To recruit pharmacists into the full study, an introductory postcard will be sent to pharmacists randomly selected to be included in our sample (Appendix 4). This postcard will introduce them to the survey, the research team, and the honoraria offered. Approximately one week after the introductory postcard, participants will receive a mailing with a cover letter introducing the research team in detail and thanking them for their participation (Appendix 5), a hard copy of the survey (Appendix 1), a $5 bill, a pre-paid return envelope, and a separate pre-paid postcard to return upon survey completion. To maximize the survey response rate, a reminder letter will be sent to the survey sample at 2 weeks (Appendix 6) and 4 weeks (Appendix 7) following the initial packet, restating the purpose of the survey, thanking those who have already participated, and reminding those who have yet to participate to return their survey. Based on available resources, a $1 additional honorarium may be included in the final mailing.

The survey will take no longer than 20 minutes, to promote participation. The cover letter will contain a link to an online survey tool for pharmacists interested in filling the survey out on-line (Appendix 8). The on-line survey will be administered using Qualtrics, which has been used by the contractors in prior research. If the survey is completed on paper, the subject will be instructed to return the survey using a provided postage-paid envelope. The survey will contain a coded numeric identifier to assure that duplicate on-line and paper surveys are not received and to allow for follow-up reminders to be sent to those who have not completed a survey. Subjects who do not respond within 2 weeks will be sent a reminder letter. After an additional two weeks, a second round of reminders will go out. The contractors have previously used this procedure in several of their studies and believe it to be effective in achieving high response rates for surveys of physicians and other health professionals.

Data from completed surveys received back via postal mail will be obtained using a scanning process using image scanning.  In this process, a digital image of the survey is captured using state-of-the-art Banctec S-series scanning equipment. The Faqss scanning software system licensed from Optimum Solutions Group is then used to review the image and capture data.  The software will attempt to recognize mark sense boxes as well as text.  Any data that the software cannot recognize with a 99.9% confidence level is displayed to a human operator working at an oversized computer screen. The operator will key the information from the image on his or her screen.  The operator has the ability to look at the entire document on their screen to ensure they understand the response. The scanning department achieves an overall accuracy rate of 99.9%, which is consistent with 100% key verified traditional data entry. Data from online surveys are automatically entered into the survey program which is stored on a secure server. Once the data are obtained from the scanning facility, Nielsen’s programmer will work with the contractors for final cleaning. They will identify outliers to the pre-determined “data rules” and make decisions about how to code responses on a case by case basis. 

Nielsen implements a number of quality control procedures. The first batch is reviewed 100% to ensure all data was recorded accurately, with the correct column placement.  This process involves comparing the actual surveys to the data captured by the system. Second, each field is programmed to only accept valid responses for that particular field. Third, the software rejects all respondent errors or confusing marks, which must then be captured by a human operator. Fourth, all data captured by a human operator is then verified 100% by a second operator. Last, hard copy surveys remain in the production area until production is completed.  Verifiers will refer to the hard copy documents if information is not available from the image (usually due to stickers, labels, etc).

Patient survey #1: The first patient survey will be conducted by Nielsen via telephone, using random-digit dialing for landlines and random samples of government lists for cell phones. Nielsen will use a web-based computer-assisted telephone interviewing program (COW), in which questionnaires are programmed into the system. In this system, the following questionnaire aspects are assessed for quality control: question and response series, skip patterns, question rotation, range checks, mathematical checks, consistency checks, and special edit procedures. The COW system reduces clerical error by eliminating the need for keypunching, since interviewers enter the respondents’ answers directly into a computer during the interview itself.  For questions with pre-coded responses, the system only permits answers within a specified range; for example, if a question has three possible answer choices (e.g., Yes, No, Not sure), the COW system will only accept coded responses corresponding to these choices.  All data are compiled and checked for internal consistency.

Nielsen will be responsible for overall project and field management, sample management, survey programming and testing, reviewing pre-test, data collection, providing field status reports, data processing and weighting, and data delivery.

Patient survey #2: For the second patient survey, patients will be selected from the Optum Research Database. Patients having recently experienced a change in physical appearance of a generic chronic medication will be identified by Optum using pharmacy claims data. The de-identified data in the research database will be linked back to a private database maintained by Optum to obtain contact information, after IRB and privacy board approval. As the health insurer for these patients, Optum maintains accurate address information.

Optum is subcontracting with Anderson, Niebuhr & Associates, Inc. (ANA), a survey vendor with whom they have worked on previous research, to facilitate the survey collection. Optum will contract, train, and manage the survey collection vendor. Under the supervision of the Optum team, ANA will print and mail the invitation letter (Appendix 9), questionnaire (Appendix 3), an informed consent statement (Appendix 10), and pre- and post-incentives to subjects. To improve the overall response rate, a copy of the entire survey packet will be mailed to survey invitees who have not yet responded 2 weeks after the initial mailing. To further maximize the response rate, non-responders will receive the entire survey packet a third time approximately 4 weeks after the initial mailing. The reminder survey packages will be identical to the initial package, with the exception of slight changes to the survey invitation letter (Appendix 11) and the omission of the $5 gift. ANA also will develop and host a web-based survey website to collect patient survey responses electronically. Participants will be given approximately 2 months to complete the survey. ANA will collect questionnaire responses and enter data from returned paper surveys into a database for analysis. ANA will manage the data, complete data quality checks, and provide Optum with a final survey dataset for analysis that accurately captures the survey responses received through the web or mail. They will provide Optum with weekly status updates and regularly correspond to keep Optum informed about any issues encountered.

There will be one final analytic dataset that will contain variables derived from responses to survey questionnaire and the claims data.  Data will be analyzed by Optum using SAS v. 9.2. The conduct and reporting of this study follows Optum Epidemiology’s Standard Operating Procedures (SOPs) that are consistent with the International Society for Pharmacoepidemiology’s Guidelines for Good Pharmacoepidemiology Practices. The SOPs in place at Optum prescribe that processes and deliverables are documented, reviewed, and validated in sufficient detail to allow for subsequent re-examination or replication.  In addition, the survey vendor (ANA) will follow their own quality control procedures for data collection and management.  At Optum, the validation of analytic work typically involves a combination of a review of program logs and lists, independent coding, a review of program processes and documentation to ensure departmental SOPs are followed, and reconciliation of program code with the study-specific statistical analysis plan to ensure populations and results are consistent with what is needed for the particular study.  Individual programs are documented and revised as needed until sign-off by a validator using the validation/programming log. Optum will provide aggregated survey results and a written report to the contractor, who will perform further data interpretation and analysis as needed on the aggregated data.

Hypotheses

The pharmacist survey is both exploratory and confirmatory. The exploratory component will provide descriptive information on the extent to which pharmacists perceive changes in pill appearance as a concern and the use of strategies to manage changes in pill appearance faced by patients. The confirmatory component will test four hypotheses:

Hypothesis 1: Greater patient confidence in the safety and effectiveness of generic pills that change their physical appearance during routine refills lead to better outcomes related to use of those pills

Hypothesis 2: Greater pharmacist involvement in managing changes in generic pill appearance during routine refills leads to greater patient confidence in the notion that the new pills will work just as safely and effectively as the previous bioequivalent version

Hypothesis 3: Greater pharmacist involvement in managing changes in generic pill appearance during routine refills leads to better patient outcomes

Hypothesis 4: Pharmacists’ concerns about the bioequivalence of generic drugs arising from changes in generic pill appearance during routine refills leads to worse patient outcomes

Both patient surveys have the same hypotheses:

Hypothesis 1: Changes in pill appearance lead patients to report experiencing less benefit and more side effects

Hypothesis 2: Changes in color lead to worse outcomes than changes in shape

Hypothesis 3: Patients who used pills that changed in appearance did not experience any differences in effectiveness or side effects

Hypothesis 4: Lack of patient confidence in pills that have changed in appearance leads to worse patient outcomes

Another key aspect of the patient survey is to identify subgroups in which there is more of an impact of product appearance. This will be exploratory in nature; there are no pre-specified hypotheses.

Analysis Plan

Pharmacist survey: Descriptive analyses will be performed for all Likert scale, closed-ended questions by characterizing the means, ranges, and distributions of all variables. Bivariable analyses will be used to explore the relationships among covariates. When variables are dichotomous or categorical, chi-square tests of independence will be employed, and for continuous variables, t-tests will be used to test for group differences. Multivariate logistic regression analyses will be used to address the key hypotheses. The primary dependent variables will come from questions about confidence in the safety and effectiveness of pills that have changed appearance and reported outcomes of changes in pill appearance. Multivariate analyses will identify potential determinants of these outcomes. The primary explanatory variables could include patient confidence in the safety and effectiveness of pills following a change in appearance, pharmacist involvement in managing changes in pill appearance, and pharmacists’ perceptions of bioequivalence of generic drugs from different manufacturers. Other independent variables could include relevant demographic data, whether and how pharmacists notify patients of potential changes in pill appearance, and the volume of prescriptions dispensed. Analyses could be stratified by several key variables, including the pharmacy dispensing volume, the frequency of pill appearance changes within a pharmacy, and respondent demographics. By linking zip code of the pharmacy to US Census data, results can be examined stratified by socioeconomic status of those living in the neighborhood of the pharmacy, and also can be interpreted in light of state laws related to generic substitution. The target sample size should be large enough for sufficient power to examine results within strata of key demographic variables.

Because little information is currently available on this topic, some of the analyses will be exploratory to investigate other potential associations among collected variables. For example, it could be explored whether pharmacists’ concerns about bioequivalence influence their perceptions regarding patient concerns about changes in generic pill appearance, or whether the information pharmacists’ provide to patients affects patient confidence in the medications that they receive or patient outcomes.

Patient surveys: The analyses of the patient surveys are similar to those for the pharmacist survey. The means, ranges, and distributions of all variables will be characterized. Bivariable and multivariable analyses will then be conducted to determine whether demographic and clinical characteristics influence patient responses to the hypothetical scenarios. Based on the results, subgroup analyses could be conducted based on some demographic variables (e.g., age, socioeconomic status) as well as by clinical conditions for which the impact of product appearance may be particularly important, including epilepsy, diabetes, hypertension, hyperlipidemia, and depression.

Power calculation

Pharmacist survey: This power calculation uses conservative assumptions to ensure adequate power for comparisons of interest even if a lower response rate than expected is obtained. The calculation used the hypothesized response distributions presented in Table 6, based on one of the questions with a 4-point Likert scale response option that will be used as a dependent variable (e.g., “Changes in pill appearance lead patients to report side effects to me”) across all four categories of response for a 4- point Likert scale question that will be used as an explanatory variable (e.g., “I notify the patient verbally, in person or on the phone”), which would support the hypothesis that greater pharmacist involvement in managing changes in generic pill appearance during routine refills leads to better patient outcomes.

Table 6. Hypothesized response distributions*

	Changes in pill appearance lead patients to report side effects to me
I notify the patient verbally, in person or on the phone	Almost always	Commonly	Rarely	Never
Almost always	40%	30%	20%	10%
Commonly	45%	25%	20%	10%
Rarely	50%	25%	15%	10%
Never	50%	30%	15%	5%

*Percentages are row percents

The power calculations are based on the linear trend statistic. For each sample size, 10,000 tables were simulated to calculate the linear trend statistic and its p-value; the power is the percentage of p-values that were less than 0.05. The power to detect these differences varies based on the number of responses:

600 responses, power = 0.58

1000 responses, power = 0.83

1200 responses, power = 0.88

1800 responses, power = 0.97

Approximately 1,000 responses would provide slightly more than 80% power to detect these small changes. As noted above, based on an 11% undeliverable rate and a 52% response rate, 2,161 questionnaires will be mailed to pharmacists to obtain the 1,000 responses required for adequate statistical power.

Other analyses will have more power as the ordinal responses will be dichotomized (e.g., better versus worse outcomes, higher versus lower patient confidence). These will also have sufficient power to detect small differences within small subgroups (e.g., working in low socioeconomic areas). For example, assuming a subgroup as small as 30% of the total target sample size and an alpha of 0.05, there is 80% power to detect a 15% difference in poorer patients outcomes associated with less involvement of the pharmacist in managing pill appearance changes.

Patient surveys: Since a key aspect of the patient surveys is to identify subgroups in which there is more of an impact of product appearance, the patient surveys are powered to ensure that important differences in subgroups can be detected. This patient survey power calculation is based on a comparison of outcomes between non-Hispanic Whites and African Americans. These groups comprise approximately 63% and 13% of Americans, respectively. Assuming that approximately 25% of non-Hispanic Whites answer “yes” to a question such as, “When your medication changed appearance, did you stop using the medication?” and 37% of African Americans respond “yes” to the same question, the normal-based two-sample test of independent proportions with an alpha of 0.05 results in 80% power to detect this statistically significant 12% difference (37% vs. 25%) in the proportion who chose to stop taking their medication with a total population of 1000 responses (630 non-Hispanic Whites, 130 African Americans, 240 other race). A sample size of 1,000 patients would provide 92% power to detect a 15% difference.

Since this power analysis was conducted comparing subgroups, it ensures that we will also have ample power for overall analyses in the marginal population.

3. Methods to Maximize Response Rates and Deal with Nonresponse

Pharmacist survey: This survey includes several design aspects intended to maximize response rates. First, the pharmacist survey will take no more than 20 minutes to complete in order to minimize the time of survey execution and respondent burden. Second, reminder letters will be sent to non-responders at 2 weeks and 4 weeks following the initial packet. Providing two reminders has been demonstrated to be effective in promoting survey response rates.⁹ Lastly, a $5 honorarium will be provided with the initial mailing to encourage pharmacists to complete the survey.

Patient survey #1: To promote response rates, the patient survey will take no more than 20 minutes to complete in order to minimize the time of survey execution and respondent burden. In addition, Nielsen’s phone center will employ several techniques to obtain a response rate as high as possible: releasing the sample in small batches, calling back each number up to eight times, calling non-responders at different times of the day on callbacks, and the use of refusal conversion techniques by phone interviewers.

Patient survey #2: Similar to the other surveys, the second patient survey will take no more than 20 minutes to complete, to minimize respondent burden and promote response rates. Participants will be given approximately 2 months to complete the survey. To improve the overall response rate, copies of the entire survey package will be mailed at 2 and 4 weeks after the initial mailing. In addition, respondents to this patient survey will receive an up to $25 honorarium ($5 to invited participants + $20 gift card to participants that complete the survey) as an incentive to complete the survey.

Since the survey invitees will have been selected from a known sampling frame with known characteristics for all invitees, the respondents can be directly compared to the sampling frame with respect to selected characteristics available in Optum’s claims data. This adjustment is done by weighting survey results for effect of differential response by patient characteristics. For each survey response, Optum will present the crude proportion responding with various answers, and the inverse probability of the response-weighted proportion of response for each possible answer. The latter will correct for non-responder bias in the survey.

All surveys: A survey invitee will be declared a nonresponder to the survey if no response is received by the end of the data collection period. The fraction of surveys returned (response rate) will be calculated using the American Association of Public Opinion Research (AAPOR), standard definition #2.¹⁰ As with all surveys, these surveys will be subject to some degree of non-response. Several analytic approaches will be used to estimate the potential for non-response bias and to account for this bias in the analyses. Initially, the distribution of observed characteristics will be compared between respondents and non-respondents, where available, to understand how well respondents represent the population of interest. For example, in the pharmacist survey it will be examined whether non-response is associated with working in certain geographic areas or with the size of the pharmacy chain. If necessary, a set of non-response weights will be developed and used in the analyses to obtain population representative estimates. Further, all analyses will include respondents personal and professional characteristics as control variables and will thus control for differences in these characteristics between responders and non-responders. These analyses can effectively control for observed differences between respondents and non-responders.

Since non-responders may differ in unobserved characteristics, including their reporting attitudes and experiences (i.e., “non-ignorable missing data”), a series of sensitivity analyses will be performed to examine the robustness of the conclusions to the possibility that non-respondents differ from respondents in important variables. The following confirmatory tests will be conducted to evaluate whether respondents were representative of the entire sample: analyses using data in the sampling frame, comparison of the survey sample with other data sources, and comparison of early vs. late survey responders.¹¹

4. Test of Procedures or Methods to be Undertaken

All three surveys will undergo cognitive testing, a key feature of high quality survey development. Cognitive testing ensures that: 1) respondents will understand the questions in the manner in which they were intended; and 2) that the questions are written in a manner answerable for respondents. Through cognitive interviewing, it can be determined whether the respondents understand the questions and can identify problems in two specific areas: potential response errors and errors in question interpretation associated with vague wording, use of technical terms, inappropriate assumptions, and item wording. The interview results will be used to ensure that the survey items are as free from error as possible.

The pharmacist survey and patient survey #1 also will be pretested. The purpose of the pre-test is to ensure that the questionnaire is clear and understandable, the instrument is the designated length, and data collection protocols function properly.

The survey format and wording of the questions will be finalized following cognitive testing and formal pretesting (when possible), since these tests are performed to make sure the results of the survey are as effective and valuable as possible. Any changes made to the survey as a result of the pretests will not change the main study design and will not increase the burden on respondents.

Pharmacist survey: The survey instrument will be tested by conducting cognitive interviews with a convenience sample of pharmacists.

The survey will be pre-tested in 9 pharmacists randomly selected from the sampling frame, distributed equally by gender and practice location. The data collection process will be assessed and the data will be reviewed to ensure their accuracy and reliability.

Patient Survey #1: Cognitive testing will be conducted in a convenience sample of local patients identified from the Brigham and Women’s Hospital Patient-centered Comparative Effectiveness Research Center, a hospital-wide center that is facilitating faculty members’ ability to directly engage patients in the research process.

A pre-test will be conducted prior to fielding the survey to the full patient sample. The pre-test will consist of initial calls to attain 9 responses, after which the data will be reviewed to ensure their accuracy and reliability.

Patient survey #2: Similar to patient survey #1, cognitive testing will be performed in a convenience sample of local patients by the contractors at Brigham and Women’s Hospital. Since the survey questions will be similar, and in many cases identical, to patient survey #1, a formal pre-test will not be conducted for patient survey #2.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

All analyses will be led by the contractors in the Division of Pharmacoepidemiology and Pharmacoeconomics at Brigham and Women’s Hospital and Harvard Medical School:

1. Aaron Kesselheim, M.D., J.D., M.P.H. (Principal investigator)

2. Joshua Gagne, Pharm.D., Sc.D. (Co-investigator)

3. Jessie Franklin, Ph.D. (Biostatistician)

4. Ameet Sarpatwari, J.D., Ph.D. (Research fellow)

Eric Campbell, Ph.D., a professor of medicine at Harvard Medical School, will assist in the statistical analysis of the surveys.

The FDA’s Office of Research and Standards/Office of Generic Drugs/Center for Drug Evaluation and Research will provide input on the analyses. Scientists at Nielsen and Optum will be consulted on aspects of the survey design and analysis. Optum scientists involved in conducting and analyzing the second patient survey include:

1. Cheryl Enger, Ph.D. (Senior Scientist)

2. Wendy Carman, Ph.D. (Epidemiologist)

3. Michael Doherty, M.S. (Validation Analyst)

4. Ling Li, M.S. (Analyst)

1 Kesselheim AS, Bykov K, Avorn J, et al. Burden of changes in pill appearance for patients receiving generic cardiovascular medications after myocardial infarction: cohort and nested case-control studies. Annals of Internal Medicine. 2014;161(2):96-103.

2 Kesselheim AS, Misono AS, Shrank WH, et al. Variations in pill appearance of antiepileptic drugs and the risk of nonadherence. JAMA Internal Medicine. 2013;173(3):202-208.

3 Midwest Pharmacy Workforce Research Consortium. 2009 National Pharmacist Workforce Survey. 2010. http://www.aacp.org/resources/research/pharmacyworkforcecenter/Documents/2009%20National%20Pharmacist%20Workforce%20Survey%20-%20FINAL%20REPORT.pdf.

4 The Pew Research Center for the People and The Press. Assessing the Representativeness of Public Opinion Surveys. May 15, 2012. Available at: http://www.people-press.org/2012/05/15/assessing-the-representativeness-of-public-opinion-surveys/.

5 The Pew Research Center for the People and The Press. Assessing the Representativeness of Public Opinion Surveys. May 15, 2012. Available at: http://www.people-press.org/2012/05/15/assessing-the-representativeness-of-public-opinion-surveys/.

6 Shrank WH, Cox ER, Fischer MA, et al. Patients' perceptions of generic medications. Health Affairs. 2009;28(2):546-556.

7 Kesselheim AS, Robertson CT, Myers JA, et al. A randomized study of how physicians interpret research funding disclosures. New England Journal of Medicine. 2012;367:1119-1127.

8 Austad KE, Avorn J, Myers JA, et al. Changing interactions between physician trainees and the pharmaceutical industry: a national survey. Journal of General Internal Medicine. 2013;28(8):1064-1071.

9 Archer TM. Characteristics associated with increasing the response rates of web-based surveys. Practical Assessment, Research and Evaluation. 2007;12(12). http://pareonline.net/getvn.asp?v=12&n=12.

10 The American Association for Public Opinion Research. 2011. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 7^th edition. AAPOR.

11 Johnson TP, Wislar JS. Response rates and nonresponse errors in surveys. Journal of the American Medical Association. 2012;307(17):1805-1806.