OMB Approval Memo

FDA DOCUMENTATION FOR THE GENERIC CLEARANCE

OF COMMUNICATION TESTING FOR DRUG PRODUCTS (0910-0695)

TITLE OF INFORMATION COLLECTION: One-on-One Interviews: Tradeoff Analysis of Risk, Benefit, and Adherence Claims in DTC and HCP Promotion

DESCRIPTION OF THIS SPECIFIC COLLECTION

1. Statement of need:

People tend to make choices about their health and medical treatments that reflect tradeoffs in their preferences for features of the options available to them. Information about product attributes, characteristics, expected beneficial and undesirable effects, and promotional claims have the potential to influence consumer and health care provider (HCP) perceptions and preferences for different treatment options.

Understanding how patients make tradeoffs among drug product characteristics—and how these tradeoffs differ from physician preferences—will provide valuable insight into the relevance and impact of various product attributes and promotional claims on treatment decisions and informed choice.

The Food and Drug Administration’s (FDA) Center for Drug Evaluation and Research (CDER), in collaboration with RTI International, is conducting research on physician and consumer preferences for treatment options related to two chronic medical conditions. We will conduct remote one-on-one video interviews with: (1) physicians who self-identify as treating patients with psoriasis and/or type 2 diabetes; and (2) consumers who self-report being diagnosed with either condition. Interviews will be designed to elicit information from participants about the factors that influence their opinions, decisions, and preferences about treatments for the respective medical conditions, with a focus on how each group makes tradeoffs among several dimensions, including efficacy, risks and side effects, dosage and administration, adherence, and marketing claims. Findings will be used to identify the most relevant treatment attributes that will be included in a future conjoint analysis study to quantify the tradeoffs that consumers and physicians make when determining preferences for prescription drug products.

2. Intended use of information:

We will use the results of this research to: (1) better understand how HCPs choose prescription drug treatments for their patients, (2) better understand what treatment factors matter to patients, and (3) inform future quantitative phases of this research project.

3. Description of respondents:

Physicians

We will conduct 35 individual, in-depth interviews with a nationwide sample of physicians (general practitioners or specialists) who care for patients with psoriasis and/or type 2 diabetes. General inclusion and exclusion requirements built into the screening protocol will ensure that all physicians are currently practicing, spend at least half of their time on direct patient care, and have not participated in a focus group or research interview within the last three months. Physicians will also need to have access to a computer with high speed Internet and a webcam for the interview, and agree to have the interview audio recorded and livestreamed. We will exclude physicians who work in the marketing, advertising, or pharmaceutical industries and those who work for the Department of Health and Human Services or RTI International because they may have specialized knowledge of FDA regulatory policies.

Diagnosed Consumers

We will conduct 70 individual, in-depth interviews with a nationwide sample of U.S. adults aged 18 years or older who self-report as having been diagnosed by an HCP with either psoriasis or type 2 diabetes (35 consumers per medical condition). Diagnosed consumers will also need to be able to read and speak English fluently, have access to a computer with high speed Internet and a webcam for the interview, and agree to have the interview audio recorded and livestreamed. General inclusion and exclusion requirements built into the screening protocol will ensure that consumers have not participated in a focus group or research interview within the last three months, and do not work for the Department of Health and Human Services, RTI International, or a pharmaceutical company or market research firm.

Demographics

Within each respondent group, we will aim to recruit individuals with diverse demographic characteristics (e.g., age, gender, race/ethnicity) to ensure that we hear from individuals with different backgrounds and perspectives. For physicians, we will apply soft quotas during screening so that the demographics of respondents are reflective of the composition of the American Medical Association.¹ For consumers, we will ensure diversity in health literacy by setting a 20% quota for low-literacy respondents using a validated, single-item screener.² For screening purposes, we will set a score 3 on the 5-point scale as the threshold for lower health literacy.

4. Date(s) to be conducted and location(s):

We plan to conduct interviews between July and October 2021. Interviews will be conducted remotely.

5. How the Information is being collected:

Recruitment Procedures

Physicians will be recruited through L&E Research’s healthcare professional panel. The healthcare professional panel consists of thousands of physicians, nurse practitioners, physician assistants, nurses, and other healthcare professionals nationwide. When joining the panel healthcare providers complete a short screener with information about themselves (medical degree, type of practice, demographics, etc.) and agree to receive invitations to participate in studies. L&E Research will use this proprietary panel to identify physicians based on the panelists’ profile information to invite for further screening.

Consumers will be recruited from L&E Research’s consumer panel. Consumers join the panel by going to L&E Research’s website or responding to ads on social media. Participants complete a short screener with information about themselves (demographics, health conditions, etc.) that L&E Research uses to identify consumers for further screening.

Potential participants will be contacted by phone to complete the screening process or they will receive an email to complete an online screener that will include a subset of the screening questions (a recruiter from L&E will then call participants who appear eligible to complete the full screener by phone). A copy of the screener with introductory text is included in Appendix A.

Once the interview has been scheduled by phone, a confirmation email letter will be sent to the recruited participant (see Appendix B) outlining the overall purpose of the research, confirming the date and time for the interview, and the Zoom link for the interview. An informed consent document (see Appendix C) will also be attached to the confirmation email letter.

L&E Research will send recruited physician’s and consumer’s first names, scheduled interview times, and responses to screener questions to RTI. RTI will provide periodic updates about scheduled interviews to FDA. One to two days prior to the scheduled interview, recruited participants will receive a reminder call and participants who do not log in for their interview will also receive a follow-up call (see Appendix D).  

Data Collection Procedures

Data collection will be conducted remotely using Zoom. Trained interviewers from RTI International will conduct individual in-depth interviews with physicians and consumers. Prior to starting interviews, interviewers will review the informed consent document sent in advance to participants with the confirmation email and request their verbal consent to participate in the study and record and livestream the interview.

Using a semi-structured guide (see Appendices E and F for the consumer and physician interview guides, respectively), the interviewer will ask participants about how they make decisions about medical treatments (consumers for themselves and physicians for their patients). No sensitive questions will be asked of participants. The expected length of each interview is approximately 60 minutes.

RTI International will audio record all interview sessions. Observers from FDA will be able to watch the interviews live. Observers will be muted once they log into the session, and thus only the RTI interviewer and notetaker will be able to interact with the participant. RTI will comply with safeguards for ensuring participant information is kept secure to the extent permitted by law. The last names of the participants will not appear on any materials. Verbatim quotes included in the final report will not be attributed to any individual.

Upon completing the interview, participants will be offered an incentive from L&E Research. L&E Research uses a vendor platform where participants can choose a gift card from a variety of merchants or a visa gift card, which can be delivered in either a physical or electronic format. Consumers will be offered an incentive valued at $50, with the possibility of offering $75 if we struggle to recruit enough participants, and physicians will be offered an incentive valued at $200.

6. Confidentiality of Respondents:

Assurance of Privacy Provided to Participants

Recruited participants will receive an informed consent document (see Appendix C) in advance of the interview. The document explains the study’s purpose, participant rights, benefits and risks (minimal) of the study, and provides them with a contact name, email, and phone number should they have questions about the study. The document also notifies participants that interviews will be audio recorded and observers will be viewing the interviews remotely.

The interviewer will review the key elements of the informed consent document (e.g., study purpose, participant rights, potential risks and benefits, presence of observers) with participants at the beginning of the interview. Participants will then be asked to provide their verbal consent to participate in the study and record the interview. In the event verbal consent for the audio recording is not given, the interview will not proceed and efforts will be made to schedule a replacement interview.

All data will be collected with an assurance that responses will remain secure to the extent permitted by law. Both the informed consent document and the interview guide contain a statement emphasizing that no one will be able to link a participant’s identity to their responses. Interviewers will not ask participants to provide identifying information beyond their first names. In addition, any quotations used in a report will not be linked to individual respondents. Further, no identifying information will be included in the data files delivered by RTI to FDA.

All interviews will be audio recorded for reporting purposes and will be livestreamed for observers. Both livestreamed and recorded interviews will only be viewed by FDA and RTI project staff. Livestreaming connections will be secure, using industry-standard firewalls and security practices. All data will be encrypted in transit using HTTPS. All equipment will be operated and maintained according to industry-standard practices, and all software validated using industry-standard quality assurance practices. Recordings will be used to create transcriptions of the interview sessions for reporting purposes and destroyed within three years after project completion.

After data collection is completed, RTI will provide FDA with copies of transcripts of all interviews. These transcripts will be provided to FDA as a written record of the sessions. To ensure participant privacy, all PII other than first names will be redacted from the transcripts before delivery to FDA.

Record Keeping and Confidentiality

The following procedures will be used to ensure participant confidentiality before, during, and after fielding:

1. L&E Research will recruit participants from their opt-in panels (HCP panel and consumer panel). Only first names of participants that have been scheduled for interviews will be provided to RTI. Any interview materials RTI shares with FDA will only contain answers to screener questions (not first names).

2. During the interviews, participants will be addressed only by their first names. Any PII (beyond the first name) shared by participants during the interview will be redacted from transcripts.

3. Respondent quotes used in reports will not be associated with any names or attributed to specific participants.

Contractors will not share personal information regarding participants with any third party without participants’ permission unless it is required by law to protect their rights or to comply with judicial proceedings, court orders, or other legal processes. This possibility will be disclosed in the informed consent document.

All identifying information, including information collected during screening, will be kept on a separate password-protected computer and/or in locked cabinets for a period of no longer than three years after the project is complete, after which they will be destroyed by securely shredding documents or permanently deleting electronic information. In the case of a breach of confidentiality, appropriate steps will be taken to notify participants.

All data will also be maintained in consistency with the FDA Privacy Act & Applicable System of Records Notices #09-10-0009 (Special Studies and Surveys on FDA Regulated Products).

7. Amount and justification for any proposed incentive:

Diagnosed Consumers

For completing the 60-minute remote interviews, diagnosed consumers will be offered an incentive valued at $50, with the possibility of offering $75 if we struggle to recruit enough participants. The proposed incentive rate is in accordance with standard practice and based on our experience with specific patient populations, the amount of time the participant spends in the study, what is required of them, recent consultation with our recruiting vendor, and OMB-approved incentives on recent FDA projects and projects for other clients. This token of appreciation is intended to provide enough incentive to participate in the study rather than another activity, improve data quality, reduce the number of cancellations, recognize the burden of childcare costs, and convey appreciation for contributing to this important activity.³ Incentives must be high enough to equalize the burden placed on respondents in respect to their time and cost of participation. Offering an incentive below these rates may result in increased costs exceeding the amount saved with a lower incentive. Consequences of insufficient incentives include increased time and cost of recruitment, and increased probability of cancelled or postponed interviews.

The proposed incentive amount is significantly below market rate for an effort of this type. Recruiting firms determine market rates for research participation based on what other comparable studies in the field are offering and what rate will incentivize the required population to participate. L&E Research and other vendors estimate that other studies being conducted with similar populations and levels of effort in this market at this time pay incentives of $100-$175. For example, it is not uncommon for companies to pay $150 for 45-minute remote consumer interviews.⁴ Online market research firms suggest paying people at least twice as much as their hourly rate to incentivize them to participate.⁵ The Bureau of Labor Statistics (BLS) calculated that the average hourly wage of employees, including benefits, in March 2021 was $39.01.⁶ At that rate, a minimum incentive for a 60-minute interview would be $78.02. But that is not the only expense to consider.

The interviews will be conducted online, and participants must have a computer and broadband Internet to participate. Additionally, we estimate that participants will spend approximately 80 minutes of their time on this task, which includes time for screening (5 minutes), time for testing the Zoom platform (10 minutes), time to participate in the interview (60 minutes), and the time involved in logging in 5 minutes early to confirm the technology is operating correctly. It is worth emphasizing that during the current pandemic, all research is remote. Thus, the convenience typically associated with virtual (remote) sessions is no longer relevant, and participants do not derive any benefit or convenience from joining a remote (versus an in-person) interview. Participants are required to join the interview from a location where there are no distractions, which may require coordinating childcare, finding a private and quiet location, or special accommodations during that time. BLS calculated in May 2020 that the median hourly wage of childcare workers is $12.24, an additional expense for some participants that will be offset by the incentive.⁷

Physicians

Historically, physicians are one of the most difficult populations to survey, partly because of the demands on their professional time. Consequently, incentives assume an even greater importance with this group. Several studies have discussed the challenges of conducting research with HCPs and have concluded that offering substantial incentives is necessary to attain high response rates.^8,9

Recruiting physicians to participate in research has been shown to be difficult for reasons related primarily to the time burden.¹⁰ Physicians’ time is limited and, thus, quite valuable. A meta-analysis on methodologies for improving response rates in physician surveys examined 21 studies published between 1981 and 2006 that investigated the effect of monetary incentives on response rates in surveys of physicians. The authors found that the odds of responding to a survey with an incentive were 2.13 times greater than responding to a survey without incentives.¹¹ Martins and colleagues conducted a review of published oncology-focused studies to investigate methods for improving response rates. Their meta-analysis also showed that monetary incentives were effective at increasing response rates.¹² Previous research also suggests that providing incentives may help reduce sampling bias by increasing rates among individuals who are typically less likely to participate in research (such as PCPs or physician specialists)^13,14 and ensuring participation from a cross section of physicians, which will improve data quality by improving validity and reliability.

In the current study, we will offer an incentive valued at $200 for physicians for one hour of interview time. The proposed incentive amount is below typical market incentive rates. Market incentive rates for physicians are approximately $250 to $350 for similar research activities, with higher rates for specialists.

Additional Considerations

In reviewing OMB’s guidance on the factors that may justify paying incentives to research participants, we have determined that the following principles apply:

Incentives help reduce costs. OMB’s guidance states that “If prior or similar surveys have devoted considerable resources to nonresponse follow-up, it may be possible to demonstrate that the cost of incentives will be less than the costs of extensive follow-up.”¹⁵

Evidence demonstrates that monetary incentives have a positive effect on encouraging survey participation across different research modes and populations.^16,17,18 Empirical studies have established that larger incentives (e.g., $100) result in significantly better response rates than smaller ones (e.g., $50, $20).^17,19 Singer and Ye¹⁶ found that each dollar of incentive paid resulted in one-third of a percentage point increase in response rate compared with no incentive.

Consequences of insufficient incentives include increased time and cost of recruitment, increased “no-show” rates, and increased probability of cancelled or postponed interviews.

During the current pandemic, many people are working from home and experiencing “Zoom fatigue” and getting them to participate in yet one more Zoom call can be difficult. For some health care providers, the pandemic brings additional guidelines for sanitization between patients, potentially reduced staff in the office due to school children being home or absences related to suspected COVID-19 infection and quarantine. All these factors increase the workload for available staff. Thus, incentives may need to be higher, to encourage physicians to set aside the time needed to participate.

Similar incentives have been approved under recent OMB packages.

According to item 76 in the Memorandum for the President’s Management Council, past experience can be utilized to justify a more elevated honorarium: “Agencies may be able to justify the use of incentives by relating past survey experience, results from pretests or pilot tests, or findings from similar studies. This is especially true where there is evidence of attrition and/or poor prior response rates”.¹⁴

Comparable Approved Incentive Rates for Consumers. Not only is the proposed incentive of $50, with the possibility of offering $75, significantly lower than market rate, but it is also consistent with what OMB approved in recent years for remote interviews or focus groups conducted by FDA and other agencies:

• FDA Boxed Warnings Study (OMB exempt because it falls under the 21^st Century Cures Act; 2020)

  • $50 incentive for 30-minute remote interviews (which would translate to $100 for 60-minute interview)

• Consumer Financial Protection Bureau Consumer Response Intake Form Improvement Study, Second Iteration (Intake Form Improvement Study II) (OMB Control number 3170-0042; 2017)

  • $75 for 60-minute remote interviews

• FDA Accelerated Approval Process (under generic OMB Control Number 0910-049; 2018)

  • $75 for 60 minutes with general population consumers

• Study of Oncology Indications in Direct-to-Consumer Television Advertising (OMB control number 0910-0885; 2020)

  • $50 for 60-minute remote cognitive interviews

• FDA Biosimilars Patient Study (OMB control number 0910-0695; 2019)

  • $75 incentive for 90-minute virtual focus groups (which translates to $50 for 60-minute virtual focus group)

• Studies to Enhance FDA Communications Addressing Opioids and Other Potentially Addictive Pain Medications (OMB control number 0910-0695; 2016)

  • $75 incentive for 90-minute virtual focus groups (which translates to $50 for 60-minute virtual focus group)

Examples of the tiered strategy of using a lower incentive amount and increasing it if recruiting proves difficult:

• A similar tiered strategy was approved by OMB in 2017 for the Centers for Disease Control and Prevention’s Formative Research to Develop HIV Social Marketing Campaigns for Healthcare Providers (OMB No. 0920-1182).

• A similar strategy was also used successfully in 2019 on the Health Care Providers’ Understanding of Opioid Analgesic Abuse-Deterrent Formulations: Focus Groups (OMB control number-010-0847) for which iTracks was the recruiter.

Comparable Approved Incentive Rates for Physicians. The proposed $200 incentive for physicians is less than has been offered on several other recent FDA studies with providers:

• FDA Multiple Indications (OMB No. 0910-0695; 2019)

  • Primary care providers received $200 and specialists $300 for participating in 60-minute interviews

• FDA Testing Communications on Biological Products (OMB No. 0910-0687; 2015)

  • Specialists received incentives of $250 for participating in 60-minute interviews

• Generic Drug Substitution in Special Populations study (OMB No. 0910-0677; 2017)

  • Specialists received $250 incentives for participating in a one-hour focus group

• Studies to Enhance FDA Communications Addressing Opioids and Other Potentially Addictive Pain Medications (OMB No. 0910-0695; 2016)

  • Specialists received $250 for participating in 60-minute telephone interviews

Incentives improve data quality. OMB’s guidance states that a “justification for requesting use of an incentive is improvement in data quality. For example, agencies may be able to provide evidence that, because of an increase in response rates, an incentive will significantly improve validity and reliability to an extent beyond that possible through other means”.¹⁴

Several studies have demonstrated that the use of gifts of gratitude are an effective method for increasing response rates, particularly among hard-to-reach populations.²⁰ Numerous empirical studies have established that providing incentives can significantly increase participation rates, and that larger incentives (e.g., $100, $150) perform significantly better than smaller incentives.^18,21,22,23 If the incentive is not adequate, participants may agree to participate and then not show up or drop out early. Low participation may result in inadequate data collection or, in the worst cases, loss of government funds associated with recruitment and interviewer and observer time.¹⁷

As well as preventing a low show rate, incentives are necessary to ensure adequate representation among harder-to-recruit populations and can help attract a reasonable cross-section of participants, reflecting diversity in age, income, and education.^24,25 Numerous studies have shown that incentives can reduce nonresponse bias for key subgroups. Griffin et al.²⁶ and Lesser et al.²⁷ found that incentives reduced nonresponse bias for gender. Incentives have also been effective in increasing participation from minority respondents.²⁸

Leverage-salience theory argues that monetary incentives can help to recruit people who otherwise might not be motivated to respond (e.g., people who do not care about the topic,²⁹ lack altruistic motives for responding, have competing obligations,¹⁶ or are typically less likely to participate in research³⁰). Using incentives to bring in a cross section of consumers can reduce nonresponse bias if these participants (those less interested in the topic, men, minorities, high income) have different responses and feedback than those who would participate without incentives.³¹

When applied in a reasonable manner, incentives are not an unjust inducement and are an approach that acknowledges respondents for their participation. The use of incentives treats participants justly and with respect by recognizing and acknowledging the effort they expend to participate. In this particular study, we are asking physician respondents to provide thought-intensive, open-ended feedback that requires a high level of engagement.

To address below-market incentive rates and ensure successful recruitment and fielding, RTI will coordinate closely with FDA to monitor recruitment status. Additionally, we will ensure that other considerations are in place to increase the likelihood of participation, such as:

1. Ensuring an adequate recruiting period before the start of fielding (as well as ongoing recruiting, as needed, during fielding period);

2. Availability of sessions at time slots that, in our experience, have been popular among HCPs—for example, early morning, evenings, lunch; and

3. Having the flexibility and appropriate staff available to run concurrent sessions to leverage popular session times.

8. Questions of a Sensitive Nature:

None.

9. Description of Statistical Methods (i.e. Sample Size & Method of Selection):

Statistical methods to recruit representative samples will not be used in this study. This study employs qualitative methods and uses convenience samples. Participants will be recruited from opt-in panels using a screener. Recruitment staff will help ensure eligible participants are recruited for the study and send reminder emails to reduce no-shows.

BURDEN HOUR COMPUTATION

(Number of responses (X) estimated response or participation time in minutes (/60) = annual burden hours):

Type/Category of Respondent	No. of Respondents	Participation Time (minutes)	Burden (hours)
Diagnosed Consumers (Psoriasis or Type 2 Diabetes)
Screener completes	140	0.08 (5 min.)	11
Number of completes	70	1.00 (60 min.)	70
HCPs
Screener completes	104	0.08 (5 min.)	8
Number of completes	35	1.00 (60 min.)	35
Total Hours			126

REQUESTED APPROVAL DATE: July 23, 2021

NAME OF PRA ANALYST & PROGRAM CONTACT:

Ila S. Mizrachi

Paperwork Reduction Act Staff

ila.mizrachi@fda.hhs.gov

301-796-7726

Kathryn Aikin, Ph.D.

Senior Social Science Analyst

Kathryn.Aikin@fda.hhs.gov

301-796-0569

FDA CENTER: Center for Drug Evaluation and Research, Office of Prescription Drug Promotion

1AMA Physician Masterfile: American Medical Association; 2020. Accessed January 29, 2021. https://www.ama-assn.org/life-career/ama-physician-masterfile.

2 Morris, N. S., MacLean, C. D., Chew, L. D., & Littenberg, B. (2006). The single item literacy screener: Evaluation of a brief instrument to identify limited reading ability. BMC Family Practice, 7(21), 1-7.

3Russell, M. L., Moralejo, D. G., Burgess, E. D. (2000). Paying research subjects: Participants’ perspectives. Journal of Medical Ethics, 26(2), 126–130

4Lai, H. C., & Wirasinghe, R. (2017, May). Applied Research for Advertising Products: Tactics for Effective Research. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (pp. 1144-1151).

5Ganon, R. How to How to Set the Right Incentive

https://help.respondent.io/en/articles/1030301-how-to-set-the-right-incentive

6U.S. Bureau of Labor Statistics (BLS), “Civilian workers by occupational and industry group,” March, 2020, Table 2, total compensation for civilian workers: http://www.bls.gov/​ncs/​ (visited June 17, 2021)

7BLS, Occupational Outlook Handbook, Childcare Workers, on the Internet at https://www.bls.gov/ooh/personal-care-and-service/childcare-workers.htm (visited June 17, 2021).

8Dykema, J., Stevenson, J., Day, B., Sellers, S. L., & Bonham, V. L. (2011). Effects of incentives and prenotification on response rates and costs in a national web survey of physicians. Evaluation & the Health Professions, 34(4), 434-447.

9Ziegenfuss, J. Y., Burmeister, K., James, K. M., Haas, L., Tilburt, J. C., & Beebe, T. J. (2012). Getting physicians to open the survey: Little evidence that an envelope teaser increases response rates. BMC Medical Research Methodology, 12(1), 41.

10Asch, S., Connor, S. E., Hamilton, E. G., & Fox, S. A. (2000). Problems in recruiting community-based physicians for health services research, Journal of General Internal Medicine, 15(8), 591-599.

11VanGeest, J., Johnson, T. & Welch, V. (2007). Methodologies for improving response rates in surveys of physicians: A systematic review. Evaluation and the Health Professions, 30, 303-321.

12Martins, Y., Lederman, R. Lowenstein, C. et al. (2012). Increasing response rates from physicians in oncology research: A structured literature review and data from a recent physician survey. British Journal of Cancer, 106(6), 1021-6.

13Converse, J. M. & Presser, S. (1986). Survey questions: Handcrafting the standardized questionnaire (No. 63). Thousand Oaks, CA: Sage.

14DeVellis, R. F. (2016). Scale development: Theory and applications (Vol. 26). Thousand Oaks, CA: Sage.

15Office of Management and Budget. (2006). Questions and Answers When Designing Surveys for Information Collections. Washington, DC: Office of Information and Regulatory Affairs. OMB. Retrieved from: https://obamawhitehouse.archives.gov/sites/default/files/omb/inforeg/pmc_sur vey_guidance_2006.pdf

16Singer, E., Van Hoewyk, J., Gebler, N., & McGonagle, K. (1999). The effect of incentives on response rates in interviewer-mediated surveys. Journal of Official Statistics, 15(2), 217.

17Singer, E., & Ye, C. (2013) The Use and Effects of Incentives in Surveys. ANNALS of the American Academy of Political and Social Science 645, 112–41

18Church, A. H. (1993). Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opinion Quarterly, 57(1), 62-79.

19Hsu, J. W., Schmeiser, M. D., Haggerty, C., & Nelson, S. (2017). The effect of large monetary incentives on survey completion: Evidence from a randomized experiment with the survey of consumer finances. Public Opinion Quarterly, 81(3), 736-747.

20Wiant, K., Geisen, E., Creel, D., Willis, G., Freeman, A., de Moor, J. & Kablunde, C., (2018). Risks and rewards of using prepaid vs. postpaid incentive checks on a survey of physicians. BMC Medical Research Methodology, 18; 104.

21Martinez-Ebers V. (1997). Using Monetary Incentives with Hard-To-Reach Populations in Panel Surveys. International Journal of Public Opinion Research, 99(1), 77-86.

22Shaghagi A., Bhopal R. S., Sheikh A. (2011). Approaches to recruiting 'hard-to-reach' populations into research: A review of the literature. Health Promotion Perspectives, 1(2), 86-94.

23Shettle, C., & Mooney, G. (1999). Monetary incentives in US government surveys. Journal of Official Statistics, 15(2), 231.

24Morgan, D. L., Scannell, A. U. (1998). Planning Focus Groups. Thousand Oaks, CA: Sage.

25Groth, S. W. (2010). Honorarium or coercion: use of incentives for participants in clinical research. Journal of the New York State Nurses Association, 41, 11-3

26Griffin, J. M., Simon, A. B., Hulbert, E., Stevenson, J., Grill, J. P., Noorbaloochi, S., & Partin, M. R. (2011). A comparison of small monetary incentives to convert survey non-respondents: a randomized control trial. BMC Medical Research Methodology, 11(1), 81.

27Lesser, V. M., Dillman, D. A., Carlson, J., Lorenz, F., Mason, R., and Willits, F. (2001) Quantifying the Influence of Incentives on Mail Survey Response Rates and Nonresponse Bias. Paper presented at the annual meeting of the American Statistical Association, Atlanta, GA.

28Singer, E. & Kulka, R. A. (2002). Paying Respondents for Survey Participation. In M. Ver Ploeg, R. A. Moffitt, & C. F. Citro (Eds.), Studies of Welfare Populations: Data Collection and Research Issues, Washington, D.C.: National Academy Press.

29Groves, R. M., Presser, S., & Dipko, S. (2004). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68(1), 2-31

30Guyll, M., Spoth, R., & Redmond, C. (2003). The effects of incentives and research requirements on participation rates for a community-based preventive intervention research study. Journal of Primary Prevention, 24(1), 25-41.

31Castiglioni, L., Pforr, K. (2007). The effect of incentives in reducing non-response bias in a multi-actor survey. Presented at the 2^nd Annual European Survey Research Association Conference, Prague, Czech Republic, June, 2007.

11