B. Statistical Methods
The Research and Evaluation Survey for the Public Education Campaign on Tobacco (RESPECT) consists of a baseline survey and follow-up cross-sectional surveys with an embedded longitudinal cohort in This Free Life campaign and control cities beginning approximately 6 months after campaign launch. The outcome evaluation data collection is occurring in 12 campaign-targeted cities and 12 similar (“comparison”) cities. The 12 campaign and 12 comparison cities were randomly selected from a list of 24 potential campaign cities identified by FDA and their media contractor. We drew a stratified random sample of 12 cities to serve as comparison cities, with the remaining 12 cities selected as campaign markets. All 24 markets were initially grouped into regions.
The primary recruitment method for this data collection involves conducting intercept interviews at LGBT social venues. Eligible respondents will be young adults who are 18 to 24 years old and who self-identify as LGBT. The sample includes young adults who self-identify as LGBT, as well as young adults who self-identify as being queer, transsexual, gender variant or pansexual (also referred to as omnisexual or trisexual). We screen potential respondents at these venues. Eligible participants are invited to complete the outcome survey online (via the participant’s personal computer or mobile device). We complement this strategy by recruiting LGBT young adults via social media from the same cities. Each wave of the seven waves of data collection consists of a sample size of approximately 3,150 18-24 year olds, with half of the sample (N=1,575) from 12 campaign-targeted cities and half (N=1,575) from comparison cities. Our goal is to recruit approximately 75% of the sample via intercept interviews and 25% via social media at baseline. Respondents are invited to complete follow-up surveys at 6-month intervals. We are assuming a 50% retention rate at each wave and recruit additional participants at each follow-up wave to keep the sample size constant at 3,150 18-24 year-olds. Once again, approximately 75% of the newly recruited participants come from intercept interviews and 25% from social media.
To ensure adequate representation of different subgroups of the young adult LGBT community (e.g., lesbian/gay females, gay males, bisexual females, bisexual males, transgender young adults), social media advertisements run in up to three simultaneous campaigns using ad targeting tools to reach three subgroups of LGBT young adults: gay and bisexual male young adults (i.e., men who are interested in men or men and women), lesbian/gay and bisexual female young adults (i.e., women who are interested in women or women and men) and transgender or genderqueer young adults (identified via keyword targeting using keywords relevant to transgender/genderqueer young adults). Quotas for LGBT subgroups are programmed in the survey instrument to ensure that adequate numbers of participants in LGBT subgroups are recruited via social media. Once social media quotas are met for each LGBT subgroup, additional members of this subgroup are not recruited to complete the survey.
For the intercept surveys, we seek input from local LGBT Outreach Coordinators at Health Departments, and from Rescue Agency, and conduct online searches to identify LGBT bars, nightclubs, community centers, Pride festivals, and other relevant LGBT-oriented venues and events. We select a convenience sample of venues and particular dates and times within each sampled bar to maximize likelihood of intercepting the targeted number of participants per city within a brief field period within each city. As needed to establish contact with venue owners, recruitment calls to venue owners and managers are supplemented by in-person contacts by local field staff. As agreed upon with the venue owners, a small team of interviewers visits the venue at an established time and approaches patrons who appear to be in the target age range. Participants who agree to participate are asked to fill out a short screening instrument (Attachment 3) on a tablet to determine eligibility. Eligible individuals are sent an email or text message to complete the baseline survey online (Attachment 2).
To supplement the field data collection strategy above, we recruit a convenience sample of additional LGBT young adults through Facebook. We post advertisements to social media and invite young adults 18 to 24 to complete a brief online screener to determine their eligibility (i.e., LGBT, 18 to 24, and living in one of the 24 selected cities). Sample selection for respondents recruited from social media involves posting advertisements on Facebook for viewing by members more likely to be 18 to 24 years old and self-identify as a LGBT young adult according to their account settings. The advertisements are geographically targeted in these social media platforms to people living in the 24 selected cities. Eligible respondents are directed to complete the baseline survey online until the required number of surveys is obtained.
Power Analysis
Statistical power estimates provide guidance on reasonable expectations for observing statistically significant change in outcomes of interest. This process requires an understanding of the study design, planned analyses (i.e., statistical model), expectations about the minimum detectable effect (MDE), as well as characteristics of the population and measures involved.
For the purpose of estimating statistical power for the RESPECT evaluation of FDA’s This Free Life campaign, we assumed data collection would reflect a repeated cross-sectional design among 24 cities, with 12 This Free Life cities and 12 cities that serve as a comparison group. Although the data collection includes an embedded longitudinal design, for the purposes of the analyses and power calculations, we are treating each wave of data collection as an independent cross-section.
The proposed impact analysis accounts for the repeated cross-sectional data collection using a generalized linear hierarchical regression model that assesses change in the proportion of young adults that agree with a belief statement related to smoking tobacco (e.g., perceived approval, perceived prevalence, and perceived popularity). The test statistic involves a two-tailed hypothesis test with a Type I error rate of 0.05 and a Type II error rate of 0.20, yielding 80% statistical power. Our parameter estimates include an intraclass correlation coefficient (ICC) of 0.01 to account for the geographic clustering of respondents and a variance inflation factor of 1.25 to account for potential imbalance across conditions. To some extent, these factors are offset by parameters that serve to reduce variation. Those parameters include over-time correlation corrections of 0.55 at the cluster levels that account for repeated measures in the same cities as well as a 0.25 variance reduction at the individual level for the inclusion of demographic and socio-economic covariates. These parameter estimates are available in the published literature and support by our experience conducting similar studies (Murray & Short 1997; Murray & Blitstein 2003; Janega, Farrelly, Davis et al. 2005).
The campaign evaluation’s goal is to be able to identify change of 10 percentage points or greater as statistically significant. There are little available data in the peer-reviewed literature on the level of agreement we can anticipate at baseline. Accordingly, we relied on the conservative assumption that 50% of young adults would agree with campaign messages at baseline.
Given the parameters and assumptions detailed above, the impact evaluation of FDA’s This Free Life campaign requires data from 1,575 LGBT young adults in total in the 12 campaign cities and 1,575 LGBT young adults total in the 12 comparison cities (N = 3,150) at each wave of data collection. This sample size is predicated on the assumption that agreement with campaign messages is 50% at baseline and increases to 60% at the time of follow up data collection. If actual agreement at baseline is either higher or lower than this value, statistical power is improved and smaller program impacts can be detected with the same sample of respondents. This effect would result in an odds ratio of approximately 1.50, meaning that young adults exposed to the campaign would be 1.50 times more likely than young adults not exposed to the campaign to agree with campaign messages about the effects of smoking tobacco.
This section describes the procedures for in-person data collection. Data will be collected in seven waves. Baseline data collection began in January 2016. The seventh wave of data collection will be conducted between May and August 2019.
RTI staff recruit venues for data collection following the Venue Recruitment Guide included as Attachment 7. As needed to establish contact with bar owners, these recruitment calls are supplemented by in-person contacts by local field staff using the same Recruitment Guide. Calls are made (and in-person visits, when possible) to the list of bars identified with input from local LGBT Outreach Coordinators at Health Departments, Rescue Agency, and online searches. Venue owners are first given brief background information about data collection procedures. They are then asked questions about when and how often LGBT young adults visit their establishments and provided with information about participant cash incentives (as a benefit of allowing for recruitment in venues). RTI staff then confirm recruitment times with venue owners who agree to allow recruitment in their establishment.
As agreed upon with bar owners, a small team of interviewers visits the bar at the established time and approaches patrons who appear to be in the target age range using the talking points and FAQs shown in Attachment 8. RTI staff introduce themselves and provide the following information to potential participants: incentive amount for completing screener survey ($10 cash), time needed to complete survey (5 minutes), and incentive amount for emailed web survey ($20-25 depending on how quickly they respond). In addition, RTI staff provide the following information based on questions asked by potential participants: study purpose, reasons for participating, study sponsor information, background information on RTI and FDA CTP, information on privacy and confidentiality, and IRB contact information.
Once a participant agrees to complete the screener, the interviewer opens the case, selects the appropriate city and bar, advances the screen to the informed consent screen (Attachment 5), and then passes the tablet to the respondent to complete. Within the screener (Attachment 3), respondents are asked their zip code, age, sexual identity, and sexual orientation (all necessary to establish eligibility including LGBT status, being within the appropriate age range, and living in the recruitment city). Respondents are also asked two questions about their tobacco use; these tobacco questions are included to allow comparison of eligible respondents who do and do not later participate in the full survey to facilitate nonresponse bias analysis.
Completed screeners automatically analyzed within the survey software to determine the young adult’s eligibility. Respondents who screen as eligible are asked for their email address, cell phone number, and first name so we can confirm we have not previously enrolled this person, and so we can send an invitation for the full web survey to those who are eligible and new to the study. GPS data are associated with each screened case to assist in detecting possible falsification by interviewers. Completed screeners that appear to be completed somewhere other than a scheduled venue are investigated further. Respondents are paid $10 in cash and asked to initial an incentive receipt (Attachment 9) at the end of the screener. Respondents who screen as eligible are provided with an information card (Attachment 10) informing them that they will receive an email or text message invitation to complete the survey within the next two weeks.
If new intercept respondents click on the provided link within the electronic invitation, they are asked to provide consent (Attachments 6a) before completing the survey online (Attachment 2).
At follow-up, young adults who participated in prior waves of data collection are re-contacted using a follow-up email (Attachment 11). If returning respondents agree to participate at follow-up, they click on the provided link within the follow-up email, which directs them to a screening questionnaire (Attachment 13), which requires informed consent (Attachment 14). If they are deemed eligible, they are asked to provide consent (Attachment 6b) before completing the survey online (Attachment 2).
Intercept respondents are paid $10 in cash for completing the screener and will bare asked to initial an incentive receipt (Attachment 9) at the end of the screener. Those who are eligible receive an email invitation to complete the full survey. If they choose to complete the baseline survey and any follow-up surveys, if invited, they receive an online gift card of $20.00, with a $5 bonus (total of $25.00) if they complete the full survey within two days of invitation.
To supplement this sample, young adult respondents are also being recruited in the same 24 U.S. cities (12 campaign and 12 comparison cities) through social media advertisements on Facebook targeted at LGBT 18- to 24-year-olds.
RTI places ads on Facebook (see Attachment 12 for several examples). As much as possible, these ads are targeted toward potentially eligible respondents, who are thought to be 18 to 24 years old, live in the data collection cities, and potentially may self-identify as LGBT. Potential respondents’ questions or comments about the ads or survey are addressed as described in Attachment 15.
When clicked, the ads direct the potential participant to a web-based
screening questionnaire (Attachment 3), which requires informed consent (Attachment 5).
Respondents who are deemed eligible then go on to provide consent (Attachment 6a) and complete the baseline instrument (Attachment 2) online. All respondents who complete this survey receive a virtual gift card valued at $20. Participants recruited via social media are not paid to complete the screener.
All follow-up data collection procedures for those recruited via social media are the same as those described above for intercept respondents at follow-up, including incentive payment procedures. Respondents may be re-contacted via email to address errors in data collection procedures, such as paying an incorrect incentive or erroneously deeming respondents as ineligible due to errors in data processing (Attachment 11).
The ability to obtain the cooperation of potential respondents in the baseline survey and maintain their participation across all survey waves is important to the success of this study. In preparation for launching the baseline data collection, we reviewed procedures for enlisting respondent cooperation across a wide range of surveys, incorporated best practices from those surveys into the data collection procedures, and adapted the procedures through continuous improvement across the survey waves.
In addition to the $20 incentive (with $5 bonus for intercept participants who complete the survey within 2 days of receipt of the survey link) and $10 incentive for intercept participants to complete the screening survey, the study uses procedures designed to maximize respondent participation. The incentive procedures and amounts for new cross-sectional sample at follow-up waves are identical to the baseline survey. For the longitudinal sample, participants receive $20 for completing the follow-up survey and an additional $5 if they complete the survey within 48 hours of the invitation to participate.
Prior intercept studies have demonstrated the importance of careful recruitment and training of field staff. Matching the characteristics of the interviewers to the study population has been found to be helpful (Spooner. et. al, 1997). To the extent possible, field interviewers are recruited from the LGBT community within each city. The ideal candidate is familiar with the venues we are using for data collection and knowledgeable about times in which 18- to 24-year olds are most likely to be present in high numbers. In addition, the ideal candidate is reliable, articulate, outgoing, confident, and non-judgmental. Interviewer training is participatory, allowing ample time to practice approaching respondents to introduce the survey, answer common questions, and overcoming objections. As noted in Section B.2.1, interviewers recruiting bar/nightclub intercept participants use a document that includes talking points and FAQs (Attachment 8) to encourage participation.
When interviewers transmit their data from completed intercept screenings, the data are summarized in daily reports posted to a web-based case management system accessed by field supervisors and RTI’s data collection managers. On a daily basis, supervisors use these reports to review response rates and production levels. This information allows supervisors to determine progress toward production goals and adjust goals for the remaining venues within each city. Supervisors discuss information and challenges with their interviewers each week.
RTI conducted rigorous internal testing of the online survey instrument prior to data collection. Evaluators reviewed the online test version of the instrument to verify that instrument skip patterns were functioning properly, delivery of campaign media materials was working properly, and that all survey questions were worded correctly and were in accordance with the instrument approved by OMB.
The following individuals inside the agency have been consulted on the design of the campaign evaluation plan, audience questionnaire development, or intra-agency coordination of information collection efforts:
Gem Benoza
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-0088
E-mail: Maria.Benoza@fda.hhs.gov
David Portnoy
Office of Science
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 301-796-9298
E-mail: David.Portnoy@fda.hhs.gov
Matthew Walker
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-3824
E-mail: Matthew.Walker@fda.hhs.gov
Leah Hoffman
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-743-1777
E-mail: Leah.Hoffman@fda.hhs.gov
Janine Delahanty
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-9705
E-mail: Janine.Delahanty@fda.hhs.gov
Ollie Ganz
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-5389
E-mail: Ollie.Ganz@fda.hhs.gov
The following individuals outside of the agency have been consulted on questionnaire development. Additionally, input has been solicited and received from FDA on the design of this study, including participation by FDA in meetings with OMB:
Matthew Farrelly
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-541-6852
E-mail: mcf@rti.org
Erik Crankshaw
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-316-3809
E-mail: ecrankshaw@rti.org
Jennifer Duke
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-485-2269
E-mail: jduke@rti.org
Jamie Guillory
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-316-3725
E-mail: jguillory@rti.org
Kristine Fahrney Wiant
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-485-5531
E-mail: fahrney@rti.org
Jane Allen
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-597-5115
E-mail: Janeallen@rti.org
Youn Lee
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-485-5536
E-mail: Younlee@rti.org
Amy Henes
RTI International
701 13th Street, NW
Washington, DC, 20005
Phone: 202-974-7821
E-mail: ahenes@rti.org
Patricia LeBaron
RTI International
230 W Monroe Suite 2100
Chicago, IL 60606
Phone: 312-777-5204
E-mail: plebaron@rti.org
Pamela Rao
Akira Technologies, Inc.
1747 Pennsylvania Ave NW Suite 600
Washington, DC 20002
Phone: (202) 517-7187
Email: prao@akira-tech.com
Xiaoquan Zhao
Department of Communication
George Mason University
Robinson Hall A, Room 307B
4400 University Drive, 3D6
Fairfax, VA 22030
Phone: 703-993-4008
E-mail: xzhao3@gmu.edu
Joseph Lee
East Carolina University
Belk Building
Greenville, NC
E-mail: leejose14@ecu.edu
Jeff Jordan
Rescue Social Change Group
3436 Ray Street
San Diego, CA 92104
Phone: 619-231-7555 x 150
Email: jeff@rescueagency.com
Mayo Djakaria
Rescue Social Change Group
660 Pennsylvania Avenue SE, Suite 400
Washington, DC 20003
Phone: 619-231-7555 x 120
Email: mayo@rescueagency.com
Dana Wagner
Rescue Social Change Group
660 Pennsylvania Ave SE, Suite 400
Washington, DC 20003
Phone: 619-231-7555 x 331
Email: dana@rescueagency.com
Brandon Tate
Rescue Social Change Group
6463 Sunset Boulevard, Suite 504
Hollywood, CA 90028
Phone: 619-231-7555 x 152
Email: brandon@rescueagency.com
References
Farrelly, M.C., Davis, K.C., Haviland, M.L., Messeri, P., Healton, C.G. (2005). Evidence of a dose-response relationship between "truth" antismoking ads and youth smoking prevalence. American Journal of Public Health, 95(3), 425-431.
Janega, J.B., Murray, D.M., Varnell, S.P., Blitstein, J.L., Birnbaum, A.S., Lytle, L.A. (2004). Assessing the most powerful analysis method for schools intervention studies with alcohol, tobacco, and other drug outcomes. Addictive Behaviors, 29(3), 595-606.
Murray, D.M., Blitstein, J.L. (2003). Methods to reduce the impact of intraclass correlation in group- randomized trials. Evaluation Review, 27(1), 79-103.
Murray, D.M., Short, B.J. (1997). Intraclass correlation among measures related to tobacco- smoking by adolescents: Estimates, correlates, and applications in intervention studies. Addictive Behaviors, 22(1), 1-12.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Matthew Farrelly |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |