Evaluation of the Food and Drug Administration’s Tobacco Public Education Campaign
0910-0753
B. Statistical Methods
The primary outcome study will consist of three probability samples: 1) two longitudinal surveys of approximately 3,300 youth each for the national (non-trier and experimenter) campaign, and 2) a longitudinal survey of 1,434 male youth for the rural smokeless campaign. This longitudinal design allows us to calculate baseline-to-follow-up changes in campaign-targeted outcomes for each study participant. We hypothesize that if the campaigns are effective, the baseline-to-follow-up changes in outcomes should be larger among individuals exposed to the campaigns more frequently (i.e., dose-response effects). Eligible youth will be aged 11 to 16 at baseline and 13 to 19 by the end of data collection. For the national sample, age is the only screening criterion. For the rural smokeless campaign, potential respondents will be screened for age and sex, as only males are eligible. As the cohorts will be aging over this time period, the data collected throughout the study will reflect information from youth aged 11 to 19. Both national and rural surveys will be conducted by RTI.
For the national sample evaluation with Cohort 1, we began by taking a probability proportional to size sample of 75 U.S. designated market areas (DMAs) that will represent the full universe of 210 DMAs, geographic areas that constitute the target population of television markets. We selected 30 DMAs for data collection for the rural smokeless campaigns. Fifteen are from the DMAs where the campaign is targeted. The other 15 will serve as comparison markets/cities. These DMAs are our Primary Sampling Units (PSUs). Within DMAs, our secondary sampling units (SSUs) are census block groups allocated roughly proportionally across the PSUs. The census block groups will serve as the areas in which our address sample will be selected. Our size measure for both primary sampling PSUs and SSUs is the number of youth aged 11 to 16. Our third stage sampling units are addresses from the Computerized Delivery Sequence file (CDS) obtained from Compact Information Systems (CIS), one of two vendors having a national license with the U.S. Postal Service. We sampled roughly 100 addresses per selected census block groups (CBG).
To obtain the 3,300 interviews at Wave 5 (i.e., fourth follow-up after baseline) of data collection of the general market (non-trier and experimenter) campaign evaluation for Cohort 1, we started with 47,817 sampled addresses from a frame of all locatable addresses within the selected CBGs (locatable addresses are those suitable for in-person data collection— post office boxes are not considered locatable). Based on prior experience with address samples for in-person surveys, we expected 95% of the selected units to be correctly geocoded into the selected census block group and about 85% of those correctly geocoded addresses to be occupied. These are determinations that RTI’s field interviewers make at the start of data collection. Of the approximately 38,612 housing units that the interviewers visit, we expected that 70%, or 27,028 households would complete the screener. Because we are using targeted address frames that have indicators for addresses with youth in our target population age range, we expected the sample to yield approximately 11,190 eligible households with at least one youth in the age range; that is, we expected the households with eligible youth to be roughly double the national average. To be conservative, we have not adjusted here for households that will have multiple youth in the age range (although all will be eligible). We expected a 50% interview completion rate at Wave 1 (i.e., baseline), which would yield 8,057 completes. Due to the incentives and the option for completing the survey via the Web, we expect 80% of those interviewed at one wave to respond at the next wave, which will yield approximately 3,300 completes by Wave 5 (i.e., fourth follow-up after baseline) for Cohort 1. Exhibit 6 outlines this progression of one-time actual numbers for all three components of the campaign evaluation.
For the new national sample evaluation (Cohort 2), we will begin by taking a sample of 100 Primary Sampling Units (PSUs) probability proportional to the number of 11-17 year olds. Our PSU will be Public Use Microdata Areas (PUMAs). PUMAs are created for the dissemination of Census public use microdata from the American Community Survey but also can serve as PSUs clusters (McMichael & Chen, 2015). The area frame of PUMAs will cover the entire lower 48 states plus the District of Columbia. Our Secondary Sampling Units (SSUs) will be Postal Carrier Route. This cluster of addresses amounts to the list of addresses a mail carrier will deliver in one day. We will select between 400 and 500 SSUs probability proportional to the number of 11-17 year olds. For our third and final stage we will select addresses from the Computerized Delivery Sequence file (CDS) leased from Compact Information Systems (CIS). We will select an approximate equal number of addresses from each SSU (Approximately 85 – 106 per SSU). Exhibit 6 details our response assumptions for National Sample Cohort 2.
The rural smokeless campaign sample differs from the national sample in one major way—only males in the age range will be considered eligible. Also, we are assuming a slightly lower rate of housing units that geocode to the correct census block group (90%) due to prior research showing that geocoding is less accurate in rural areas. Aside from those two modifications, all other assumptions are the same as the national components. We will need to select 24,668 addresses to yield 1,434 completes by Wave 5 (i.e., fourth follow-up after baseline) of the rural smokeless sample. Exhibit 6 details our response assumptions for the Rural Smokeless Sample.
For the ExPECTT Cohort 2 baseline, the sample will include addresses. By doing batch tracing, we can potentially identify phone numbers associated with those addresses. Batch tracing is an automated service in which an address is matched to information in multiple databases. The goal will be to identify telephone numbers associated with each sample address. The batch tracing vendor returns the most current address information as determined by hard logic algorithms. By using these services, we can match addresses and telephone numbers quickly at a low cost. For households that don’t return a mail screener, the field interviewer may be able to use batch tracing to identify a phone number for the household. The field interviewer can then call the address to find out if they received the mailer, to schedule an appointment, etc. If the interviewer confirms the phone number they are calling is associated with the sampled addresses, they may be able to do the screener over the phone to save in travel costs if the household is not eligible. It is also helpful to do batch tracing between waves so that we can locate new phone numbers and possible new addresses if the parent has changed their number or the family has moved. Having this additional information will increase our ability to contact households more quickly, thus reducing field costs.
Prior to the start of field work we will contact all selected housing units by mail asking them to complete a short mail screener. A small $2 token of appreciation will be sent along with the lead letter. The lead letter will briefly explain the purpose of the survey and request the cooperation of a parent or legal guardian aged 18 or older in each household. This letter will be printed on project-specific letterhead with the signature of FDA’s Project Director and RTI’s Project Director. For the second cohort general market campaign evaluation, we will use two slightly different mail screeners. Screener B consists of 3 fewer items than the primary mail screener, Screener A. Half of the households sampled will receive Screener A and half will receive Screener B. We will analyze the data generated by the screeners to determine whether the briefer screener (Screener B) is as effective as our current primary screener (Screener A). If it is as effective, we will be able to use the briefer screener in the future. The tablet screener will include the measures from Screener A. The mail screener includes several questions designed to assess the presence of an eligible youth in the home, and an envelope in which to return the screener to RTI. A reminder postcard will be sent a week to 10 days after the first mailing. Two to three weeks later, a second screener will be mailed to screener non-respondents. The second screener will not contain an additional $2.
Exhibit 6. Addresses and the Associated Assumptions to Yield the Needed Number of Completes
Activity |
Rural Sample |
National Sample |
National Sample Cohort 2 (All Youth) |
Selected addresses |
24,668 |
47,817 |
42,510 |
Correctly geocoded housing units |
22,201(90%) |
45,426 (95%) |
NA |
Occupied housing units |
18,871 (85%) |
38,612 (85%) |
36,134 (85%) |
Screened households |
13,210 (70%) |
27,028 (70%) |
27,100 (75%) |
Eligible households |
5,469 (41%) |
11,190 (41%) |
11,111 (41%) |
Eligible persons |
2,734 (50%) |
11,190 (100%) |
11,111 (100%) |
Baseline completes |
1,969 (72%) |
8,057 (72%) |
8,000 (72%) |
Wave 2 (1st follow-up) completes |
1,575 (80%) |
6,445 (80%) |
6,400 (80%) |
Wave 3 (2nd follow-up) completes |
1,260 (80%) |
5,156 (80%) |
5,120 (80%) |
Wave 4 (3rd follow-up) completes |
1,593 (90%) |
4,125 (80%) |
4,096 (80%) |
Wave 5 (4th follow-up) completes |
1,434 (90%) |
3,300 (80%) |
3,277 (80%) |
Note: The 50% response rate at the first time point is a product of the person completion rate and the household screening rate (72% * 70%).
For the purposes of estimating statistical power for the national samples, we assume that the test statistic evaluating campaign impact will involve a two-tailed hypothesis test with a Type I error rate of 0.05 and a Type II error rate of 0.20, yielding 80% statistical power. Our estimates include an intraclass correlation coefficient (ICC) of 0.01 to account for the geographic clustering of respondents and a variance inflation factor of 1.25 to account for potential imbalance across conditions. To some extent, these factors are offset by parameters that serve to reduce variation. Those parameters include over-time correlation corrections of 0.75 at the cluster and individual levels as well as a 0.20 reduction at the individual level for the inclusion of demographic and economic covariates that will reduce between-person variations. These parameter estimates are available in the published literature and supported by our experience conducting similar studies (Murray & Short, 1997; Murray & Blitstein, 2003; Janega et al., 2004; Farrelly et al., 2005). Statistical models will be used to assess change in the prevalence of past 30-day tobacco use among youth aged 12 to 17 as the primary impact of the media campaign. An analysis of data from the National Youth Tobacco Survey (NYTS) shows that the prevalence of tobacco use (i.e., cigarette, cigar, and smokeless tobacco) among this group was 16.7% prior to campaign launch. A 3-percentage point reduction in this rate of tobacco use would result in an 18% decline in tobacco use prevalence, or an odds ratio of approximately 0.80. This expectation is reasonable and similar to the change observed as a result of a previous national youth tobacco campaign (Farrelly et al., 2007). Based on these parameters, we anticipate data collection will include complete and repeated measures on approximately 3,300 youth in the target age range in each national sample cohort (see Exhibit 6). Consistent with theories of behavior change and previous campaign evaluations, we expect larger changes in intermediate outcomes such as intentions to use tobacco, and tobacco-related attitudes and beliefs. As a result, we should have sufficient power to also detect changes in these intermediate outcomes.
Expected effect sizes for the rural smokeless campaign were calculated with relatively little available data for comparative purposes. Data from the 2015 Youth Risk Behavior Survey indicates that the prevalence of smokeless tobacco current use among Caucasian (non-Hispanic) high school students is approximately 8%. The markets selected for the rural smokeless campaign are in states with higher than average smokeless tobacco use rates. Additionally, smokeless tobacco use rates are significantly higher among boys than among girls (CDC, 2012). Accordingly, restricting the sampling frame to boys aged 12 to 17 from these rural communities increases the nominal base rate; for our calculations, we assume a 15% base rate for smokeless tobacco use.
For the purposes of estimating statistical power, our primary outcome variable is 30-day prevalence of using smokeless tobacco, and our models estimate the change (pre-test/post-test) in prevalence among rural, male youth aged 12 to 17. We anticipate that the test statistic evaluating the campaign impact will involve a two-tailed hypothesis test with a Type I error rate of 0.05. Our estimates include an ICC of 0.005 to account for the geographic clustering of respondents and a variance inflation factor of 1.5 to account for unequal weighting effects. To some extent, these factors are offset by parameters that will serve to reduce variation. These factors include over-time correlation parameters of 0.75 at the cluster and individual levels as well as a 0.20 reduction at the individual level for the inclusion of demographic and economic covariates that will reduce between-person variations. These parameter estimates are available in the published literature and supported by our experience conducting similar studies (Murray, 1997; Murray & Blitstein, 2003; Janega et al., 2004; Farrelly et al., 2005).
At 80% statistical power, the rural smokeless campaign evaluation will include 34 rural youth respondents in each of the 30 media markets (N = 1,008), and the resulting analyses will be able to identify changes of 4 percentage points or larger. This effect would result in an odds ratio of approximately 0.70, meaning that boys exposed to the campaign would be 0.70 times less likely to use smokeless tobacco compared with boys not exposed to the campaign. This odds ratio represents a standardized effect size estimate (Cohen’s D) of 0.19, which is similar to the effects for media campaigns noted by Snyder and colleagues (2004) in their meta-analysis on the effects of health messages in mass media.
In addition to the primary outcome evaluation, we will conduct a series of Web-based, cross-sectional media tracking surveys, with a non-probability sample of 18,000 youth. The media tracking surveys are designed to assess youth’s awareness of and receptivity to campaign advertising over time. Surveys will be conducted periodically after campaign launch and will be timed to measure reactions to new campaign advertising, throughout the evaluation period. RTI will conduct this study using a sample of youth purchased from the digital data collection company Global Market Insite, Inc. (Lightspeed). Lightspeed will program the surveys, distribute a link to the Web-based surveys to approximately 180,000 members of their youth panel, collect the data, and track the number of completed surveys until we reach our target of 6,000. Parents or guardians of youth will provide their permission, following which youth will be advised of the privacy of their data and asked to provide their assent to participate before encountering the first survey question. All data will be disassociated with names, addresses, and other identifying information to ensure respondent privacy to the fullest extent of the law, and all data will be entered directly into secure Lightspeed servers.
Although the primary outcome evaluation will be the source for all public information about the campaign, the media tracking surveys allow FDA to track campaign awareness and reactions to advertising more frequently. Consistent tracking of campaign awareness and receptivity to its messages is necessary because campaign effects may diminish based on media purchase patterns (Wakefield et al., 2011). In addition, this sample allows FDA to assess awareness at more granular levels, particularly within subpopulation groups. The sample size is calculated based on CTP requirements to assess campaign effects by gender, age (aged 12 to 14 and aged 15 to 17), geographic area (rural and non-rural), and tobacco use susceptibility. Each media tracking survey will provide approximately 250 youth in each of these segments. This sample will yield 80% power within each subgroup to detect changes in campaign awareness and ad-level awareness between waves of 12%.
This section describes the procedures for baseline data collection. The procedures for the three data collections (the first and second cohort general market (non-trier and experimenter) campaign evaluations and the rural smokeless evaluation) differ slightly. The rural smokeless campaign evaluation and the second cohort general market campaign evaluation both make use of a mail screener to improve cost effectiveness. All baseline data will be collected by field interviewers at respondents’ homes. To be eligible, youth must be aged 11 to 16 at the time of the baseline data collection. Parents or legal guardians of selected youth will be asked to complete a brief survey about their household to provide context for the data collected from youth.
Before the interviewer’s arrival at the sampled dwelling unit (SDU), a lead letter (see Attachment 10_E2) will be mailed to the selected addresses. In the case of the rural smokeless campaign evaluation a small $2 incentive was sent along with the lead letter. The same will be done for the second cohort general market campaign evaluation mail screener. The lead letter briefly explains the purpose of the survey and requests the cooperation of a parent or legal guardian aged 18 or older in each household. This letter is printed on project-specific letterhead with the signature of FDA’s Project Director and RTI’s Project Director. The mail screener includes several questions designed to assess the presence of an eligible youth in the home, and an envelope in which to return the screener to RTI.
Interviewers visited all sampled homes for the first cohort general market campaign evaluation. For the rural smokeless campaign evaluation, interviewers visited all homes which were not explicitly excluded from the study on the basis of the mail screener. The same will be done for the second cohort general market campaign evaluation. In other words, they visit households known to have eligible youth, as well as households which did not return the mail screener. Upon arrival at each SDU, the interviewer refers an adult resident to the lead letter, and screener if applicable, and answers any questions that person might have about the study. If the resident has no knowledge of the lead letter, the interviewer provides another copy, explains that one was previously sent, and then answers any questions the person might have. If someone in the household returned a screener, the interviewer works to identify that adult individual. Regardless of whether the adult parent or guardian of an eligible youth has completed the mail screener, the interviewer will administer the tablet screener at that time. If no one is home during the initial visit to the SDU, the interviewer has the option to leave a card (see Attachment 13_E2) to inform the residents that the interviewer plans to visit the household at a different time. Further visits are made as soon as feasible after the initial visit. Interviewers make at least four additional visits beyond the initial visit to each SDU to complete the screening process and up to another four visits to complete interviews with selected youth, if at least one youth is selected for an interview.
If the interviewer is unable to contact a parent or legal guardian aged 18 or older at the SDU after repeated attempts, the field supervisor may send an unable-to-contact letter (see Attachment 13_E2) to reiterate information provided in the lead letter and ask for participation in the study. If the interviewer is still unable to contact anyone at an SDU, the interviewer might send an additional call-me letter (see Attachment 13_E2) to the SDU. The call-me letter will request that the residents call the field supervisor to set up a screening appointment.
When contact is made with an adult member of an SDU and introductory information about the study is communicated, the interviewer will present a Question & Answer Fact Sheet (see Attachment 11_E2) that provides answers to commonly asked questions. When a potential respondent refuses to complete the household screening or interviewing procedures, the interviewer will rely on their training and experience to accept the refusal in a positive manner. This technique will reduce the potential for creating an adversarial relationship between the residents and the interviewer that could preclude future visits. The supervisor might then request a refusal letter (see Attachment 13_E2) be sent to the residence. The refusal letter will be tailored to the specific concerns expressed by the potential respondent and ask him or her to reconsider participating in the study. Refusal letters will also include the supervisor’s telephone number, in case the potential respondent has questions or would like to set up an appointment with the interviewer. Unless the respondent calls the supervisor or RTI’s office to refuse participation in the study, one further attempt to enlist the household’s cooperation will be made by specially selected interviewers with experience in addressing initial refusals. Specially trained interviewers will also be selected based on their proximity to the case to minimize travel costs.
When an adult resident of a household agrees to cooperate with the study procedures, the interviewer will begin the SDU screening procedures using a tablet computer. The tablet screener will be administered even if a mail screener has been returned for the household. For the rural smokeless and second cohort general market campaign evaluations, the interviewer administers the screener if necessary, or if completed by mail, proceeds with the baseline interview. Survey data are collected from adult respondents immediately following in-person completion of the screener (Attachment 3_E2a), after respondents have been found eligible and have consented to participate. As many eligible youth aged 11 to 16 as currently reside in the household will be selected to complete the interview. A maximum of two adult parents or legal guardians will be selected to complete the interview, should there be eligible youth with different parents or legal guardians.
For each youth selected to complete the baseline interview, the interviewer will follow these steps:
The interviewer will obtain verbal consent from the parent or legal guardian for themselves and for the selected youth before approaching the youth for participation in the study.
The interviewer will verbally administer a brief screener and the parent survey using a tablet computer, to collect data specific to youth media use and household characteristics. For the second cohort general market campaign evaluation, the parental permission text will also be administered by tablet. A subset of the adult questions will be asked of each participating youth; the household questions will be asked once. If there are multiple adult respondents, the first adult to complete the survey will complete the household questions on behalf of the household; these questions will be omitted from the survey of the second adult respondent.
When parent or guardian permission has been obtained, the interviewer will obtain verbal assent from selected youth respondents. The assent form, which will appear as the first visible screen on the laptop, will be designed to communicate the goals and procedures to youth aged 11 to16 (Attachments 6_E2a, 6_E2b, 6_E2c and 6_E2d). The interviewer will also read the assent language to the youth before beginning the interview, to assure them that what they report will be kept confidential and to communicate the voluntary nature of participation and their right to refuse to answer any question asked.
When both consent and assent have been obtained, the interviewer will administer the first portion of the youth interview in a prescribed and uniform manner. For the self-completed portion of the youth interview, the interviewer will then turn the computer over to the youth to read the survey questions and enter responses to the questions directly into the computer.
After the interview is completed and before the verification information is collected, youth respondents will be offered an incentive of $20.00 each for participation. They will receive a receipt for this incentive (Attachments 13_E2 and 14_R). For verification purposes, a random subset of approximately 10% of the households of youth respondents will be contacted via telephone after the interview. The verification interviews will ask the parent to answer a few questions confirming that the interview took place, that proper procedures were followed, and that the amount of time required to administer the interview was within the expected duration. Verification letters will be mailed to respondent addresses when telephone numbers are unavailable (see Attachment 12_E2 and 12_R). Additionally, a random subset of approximately 10% of the remaining completed interviews will be verified using computer audio-recorded interviewing (CARI). This verification will confirm that the tablet screener, parental consent and parental permission process was adequately implemented and conducted. Consent for recording will be requested of the respondent as part of the general informed consent procedures. Any adverse events related to consent issues would be reported to both IRBs.
All interview data will be transmitted at least daily via secure encrypted data transmission to RTI’s offices, where the data will be subsequently processed and prepared for analysis, reporting, and data file delivery. Upon transmission to RTI, all data will be automatically wiped from all data collection devices used in the field.
Baseline data collection for Cohort 1will be followed by four follow-up surveys, and for Cohort 2 by three follow-up surveys conducted at approximately 8-month intervals. This design will produce data for the same youth over a 2-year period or longer. This study design will provide a more accurate and thorough understanding of tobacco initiation, prevalence, and cessation among the campaign’s target audience of youth aged 12 to 17. Eligible youth will be aged 11 to 16 at the baseline survey and 13 to 19 at the final survey wave. As the cohort will be aging over this time period, the data collected throughout the study will reflect information from youth aged 11 to 19. Like the baseline survey, most of the follow-up surveys will be conducted in person by interviewers. Youth respondents will also be offered the option to complete the follow-up surveys via a Web-based application. The expected proportion of in-person surveys for each follow-up wave is 70%, with the remaining 30% completed via the Web.
Panel maintenance letters will be sent out in advance of follow-up data collections to update contact information to the degree possible (Attachments 15_E1, 15_E2, 15_R and 16_R). Before interviewers make any in-person contact with sample members and their parents, lead letters with Web-survey log-in credentials will be sent to respondents to invite them to participate in the follow-up by Web or in-person (Attachments 10_E1 and 10_R). These advance letters will inform parents and youth about the study’s purpose and background, explain the survey procedures, and provide information to the respondent on participating via the Web or with an interviewer in their home. The letters will provide the Web address for the online version of the survey and the user ID and password each sample member will need to enter the survey application. Respondents who provide an e-mail address in the baseline survey will also receive an e-mail invitation to complete each follow-up surveys via the Web. The follow-up lead letter and text for the follow-up e-mail invitations are shown in Attachments 10_E1, 10_R, 18_E1, 18_R, 19_E1, 19_R, 20_E1, 20_R, 21_E1 and 21_R. Participation via the Web will provide flexibility and convenience to sample members who can complete the survey online. Completion of the Web-based survey will be tracked closely during each follow-up wave, to identify respondents who will need an interviewer visit to their home to complete the interview in-person. Parents will also receive fact sheets and a copy of the parent permission form before the start of each follow-up wave of data collection.
During the data collection, if an interviewer is unable to locate the participating family, the interviewer will request that the case be sent to interactive tracing through the project control system. Interactive tracing is conducted by one of RTI’s tracing specialists who reviews the case contact information in the control system and accesses resources and databases to search for additional or more current contact information to locate the parent. Once located, the contact information is shared with the interviewer through an update made within the project control system so that the interviewer can attempt to complete the associated case(s).
The outcome surveys will include the same set of items at baseline and in all follow-up surveys with the exception of items regarding each campaign and its materials, which will vary over the course of the campaigns (e.g., television ads, print materials) (Attachments 2_E1, 2_E2a, 2_E2b and 2_R). Minor revisions to surveys may be necessary given the media development process and the possibility of changes in campaign implementation, but every effort has been made to minimize the possibility of instrument changes. The youth survey instrument includes measures of demographics; tobacco use behavior; intentions to use tobacco; self-efficacy; cessation intentions; cessation behaviors; tobacco-related attitudes, beliefs, and risk perceptions; social norms; media use and awareness; and environmental questions. The youth surveys will include measures of audience awareness of and exposure to the campaigns’ advertisements as well as the aforementioned outcome variables of interest (see Attachment 2a). There is no parent survey at follow-up.
This section describes the procedures for the media tracking data collection. Cross-sectional media tracking data will be collected through a Web-based system with sample purchased from the digital data collection company Lightspeed. The survey will be administered three times over the course of the campaign evaluation and will be timed to provide feedback on youth awareness of and reactions to new campaign advertising. Data will be collected from total of 18,000 unique respondents over the course of the campaign evaluation. This sample size will make it possible to produce estimates for sixteen youth segments. For the purposes of estimating statistical power, we assume that the test statistic evaluating subpopulation differences in campaign awareness will involve a two-tailed hypothesis test with a Type I error rate of 0.05 and a Type II error rate of 0.20, yielding 80% statistical power. The sample will be a non-random convenience sample consisting of youth members of the Lightspeed online panel. Respondents will be aged 13 through 17.
Lightspeed will program the media tracking survey instruments, which, as noted above, will differ only when necessary, to measure the specific advertisements they are designed to measure. The sample will be purchased from Lightspeed. Lightspeed invites youth panel participants to complete the survey through an invitation to their parents, who provide consent for their child to participate. Respondent anonymity is guarded by assigning each a unique alphanumeric variable. Participants log onto Lightspeed’s secure server to complete the survey, using a link provided by Lightspeed and their unique identifier. No name or other identifying information is associated with survey data. Survey data are entered directly into and housed in Lightspeed’s server. Upon completing the survey, respondents will receive reimbursement from Lightspeed. Respondents are reimbursed by Lightspeed using “MarketPoints,” a non-monetary incentive that may be exchanged for goods with certain Lightspeed partner vendors, and has a value of approximately $10 per survey.
The primary purpose of the media tracking survey is to monitor youth awareness of and receptivity to the campaigns and specific campaign advertisements (see Attachments 4_E2 a1, 4_E2a2, 4_E2a3 and 4_E2b). For this reason, the survey instruments will measure the following: awareness of any tobacco-focused mass media campaign, including the FDA’s Tobacco Public Education campaign; awareness of specific campaign advertisements; reactions to the campaigns and to specific advertisements; tobacco-related beliefs, attitudes, intentions, and behavior; and demographic information. Items used to measure campaigns and ad awareness will necessarily change depending on what media is airing at the time of the survey. We expect to measure exposure to advertisements from other tobacco-related media campaigns and possibly other tobacco-related pharmaceutical company ads. Items used to measure tobacco-related beliefs, attitudes, intentions, and behaviors and demographic variables are subsets of those used in the outcome baseline survey (see Attachments 4_E2a1, 4_E2a2, 4_E2a3 and 4_E2b).
Data will be used to determine whether the campaigns are functioning as expected and may inform mid-campaign adjustments. For example, these data will provide information about whether the campaigns are reaching population subgroups of interest and may provide insights into which specific advertisements are most noticed and liked by youth. This information may allow media buyers to air more effective advertisements in heavier rotation, while phasing out less effective ads. Media tracking data are not nationally representative and will not be used to evaluate overall campaign effectiveness.
The ability to obtain the cooperation of potential respondents in the baseline survey and maintain their participation across all survey waves will be important to the success of this study. In preparation for launching the baseline data collection, we will review procedures for enlisting respondent cooperation across a wide range of surveys, incorporate best practices from those surveys into the data collection procedures, and adapt the procedures through continuous improvement across the survey waves. For the second cohort general market campaign evaluation, we will use two slightly different mail screeners. Screener B consists of 3 fewer items than the primary mail screener, Screener A. Half of the households sampled will receive Screener A and half will receive Screener B. We will analyze the data generated by the screeners to determine whether the briefer screener (Screener B) is as effective as our current primary screener (Screener A). If it is as effective, we will be able to use the briefer screener in the future. The tablet screener will include the measures from Screener A.
The incentive for completion of the youth baseline survey is $20. At follow-up youth respondents will be offered a $25 incentive to complete the survey online during an early release period that will run for approximately three weeks. Subsequently, youth respondents will be offered a $20 incentive to complete the survey either online or in person. Studies suggest that this incentive approach can increase response rates and reduce costs and nonresponse. In addition, the study will use procedures designed to maximize respondent participation. E-mails reminders will be sent to encourage sample members to complete the survey via the Web and remind them about the option of having an interviewer visit their homes to complete the survey. Data collection procedures will begin with assignment of SDUs to specific interviewers at the start of data collection. When assigning cases, supervisors will take into account which interviewers are in closest proximity to the work, interviewer skill sets, and basic information such as demographics and size of each sampled area. Supervisors will assign cases to interviewers in ways designed to maximize production.
When interviewers transmit their data from completed household screenings and interviews, the data will be summarized in daily reports posted to a Web-based case management system accessed by field supervisors and RTI’s data collection managers. On a daily basis, supervisors will use these reports to review response rates, production levels, and record of call information. This information will allow supervisors to determine each interviewer’s progress toward weekly production goals, when interviewers should attempt further contacts with SDUs, and how to handle challenging situations such as households that initially refuse to participate or households where the interviewer has been unable to contact anyone. Supervisors will discuss information and challenges with their interviewers each week. When feasible, cases will be transferred to other interviewers with different skill sets to assist with converting initial refusals into participating households. Cases might also be transferred among interviewers to improve production in areas where the original interviewer is not meeting response rate goals.
As noted in Section B.2, interviewers will use a Sorry I Missed You Card (Attachment 13_E2 and 14_R) and the Question and Answer Fact Sheet (Attachment 11_E2 and 11_R) when needed to contact respondents and encourage participation. To assist efforts to convert households that initially refuse to participate, refusal letters (Attachment 13_E2 and 14_R) tailored to specific refusal reasons will be used. Similarly, an Unable-to-Contact Letter (Attachment 13_E2 and 14_R) will be sent to an SDU if the interviewer has been unable to contact an adult resident after multiple attempts. When interviewers have been unable to gain access one or more SDUs due to an access barrier, such as a locked gate or doorperson, Controlled Access Letters (Attachment 13_E2 and 14_R) will be sent to the appropriate person or organization to obtain assistance in gaining access to these SDUs.
Prior to launching the baseline survey, we fielded an eight-case pretest of the survey instrument. This survey was identical to the instrument being used in this evaluation and approved by OMB with the exception of a few additional questions to assess overall clarity of instrument questions and respondents’ opinions on any aspects of the survey that were not clear. The purpose of the pretest was twofold: (1) to assess technical aspects and functionality of the survey instrument, and (2) to identify areas of the survey that were either unclear or difficult to understand. We reviewed diagnostic data on average time of survey completion, survey completion patterns (e.g., are there any concentrations of missing data?), and other aspects related to the proper function of the survey. We also examined data on pilot test measures that used to assess the clarity of item wording and ease of understanding.
In addition to the aforementioned 8-case pretest, the evaluation contractor RTI conducts rigorous internal testing of the online survey instrument prior to its fielding. Evaluators review the online test version of the instrument that we use to verify that instrument skip patterns are functioning properly, delivery of campaign media materials is working properly, and that all survey questions are worded correctly and are in accordance with the instrument approved by OMB.
The following individuals inside the agency have been consulted on the design and statistical aspects of this information collection as well as plans for data analysis:
April Brubach
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
9200 Corporate Boulevard
Rockville, MD 20850
Phone: 301-796-9214
E-mail: april.brubach@fda.hhs.gov
Gem Benoza
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
9200 Corporate Boulevard
Phone: 240-402-0088
E-mail: Maria.Benoza@fda.hhs.gov
David Portnoy
Office of Science
Center for Tobacco Products
Food and Drug Administration
9200 Corporate Boulevard
Phone: 301-796-9298
E-mail: David.Portnoy@fda.hhs.gov
Janine Delahanty
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-9705
E-mail: Janine.Delahanty@fda.hhs.gov
Matthew Walker
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-3824
E-mail: Matthew.Walker@fda.hhs.gov
Alexandria Smith
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-2192
E-mail: Alexandria.Smith@fda.hhs.gov
The following individuals outside of the agency have been consulted on the questionnaire development, statistical aspects of the design, and plans for data analysis:
Michelle O’Hegarty
Centers for Disease Control and Prevention
4770 Buford Highway NE, Mailstop F79
Atlanta, GA 30341
Phone: 770-488-5582
E-mail: mohegarty@cdc.gov
Pamela Rao
Akira Technologies, Inc.
1747 Pennsylvania Ave NW Suite 600
Washington, DC 20002
Phone: (202) 517-7187
Email: prao@akira-tech.com
Xiaoquan Zhao
Department of Communication
George Mason University
Robinson Hall A, Room 307B
4400 University Drive, 3D6
Fairfax, VA 22030
Phone: 703-993-4008
E-mail: xzhao3@gmu.edu
The following individuals will conduct data collection and analysis:
Matthew Farrelly
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-541-6852
E-mail: mcf@rti.org
Jennifer Duke
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-485-2269
E-mail: jduke@rti.org
Jane Allen
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-597-5115
E-mail: Janeallen@rti.org
Kevin Davis
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-541-5801
E-mail: kdavis@rti.org
James Nonnemaker
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-541-7064
E-mail: jnonnemaker@rti.org
Abreu, D. A., & Winters, F. (1999). Using monetary incentives to reduce attrition in the survey of income and program participation. Proceedings of the Survey Research Methods Section of the American Statistical Association.
Castiglioni, L., Pforr, K., & Krieger, U. (2008). The effect of incentives on response rates and panel attrition: Results of a controlled experiment. Survey Research Methods, 2(3), 151–158.
Centers for Disease Control and Prevention. (2012). Youth Risk Behavior Surveillance–United States, 2011. Morbidity and Mortality Weekly Report, 61(4), 1–162.
Davis, K. C., Nonnemaker, J., Duke, J., & Farrelly, M. C. (2013). Perceived effectiveness of cessation advertisements: The importance of audience reactions and practical implications for media campaign planning. Health Communication, 28(5), 461–472. doi:10.1080/10410236.2012.696535
Davis, K. C., Uhrig, J., Bann, C., Rupert, D., & Fraze, J. (2011). Exploring African American women’s perceptions of a social marketing campaign to promote HIV testing. Social Marketing Quarterly, 17(3), 39–60.
Dillard, J. P., Shen, L., & Vail, R. G. (2007). Do perceived message effectiveness cause persuasion or vice versa? Seventeen consistent answers. Human Communication Research, 33, 467–488.
Dillard, J. P., Weber, K. M., & Vail, R. G. (2007). The relationship between the perceived and actual effectiveness of persuasive messages: A meta-analysis with implications for formative campaign research. Journal of Communication, 57, 613–631.
Farrelly, M. C., Davis, K. C., Haviland, M. L., Messeri, P., & Healton, C. G. (2005). Evidence of a dose-response relationship between “truth” antismoking ads and youth smoking prevalence. American Journal of Public Health, 95(3), 425–431. doi: 10.2105/AJPH.2004.049692
Jäckle, A., & Lynn, P. (2008). Respondent incentives in a multi-mode panel survey: Cumulative effects on nonresponse and bias. Survey Methodology, 34(1), 105–117.
Janega, J. B., Murray, D. M., Varnell, S. P., Blitstein, J. L., Birnbaum, A. S., & Lytle, L. A. (2004). Assessing the most powerful analysis method for schools intervention studies with alcohol, tobacco, and other drug outcomes. Addictive Behaviors, 29(3), 595–606.
McMichael, J., & Chen, P. (2015). Using census public use microdata areas (PUMAs) as primary sampling units in area probability household surveys. In JSM Proceedings, Survey Research Methods Section, pp. 2281–2288. Alexandria: American Statistical Association.
Murray, D. M., & Blitstein, J. L. (2003). Methods to reduce the impact of intraclass correlation in group-randomized trials. Evaluation Review, 27(1), 79–103.
Murray, D. M., & Short, B. J. (1997). Intraclass correlation among measures related to tobacco-smoking by adolescents: Estimates, correlates, and applications in intervention studies. Addictive Behaviors, 22(1), 1–12.
Shettle, C., & Mooney, G. (1999). Monetary incentives in U.S. government surveys. Journal of Official Statistics, 15, 231–250.
Singer, E. (2002). The use of incentives to reduce nonresponse in household surveys. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey Nonresponse (p. 163–177). New York, NY: Wiley.
Snyder, L. B., Hamilton, M. A., Mitchell, E. W., Kiwanuka-Tondo, J., Fleming-Milici, F., & Proctor, D. (2004). A meta-analysis of the effect of mediated health communication campaigns on behavior change in the United States. Journal of Health Communications, 9, 71–96.
Substance Abuse and Mental Health Services Administration (SAMHSA). (2012). Results from the 2011 National Survey on Drug Use and Health: Summary of national findings. NSDUH Series H-44, HHS Publication No. (SMA) 12-4713. Rockville, MD: Substance Abuse and Mental Health Services Administration.
U.S. Department of Health and Human Services (USDHHS). (2006). The health consequences of involuntary exposure to tobacco smoke: A report of the Surgeon General. Atlanta, GA: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, Coordinating Center for Health Promotion, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health.
Wakefield, M. A., Spittal, M. J., Yong, H-H., Durkin, S. J., & Borland, R. (2011). Effects of mass media campaign exposure intensity and durability on quit attempts in a population-based cohort study. Health Education Research, 26(6), 988–997.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | PR* |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |