Attach 4E FLASHE Cognitive Testing Plan

Attachment 4E: Cognitive Testing Plan for the Family Life, Activity, Sun, Health, and Eating (FLASHE) Survey

Background and Task Purpose

The Family Life, Activity, Sun, Health, and Eating (FLASHE) Survey is a new National Cancer Institute survey that seeks to examine psychosocial, generational (parent-child), and environmental correlates of cancer preventive behaviors. The survey’s goal is to advance understanding of the dynamic relationship among those factors by providing a public use dataset as a resource to the research community.

For this task, Westat will conduct cognitive testing in support of development of the FLASHE survey. Cognitive testing will consist of individual one-hour interviews with 20 parent-teenager (ages 11-18) pairs, for a total of 40 interviews (parents will be interviewed separately from their children). Westat will perform the main activities associated with cognitive testing, including recruiting respondents, developing an interview guide, conducting the interviews, analyzing the data and providing recommendations for revising the survey instruments. A total of eight instruments will be included in the testing.

• Parent Demographics Survey

• Parent Diet Survey

• Parent Physical Activity Survey

• Parent Physical Activity Checklist

• Teenager Demographics Survey

• Teenager Diet Survey

• Teenager Physical Activity Survey

• Teenager Activity Recall

The remainder of this document provides detailed information about the FLASHE survey cognitive testing plan.

Section 1. Rationale for Cognitive Testing Approach

Cognitive testing of surveys involves using an in-depth, semi-structured interview to gather insights into the cognitive sources of potential misinterpretation (Willis, 2005; Groves et al. 2004; Sirkin et al., 1999). This qualitative methodology focuses on examining the question-answer process using a cognitive psychological perspective. According to Tourangeau et al. (2000), survey response unfolds through four stages – comprehension; retrieval, judgment, and responding. Using this information about respondents’ thought processes, survey researchers can identify and refine question wordings that are either misunderstood or understood differently by different respondents; instructions that are insufficient, overlooked, misinterpreted, or difficult to understand; vague definitions or ambiguous instructions that may be interpreted differently; items that ask for information that the respondent does not have access to; and confusing response options or response formats.

There are multiple approaches to uncovering this information in the cognitive interview, including “think aloud”, concurrent, and retrospective probing, and often multiple methods are applied within the same interview (Beatty and Willis, 2007). Cognitive interviewers must also be able to probe any unexpected or otherwise provocative utterances by the respondent that suggest difficulty understanding or answering questions. By applying these methods to the draft FLASHE survey, we will be able to determine how well potential participants understand certain words, phrases, and concepts used in the survey items as well as assess any difficulties they have answering the questions.

Section 2. Recruiting Respondents

Westat will recruit respondents using a variety of methods including use of Westat’s current database of potential participants and advertising on CraigsList under the Jobs postings within the subheading Et Cetera. Westat's digital media team will reach out to 100 local DC area (including lower Pennsylvania) publishers such as mommy and papi bloggers, health websites, giveaway publishers, and Facebook and Twitter users. The team will also look at local MeetUps. The goal will be to encourage these publishers to post about the FLASHE program, the request for parent/caregiver and child participants, and urge their readers to visit the website to sign up. Westat, along with NCI and the FAAST team, will develop a list of the 100 publishers for outreach as well as an internal messaging document to guide the team's communications with publishers about the outreach campaign. Recruitment outreach will be conducted using a phased approach by location starting with publishers closest to the DC area and reaching out to publishers further away to secure more participants if necessary. Publishers will be offered a $100 incentive to post and the messaging will also highlight a $75 incentive for parent participants and a $50 incentive for teenager participants. As recruitment outreach gets underway, and we have data on which publishers are participating, we can iteratively adjust our outreach strategy to incorporate more of the interested publishers. If needed, additional recruitment outreach measures will include advertising in free local publications in the Rockville and Fredrick, MD area, as well as implementing paid radio and mainstream newspaper ads.

Section 3. Screening and Scheduling Respondents

Interested parents and teenagers will be instructed to call Westat and speak to staff that will be trained to administer the recruiting screener. Potential interviewees will also have the option to leave a message providing their name and contact information. Westat will screen potential respondents with a recruiting questionnaire and the screening criteria will capture a mix of ages, education levels, race/ethnic backgrounds, and gender. The interviews will be conducted in two rounds with each round consisting of 10 parents and 10 teenagers. Each round will be conducted over approximately a 2 week period in order to allow for flexibility in scheduling with participants. We will also schedule 2 additional pairs of interviews per round to serve as alternatives in the event that previously scheduled respondents cannot participate as planned. All interviews will be conducted in-person at Westat’s cognitive testing facility in Rockville, MD or at Westat’s Telephone Research Center located in Fredrick, MD. Interviewees will be contacted by phone or email 1 to 2 days prior to the scheduled interview day to remind them of their scheduled interview.

Section 4. Developing the Interview Guide

The interview guide explores comprehension of survey items, as well as the respondents’ overall navigation and interaction with the survey instrument. The cognitive interview probes will be administered concurrently and retrospectively, whereby the interviewer will ask them immediately after each item, or section of questions, that has been selected for testing. The cognitive interview guide includes a series of pre-determined general probes, such as “What, in your own words, is this question asking?” and “What, if anything, did you find confusing or unclear about this question?” as well as more tailored probes that focus on assessing comprehension of specific words, phrases, or concepts of concern.

Section 5. Cognitive Interviewer Training

The interviewing staff will consist of senior survey methodologists from Westat with extensive experience using cognitive interview methods to test survey questionnaires and materials. The interviewers will also receive a 4-hour training specific to the FLASHE survey and the cognitive interview guide that has been designed specifically for this project. The lead researchers will design the training to ensure that all of the interviewers are thoroughly aware of the objectives underlying the questions and topics covered in the interview guide.

Section 6. Conducting the Cognitive Interview

Of the 10 interview pairs in each round, half will receive the physical activity items and half will receive the diet items. The structure of the interview will involve administering the study introduction, with an explanation of the study purpose and the respondent’s rights as a research subject. Immediately prior to the start of each interview, participants will be asked to read and sign a consent form to document their awareness of the voluntary nature of their participation, agreement with the discussion being recorded, and so forth. The survey will be handed to the respondent at the time of the interview (it will not be sent to them ahead of the interview) and will be self administered by the interviewees. For each item that is targeted for testing, we will administer verbal probes that focus on one or more of respondents’ comprehension, confidence in their answers, or ability to recall information. As needed, we will also tailor probes to those issues that are specific to a particular item or the particular questionnaire. Along with the survey items, we may also test usability issues such as layout, appearance and any other potential concerns related to user interactions with the survey instrument. As needed, interviewers will also spontaneous probes that explore unanticipated issues respondents raise on their own during the interview. Note that interviewers will not probe on, nor will respondents be asked to answer the questions in the sections on tobacco use or sun safety because these are well estbliashed items that have already been tested extensively elsewhere.. At the conclusion of the interview, respondents will be provided an opportunity to offer any other feedback or reactions they did not have a chance to share previously. Each interview will last 90 minutes. For their participation, parents will be provided with $75 and teenagers $50.

Section 7. Cognitive Interview Analysis

Westat will analyze the interview data in a series of steps. At the first step, each interviewer will review and synthesize the data (from the audio-recordings and from any notes they may have taken) from their own interviews. During this step, the interviewer will identify themes within each respondent’s interview and across all those interviews. The themes will be organized by overall survey issues, individual survey items and sections, and respondents’ overall reactions to the survey. In the second step, all interviewers will meet and discuss the themes they have identified. As a group and with the guidance of the lead analyst, the team will identify common themes (as well as unique situations worthy of notice) within and across all interviews. The team will also began to identify recommendations for revising the survey based on the cognitive interview results. In the final step, the lead analyst will conduct another review of all interview data and organize the themes into the report of findings and recommendations. This final review will ensure that the recommendations are thoroughly grounded in the cognitive interview data.

At a minimum, the final report will consist of a brief description of methods, including a table of respondent characteristics and any noteworthy issues related to protocol development; a summary of key findings and recommendations; and, if appropriate, a more detailed presentation of item-by-item results and recommendations. After we deliver the report to the FLASHE team, we will mutually decide which recommendations to implement so as to finalize the survey.

References

Beatty, P.C. and Willis, G.B. (2007.) The practice of cognitive interviewing. Public Opinion Quarterly 71(2):287-311

Groves, R.M., Fowler, F.J., Couper, M.P., Lepkowski, J.M., Singer, E., & Tourangeau, R. (2004). Survey Methodology. New York: John Wiley & Sons.

Sirken, M.G., Herrmann, D.J., Schechter, S., Schwarz, N. Tanur, J., & Tourangeau, R. (1999). Cognition and Survey Research. New York: John Wiley & Sons.

Tourangeau, R., Rips, L.J. and Rasinski, K. (2000). The Psychology of Survey Response. New York, NY: Cambridge University Press.

Willis, G.B. (2005), Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage.

1