Supporting Statement B
for Request for Extension without Change of a Currently Approved Collection
Generic Clearance:
Questionnaire Testing and Methodological Research for the Medicare Current Beneficiary Survey (MCBS)
OMB Control No: 0938-1275
Contact Information:
William S. Long
Contracting Officer’s Representative, Medicare Current Beneficiary Survey
Office of Enterprise Data and Analytics/CMS
7500 Security Boulevard, Mail Stop B2-29-04
Baltimore, MD 21244
(410) 786-7927
william.long@cms.hhs.gov
(410) 786-5515 (fax)
October 17, 2017
B1. Universe and Respondent Selection 3
B2. Procedures for Collecting Information 3
B3. Methods to Maximize Response 6
B5. Individuals Consulted on Statistical Aspects and Individuals Collecting/Analyzing Data 7
The data collected under this clearance request will be used for survey research and testing activities rather than to produce estimates about populations. Consequently, many of the small-scale methods proposed (cognitive testing, focus groups, usability testing) will involve respondents selected either to cover a broad range of demographic subgroups of interest to the MCBS, or for specific characteristics related to the specific content being tested. For other methods, including field tests, split ballot experiments or other methodological experiments, statistically representative subsamples of existing MCBS panel respondents will be selected for inclusion. A description of the plans for selecting respondents for each individual test will be provided to OMB at the time the testing plans are submitted.
While survey research employs a quantitative methodology and relies on a relatively large population-based probability sample to support statistical inference and representativeness, cognitive testing, usability testing and focus groups conducted in a laboratory usually employ a qualitative methodology and generally rely on relatively small samples. Unlike survey research, the primary objective of such testing is not to produce statistical data that can be generalized to an entire population. Rather, the objective is to provide an in-depth exploration of particular concepts, processes and/or patterns of interpretation. Cognitive interviewing samples generally do not achieve full inclusivity of all social and demographic groups. However, as a general rule, respondents are selected for inclusion based on similarity of characteristics to MCBS respondents.
For field tests, split ballot experiments and other methodological experiments, the purpose, scope and size of the testing require the use of subsamples of MCBS respondents currently enrolled in the survey.
Most of the data collection anticipated under this clearance request will be conducted through in-person interviews, whether through field tests, cognitive interviews, focus groups, or usability testing. CMS will submit individual collections under this generic clearance to OMB, and will provide OMB with a copy of recruitment materials, questionnaires, consent forms, protocols, and debriefing materials in advance of any testing activity. The general recruitment, screening and interviewing procedures are described below.
Recruitment
Laboratory respondents for cognitive testing, focus groups and usability testing will usually be recruited by means of flyers and other advertisements posted in public places, newspaper advertisements, or word-of-mouth. Prior experience at federal and private cognitive laboratories has shown that advertisements in local newspapers and flyers attract a large pool of potential respondents. These recruitment mechanisms have been productive in the past for obtaining a diverse group of respondents to help us determine potential sources of error in survey questions. To test questions that are targeted toward specific subgroups (i.e. by age, race/ethnicity, or other factors), the advertisement or flyer may be developed to identify appropriate respondents. Direct contact to solicit support from senior centers, employers, and/or social or service organizations is occasionally used as a way to recruit potential participants. In these cases, a flyer is provided to a contact person who then either posts the flyer or distributes it to members of the organization.
For field tests and methodological experiments, cases to be included will be pre-identified in the MCBS sample, based on the desired sampling mechanism for each test. MCBS interviewers will use the same procedures as used in the actual MCBS interview to explain the survey and acquire informed consent.
Screening and Scheduling Procedures
The first contact with potential laboratory research respondents occurs in response to flyers or advertisements. Interested persons leave contact information (name and telephone number) on voicemail. The MCBS staff person then calls the person back, gives a brief description of the nature of the study, i.e., one-on-one interview or focus group, where the interview/focus group takes place, video/audio taping procedures, and the remuneration to be offered. The MCBS staff person also determines through a brief series of questions whether the volunteer possesses the desired research characteristics (e.g., age, sex, and other demographic characteristics). If the person does meet the selection criteria and agrees to participate in the cognitive interview or focus group, he/she is scheduled for an interview or focus group session. Telephone numbers and minimal demographic information (e.g. name, age, sex, etc.) are obtained for all scheduled participants. For those callers who are ineligible for the study because they are not members of a subgroup of interest, no identifying information will be retained. Recruitment data will be maintained in separate, encrypted files with password protection. Any hardcopy materials will be stored in locked files and access limited to authorized study personnel.
For field tests and methodological experiments, cases to be included will be pre-identified in the MCBS sample based on the desired sampling allocation for each test, so no separate respondent screening should be necessary. MCBS interviewers will use the same interview scheduling procedures as are currently in place for the main MCBS data collection.
Interviewing Procedures
Cognitive Testing Interviews. Participants scheduled for in-person cognitive testing interviews will usually travel to the contractor facilities. On rare occasions, a participant may be unable to travel to the intended location (e.g. an individual may be housebound or have limited mobility). In such cases, the interview may be conducted in his/her home or at a location normally frequented by the participant, such as a senior center. To reduce the number of "no shows" for cognitive interviews, participants scheduled more than a week in advance receive a reminder telephone call by the MCBS staff the day prior to the scheduled interview.
When the respondent arrives for their scheduled interview, he/she is greeted by the MCBS staff person. The participant will then be brought to the interview room and asked to read (or have read aloud to) a consent form. The form contains a brief description of the study, assurances of confidentiality for the participant, explains the voluntary nature of the study, and describes the risks and benefits. The need for recording the interview (audio or video) is explained and the respondent is asked to sign the consent form. In the rare instance that the participant consents to the cognitive interview, but not to recording it, the session will be carried out but not recorded. If the respondent grants consent to record the interview but changes his/her mind while the session is being recorded, the interviewer will ask for verbal consent to retain the interviewing materials and the portion already taped.
The interviewer, usually a survey methodologist or other project staff, will begin the cognitive interview by reading a more detailed explanation of the purpose of the interview and the procedures to be used. Interviewing procedures vary depending on the specific testing technique to be applied. The selection of the technique is determined by the nature of the project, or the stage of development of the questionnaire or set of questions under study. The most commonly used method is the cognitive interview with concurrent probing. In these interviews, respondents are presented draft survey questions and asked to “think aloud” about how and why they answered as they did. The interviewer usually probes extensively to ascertain the degree of comprehension and the recall processes involved.
If possible, the cognitive interview will be conducted in the mode intended for the instrument, either face-to-face, telephone, or self-administered using a paper or web-based instrument. For a telephone interview, the respondent is called from one room to another and in-person debriefing follows.
As described in Supporting Statement A. section A.9, participants will receive a small incentive. They will be asked to sign a receipt form indicating receipt of the remuneration. Immediately following the interview, any hardcopy interviewing data (e.g. questionnaires) will be separated from the consent form and signed receipt form, so that no demographic information will be associated with the individual’s name. Signed consent forms will be stored separately from any data collection from the participant. Audio and video recording files will be stored on NORC’s secure servers and the files deleted from the recording device.
Focus Groups. Participants scheduled for focus groups will usually travel to the contractor facilities. To reduce the number of "no shows" for focus groups sessions, participants scheduled more than a week in advance receive a reminder telephone call by the MCBS staff the day prior to the scheduled session. When participants arrive they are greeted by MCBS project staff and directed to the focus group room. Participants are given a consent form to read (or to have read to them by project staff). The consent form contains a brief description of the study, assurances of confidentiality for the participant, explains the voluntary nature of the study, and describes the risks and benefits. The need for recording the focus group sessions (audio and/or video) is explained and the participant is asked to sign the consent form.
MCBS staff person(s) will moderate the focus group. Before discussion begins, the moderator will distribute name tags and will tell respondents to pick a name to put on the name tag. Respondents will be told that they do not have to use their real names. The moderator will then describe the process of the focus group and ask if there are any questions. After all questions are answered, the moderator will then begin the focus group discussion following the moderator guide designed for that particular study.
Once the focus group has concluded, participants will receive a small incentive. They will be asked to sign a receipt form indicating receipt of the remuneration. Immediately following the interview, any hardcopy materials completed by the participant will be separated from the consent form and signed receipt form, so that no demographic information will be associated with the individual’s name. Signed consent forms will be stored separately from any data collection from the participant. Audio and video recording files will be stored on NORC’s secure servers and the files deleted from the recording device.
Usability Testing. Participants scheduled for usability testing will usually travel to the contractor’s facilities. When the participant arrives for their scheduled interview, he/she is greeted by the MCBS staff person. The participant will then be brought to the testing room and asked to read (or have read aloud to) a consent form. The form contains a brief description of the study, assurances of confidentiality for the participant, explains the voluntary nature of the study, and describes the risks and benefits. The need for recording the interview (audio or video) is explained and the respondent is asked to sign the consent form. In the rare instance that the participant consents to the cognitive interview, but not to recording it, the session will be carried out but not recorded. If the respondent grants consent to record the interview but changes his/her mind while the session is being recorded, the interviewer will ask for verbal consent to retain the interviewing materials and the portion already taped.
The MCBS project staff person will begin the testing session by reading a more detailed explanation of the purpose of the interview and the procedures to be used. Interviewing procedures vary depending on the specific testing technique to be applied. Usability could be evaluated with cognitive testing (think aloud, debriefing questions), behavior coding (e.g., coding errors made by the interviewer or respondent while trying to navigate the instrument), and analysis of interview completion time.
As described in Supporting Statement A. section A.9, participants will receive a small incentive. They will be asked to sign a receipt form indicating receipt of the remuneration. Immediately following the interview, any hardcopy interviewing data (e.g. questionnaires) will be separated from the consent form and signed receipt form, so that no demographic information will be associated with the individual’s name. Signed consent forms will be stored separately from any data collection from the participant. Audio and video recording files will be stored on NORC’s secure servers and the files deleted from the recording device.
Field Tests. Field tests will be conducted with subsamples of the main MCBS cohorts, as part of regular survey data collection cycle. The test questions will be incorporated into the questionnaires and administration will follow MCBS protocols.
Field testing will be conducted by MCBS field interviewers. Cases to be included will be pre-identified in the MCBS sample, based on the desired sampling allocation. For all interviews administered in this manner, the field interviewer will follow approved MCBS survey administration procedures. As time and resources allow, a subset of the interviews may be observed by CMS staff or MCBS project staff and observations manually recorded to allow for systematic analysis. In addition, CMS or project staff may conduct analysis of outcome data such as response rates and response distributions to key items, paradata, interviewer observations, and respondent debriefing data. Subject matter staff are debriefed, and findings are used to modify the questionnaire for follow-up field tests or incorporation into the main MCBS questionnaires.
For the purpose of conducting cognitive interviews and focus groups, CMS will rely on our contractor to advertise in local newspapers and flyers as well as reaching out to former survey participants that have been retired from the MCBS, to attract a large enough pool of potential laboratory research respondents. These recruitment mechanisms have been productive in the past for obtaining a diverse group of respondents to help us determine potential sources of error in survey questions. For those questionnaires that target specific subgroups, special recruitment procedures will be developed to identify respondents. After laboratory volunteers have been recruited, the probability of the respondent failing to show is minimized by making reminder phone calls to volunteers.
As separate evaluation is proposed to examine the current methods to maximize response rates within the full MCBS collection. Once identified, proposed strategies will be cleared by OMB before implementation. In general, callbacks will be used to maximize response rates in field tests and methodological experiments. For cognitive interviews and focus groups, participants will receive an incentive for their participation. Reminder phone calls and/or letters to participants will be used to encourage them to keep their appointments. Tallies will be kept of the number of non-respondents to all testing activities. More specific information will be contained in the description provided to OMB at the time the individual testing plans are submitted.
This submission is a request for authorization to conduct tests of questionnaires, instrumentation and methodologies typical in survey research programs and to build on this research through field tests and methodological experiments. We expect that the testing and experiments conducted under this clearance will result in improved data quality and lower respondent burden.
Kirk Wolter, Ph.D.
Executive Vice President
NORC at the University of Chicago
55 East Monroe Street
30th Floor
Chicago, Illinois
(312) 759-4206
wolter-kirk@norc.org
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement for Request for Clearance: |
| Author | Karen Whitaker |
| File Modified | 0000-00-00 |
| File Created | 2021-01-12 |