for Request for Revision to a Currently Approved Collection
Questionnaire Testing and Methodological Research for the Medicare Current Beneficiary Survey (MCBS)
OMB Control No: 0938-1275
Contact Information:
William S. Long
Contracting Officer’s Representative, Medicare Current Beneficiary Survey
Office of Enterprise Data and Analytics/CMS
7500 Security Boulevard, Mail Stop Mailstop B2-04-12
Baltimore, MD 21244
(410) 786-7927
william.long@cms.hhs.gov
(410) 786-5515 (fax)
December 18, 2020
B1. Universe and Respondent Selection 3
B2. Procedures for Collecting Information 3
B3. Methods to Maximize Response 7
B5. Individuals Consulted on Statistical Aspects and Individuals Collecting/Analyzing Data 8
The data collected under this clearance request will be used for survey research and testing activities. Consequently, many of the small-scale methods proposed, especially cognitive testing, usability testing, or conducting focus groups, will involve respondents selected either to cover a broad range of demographic subgroups of interest to the MCBS, or for specific characteristics related to the specific content being tested. For other methods, including field tests within the MCBS production environment or outside the MCBS production environment, statistically representative subsamples of existing MCBS panel respondents or prior MCBS respondents who have ended their participation in the survey (e.g., completed their 11th interview) will be selected for inclusion. A detailed description of the plans for selecting respondents for each individual test will be provided to OMB at the time the testing plans are submitted.
While survey research employs a quantitative methodology and relies on a relatively large population-based probability sample to support statistical inference and representativeness, cognitive testing, usability testing, focus groups and debriefing interviews usually employ a qualitative methodology and generally rely on relatively small samples. Unlike survey research, the primary objective of such testing is not to produce statistical data that can be generalized to an entire population. Rather, the objective is to provide an in-depth exploration of particular concepts, processes and/or patterns of interpretation. Cognitive interviewing samples generally do not achieve full inclusivity of all social and demographic groups. However, as a general rule, respondents are selected for inclusion based on similarity of characteristics to MCBS respondents; depending on the purpose of the test, additional screening information is used to ensure that the respondent has characteristics related to the content tested. For example, in a test of self-management of chronic conditions approved by OMB on 9/27/18 (CMS-10549-GenIC#3), respondents who had at least one specific chronic condition were recruited to participate in the testing of questions.
For field tests within the production environment, the purpose, scope and size of the testing require the use of subsamples of MCBS respondents currently enrolled in the survey; the advanced mail test approved by OMB on 12/3/18 (CMS-10549-GenIC#4) is an example of a test conducted within production during Fall 2019 Round 85. Field tests conducted outside the MCBS production environment will use small subsamples of MCBS respondents and can include those from retired panels who are no longer actively enrolled in the survey; the test of the COVID-19 Rapid Response Supplement approved by OMB on 5/7/2020 (CMS-10549-GenIC#7) is an example of a test conducted outside the production environment that used both current MCBS respondents as well as those who had recently completed their 11th interview.
Data collection anticipated under this clearance request will be conducted through in-person interviews or telephone interviews, whether through field tests, cognitive interviews, focus groups, or usability testing. CMS will submit individual collections under this generic clearance to OMB, and will provide OMB with a copy of recruitment materials, questionnaires, consent forms, protocols, and debriefing materials in advance of any testing activity. The general recruitment, screening and interviewing procedures are described below.
Respondents for cognitive testing, focus groups, and usability testing will usually be recruited by means of flyers and other advertisements posted in public places, newspaper advertisements, or word-of-mouth. Prior experience at federal and private cognitive laboratories has shown that advertisements in local newspapers and flyers attract a large pool of potential respondents. These recruitment mechanisms have been productive in the past for obtaining a diverse group of respondents to help us determine potential sources of error in survey questions. To test questions that are targeted toward specific subgroups (i.e. by age, race/ethnicity, or other factors), the advertisement or flyer may be developed to identify appropriate respondents. Direct contact to solicit support from senior centers, employers, and/or social or service organizations is occasionally used as a way to recruit potential participants. In these cases, a flyer is provided to a contact person who then either posts the flyer or distributes it to members of the organization.
For field tests and methodological experiments within the production environment, cases to be included will be pre-identified in the MCBS sample, based on the desired sampling mechanism for each test. MCBS interviewers will use the same procedures as used in the actual MCBS interview to explain the survey and acquire informed consent.
For field tests conducted outside the production environment, respondents to be recruited will be pre-identified either in the current MCBS sample or from recently retired survey panels, including respondents who have completed their final, 11th MCBS interview within the past year. Occasionally, new respondents will be recruited only for the purpose of the test. All recruited respondents will usually receive an advance letter mailing describing the purpose of the questionnaire testing to be conducted under the survey system, the expected length of the interview, and data collection procedures. Interviewers will obtain verbal consent to conduct the interviews before administering the survey items. Specific notification and recruitment procedures will be described in the proposed information clearance requests for OMB approval.
The first contact with potential laboratory research respondents occurs in response to flyers or advertisements. Interested persons leave contact information (name and telephone number) on voicemail. The MCBS staff person then calls the person back, gives a brief description of the nature of the study, i.e., one-on-one interview or focus group, where the interview/focus group takes place, video/audio taping procedures, and the remuneration to be offered. The MCBS staff person also determines through a brief series of questions whether the volunteer possesses the desired research characteristics (e.g., age, sex, and other demographic characteristics). If the person does meet the selection criteria and agrees to participate in the cognitive interview or focus group, he/she is scheduled for an interview or focus group session. Telephone numbers and minimal demographic information (e.g. name, age, sex, etc.) are obtained for all scheduled participants. For those callers who are ineligible for the study because they are not members of a subgroup of interest, no identifying information will be retained. Recruitment data will be maintained in separate, encrypted files with password protection. Any hardcopy materials will be stored in locked files and access limited to authorized study personnel.
For field tests and methodological experiments within the production environment, cases to be included will be pre-identified in the MCBS sample based on the desired sampling allocation for each test, so no separate respondent screening should be necessary. MCBS interviewers will use the same interview scheduling procedures as are currently in place for the main MCBS data collection.
For field tests conducted outside the production environment, the pre-identified respondents will be contacted by an MCBS interviewer and asked whether they agree to participate. For respondents who agree to participate, interviewers will then administer the survey or schedule an interview time that is more convenient for the respondent. Because MCBS respondents participate in the survey over a four-year period, there is a chance that some respondents might participate in more than one field test, depending the number of tests per year and the participation of the respondent in a given round.
Cognitive Testing Interviews. Participants scheduled for in-person cognitive testing interviews will usually travel to the contractor facilities. Occasionally, interviewers may be conducted in homes or at a location normally frequented by the participant, such as a senior center. To reduce the number of "no shows" for cognitive interviews, participants scheduled more than a week in advance receive a reminder telephone call by the MCBS staff the day prior to the scheduled interview.
For cognitive testing in person, when the respondent arrives for their scheduled interview, he/she is greeted by the MCBS staff person. The participant will then be brought to the interview room and asked to read (or have read aloud to) a consent form. If the interview is conducted by phone, the interviewer reads the consent form aloud. The form contains a brief description of the study, assurances of confidentiality for the participant, explains the voluntary nature of the study, and describes the risks and benefits. The need for recording the interview (audio or video) is explained and the respondent is asked to sign the consent form or indicate consent on the phone). In the rare instance that the participant consents to the cognitive interview, but not to recording it, the session will be carried out but not recorded. If the respondent grants consent to record the interview but changes his/her mind while the session is being recorded, the interviewer will ask for verbal consent to retain the interviewing materials and the portion already taped.
The interviewer, usually a survey methodologist or other project staff, will begin the cognitive interview by reading a more detailed explanation of the purpose of the interview and the procedures to be used. Interviewing procedures vary depending on the specific testing technique to be applied. The selection of the technique is determined by the nature of the project, or the stage of development of the questionnaire or set of questions under study. The most commonly used method is the cognitive interview with concurrent probing. In these interviews, respondents are presented draft survey questions and asked to “think aloud” about how and why they answered as they did. The interviewer usually probes extensively to ascertain the degree of comprehension and the recall processes involved.
If possible, the cognitive interview will be conducted in the mode intended for the instrument, either face-to-face, telephone, or self-administered using a paper or web-based instrument. For a telephone interview, the respondent is called from one room to another and in-person debriefing follows.
As described in Supporting Statement A. section A.9, participants will receive a small incentive. They will be asked to sign a receipt form indicating receipt of the remuneration. Immediately following the interview, any hardcopy interviewing data (e.g. questionnaires) will be separated from the consent form and signed receipt form, so that no demographic information will be associated with the individual’s name. Signed consent forms will be stored separately from any data collection from the participant. Audio and video recording files will be stored on NORC’s secure servers and the files deleted from the recording device.
Focus Groups. Participants scheduled for focus groups will usually travel to the contractor facilities. To reduce the number of "no shows" for focus groups sessions, participants scheduled more than a week in advance receive a reminder telephone call by the MCBS staff the day prior to the scheduled session. When participants arrive they are greeted by MCBS project staff and directed to the focus group room. Participants are given a consent form to read (or to have read to them by project staff). The consent form contains a brief description of the study, assurances of confidentiality for the participant, explains the voluntary nature of the study, and describes the risks and benefits. The need for recording the focus group sessions (audio and/or video) is explained and the participant is asked to sign the consent form.
MCBS staff person(s) will moderate the focus group. Before discussion begins, the moderator will distribute name tags and will tell respondents to pick a name to put on the name tag. Respondents will be told that they do not have to use their real names. The moderator will then describe the process of the focus group and ask if there are any questions. After all questions are answered, the moderator will then begin the focus group discussion following the moderator guide designed for that particular study.
Once the focus group has concluded, participants will receive a small incentive. They will be asked to sign a receipt form indicating receipt of the remuneration. Immediately following the interview, any hardcopy materials completed by the participant will be separated from the consent form and signed receipt form, so that no demographic information will be associated with the individual’s name. Signed consent forms will be stored separately from any data collection from the participant. Audio and video recording files will be stored on NORC’s secure servers and the files deleted from the recording device.
Usability Testing. Participants scheduled for usability testing will usually travel to the contractor’s facilities. When the participant arrives for their scheduled interview, he/she is greeted by the MCBS staff person. The participant will then be brought to the testing room and asked to read (or have read aloud to) a consent form. The form contains a brief description of the study, assurances of confidentiality for the participant, explains the voluntary nature of the study, and describes the risks and benefits. The need for recording the interview (audio or video) is explained and the respondent is asked to sign the consent form. In the rare instance that the participant consents to the cognitive interview, but not to recording it, the session will be carried out but not recorded. If the respondent grants consent to record the interview but changes his/her mind while the session is being recorded, the interviewer will ask for verbal consent to retain the interviewing materials and the portion already taped.
The MCBS project staff person will begin the testing session by reading a more detailed explanation of the purpose of the interview and the procedures to be used. Interviewing procedures vary depending on the specific testing technique to be applied. Usability could be evaluated with cognitive testing (think aloud, debriefing questions), behavior coding (e.g., coding errors made by the interviewer or respondent while trying to navigate the instrument), and analysis of interview completion time.
As described in Supporting Statement A. section A.9, participants will receive a small incentive. They will be asked to sign a receipt form indicating receipt of the remuneration. Immediately following the interview, any hardcopy interviewing data (e.g. questionnaires) will be separated from the consent form and signed receipt form, so that no demographic information will be associated with the individual’s name. Signed consent forms will be stored separately from any data collection from the participant. Audio and video recording files will be stored on NORC’s secure servers and the files deleted from the recording device.
Field Tests within the MCBS production environment. Field tests within production will be conducted with subsamples of the active MCBS cohorts, as part of regular survey data collection cycle. The test data collection protocols or test questions will be incorporated into the questionnaires and administration will follow MCBS protocols.
Field testing will be conducted by trained MCBS field interviewers. Cases to be included will be pre-identified in the MCBS sample, based on the desired sampling allocation. For all interviews administered in this manner, the field interviewer will follow approved MCBS survey administration procedures. As time and resources allow, a subset of the interviews may be observed by CMS staff or MCBS project staff and observations manually recorded to allow for systematic analysis. In addition, CMS or project staff may conduct analysis of outcome data such as response rates and response distributions to key items, paradata, interviewer observations, and respondent debriefing data. Subject matter staff are debriefed, and findings are used to modify the questionnaire for follow-up field tests or incorporation into the main MCBS questionnaires.
Field testing outside of MCBS production environment. Field tests conducted outside of the MCBS production environment will be conducted using MCBS standard field or telephone interviewing procedures. Each short survey, containing the test data collection protocol or test questionnaire items as well as other MCBS survey items needed for context, will be programmed for CAPI and phone administration. As time and resources allow, a subset of the interviews may be observed by CMS staff or MCBS project staff and observations manually recorded to allow for systematic analysis. In addition, CMS or project staff may conduct analysis of outcome data such as interviewer observations and respondent debriefing data. Subject matter staff are debriefed, and findings are used to modify the questionnaire for follow-up field tests or incorporation into the main MCBS questionnaires.
For the purpose of conducting cognitive interviews and focus groups, recruitment will be conducted through advertisements in local newspapers or flyers in senior centers or other neighborhood community centers. These recruitment mechanisms have been productive in the past for obtaining a diverse group of respondents to help us determine potential sources of error in survey questions. For those questionnaires that target specific subgroups, special recruitment procedures will be developed to identify respondents. After volunteers have been recruited, the probability of the respondent failing to show is minimized by making reminder phone calls to volunteers. For cognitive interviews and focus groups, participants will receive an incentive for their participation. Reminder phone calls and/or letters to participants will be used to encourage them to keep their appointments. Tallies will be kept of the number of non-respondents to all testing activities.
For field surveys within production or outside of production, standard MCBS methods for maximizing response rates with existing MCBS participants will be used. In addition to the use of advance mail materials, MCBS field interviewers are highly trained in outreach and gaining cooperation. Appointments are scheduled at the convenience of respondents, with regular follow up used to ensure that appointments are kept.
More specific information will be contained in the description provided to OMB at the time the individual testing plans are submitted.
This submission is a request for authorization to conduct tests of questionnaires, instrumentation and methodologies typical in survey research programs and to build on this research through field tests and methodological experiments. We expect that the testing and experiments conducted under this clearance will result in measures that close data gaps or address emerging health issues. Research under this generic clearance also seeks to improve data quality and lower respondent burden by ensuring that new data collection protocols and questions are thoroughly tested prior to being incorporated into future collections.
Edward Mulrow, Ph.D. Vice President
NORC at the University of Chicago
4350 East-West Highway, 8th Floor
Bethesda, MD 20814
(301) 634-9441
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | CMS-10549_0938_1275_SSB |
| Author | CMS |
| File Modified | 0000-00-00 |
| File Created | 2021-01-11 |