LTCSB 10-day Letter

DEPARTMENT OF HEALTH & HUMAN SERVICES Public Health Service

Centers for Disease Control and Prevention

National Center for Health Statistics

3311 Toledo Road

Hyattsville, Maryland 20782

May 14, 2018

Margo Schwab, Ph.D.

Office of Management and Budget

725 17th Street, N.W.

Washington, DC 20503

Dear Dr. Schwab:

The staff of the NCHS Collaborating Center for Questionnaire Design and Evaluation Research (CCQDER) (OMB No. 0920-0222, exp. 07/31/2018) plans to conduct in-depth interviewing of respondents of the National Study of Long-Term Care Providers (NSLTCP) (OMB No. 0920-0943, Exp. Date 05/13/2019) to determine whether there are data quality (measurement and response generation) challenges with the services users surveys questions that are related to administrative record keeping, and if there are, how this can inform future study design and questionnaire development through a better understanding of record keeping practices among providers.

We propose to start recruiting for volunteer participants as soon as we receive clearance and to start testing as soon as possible after that.

Background Information about Cognitive Testing of Questionnaires

Consistent with previous NCHS/CCQDER cognitive interviewing studies, the current study will investigate the cognitive processes respondents go through when responding to survey questions. Qualitative data will be collected through in-depth, semi-structured interviews. Findings from the analysis of this data has the potential to reveal varying patterns of interpretation and judgement for each survey item as well as differences across groups of respondents all of which can contribute to response error. Findings from all CCQDER studies are documented in a final report and made publicly accessible on a searchable website at https://wwwn.cdc.gov/QBank.

Proposed project: Cognitive Interviewing Study to Evaluate Record Linkage in the National Study of Long-Term Care Providers

The National Study of Long-term Care Providers (NSLTCP) is a biennial study begun in 2012 that is conducted by the Long-Term Care Statistics Branch (LTCSB) in the Division of Health Care Statistics (DHCS) and is designed to provide national and state representative statistical information about the supply and use of long-term care services providers in the United States. NSLTCP includes five provider sectors: residential care communities, adult day services centers, nursing homes, home health agencies, and hospices. NSLTCP comprises two components: (1) primary data collected by the National Center for Health Statistics (NCHS) through surveys of residential care communities (RCCs) and adult day services centers (ADSCs), and (2) administrative data on nursing homes, home health agencies, and hospices obtained from the Centers for Medicare & Medicaid Services.

For the primary data collection survey component, NSLTCP used a set of eligibility criteria for ADSCS and RCCs to be eligible to participate in the study. ADSCs have to be included in the National Adult Day Services Association’s database as well as conform to other additional criteria. Among other criteria, all RCCs that participate in the survey are to be licensed, registered, listed, certified, or otherwise regulated by the state. RCCs licensed to exclusively serve the mentally ill or the intellectually disabled/developmentally disabled populations are excluded from NSLTCP.

NSLTCP uses a multi-mode survey protocol with mail, web, and computer-assisted telephone follow-up for non-response. The questionnaires include survey items on provider characteristics such as ownership, size, number of years in operation, services offered, selected practices, and staffing in addition to aggregate user characteristics, such as age, sex, race, and the number of residents/participants needing assistance with activities of daily living. For the services user questions, respondents are asked to provide the number of services users (ADSC participant or RCC resident) who have a characteristic of interest. For instance, one of the services user questions on health conditions asks, “Of the residents currently living in this residential care community, about how many have been diagnosed with each of the following conditions?” For the 2016 NSLTCP, a list of 18 conditions were provided as response categories. On the cover page of the mail questionnaire and the introductory page of the web questionnaire, and later during CATI follow-up if needed, respondents are specifically asked to “Please consult records and other staff as needed to answer questions.” However, it is unclear if (1.) resident/participant records are maintained by all providers, (2.) whether record-keeping practices vary by provider characteristics, such as size of provider, and respondent characteristics, such as professional tenure and/or official position of respondent, (3.) whether all respondents have access to and are willing to spend time looking up records and calculating numbers of residents/participants when responding to the questions, (4.) whether respondents consider it burdensome to answer the aggregate services user questions, and (5.) how these factors may affect the quality of the data.

The overall purpose of this project is to determine whether there are data quality (measurement and response generation) challenges with the services users surveys questions that are related to administrative record keeping, and if there are, how this can inform future study design and questionnaire development through a better understanding of record keeping practices among providers. Specifically, this project will assess 1) record keeping practices and response generation (data quality) for current aggregate-level services user survey questions and 2 ) their (variation among providers (e.g., by sector, by size, by social vs medical model for adult day, by other relevant characteristics), to determine the extent to which the current aggregate-level services user survey questions fit well or cause undue burden or data quality challenges, and whether this varies by survey question or survey question type.

The National Study of Long-Term Care Providers questions we are evaluating are included as Attachments 1a (adult day services) and 1b (residential care community). The testing procedure conforms to the cognitive interviewing techniques that have been described in CCQDER’s generic OMB clearance package (No. 0920-0222, Exp. Date 07/31/2018).

Based on NCHS’ experience with the 2016 surveys, testing is needed for three reasons—item non-response, unit non-response, and feedback from some respondents about the time it took (burden) to calculate responses to some of the services user questions. Specifically, among the services user questions, about 20 items had item-missing rates of 3%-10% missing and about 20 items had item-missing rates of 11%-16%. The weighted response rate was 51% for the residential care sector, and 62% for the adult day sector. Because respondents could see the mail questionnaire in its entirety, it may be that some respondents looked at the questionnaire and decided not to complete or return it when they saw questions, because of concern about burden. For those completing the questionnaire by web, some respondents may have broken off if they had challenges completing services user questionnaires. Finally, respondents were given the opportunity in 2016 to submit comments to NCHS about their experience with the survey. Multiple respondents commented about challenges in calculating accurate responses to some of the services user questions, most commonly but not limited to the diagnoses. NCHS seeks to understand what factors contribute to these challenges and burden, including the extent to which there may be a misalignment between the question wording and common record keeping practices among providers.

We propose to recruit as many as 40 English speaking adults (aged 18 and over) who are most knowledgeable about the operations, staff, services, and resident/client characteristics of the adult day services centers or residential care facilities, (i.e., directors, assistant directors, administrators, assistant administrators, etc.). We purposely intend to recruit among previous respondents to NSLTCP, as we think they will have a keen sense of what it was like to complete the questionnaire and may have the chance to be sampled in the future. In fact, we intend to use that as a benefit for participating in the cognitive testing. Ultimately, we anticipate that this approach will help NCHS better understand the respondent experience, which may be used to inform future versions of the NSLTCP surveys.

Directors, Assistant Directors, Administrators, Assistant Administrators, etc. in the sample area will be recruited from a list provided by the Division of Healthcare Statistics. DHCS Staff will hand carry contact information to the CCQDER. An advance letter (see Attachment 2) will be sent to all organizations from DHCS’ list. As a follow-up to the invitational letter CCQDER staff will call individuals to talk to them about the study, what they will be asked to do, and to ascertain their interest in participating in the study. There will be no coercion. Individuals will be told that their participation in the study is entirely voluntary. The 5 minutes screener used to verify eligibility of the most knowledgeable person from adult day services centers and residential care communities are shown in Attachments 3a & 3b. Note that wording of the template has been approved and is contained within our umbrella package. Only project specific information has been added to the document. It is anticipated that as many as 72 individuals may need to be screened in order to recruit 40 participants.

Interviews averaging 60 minutes (including the completion of a Respondent Data Collection Sheet) will be conducted by CCQDER staff members with English speaking respondents. Interviews will be conducted by CCQDER staff members in a private room in the residential care facility/adult day services center or a mutually agreeable location. Interviews will be audio recorded. These recordings will allow researchers to ensure the quality of their interview notes. In the rare case that a study participant initially agrees to audio recording during the telephone screening, but changes their mind and checks “no” to allowing the interview to be recorded on the informed consent document the interview will proceed without audio recording. In this case the interviewer will depend on their handwritten notes when conducting analysis. In addition, individuals who select “yes” for allowing the audio recording on the informed consent form, but “no” for retaining the recording for future research (final text before signatures on informed consent form), will be allowed to participate in the study.

After respondents have been briefed on the purpose of the study and the procedures that CCQDER routinely takes to protect human subjects, respondents will be asked to read and sign an Informed Consent (Attachment 4). Only project specific information has been added to the document. Respondents will also be asked to fill in their demographic characteristics on the Respondent Data Collection Sheet (Attachment 5). This document is contained in our umbrella package. The burden for completion of this form is captured in the interview.

The interviewer will then ask the respondent to confirm that he/she understands the information in the Informed Consent, and then state that we would like to record the interview. The recorder will be turned on once it is clear that the procedures are understood and agreed upon.

The interviewer will then orient the respondent to the cognitive interview with the following introduction:

[fill staff name] may have told you that we will be working on some questions that will eventually be added to national surveys. Before that happens, we like to test them out on a variety of people. The questions we are testing today are about administrative data at long-term care facilities. We are interested in your answers, but also in how you go about making them. I may also ask you questions about the questions—whether they make sense, what you think about when you hear certain words, and so on.

I would first like you to fill out this questionnaire, and then we will discuss the questions and how you formulated your answers. Please answer the best you can, and mark any questions that you don’t understand or are having difficulty answering and we will discuss them once you have finished, and I’d like you to answer as best you can. Please note on the instrument if:

there are words you don’t understand,

the question doesn’t make sense to you,

you could interpret it more than one way,

it seems out of order,

or if the answer you are looking for is not provided.

The more you can tell us, the more useful it will be to us as we try to develop better questions. Okay? Do you have any questions before we start?

After the interview, respondents will be given the thank-you letter (document contained in umbrella package) signed by the Charles J. Rothwell, Director of NCHS (Attachment 6), a copy of the informed consent document, and $100.

After the interview, respondents will be asked to read the Special Consent for Expanded Use of Recordings (Attachment 7). There will be no coercion and the respondents will be told that they can call and reverse the decision at any time if they change their minds. If respondents do sign the special consent form they will be given a copy of that as well.

Extreme care will be taken with all recordings and paperwork from the interviews conducted off-site. Recordings and identifying paperwork will be stored in a secured travel case until returned to NCHS, at which point they will be transferred to the usual secured locked storage cabinets.

We propose giving participants $100 for their participation. Since the recruitment of directors, assistant directors, administrators, assistant administrators of residential care facilities and adult day services centers are necessary for this study to take place and since these individuals are extremely busy, and because there are a limited number of eligible facilities within the study area, the incentive has been increased over and above our normal $40 incentive to increase participation, reduce the number of cancelations, and maximize time and travel in a particular location. Because NSLTCP respondents typically do not receive an incentive for their participation, it will be made clear that the token of appreciation offered in this instance is being provided (by CCQDER) for their additional time and effort to answer cognitive testing questions, which are not part of the NSLTCP regular surveys. If indeed a future NSLTCP respondent is selected for one of the regular surveys and questions why an incentive is not being paid, it will be explained that the previous incentive was given for their part in making the future survey experience more positive.

In total, for this project, the maximum respondent burden will be 46 hours. A burden table for this project is shown below:

Form Name	Number of Respondents	Number of Responses/ Participant	Average Burden per response (in hours)	Total Burden Hours
Screener: Residential Care Communities (ADSCs)	36	1	5/60	3
Screener: Adult Day Services Centers (RCCs)	36	1	5/60	3
Questionnaire: Residential Care Communities (ADSCs)	20	1	1	20
Questionnaire: Adult Day Services Centers (RCCs)	20	1	1	20
Total	46

Attachments (9)

cc:

V. Buie

J. Zirger

DHHS RCO

5 | Page