Application for New Data Collection:
Supporting Statement for
Outcome Evaluation of
Older Americans Act Title III-E
National Family Caregiver Support Program
Section B
August 31, 2015
Submitted to:
U.S. Administration for Community Living Administration on Aging 1 Massachusetts Avenue, NW Washington, DC 20001
|
Submitted by:
WESTAT 1600 Research Boulevard Rockville, Maryland 20850-3195 (301) 251-1500
|
Chapter Page
B. Collection of Information Employing Statistical
Methods
B.1 Respondent Universe and Sampling Methods
B.2 Procedures for the Collection of Information
B.3 Methods to Maximize Response Rates and Deal with
Nonresponse
B.4 Test of Procedures or Methods to be Undertaken
B.5 Individuals Consulted on Statistical Aspects and Individuals
Collecting and/or Analyzing Data
Tables Page
B-1 Margin of error for the difference between two estimates
and estimates of target characteristics
B-2 Average frame counts of Caregivers among sampled AAAs
Appendices
F Caregiver Pre-Interview Notification Letter & Sample Six-Month Reminder Card
G-P Survey Instruments [see separate zip file/document]:
G NFCSP Caregiver Participant Group Survey: Baseline
H NFCSP Caregiver Participant Group Survey: 6-month follow-up
I NFCSP Caregiver Participant Group Survey: 12-month follow-up
J Caregiver Comparison Group Survey: Baseline
K Caregiver Comparison Group Survey: 6-month follow-up
L Caregiver Comparison Group Survey: 12-month follow-up
M NFCSP Care Recipient Survey: Baseline
N NFCSP Care Recipient Survey: 12-month follow-up
O Comparison Group Care Recipient Survey: Baseline
P Comparison Group Care Recipient Survey: 12-month follow-up
Introduction
The contractor (Westat) is developing the study sample for the NFCSP outcome evaluation as an outgrowth of work Westat has done for ACL on the National Survey of Older Americans Act Participants (NSOAAP)/National Survey of OAA Title III Service Recipients, OMB Control Number 0985-0023 since 2002. Several of those surveys have included samples of NFCSP caregiver clients as part of the study. Details of a proposed sampling plan are provided in the paragraphs below, concerning such issues as the sampling frame, stratification of the population of Area Agencies on Aging (AAAs) before sampling, precision requirements, client sampling, and identification of a comparison group of caregivers that do not receive services from NFCSP providers.
Sampling of Area Agencies on Aging (AAAs)
The sample design for the NFCSP outcome evaluation will be based on a two-stage stratified probability sample of NFCSP participants, with the first stage being a stratified sample of AAAs, chosen with a modified version of sampling with probability proportional to size. The second stage consists of random sampling of NFCSP caregivers within the sampled AAAs, as has been done for several years in the ACL NSOAAP. This type of sample design is consistent with how AAAs maintain their client lists, which will ensure a realistic sampling approach. The sampling frame of AAAs used in the most recent national surveys is available for use in sampling AAAs for this study, with any updates incorporated into the frame development process. Prior experience with the NSOAAP indicates that approximately 75 percent of AAAs are expected to offer services under the NFCSP, and approximately 80 percent of sampled AAAs have participated in prior surveys.
To ensure proper representation of AAAs by size and geographic region, the sample will be stratified by a size measure based on the budget sizes of the AAAs and by Census region. These data are available for AAAs providing registered caregiver services on the sampling frame constructed by Westat for the 10th NSOAAP. After stratification, the sample will be selected in such a way that larger AAAs will have a higher probability of selection, but the selection will be modified to account for the fact that the size measure does not perfectly correlate with the number of clients served by the AAAs. After AAAs offering NFCSP services have been identified, client lists from those AAAs will be used as a sampling frame of clients from which a sufficient sample size of clients will be obtained for the survey. Considerations of sample size requirements are given below.
Because the objectives of this study include making comparisons between the NFCSP and non-NFCSP groups, the sample size requirements are larger than what would be needed for making point estimates of a specified precision in a cross-sectional survey. For example, for a cross-sectional estimate of a characteristic affecting 50 percent of the population (the “worst case” scenario) with a sample size of 1,250 and an assumed two-stage sample design effect of 1.2, the margin of error for the estimate would be 3.1 percent. The sample size requirements to compare estimates of one subgroup with another subgroup are different from the requirement for obtaining individual estimates at that level. The standard error (SE) of the difference between two estimates (for example, A and B) can be obtained by and the margin of error for the difference is approximately . Since the variance of the estimate of a difference between estimates is the sum of the variances of the relevant individual estimates, the required sample size for estimating a difference or change is larger than would be needed for a single point estimate. For the example above, if we are comparing two estimates from independent surveys, the variance of the difference given above applies. If the average of the two estimates is 50 percent (say, 45 percent and 55 percent) with a sample size of 1,250 in each group and an assumed design effect of 1.2, the margin of error around the difference would be 4.29 percent. This means that the minimum detectable difference between two estimates whose average value is 50 percent would be 4.29 percentage points. For a full range of sample sizes and detectable differences, see Table 1.
It should be pointed out, however, that for comparisons across time where the same respondents participate in the surveys at two time points, the standard error of the difference is likely to be smaller than for the situation described above (that is, for comparing independent samples). The reason is that we would expect positive correlation between responses for the same individual. In such a case the standard error of the difference would include a term that subtracts a function of the correlation between individual responses, thus leading to the smaller standard error and greater power to detect differences.
Table B-1 Margin of error for the difference between two estimates by various sample sizes and for various averages of the two estimates (computed for a two-stage design with a design effect of 1.20)
Sample size |
Average of the estimates to be compared |
||||
10 percent |
20 percent |
30 percent |
40 percent |
50 percent |
|
2,500 |
1.86 |
2.48 |
2.84 |
3.04 |
3.10 |
2,000 |
2.08 |
2.77 |
3.17 |
3.39 |
3.46 |
1,500 |
2.40 |
3.20 |
3.67 |
3.92 |
4.00 |
1,000 |
2.94 |
3.92 |
4.49 |
4.80 |
4.90 |
750 |
3.39 |
4.53 |
5.18 |
5.54 |
5.66 |
500 |
4.16 |
5.54 |
6.35 |
6.79 |
6.93 |
400 |
4.65 |
6.20 |
7.10 |
7.59 |
7.75 |
300 |
5.37 |
7.16 |
8.20 |
8.76 |
8.94 |
250 |
5.88 |
7.84 |
8.98 |
9.60 |
9.80 |
200 |
6.57 |
8.76 |
10.04 |
10.73 |
10.95 |
100 |
9.30 |
12.39 |
14.20 |
15.18 |
15.49 |
Because the sample design will be complex, Westat will calculate necessary sampling weights with appropriate weighting adjustments, such as nonresponse adjustments at both levels of sampling, variance estimation methods that take into account the complex sample design, any necessary weight trimming adjustments and post-stratification adjustments to available known population totals. All necessary adjustments to the weights will be completed prior to the preparation of final data for delivery to ACL. These steps are taken to ensure that proper inferences can be drawn from the sample to the population under study.
Sampling of NFCSP Clients
To minimize reporting burden on the sampled AAAs, the same sample of AAAs will be used for both the 10th NSOAAP and the NFCSP outcome evaluation, with independent samples of clients selected within the sampled AAAs. The use of the same set of first-stage sampling units for more than one survey is common practice in the survey research community. The timing of the two surveys is such that the 10th NSOAAP will be conducted first, although the samples of NFCSP clients can be selected at the same time, thus insuring that both samples of caregiver clients are representative of the population of caregivers. This can be accomplished by selecting a large enough sample of caregiver clients using the standard NSOAAP protocol (systematic random sampling within each sampled AAA) to meet the requirements of both surveys. Operationally this procedure is more efficient than selecting separate samples since the client sampling software is already in place and would require no additional developmental work. Once the client sample is selected, the appropriate number of clients will be randomly assigned to each of the two surveys. This procedure is similar to the random assignment of participants to the treatment and control groups in a randomized clinical trial and ensures that both client samples are representative of the client population from which they were sampled.
The NSOAAP design currently calls for the sampling of 32 caregiver clients per AAA on average to meet the overall eligible responding sample size of 2000 caregivers. In order to obtain the desired eligible responding sample of caregivers for the NFCSP outcome evaluation, prior experience with the NSOAAP suggests that we would need to sample an additional 18-20 clients per AAA. This means that we need to sample a total of approximately 50 clients per AAA on average for the two studies. This is no problem for the larger AAAs, as many of them have more than 50 caregiver clients on their client lists. However, it does raise an issue for the smaller AAAs, as indicated in the table below, which is based on the most recent data from NSOAAP from 2014. The size groups in the table are arranged from largest (Size Group 0) to smallest (Size Group 4) in terms of their budget sizes. This will allow us to produce unbiased estimates from both surveys.
Table B-2 Average frame counts of Caregivers among sampled AAA's
|
|
|||
|
|
|
|
|
SIZE GROUP OF AAA (based on AAA budget) |
Average frame counts of caregivers per AAA |
Median number of caregivers per AAA |
2014 AAA Sample Size |
Number of AAAs with 50+ caregivers |
0 |
334.5* |
177 |
42 |
39 |
1 |
322.6* |
170 |
67 |
50 |
2 |
201.1* |
90 |
67 |
42 |
3 |
152.6* |
53.5 |
68 |
29 |
4 |
56.5 |
47 |
68 |
26 |
*Slightly skewed due to a handful of AAAs with a very large number of caregivers.
On average, AAAs in all the size groups should have enough caregivers to meet the sample size requirement. However, as indicated in columns 3, 4 and 5 of the table, there is considerable variability among AAAs in the number of caregivers participating in the NFCSP. This is not unlike the NSOAAP, in which a higher proportion of clients for all services come from the larger AAAs. It is simply the reality of some AAAs’ not serving a large enough number of clients to allow equal-sized samples from all AAAs. We expect to obtain the required 1,250 eligible responding caregivers. They just won’t be equally distributed among large and small AAAs.
Selection of the Comparison Group of Non-NFCSP Caregivers
Recruitment of a good comparison group is essential for the success of this study. The contractor (Westat) will recruit non-NFCSP caregivers using an approach used successfully in the 2003 NSOAAP before the NFCSP was fully operational.
First, we know from our surveys of OAA participants that virtually all home care clients also have a family caregiver who provides hands-on assistance, especially during evenings and weekends when formal services are often unavailable. Identifying and screening these caregivers from the 10th NSOAAP as not participating in the NFCSP is the way we plan to select the comparison group from the same locations as the participants. We also know from the NSOAAP that virtually all their care recipients receive a range of HCBS support. To be truly comparable, therefore, the caregiver comparison group should involve caregivers whose care recipients receive some level of HCBS support. The non-NFCSP clients will be identified from sampled clients for the non-NFCSP services in the 10th NSOAAP (home delivered meals, homemaker, case management, congregate meals, and transportation).
As stated above, for the 2003 NSOAAP the NFCSP was in its infancy, and there were not sufficient numbers of caregivers in the program for sampling and surveying purposes. Instead, we sampled OAA HCBS clients (homemaker, transportation, and home delivered meals), and as part of a survey of these individuals asked them if they had a family member (or other person) who provided assistance with activities of daily living, such as bathing and dressing. If they said yes, we asked who that person was, if we could have permission to contact and interview that caregiver, and requested the name and phone number of that person. Virtually all the OAA HCBS clients agreed to provide the contact information about the caregiver. We programmed our CATI system to automatically create a call record for interviewing that caregiver, which was very successful. For the NFCSP evaluation comparison group we will expand the scope of screening NSOAAP clients to include those who are sampled for case management and congregate meals services as well as homemaker, transportation and home delivered meals. Results from the most recently available NSOAAP data suggest that approximately 60% of clients (for the five non-caregiver services) report having a caregiver. With a responding sample of approximately 4,000 clients for non-caregiver services, we could expect approximately 2,400 caregivers as potential comparison group members. We will have to eliminate the collected caregivers who are in the NFCSP, and we will not know in advance how many of these there might be. However, the number of caregivers in the NFCSP is small compared to the size of the total OAA service client population, so we do not expect to have to eliminate very many caregivers from the comparison group. All of these considerations lead us to believe that we can successfully recruit a sample of non-NFCSP caregivers of sufficient size to meet the requirements for the comparison group for the evaluation study.
Several data collection activities will be conducted to support the survey. They are designed to ensure as complete as possible a sample stage one (sample of AAAs) and the stage two samples: 1) NFCSP Caregivers and their care recipients and 2) non-NFCSP care recipients and their caregivers. . This will provide a representative sample for the analyses.
The first stage sample of AAAs and the second stage sample of caregivers and care recipients will be collected during the recruitment phase of the 10th NSOAAP. The proposed design for the 10th NSOAAP will employ a probability sample of all AAAs proportional to size (PPS) of the total annual budget. All AAAs selected for the NSOAAP will receive a Federal Express package that contains an introductory letter from ACL along with detailed instructions for the AAA. Approximately three days later, a researcher will call the agencies to review the contents of the package and to provide instructions for sampling the service recipients. The researcher will explain the numbered participant lists the agency needs to generate from which to select the random sample of service recipients. In addition, the package will provide detailed instructions specific to the client tracking software used by the AAA. Previous experience has enabled Westat, the contractor, to streamline the data collection procedures for the AAAs. AAAs will submit OAA service-specific lists of clients or client IDs from which Westat will use a computerized line-sampling program to randomly select clients for the NSOAAP. After approximately 32 NFCSP caregiver clients per AAA have been randomly selected to receive the caregiver module of the NSOAAP, an additional 20 NFCSP caregiver clients who receive respite care or counseling/training services will be randomly selected for the NFCSP outcome evaluation.
Pre-notification Advance Letters
Potential NFCSP Caregiver respondents selected to receive the telephone survey will receive a letter from their respective AAAs on the agency’s letterhead. The letter contains an introduction to the study, explanation about the nature of participation, and a toll-free number to call if they do not wish to participate. Those who opt out of the study are not contacted further. Care Recipients of NFCSP Caregivers will be identified during the interview with the NFCSP Caregiver and will receive a similar letter about the study and instructions for opting out. A comparison group of non-NFCSP Caregivers will be identified during the 10th NSOAAP telephone interview with non-NFCSP service recipients and will also receive a letter about the study.
Telephone Interview
The data collectors conducting the telephone interview participate in intensive training sessions prior to data collection and are monitored during data collection to ensure the protocol is properly followed. The training covers general interviewing techniques, topics specific to administering the specific Caregiver and Care Recipient survey instruments, and practice sessions.
The study sample includes people who are elderly and who may be living with disabilities. With that in mind, the training designed and conducted for the data collectors/telephone interviewers includes special guidance for interviewing and accommodating respondents who are elderly and who may have disabling conditions and/or communication problems (hearing impairments, speech disorders, cognitive impairments, memory disorders, non-native English speakers.) In certain instances, an interview will be conducted with assistance of the caregiver’s interpreter.
Data collectors are advised to be alert to the respondent’s fatigue and to suggest calling back and completing the interview during another session. For Spanish-speaking respondents, trained bilingual data collectors conduct the interview in Spanish.
At all three data collection points (baseline survey in 2015 and two follow-ups at 6 month and 12 month intervals in 2016), data collectors will conduct the following telephone surveys:
40 minute telephone survey of a representative sample of OAA Title III-E NFCSP caregivers receiving (a) respite care and/or (b) caregiver counseling or education & training.
10 minute telephone survey of care recipients identified by the NFCSP caregivers during their survey (see #1 above).
35 minute telephone survey of a comparison sample of non-NFCSP caregivers.
10 minute telephone survey of 10th NSOAAP respondents of non-caregiver services who indicated that they have a caregiver
Westat has quality control procedures in place for every phase of the project. Interviewers participate in rigorous training that includes general interviewer training and project specific training. Trainers observe interviewers conducting practice interviews and they monitor interviewers during data collection. During data collection, data are checked to ensure that there are no outliers in the dataset. In addition, when questions are raised during an interviewer, interviewers complete a form explaining an ambiguous or inconsistent response. Researchers review the forms and make any necessary adjustments.
The survey consists of telephone interviews with NFCSP caregivers, care recipients of NFCSP caregivers, a comparison group of non-NFCSP caregivers, and 10th NSOAAP respondents of non-caregiver services who indicated that they have a caregiver. The interview is structured and will contain specific questions about the services the caregiver has received, as well and the questions about the status of the care recipient. Probes will be used to facilitate obtaining complete responses to all the questions. Please refer to Appendices G, H, I, J, K, L for the baseline, 6-month, and 12-month NFCSP caregiver participant and comparison group survey instruments. Copies of the survey instruments for the care recipients are provided in Appendices M, N, O, and P. Please note that care recipient survey instruments will be used for Baseline and 12-month follow-up.
The caregiver interviews will last approximately 35-40 minutes and cover the topics discussed below:
Questions to verify and determine use of caregiver services. If a NFCSP caregiver receives respite care, they will be asked about the type and frequency of respite care provided. If the caregiver receives counseling or caregiver education and training, they will be asked about the type and frequency of counseling/training provided.
Caregiving Tasks. The caregiver will be asked about the nature, frequency, and intensity of the caregiving tasks that they provide for their family member.
Knowledge and use of OAA Title III-E caregiver services. [Not asked of the Comparison Group]
Delayed institutionalization & continued caregiving
Caregiving burden; self-efficacy and beliefs
Impact of caregiving (physical, social/emotional, financial)
Care recipient’s health and physical functioning
Caregiver and household demographics: Demographic information about the caregiver will be collected
The care recipient interview will last approximately 5 minutes and cover the topics discussed below:
Care recipient’s quality of life and emotional-wellbeing.
Care recipient’s perception of caregiver’s need for help
The survey instruments for this proposed information collection are based questions from the following sources:
National Health and Aging Trends Study (NHATS)/National Survey of Caregiving (NSOC)
NIH PROMIS Measures
Zarit Burden Interview (ZBI)
National Survey of Older Americans Act Participants (survey questions developed under ACL/AoA’s Performance Outcomes Measurement Project). Additional national survey questions come from such commonly used vehicles as the Survey of Income and Program Participation (SIPP), the 12-Item Short Form Health Survey (SF-12), and the Behavioral Risk Factor Surveillance System (BRFSS) surveys conducted within each state using HHS/CDC standard questions.
Procedures for Eliciting Cooperation and Maximizing Response Rates among AAAs
To minimize reporting burden on the sampled AAAs, the same sample of AAAs will be used for both the 10th NSOAAP and the NFCSP outcome evaluation, with independent samples of clients selected within the sampled AAAs. Following OMB approval for this Caregiver outcome Evaluation, Westat will send inform the AAAs of the additional study and obtain their consent to randomly select additional NFCSP caregiver participants from the lists of caregiver clients they submitted as part of the 10th NSOAAP.
To ensure a high participant response rate, with the approval of the participating AAAs, Westat will utilize individual AAA letterhead to send a customized letter to all sampled caregivers and care recipients who are eligible for the telephone survey before they are contacted by an interviewer. Westat will attempt to contact participants at different times of the day and different days of the week to maximize the possibility of contact. Westat is also experienced in refusal conversion procedures, having achieved a refusal conversion rate of 40 percent for the earlier surveys.
Procedures for Maintaining Cooperation for the Second and Third Data Collection Points
As described above, we will send reminder cards to the respondents 5-6 months after each data collection wave. Whenever reminder cards have been returned, a researcher will contact the next of kin or contact person and follow the procedures discussed in Section B2.2.2.
B. 4 Tests of Procedures or Methods to Be Undertaken
As discussed in earlier sections, the questions in the survey instruments come from other sources.
The use of statistical sampling methods is critical to this survey. Under the supervision of ACL, Westat is responsible for selecting the sample, conducting the interviews, data weighting and data analysis. Below are the names and contact information of individuals responsible for the statistical aspects of the study and individual collection and/or analysis of the data.
Administration for Community Living/Administration for Aging Personnel Responsible for Deliverables
Alice-Lynn Ryssman
Social Science Analyst; ACL Evaluation Project Lead
Office of Performance and Evaluation
Administration for Community Living/Administration on Aging
U.S. Department of Health and Human Services
1 Massachusetts Avenue, NW
Washington, DC 20001
Tel: 202-357-3491
Alice-Lynn.Ryssman@acl.hhs.gov
Westat Staff
Dwight Brock, Ph.D. – Role: statistician with involvement in study design, development of the sampling plan, weighting, and data analysis
Senior Statistician
Westat
1600 Research Blvd.
Rockville, Maryland 20850
301-517-4026
Joanne Campione, Ph.D. – Role: Project Director with involvement in design, sampling, and data analysis
Senior Study Director
Westat
1009 Slater Road, Suite 110, Stratford Building
Durham, NC 27703
(919) 474-3506
Kathryn Engle – Role: Data collection supervisor
Westat
1600 Research Blvd.
Rockville, Maryland 20850
301-610-4911
Katie Hubbell
Systems Analyst – Role: systems analyst with involvement in sampling data weighting, data analysis, and reports.
Senior Systems Analyst
Westat
1600 Research Blvd.
Rockville, Maryland 20850
301-294-2020
Robin Ritter – Role: Manager of Survey Operations including agency recruitment
Senior Study Director
Westat
1600 Research Blvd.
Rockville, Maryland 20850
301-240-314-5804
Jon Ratner, Ph.D. – Role: senior analyst/health economist with involvement in study design and analysis
Senior Statistician
Westat
1600 Research Blvd.
Rockville, Maryland 20850
(301) 610-8829
Jacqueline Severynse – Role: design of the sampling plan, weighting, and data analysis.
Senior Statistician
Westat
1600 Research Blvd.
Rockville, Maryland 20850
301-517-8058
JacquelineSeverynse@westat.com
Appendix F
Pre-notification Client Information Letter and Sample Six-Month Reminder Card
|
Sample Letter for NFCSP Caregivers Selected to be Interviewed
Tri-County Area Agency on Aging
123 Anywhere St.
Anywhere, XX 12345
<DATE>
Dear Mr./Mrs. ______________________________ ,
We are writing to ask for your help. <<INSERT AAA NAME>> is taking part in a study for the Administration for Community Living (ACL), part of the U.S. Department of Health and Human Services. ACL wants to know more about caregivers around the country and the support they get.
You have been chosen to be part of this national study. An interviewer from Westat, a research firm working for ACL, will call you in the near future to ask you some questions about the caregiver respite and/or counseling and training services that you receive from <<INSERT AAA NAME & PROVIDER NAME>>. Your answers will be kept private. Your name won’t be given to anyone else.
Your answers will be combined with answers from other clients and will be part of a report to ACL about how well these services are meeting the needs of older Americans. ACL and Congress will use this information to help them decide how well the programs are working and what improvements to make. Your answers will be very important in helping ACL and Congress decide what to do.
I hope you will be part of this important study. It’s your choice. Whatever you decide, it won’t affect the services you get now or in the future. If you do not want to take part in this study, please call Westat toll-free at 1-888-204-0046 by <<INSERT DATE>>.
If you have any questions about this study, please call Westat at 1-888-204-0046 or the Administration on Aging toll-free at 1-888-204-0271. If you have any questions about the services you get, please call our office at <<INSERT AAA PHONE NUMBER>>.
Thank you for your help and support.
Sincerely,
______________________________,
Executive Director
An agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a currently valid OMB control number. The OMB control number for this information collection is xxxx-xxxx. Public reporting burden for this information collection is estimated to average 40 minutes per response. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to the Administration for Community Living, Washington, DC 20201 Attn: Alice-Lynn Ryssman.
Sample Six-Month Reminder Card for Follow-Up Interviews
<DATE>
Dear <NAME OF RESPONDENT>:
Thank you very much for participating in a telephone interview about the National Family Caregiver Support Program. We would like to call you back in approximately ___ (weeks) [6 months from previous interview] for a follow-up interview. At that time, an interviewer will call to ask if there are any changes in the caregiver services that you receive or any changes to your health or that of the person you provide care for.
We urge you to continue your participation in this important study about the caregiver services that you receive from the <NAME OF AGENCY/PROVIDER>. Your participation will help us determine how to best meet the needs of caregivers caring for elderly family members.
If your telephone number has changed, please complete the enclosed postcard and return it to Westat.
If you have any questions about the study, please call <NAME OF CONTACT PERSON> at <TOLL-FREE NUMBER>. She will be glad to answer any questions.
Again, thank you very much for your participation in this important study.
Yours truly,
NAME OF PARTICIPANT___________________________________________.
My phone number has changed. Please call me at ________________________.
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
Author | Robin Ritter |
File Modified | 0000-00-00 |
File Created | 2021-01-24 |