OMB Revised Supporting Statement B_NFCSP_6_14_2015_ACL

Part B of the Justification for this information collection activity, the Evaluation of Older Americans Act Title III-E National Family Caregiver Support Program, addresses the five points outlined in Part B of the OMB guidelines.

B.1. Respondent Universe and Sampling Methods

The universe of potential respondents includes all State Units on Aging (SUAs) and Area Agencies on Aging (AAAs)^¹. This will ensure that this first process evaluation of the National Family Caregiver Support Program (NFCSP) will be based on a comprehensive set of results for analysis of the similarities and differences across all states and localities. Based on the AAA survey results, a phased, stratified sample of 274 AAAs will be selected, stratified based on organizational size and geography. The additional criterion for selection will be that the AAA provides, at a minimum, respite and/or caregiver training/education services; the AAA may provide other services as well. Each of the selected 274 AAAs will be asked for a list of local providers of respite and/or caregiver training/education, from which up to five local providers will be randomly selected for the local provider survey. In addition, all 10 of the single state PSAs will be included in this phase of the study; they will be asked for a list of their local services providers. More information is provided in Section B.1.1-B.1.3.

State Unit on Aging (SUA) Survey

The web-based SUA survey will be administered to the universe of SUAs (N=56) providing NFCSP services and is available for review at: http://www.aoa.gov/Program_Results/Program_survey.aspx.

ACL will send SUA Directors a letter providing an overview of the study, followed by an e-mail notification of the survey. The Directors will be encouraged to share the survey solicitation with the most appropriate person(s) in the agency for completion. We will use web-based format and we will also prepare a PDF version of the survey for individuals who do not have the capacity to complete a web-based survey or who prefer a paper version. The research team will employ various methods to maximize survey participation, which could include telephone follow-up, postcard follow-up, email follow-up, etc.

Area Agencies on Aging (AAA) Survey

The AAA survey will be administered to the universe of 619 AAAs providing NFCSP services. All 619 of the AAAs will receive a single in-depth survey. The AAA survey is available for review at: http://www.aoa.gov/Program_Results/Program_survey.aspx.

This approach will produce the maximum number of responses from AAAs, which will provide greater ability to compare results across different groupings of AAAs.

The comprehensive AAA survey will also be offered in a PDF version for respondents preferring to complete a paper version of the survey. The survey will solicit information about program models, operational procedures, role of consumer direction/self-directed support, policies for prioritization and cost containment, the role of care management, and activities undertaken in service delivery and quality assurance.

LSP Surveys

The LSP survey will be administered as a web-based survey by UCHC Center on Aging, with an optional PDF version. A total of 1,000 LSPs will be sampled. The LSP survey is available for review at: http://www.aoa.gov/Program_Results/Program_survey.aspx

The rest of Section B.1 covers the sampling approach for the LSP surveys:

Universe of Potential Respondents
Stratified Sampling
Sample Development and Selection of AAAs and LSPs

B.1.1. Universe of Potential Respondents

At the local service provider (LSP)^² level, the universe of potential respondents includes LSP organizations delivering the key NFCSP-funded services of respite and/or caregiver training/education. Together, these two types of NFCSP services account for more than half of the program funding^³ and serve almost 200,000 clients annually. In Fiscal Year 2011, there were 11,309 total providers across all types of Title III-E services. Of the total providers nationally, as many as 85 percent provided either respite (n=8,368) or caregiver training/education and (n= 1,247). Thus, the results of the LSP survey will be generalizable to other programs providing these services and represent a substantial majority of all LSPs.

B.1.2. Stratified Sampling

The sample of the LSPs will be based on a stratified sample of all AAAs that both complete the AAA survey and provide either respite and/or caregiver training/education (from AAA survey question #19e and #19b, respectively). From the AAA survey, we will draw a sample of 274 AAAs based on their responses to the provider module on the AAA survey. We will stratify the sample of AAAs by the following:

Size of AAA budget (small, medium, and large from AAA survey question #100);
Population density of region served by the AAA (from AAA survey question #4)
- Rural (Predominantly rural; predominantly remote or frontier)
- Urban/suburban (Predominantly urban; predominantly suburban)
- Mixed (mix of urban and rural; mix of suburban and rural; mix of urban, suburban and rural)

B.1.3. Sample Development and Selection of AAAs and LSPs

The specific purpose of sample size calculation, crucial in collecting valid and generalizable information, varies depending on the goal of the research. Although commonly used in studies that generate inferential statistics, power analysis, which is used to determine the sample size required to detect statistically significant effect sizes, is not an appropriate statistical technique for determining the sample size in this descriptive study. Our goal is to generate descriptive information about the structure and operation of NFCSP programs within AAAs and their affiliated providers, rather than looking for minimum detectable differences in client outcomes based on AAA or provider characteristics. Thus, for this process analysis, we need to select the appropriate number of AAAs and LSPs to generate descriptive statistics with reasonable confidence intervals and acceptable levels of confidence. Depending on sample size calculation formulas and assumptions, there may be a range of AAAs. For the purposes of this discussion, we will use the following formula and assumptions.

While the total population of AAAs is 619, the target population for the study includes only those AAAs which contract with local service providers for either caregiver training & education or respite services, since these are the target services for the evaluation. If the AAA only provides these services directly and does not have LSPs, they will also be excluded from the sampling frame because they will have already provided the information through the AAA survey. Based on data from the 362 AAAs who have completed the survey to date (June 1, 2015), 18.8% do not provide these services with outside LSPs. The sample size calculation for AAAs is based on a total population of 619 AAAs, adjusted down to a target population of 503 based on the estimated 81.2% who do contract with outside LSPs for training or respite.

Parameters for sample size estimation:

Population size: 503
Confidence level: 95% (Z=1.96)
Variable(s) of interest: dichotomous questions, assuming a conservative / worst case assumption that the percentage answering “yes” or “no” is 50%) (p = 0.5)
Half width of confidence interval: d = 0.04
Sample size = 274

All 10 states operating as a “single state Planning and Service Areas (PSA)” (as defined in 45 CFR 1321.41 as states “so small or rural that the purposes of this part would be impeded if the State were divided into planning and services areas”) will be selected with certainty and added to this sample, resulting in a total sample size of 284.

This sample size estimate is based on the total target population of 503 AAAs who contract with LSPs. Because the sampling frame elements are the AAAs who participated in the survey, it is smaller than 503. At this point, we expect the sampling frame size to be about 400.

In order to capture the variability among these organizations, the desired sample size will be achieved through proportional stratified sampling based on budget and geographic area served. AAAs will be stratified into terciles based on their total operating budgets, categorized as small at $2.5 million or less, medium between $2.5 and $6.3 million, and large budgets of more than $6.3 million. These category boundaries are calculated from the responses received to date to create equal size groups.

Based on the AAA’s NFCSP survey self-report of the rural/urban nature of the geographic area served, the sampling frame of AAAs will be further grouped into three strata: urban/suburban, rural and/or frontier, or mixed.

In total, the sampling frame of AAAs will be grouped into 9 categories (3 budgetary sizes by 3 geography strata) to allow a sample of 274 AAAs to be drawn proportional to the distribution of AAAs among the categories. For example, if 10% of the AAAs are mixed geography with medium budgets, 10% of the 274 (n=27) AAAs will be drawn from this group.

The LSPs will be selected from these 274 AAAs. Assuming a half-width confidence interval of d=0.04, 50% population distribution on dichotomous questions (p=0.5), a 95% confidence level (Z=1.96), and a total population of 8,000 LSPs which provide training and/or respite (derived from 2012 State Program Report numbers of 10,500 total NFCSP providers and adjusted downward based on an estimate of the number of respite providers and caregiver education providers), the minimum LSP sample size is 559. However, since very little is known about the population of training and respite LSPs, and to decrease confidence intervals on subpopulation estimates, we will oversample them for the process evaluation. The average number of LSPs per AAA is 4.6 according to our preliminary data, but some AAAs have as few as 1 and some have as many as 30, and the degree of variation among LSPs in terms of clients served is unknown at this point. To provide the fullest picture of the LSP network, and to insure the richest possible sampling frame for the NFCSP participant outcome study, we will recruit 1,000 LSPs, aiming for a response rate of 80% to yield 800 participating organizations. AAAs in our sample will be asked to provide lists of all providers; from some lists we will draw random samples, from others we will recruit all providers. For the approximately 70% of participating AAAs that are expected, based on preliminary data, to have three or fewer LSPs, all providers will be recruited to participate. For those with 4 LSPs (an estimated 15%), three will be randomly selected. For the remaining 15% of the participating AAAs with 5 or more providers, 4 providers will be randomly selected. Whenever we are able to randomly sample, other providers will be held in reserve to replace providers who do not agree to participate.

Considerations for the LSP sampling frame include:

The sample will represent providers of services to the major caregiver population group, which is caregivers of adults 60 years and older.
The LSPs included on the AAAs’ contact lists must provide respite and/or training and education among their core services;
Both AAAs that directly provide services and the local service providers under contract to AAAs will be included in the LSP sampling frame;
All ten single state PSAs that directly provide services will also be included.

B.1.4. Recruitment Strategy

To enhance the likelihood of a high participation rate in the process evaluation surveys at each organizational level, we have developed a contact/recruitment strategy that includes advance mailings from the NFCSP funding agency ACL to the SUA grantees and AAA sub-grantees. As well, written and online (video) promotional materials have been prepared to introduce the survey to SUAs, AAAs, and LSPs. Finally, we will obtain, where possible, endorsements from other relevant organizations such as the National Association of States United for Aging and Disabilities (NASUAD). The outline of the NFCSP process evaluation outreach is provided in the “NFCSP Data Collection Activities” under Study Recruitment Materials at: http://www.aoa.gov/Program_Results/Program_survey.aspx. The documentation details the advance contacts, invitations to participate in the survey, and follow ups to be conducted with the SUA, AAA, and LSP surveys.

B.1.5. Sample Size and Response Rates

The sample sizes for each respondent group are discussed in subsections B.1.2 through B.1.3. Exhibit 1 summarizes the planned census/sample sizes and the expected number of responses. A 100 percent response is anticipated at the SUA level. For the AAA and LSPs,, we expect 80 percent cooperation.

Exhibit 1 Summary of Sample Sizes

Respondent Group	Census/ Sample Selected	Number of Responses
State and Territorial Units on Aging (census)	56	56^⁴
Area Agency on Aging (census)	619	494^a
Area Agency on Aging (stratified sample)	274	274
Local Service Provider (random sample)	1000	800^a

^aApproximate.

The investigators will use a number of proven methods to maximize participation in the SUA, AAA, and LSP surveys. These include:

Use of survey instruments with clear instructions for completion;
Limited number of open-ended questions in the survey instruments;
Flexibility about the time and administration mode (e.g., online, fax, e-mail, mail) for the SUA, AAA and LSP surveys; and,
Tracking responses and conducting follow-ups with non-respondents via email, phone, and postcard mailings.

B.2. Procedures for the Collection of Information

Procedures for the collection of information addressed below include:

Statistical methodology for stratification and sample selection
Estimation procedure
Degree of accuracy needed for the purpose described in the justification
Unusual problems requiring specialized sampling procedures
Any use of periodic (less frequent than annual) data collection cycles to reduce burden

B.2.1. Statistical Methodology for Stratification and Sample Selection

This is described in subsections B.1.1 through B.1.6.

B.2.2. Estimation Procedures

See Section B.1.3 above.

B.2.3 Degree of accuracy needed for the purpose described in the justification

As noted above in B.1.3, the specific purpose of sample size calculation, crucial in collecting valid and generalizable information, varies depending on the goal of the research. Although commonly used in studies that generate inferential statistics, power analysis, which is used to determine the sample size required to detect statistically significant effect sizes, is not an appropriate statistical technique for determining the sample size in this descriptive study. Our goal is to generate descriptive information about the structure and operation of NFCSP programs within AAAs and their affiliated providers, rather than looking for minimum detectable differences in client outcomes based on AAA or provider characteristics. Thus, for this process analysis, we need to select the appropriate number of AAAs and LSPs to generate descriptive statistics with reasonable confidence intervals and acceptable levels of confidence. As previously described, we assumed a half-width confidence interval of d=0.04, 50% population distribution on dichotomous questions (p=0.5), and a 95% confidence level (Z=1.96), for determining the sample sizes for both AAAs and LSPs. These sample size calculations insure that the descriptive results of the AAA and LSP surveys will be adequately representative of the population of AAAs and LSPs, respectively.

B.2.4. Data Collection Methods

SUA Process Survey

The ACL task order officer will elicit the support of the ACL Regional Offices. Before telephone contact, we will send an outreach package by Federal Express to the ACL regional contacts and make a courtesy telephone call to seek their support. After we have established contact with the 10 ACL regions, we will send the 56 SUA outreach packages by FedEx and begin recruitment calling. We will enlist the support of the SUA director and request name and contact information of the designated respondent if it is someone other than the director. If an alternate proxy respondent is identified, we will request that the SUA director give the survey materials to that person, and a survey specialist will attempt to contact that person.

The Outreach Packages Will Include:

A cover letter
A brochure
A survey preparation worksheet

These materials for the SUA Survey Outreach Package are described in detail in subsection B.3.1 and provided in the “NFCSP Process Evaluation Activities” documentation under the heading of Study Recruitment Materials at: http://www.aoa.gov/Program_Results/Program_survey.aspx.

After the respondent is identified, we will contact them and urge them to complete the survey. In some instances, we will complete the survey with the respondent by telephone. Other respondents will complete the paper survey and return it to the contractor.

AAA and LSP Process Surveys

The AAA and LSP process surveys will be web-based. Web surveys offer maximum flexibility to respondents and minimize errors associated with data entry of hard-copy surveys, although a PDF version of each survey will be made available to any organization requesting this option. High response rates are achievable when support is available to help respondents during the field period. For this purpose, AAAs and LSPs not responding to the initial mailing will be contacted to (1) identify appropriate respondent(s), (2) provide technical assistance to complete the survey, and (3) monitor completion. Reminder emails will be sent to encourage timely submission of completed surveys.

Recruitment for the AAA data collection will be similar to that for the SUA process survey; however, it will be more reliant on electronic communication. Recruitment materials will be sent by email that includes (1) a brochure; (2) a list of frequently asked questions (FAQ) about the study’s purpose, the role of the AAAs and LSPs and information on whom to contact with questions; and (3) a letter inviting their participation in this study of the National Family Caregiver Support Program. AAAs selected for the sample of LSPs will be asked to make the first contact with the five LSPs through an email drafted by the contractor.

These materials for both the AAA Survey Outreach Package and the LSP Survey Outreach Package are described in detail in subsection B.3.1 and are provided in the “NFCSP Process Evaluation Activities” documentation under the heading of Study Recruitment Materials at: http://www.aoa.gov/Program_Results/Program_survey.aspx.

B.3. Methods to Maximize Response Rates and Deal with Nonresponse

To maximize response rate for this study, we will develop multimode data collection systems that ensure high quality data collection while minimizing burden on respondents. Exhibit 1 summarizes the data collection mode and number of responses for each survey. We will encourage greater participation through contact and recruitment materials that are relevant to each sample group. Here, we present our strategies for maximizing response rates by survey.

B.3.1. Process Surveys

The process survey will examine the strategies, activities and resources of the Title III-E organizations at each of the three levels of the Aging Network: (1) SUAs, (2) AAAs, and (3) LSPs. We will initiate the contacts at the ACL region level and proceed to the SUA level and from there to the AAAs and LSPs. At each level in the Aging Network, we will not only request endorsement for the next level but also ask the respondent to directly communicate that support to the next level in the Aging Network.

A key element in a high recruitment success rate will be the recruitment materials. Dillman (2000) showed that clear, well-written, and persuasive survey materials assist in higher response rates. Recruitment materials include a cover letter, project brochure, and survey worksheet.

Cover letter. The cover letter will explain the purpose of the NFCSP evaluation and will contain endorsements from other agencies or individuals that support the evaluation.

Brochure. The trifold brochure will contain information on the purpose and importance of the study, key components of the study, contact information for the sponsoring and contracting agencies, and responses to frequently asked questions, with a toll-free number and email address in case the recipient has additional questions.

Survey preparation worksheet. The survey preparation worksheet will list the specific types of information that should be gathered, e.g., prior annual report(s) on clients and budgets. By knowing in advance the types of information we seek, the respondents will be able to identify the best respondent for the survey (him- or herself or another staff member). This will provide time for the selected respondent to prepare for the survey and help reduce the burden on the respondents.

B4. Tests of procedures or methods to be undertaken

Instruments to be used in this study were pre-tested on small numbers of appropriate respondents (2 SUA representatives, 9 AAA representatives, and 5 LSP representatives). Time burden estimates were derived and refinements made to question wording and ordering based on this pilot testing.

In developing the study protocol, every attempt was made to replicate or adapt existing instrumentation.

The results of the three pretests are summarized below:

SUA. The feedback on pre-testing the SUA survey was generally positive. Both pre-testers preferred taking the survey online, but would like to have the paper survey on hand when completing the survey. The pre-testers suggested changes to specific questions to minimize confusion, which were incorporated into the final SUA survey.
AAA. The respondents’ opinion of the survey was positive overall. A number of changes were recommended. These included reordering questions so that more efficient skip patterns could be used, adding skip patterns or “do not apply” responses, adding definitions, clarifying terms, by adding examples or changing terminology. Most of these changes are straightforward and will assist other AAA directors when they complete the final survey. These changes were made in the survey, along with some recommendations to change specific questions and reduce the length of the survey.

After incorporating the changes to the AAA survey after the first round of pre-testing, Scripps completed a smaller pre-test for the edited survey. The individuals were from the same pool that pre-tested the survey the first time. General comments about the new version were generally positive and respondents reported that the changes “made it easier to complete” and it was “better this time.”

LSP. The majority of the survey was well-received by pre-testing participants. They felt that the information requested would be available to the person being surveyed or in collaboration with colleagues. There were a few areas where questions appeared unclear or where the order of questions could be improved. This feedback was incorporated into the survey.

B5. Individuals Consulted on Statistical Aspect and Individuals Collecting and/or Analyzing Data

The investigators chosen by ACL to conduct this study include the following individuals:

Lisa Alecxih, Senior Vice President, The Lewin Group, 703-269-5542, Project Director
Cindy Gruman, Vice President, The Lewin Group, 703-269-5506, Program Manager
Ashley Tomisek, Research Consultant, The Lewin Group, 703-269-5632, Research Assistant
Jane Straker and Suzanne Kunkel, Scripps Gerontology Center
Julie Robinson and Noreen Shugrue, UConn Health Center

In addition, Greg Link, the program manager for the National Family Caregiver Support Program in the Administration on Aging/Administration for Community Living (ACL), provided extensive expertise and helpful review in the development of the study design and instruments.

The ACL task order officer for this study is:

Alice-Lynn Ryssman

Office of Performance and Evaluation, Center for Disability and Aging Policy

Administration for Community Living, US Department of Health and Human Services

1 Massachusetts Avenue, NW, Washington, DC 20201

Telephone-202.357.3491

Fax-202.357.3549

Email: alice-lynn.ryssman@ACL.hhs.gov

1 AAAs plan, coordinate, and advocate for the development of a comprehensive service delivery system to meet the needs of older adults in a specific geographic area. They administer state and federal funds for community-based services. http://www.tjaaa.org/glossary-of-terms.aspx.

2 Area agencies normally contract with local for-profit or nonprofit or public providers (LSPs) to deliver benefits. The contract service providers nationwide, providing care under the act, are the largest single network of long-term care providers in the country. An agency may be allowed to directly provide supportive services, nutrition services, or in-home services if it can prove that it can provide these services more effectively. http://www.longtermcarelink.net/eldercare/area_agencies_on_aging.htm.

3 National and state Title III-E clients and expenditures by NFCSP service type are collected through ACL’s State Program Report (SPR) and reported through ACL’s online Aging Integrated Database (AGID) at http://www.agid.acl.gov/ .

4 A 100% response rate is based on the 100% response rate to the SUA survey administered in early 2014 as part of the Title III-C Elderly Nutrition Services Program.

File Type	application/msword
File Title	National Family Caregiver Support Program (NFCSP) Evaluation
Author	bernadette.wright
Last Modified By	ALRyssman
File Modified	2015-06-29
File Created	2015-06-29