Supporting Statement Part B - 6-14-2011

B. Statistical Methods (used for collection of information employing statistical methods)

1. Respondent Universe and Sampling Methods

1. Full-scale Study

Programs will be selected using stratified random sampling procedures. The only stratification variable will be four geographic region designated by the U.S. Census: Northeast, Midwest, South, and West. Three territories will be included – District of Columbia, Puerto Rico, and Guam – because they each have all three DD Network programs. District of Columbia and Puerto Rico will be included in Region 3 (South). Guam is included in Region 4 (West) for sampling

Table B-1 below shows the individuals that will be interviewed within each selected program.

Table B-1. Number sampled in each respondent category and potential respondent universe

Respondent Category	Potential Respondent Universe	Sample Number
DD Council: Executive Director Interview	55	20
DD Council: Interview with Council Chair/Council Members	440	60
DD Council: Group Interview with Policymakers, Collaborators, and Grantees	5,500	160
DD Council: Group Interview with Recipients of Self-Advocacy and Leadership Education and Training	2,750	100
DD Council: Group Interview with Recipients of Education and Training to Improve Community Capacity	2,750	100
DD Council: Self-administered Form	55	20
P&A: Executive Director Interview	57	20
P&A: Staff Interview	855	60
P&A: Board of Directors (Commissioners)-Chair and Members	1,140	60
P&A: Group Interview with Policymakers and Collaborators	5,700	160
P&A: Interview with Recipient of Community Education	5,700	100
P&A: Interview with Clients	5,700	100
P&A: Self-administered Form	57	20
UCEDD: Interview with Director	68	20
UCEDD: Telephone Interview with Current and Graduated Students	3,400	100
UCEDD: Interview with the Consumer Advisory Committee	1,360	60
UCEDD: Interview with Peer Researchers and Colleagues	3,400	100
UCEDD: Interview with Recipients of Community Services or Members of Organizations/Agencies that are Trained to Provide Community Services	500,000	100
UCEDD: Self-administered Form	68	20

2. ADD Assessment

The ADD assessment consists of interviews with ADD staff and stakeholders. Representatives from the following groups will be interviewed:

• ADD staff

• Federal disability partners

• National disability organizations

• Projects of National Significance

• Executive Directors from all DD Network programs in each state and territory

The ADD Commissioner and all ADD program staff and management were interviewed. The contractor worked with the ADD Commissioner, Deputy Commissioner, and Project Officer of the contract to identify appropriate individuals in Federal disability-related agencies and National disability organizations (thus, no sampling). Up to 9 individuals will be interviewed in each group. Projects of National Significance were grouped into categories (Emergency Preparedness; Family Support and Community Access Demonstrations; National Autism Resource and Information Center; Voting Project; Family Support 360; Youth Information, Training, and Resource Center; Ongoing Data Collection). Two projects were sampled from Family Support 360 and one project was sampled from each of the remaining categories. The project director of each selected project will be contacted for an interview.

All Executive Directors (180) will be requested to complete a web-based survey, so there will be no sampling for this group.

2. Procedures for the Collection of Information

1. Full-scale study

Data was collected using semi-structured interview guides to interview key informants identified with the help of Executive Directors of each program. Interviews were transcribed and summarized.

Analysis will consist of examining both quantitative and qualitative data collected on the 60 programs in 20 states and summarizing findings regarding structural, process, output and outcome indicators for each program and collaboration among the three programs. The focus of data analysis and synthesis will be on the extent to which DD Network programs are achieving outcomes. However, the contractor will also examine inputs, processes, and outputs to determine some of the factors that may contribute to falling short of achieving these outcomes and the factors that may be facilitating their achievement.

Analysis of in-person interview data, telephone interviews, materials obtained from programs will be qualitative. Data from the self-administered questionnaires are both quantitative and qualitative. In either case, a descriptive analysis, focusing on the research questions (Exhibit B-1) will take place.

Exhibit B-1. Research questions for the full-scale study

Structural Indicator questions:

1. What are the typical structural elements that enable and/or impede the individual entities of the DD Network (e.g., State Council, P&A, and UCEDD) to engage in activities that help meet the purpose and principles of the DD Act?

2. What are the typical structural elements that enable and/or impede the collaborative activities of the State DD Networks that support achievement of the DD Act purpose and principles?

Process Indicator questions:

1. What are the typical activities of the DD Network programs (e.g., State Council, P&A, and UCEDD) that contribute to achievement of the purpose and principles of the DD Act?

2. What are the typical collaborative activities of the State DD Networks that support achievement of the purpose and principles of the DD Act?

Output Indicator questions:

1. What are the typical immediate results of the activities of the DD Network programs (e.g., State Council, P&A, and UCEDD) that contribute to achievement of the purpose and principles of the DD Act?

2. What are the typical immediate results of the collaborative activities of the State DD Networks that support achievement of the DD Act purpose and principles?

Outcome Indicator questions:

1. What are the typical short term outcomes achieved through the activities of the DD Network programs (e.g., State Council, P&A, and UCEDD)?

2. What are the typical short term outcomes achieved through the collaborative activities of the State DD Networks?

Effectiveness questions:

1. What is the relationship between the outcomes achieved by the DD Network programs (e.g., State Council, P&A, and UCEDD) and the processes they used to affect those outcomes?

2. What is the relationship between the outcomes achieved by the State DD Networks and the processes used to affect those outcomes?

3. To what extent have the DD Network programs (e.g., State Council, P&A, and UCEDD) been effective in achieving the purposes and principles of the DD Act?

4. To what extent have the State DD Networks been effective in achieving the purposes and principles of the DD Act?

5. What efficiencies are being used by DD Network programs (e.g., State Council, P&A, and UCEDD) to obtain high quality outputs and achievements?

6. What efficiencies are being used by the State DD Network obtain high quality outputs and achievements?

2. ADD Assessment

The design of the ADD assessment is essentially a collection of retrospective and current data that relate to ADD’s structures, processes, outputs, and outcomes to draw conclusions about influences on the efficiency and effectiveness of DD Network programs. We define “effectiveness” as the ability to meet objectives. Thus, the ADD assessment will examine ADD’s ability to meet its objectives, from the perspective of a variety of stakeholders (including Executive Directors of DD Network programs), the DD Network programs’ ability to meet their objectives, and the relationship between the two.

Westat will use a semi-structured interview guide to obtain information from ADD staff, Federal agency partners, national disability organization staff, and staff from Projects of National Significance (discretionary grants funded by ADD. Up to nine individuals per category will be identified with assistance from ADD. Interviews will be conducted by telephone.

The contractor, Westat, will obtain budget and cost allocation data from ADD at three points in time (e.g., 2005, 2007, and 2009) to assess changes in some of the key structures (inputs) for ADD. Data collected through interview and by web-based questionnaire will relate to a recent period (e.g., 2009 and 2010).

Program materials will be obtained from ADD or the ADD website and reviewed by Westat staff.

To administer the web-based questionnaire, the Commissioner of ADD will send a letter to Executive Directors informing them that they will be contacted by Westat to participate in a web-based survey. Westat will then send an initial email to all Executive Directors to explain the purpose of the ADD assessment and provide a username and password to gain access to the questionnaire (Appendix A). Westat will also attach a separate file that contains instructions.

Data from each stakeholder group will be analyzed separately and then integrated to answer specific research questions (Table 4). Analysis of this qualitative data will primarily be descriptive. We will also conduct a descriptive analysis of the data extracted from background materials. Some data will be transformed into tabular format (e.g., to demonstrate budget appropriations or cost allocations over time).

Table B-2. ADD Assessment Analysis

Research questions	Source of data	Analysis
What structures does ADD have in place to administer the DD Act and compliance with the Act (e.g., policies, standards, data collection requirements, technical assistance, funding, and membership on Federal disability-related interagency committees)?	ADD documents Interviews with ADD staff	Document review Review and consolidation of interview summaries
What are the strengths and weaknesses of these structures?	Interviews with ADD staff Executive Director (ED) web-based survey	Review and consolidation of interview summaries Descriptive analysis of web-based survey
How have the ADD structures (e.g., appropriations from Congress) changed over time?	ADD documents Interviews with ADD staff	Review of ADD documents Review and consolidation of interview summaries
How are costs allocated to each ADD process?	ADD documents Interviews with ADD staff	Review of ADD documents Review and consolidation of interview summaries
How does ADD develop policies?	Interviews with ADD staff Federal disability partners	Review and consolidation of interview summaries
How does ADD monitor compliance with its policies?	ADD documents Interviews with ADD staff ED web-based survey	Document review Review and consolidation of interview summaries Descriptive analysis of web-based survey
How does ADD disperse funds?	Interviews with ADD staff	Review and consolidation of interview summaries
What factors have an effect on ADD processes?	Interviews with ADD staff ED web-based survey	Review and consolidation of interview summaries Descriptive analysis of web-based survey
How does ADD provide technical assistance?	Interviews with ADD staff ED web-based survey	Review and consolidation of interview summaries Descriptive analysis of web-based survey
What are the strengths and weaknesses of ADD processes?	ADD staff National disability organizations ED web-based survey	Review and consolidation of interview summaries Descriptive analysis of web-based survey
What are the products of ADD monitoring (e.g., reports to ADD)?	ADD documents Interviews with ADD staff	Document review Review and consolidation of interview summaries
To what extent are DD Network programs prepared to meet their goals and comply with the DD Act?	Program visits to 60 DD Network programs Telephone interviews with DD network program stakeholders Self-administered questionnaires Program materials Web-based survey	Integration of findings from all four sources of data in the full-scale study Descriptive analysis of web-based survey
What are the strengths and weaknesses of each product?	ADD documents Interviews with ADD staff ED web-based survey	Review of documents Review and consolidation of interview summaries Descriptive analysis of web-based survey
To what extent is ADD effective in being publicly accountable?	Interviews with ADD staff ED web-based survey Federal disability-related partners National Disability organizations Projects of National Significance	Review and consolidation of interview summaries of ADD staff interviews Descriptive analysis of web-based survey Review and consolidation of interview summaries of Federal disability-related partner interviews; descriptive analysis Review and consolidation of interview summaries from PNSs; descriptive analysis
To what extent are DD Network programs receiving the leadership they need from ADD?	ADD staff ED web-based survey	Review and consolidation of interview summaries of ADD staff interviews Descriptive analysis of web-based survey
To what extent are reports to ADD useful in monitoring the DD Network programs in compliance with the Act?	ADD staff ED web-based survey	Review and consolidation of interview summaries of ADD staff interviews Descriptive analysis of web-based survey
What are the factors that facilitate/impede ADD in achieving effective outcomes (including the effectiveness of the DD Network programs)?	Interviews with ADD staff ED web-based survey Federal disability-related partners National Disability organizations Projects of National Significance	Review and consolidation of interview summaries of ADD staff interviews Descriptive analysis of web-based survey Review and consolidation of interview summaries of Federal disability-related partner interviews; descriptive analysis Review and consolidation of interview summaries from PNSs; descriptive analysis

3. Methods to Maximize Response Rates and Deal with Nonresponse

Identification of individuals to interview have, for the most part been in collaboration with Executive Directors or ADD. There has been excellent response, but when the contractor has been unable to interview a person who was selected, Executive Directors and ADD were asked for another name.

In the case of the web-based survey, Executive Directors have been asking ADD to include an assessment of ADD as part of the National Study of the ADD Programs for some time. Thus, ADD expects very good response to this survey. Nevertheless, within two weeks of distributing the questionnaire, Westat will follow up non-respondents by email to remind them to complete the survey. After two more weeks, another email will be sent, and after one more week, Westat will follow up with a telephone call.

We will keep track of those who do not respond to determine whether there are types of programs that have low response (e.g., region, allotment size, medical schools vs. non-medical schools for the UCEDDs). If we find significant differences in the types of programs that respond, we will control for this in our analysis by analyzing data separately according to the types of programs with low and high response. There are no plans to impute data.

4. Test of Procedures or Methods to be Undertaken

Westat staff worked with three DD Network program Executive Directors to develop the web-based questionnaire. Thus, no cognitive testing was undertaken.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

ADD

Jennifer Johnson

202-690-5982

Jennifer.johnson@acf.hhs.gov

Westat

Lynn Elinson

412-421-8610

lynnelinson@westat.com