Supporting Statement Part B - 6-14-2011

Supporting Statement Part B - 6-14-2011.docx

Developmental Disabilities Program Independent Evaluation Project

OMB: 0970-0372

Document [docx]
Download: docx | pdf

B. Statistical Methods (used for collection of information employing statistical methods)

  1. Respondent Universe and Sampling Methods



    1. Full-scale Study


Programs will be selected using stratified random sampling procedures. The only stratification variable will be four geographic region designated by the U.S. Census: Northeast, Midwest, South, and West. Three territories will be included – District of Columbia, Puerto Rico, and Guam – because they each have all three DD Network programs. District of Columbia and Puerto Rico will be included in Region 3 (South). Guam is included in Region 4 (West) for sampling



Table B-1 below shows the individuals that will be interviewed within each selected program.



Table B-1. Number sampled in each respondent category and potential respondent universe


Respondent Category

Potential Respondent Universe

Sample Number

DD Council: Executive Director Interview

55

20

DD Council: Interview with Council Chair/Council Members

440

60

DD Council: Group Interview with Policymakers, Collaborators, and Grantees

5,500

160

DD Council: Group Interview with Recipients of

Self-Advocacy and Leadership Education and Training

2,750

100

DD Council: Group Interview with Recipients of

Education and Training to Improve Community

Capacity

2,750

100

DD Council:

Self-administered Form

55

20

P&A: Executive

Director Interview

57

20

P&A: Staff Interview

855

60

P&A: Board of Directors (Commissioners)-Chair and Members

1,140

60

P&A: Group Interview with Policymakers and

Collaborators

5,700

160

P&A: Interview with Recipient of Community

Education

5,700

100

P&A: Interview with Clients

5,700

100

P&A: Self-administered Form

57

20

UCEDD: Interview with Director

68

20

UCEDD: Telephone Interview with Current

and Graduated Students

3,400

100

UCEDD: Interview with the Consumer Advisory

Committee

1,360

60

UCEDD: Interview with Peer Researchers and

Colleagues

3,400

100

UCEDD: Interview with Recipients of Community Services or Members of

Organizations/Agencies that are Trained to Provide Community Services

500,000

100

UCEDD: Self-administered Form

68

20





    1. ADD Assessment

The ADD assessment consists of interviews with ADD staff and stakeholders. Representatives from the following groups will be interviewed:

  • ADD staff

  • Federal disability partners

  • National disability organizations

  • Projects of National Significance

  • Executive Directors from all DD Network programs in each state and territory

The ADD Commissioner and all ADD program staff and management were interviewed. The contractor worked with the ADD Commissioner, Deputy Commissioner, and Project Officer of the contract to identify appropriate individuals in Federal disability-related agencies and National disability organizations (thus, no sampling). Up to 9 individuals will be interviewed in each group. Projects of National Significance were grouped into categories (Emergency Preparedness; Family Support and Community Access Demonstrations; National Autism Resource and Information Center; Voting Project; Family Support 360; Youth Information, Training, and Resource Center; Ongoing Data Collection). Two projects were sampled from Family Support 360 and one project was sampled from each of the remaining categories. The project director of each selected project will be contacted for an interview.


All Executive Directors (180) will be requested to complete a web-based survey, so there will be no sampling for this group.



  1. Procedures for the Collection of Information


    1. Full-scale study


Data was collected using semi-structured interview guides to interview key informants identified with the help of Executive Directors of each program. Interviews were transcribed and summarized.


Analysis will consist of examining both quantitative and qualitative data collected on the 60 programs in 20 states and summarizing findings regarding structural, process, output and outcome indicators for each program and collaboration among the three programs. The focus of data analysis and synthesis will be on the extent to which DD Network programs are achieving outcomes. However, the contractor will also examine inputs, processes, and outputs to determine some of the factors that may contribute to falling short of achieving these outcomes and the factors that may be facilitating their achievement.


Analysis of in-person interview data, telephone interviews, materials obtained from programs will be qualitative. Data from the self-administered questionnaires are both quantitative and qualitative. In either case, a descriptive analysis, focusing on the research questions (Exhibit B-1) will take place.


Exhibit B-1. Research questions for the full-scale study


Structural Indicator questions:

  1. What are the typical structural elements that enable and/or impede the individual entities of the DD Network (e.g., State Council, P&A, and UCEDD) to engage in activities that help meet the purpose and principles of the DD Act?

  2. What are the typical structural elements that enable and/or impede the collaborative activities of the State DD Networks that support achievement of the DD Act purpose and principles?


Process Indicator questions:

  1. What are the typical activities of the DD Network programs (e.g., State Council, P&A, and UCEDD) that contribute to achievement of the purpose and principles of the DD Act?

  2. What are the typical collaborative activities of the State DD Networks that support achievement of the purpose and principles of the DD Act?


Output Indicator questions:

  1. What are the typical immediate results of the activities of the DD Network programs (e.g., State Council, P&A, and UCEDD) that contribute to achievement of the purpose and principles of the DD Act?

  2. What are the typical immediate results of the collaborative activities of the State DD Networks that support achievement of the DD Act purpose and principles?


Outcome Indicator questions:

  1. What are the typical short term outcomes achieved through the activities of the DD Network programs (e.g., State Council, P&A, and UCEDD)?

  2. What are the typical short term outcomes achieved through the collaborative activities of the State DD Networks?


Effectiveness questions:

  1. What is the relationship between the outcomes achieved by the DD Network programs (e.g., State Council, P&A, and UCEDD) and the processes they used to affect those outcomes?

  2. What is the relationship between the outcomes achieved by the State DD Networks and the processes used to affect those outcomes?

  3. To what extent have the DD Network programs (e.g., State Council, P&A, and UCEDD) been effective in achieving the purposes and principles of the DD Act?

  4. To what extent have the State DD Networks been effective in achieving the purposes and principles of the DD Act?

  5. What efficiencies are being used by DD Network programs (e.g., State Council, P&A, and UCEDD) to obtain high quality outputs and achievements?

  6. What efficiencies are being used by the State DD Network obtain high quality outputs and achievements?




    1. ADD Assessment


The design of the ADD assessment is essentially a collection of retrospective and current data that relate to ADD’s structures, processes, outputs, and outcomes to draw conclusions about influences on the efficiency and effectiveness of DD Network programs. We define “effectiveness” as the ability to meet objectives. Thus, the ADD assessment will examine ADD’s ability to meet its objectives, from the perspective of a variety of stakeholders (including Executive Directors of DD Network programs), the DD Network programs’ ability to meet their objectives, and the relationship between the two.


Westat will use a semi-structured interview guide to obtain information from ADD staff, Federal agency partners, national disability organization staff, and staff from Projects of National Significance (discretionary grants funded by ADD. Up to nine individuals per category will be identified with assistance from ADD. Interviews will be conducted by telephone.


The contractor, Westat, will obtain budget and cost allocation data from ADD at three points in time (e.g., 2005, 2007, and 2009) to assess changes in some of the key structures (inputs) for ADD. Data collected through interview and by web-based questionnaire will relate to a recent period (e.g., 2009 and 2010).


Program materials will be obtained from ADD or the ADD website and reviewed by Westat staff.


To administer the web-based questionnaire, the Commissioner of ADD will send a letter to Executive Directors informing them that they will be contacted by Westat to participate in a web-based survey. Westat will then send an initial email to all Executive Directors to explain the purpose of the ADD assessment and provide a username and password to gain access to the questionnaire (Appendix A). Westat will also attach a separate file that contains instructions.


Data from each stakeholder group will be analyzed separately and then integrated to answer specific research questions (Table 4). Analysis of this qualitative data will primarily be descriptive. We will also conduct a descriptive analysis of the data extracted from background materials. Some data will be transformed into tabular format (e.g., to demonstrate budget appropriations or cost allocations over time).



Table B-2. ADD Assessment Analysis


Research questions

Source of data

Analysis

  1. What structures does ADD have in place to administer the DD Act and compliance with the Act (e.g., policies, standards, data collection requirements, technical assistance, funding, and membership on Federal disability-related interagency committees)?

ADD documents


Interviews with ADD staff

Document review


Review and consolidation of interview summaries

    1. What are the strengths and weaknesses of these structures?

Interviews with ADD staff


Executive Director (ED) web-based survey

Review and consolidation of interview summaries


Descriptive analysis of web-based survey

  1. How have the ADD structures (e.g., appropriations from Congress) changed over time?

ADD documents


Interviews with ADD staff

Review of ADD documents


Review and consolidation of interview summaries

  1. How are costs allocated to each ADD process?

ADD documents


Interviews with ADD staff

Review of ADD documents


Review and consolidation of interview summaries

  1. How does ADD develop policies?

Interviews with ADD staff


Federal disability partners

Review and consolidation of interview summaries

  1. How does ADD monitor compliance with its policies?

ADD documents


Interviews with ADD staff


ED web-based survey

Document review


Review and consolidation of interview summaries


Descriptive analysis of web-based survey

  1. How does ADD disperse funds?

Interviews with ADD staff

Review and consolidation of interview summaries

  1. What factors have an effect on ADD processes?

Interviews with ADD staff


ED web-based survey

Review and consolidation of interview summaries


Descriptive analysis of web-based survey

  1. How does ADD provide technical assistance?

Interviews with ADD staff


ED web-based survey

Review and consolidation of interview summaries


Descriptive analysis of web-based survey

  1. What are the strengths and weaknesses of ADD processes?

  • ADD staff


National disability organizations


  • ED web-based survey


Review and consolidation of interview summaries


Descriptive analysis of web-based survey

  1. What are the products of ADD monitoring (e.g., reports to ADD)?

ADD documents


Interviews with ADD staff

Document review


Review and consolidation of interview summaries

  1. To what extent are DD Network programs prepared to meet their goals and comply with the DD Act?

Program visits to 60 DD Network programs


Telephone interviews with DD network program stakeholders


Self-administered questionnaires


Program materials


Web-based survey

Integration of findings from all four sources of data in the full-scale study


Descriptive analysis of web-based survey

  1. What are the strengths and weaknesses of each product?

ADD documents


Interviews with ADD staff


ED web-based survey

Review of documents


Review and consolidation of interview summaries


Descriptive analysis of web-based survey

  1. To what extent is ADD effective in being publicly accountable?

Interviews with ADD staff


ED web-based survey


Federal disability-related partners


National Disability organizations


Projects of National Significance

Review and consolidation of interview summaries of ADD staff interviews


Descriptive analysis of web-based survey


Review and consolidation of interview summaries of Federal disability-related partner interviews; descriptive analysis


Review and consolidation of interview summaries from PNSs; descriptive analysis

  1. To what extent are DD Network programs receiving the leadership they need from ADD?

ADD staff


ED web-based survey


Review and consolidation of interview summaries of ADD staff interviews


Descriptive analysis of web-based survey

  1. To what extent are reports to ADD useful in monitoring the DD Network programs in compliance with the Act?

ADD staff


ED web-based survey


Review and consolidation of interview summaries of ADD staff interviews


Descriptive analysis of web-based survey

  1. What are the factors that facilitate/impede ADD in achieving effective outcomes (including the effectiveness of the DD Network programs)?

Interviews with ADD staff


ED web-based survey


Federal disability-related partners


National Disability organizations


Projects of National Significance

Review and consolidation of interview summaries of ADD staff interviews


Descriptive analysis of web-based survey


Review and consolidation of interview summaries of Federal disability-related partner interviews; descriptive analysis


Review and consolidation of interview summaries from PNSs; descriptive analysis





  1. Methods to Maximize Response Rates and Deal with Nonresponse

Identification of individuals to interview have, for the most part been in collaboration with Executive Directors or ADD. There has been excellent response, but when the contractor has been unable to interview a person who was selected, Executive Directors and ADD were asked for another name.

In the case of the web-based survey, Executive Directors have been asking ADD to include an assessment of ADD as part of the National Study of the ADD Programs for some time. Thus, ADD expects very good response to this survey. Nevertheless, within two weeks of distributing the questionnaire, Westat will follow up non-respondents by email to remind them to complete the survey. After two more weeks, another email will be sent, and after one more week, Westat will follow up with a telephone call.

We will keep track of those who do not respond to determine whether there are types of programs that have low response (e.g., region, allotment size, medical schools vs. non-medical schools for the UCEDDs). If we find significant differences in the types of programs that respond, we will control for this in our analysis by analyzing data separately according to the types of programs with low and high response. There are no plans to impute data.


  1. Test of Procedures or Methods to be Undertaken

Westat staff worked with three DD Network program Executive Directors to develop the web-based questionnaire. Thus, no cognitive testing was undertaken.


  1. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

ADD

Jennifer Johnson

202-690-5982

Jennifer.johnson@acf.hhs.gov


Westat

Lynn Elinson

412-421-8610

lynnelinson@westat.com


File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorDepartment of Health and Human Services
File Modified0000-00-00
File Created2021-02-01

© 2024 OMB.report | Privacy Policy