Part B

Health Resources and services administration

maternal and child health bureau

Combating Autism Act Initiative Data Collection

Part B

B. Collection of Information Employing Statistical Methods

B.1. Respondent Universe and Sampling Methods

The respondent universe for the collection of data is all grantees receiving Health Resources and Services Administration (HRSA) grant support under the Combating Autism Act Initiative (CAAI). This includes:

• 43 Leadership Education in Neurodevelopmental Disabilities (LEND) grantees

10 Developmental Behavioral Pediatrics (DBP) training program grantees

• 23 Research Program grantees

  • 3 Research Networks (one focused on physical/medical health; one focused on behavioral, mental, social and/or cognitive health, and one focused on developmental pediatrics)

  • 20 R40 Grantees

• 13 State Implementation Program grantees

• 2 Resource Centers (the resource centers provide technical assistance to the grant programs and disseminate information from CAAI.

Because of the small number and unique nature of each grant program, a census of all programs will be conducted. The findings will not be generalized beyond the respondents.

B.2. Procedures for the Collection of Information

Information will be collected from each grantee in a number of ways. First, each grantee will submit specific program performance measures on an annual basis to the HRSA program officer. These measures have already been cleared through OMB. Second, information in the form of qualitative data will be garnered from semi-structured interviews with the Project Director from each grant program. Third, grantee project directors will complete the study questionnaires which provide detailed quantitative information regarding project implementation.

The interviews will provide data pertaining to the grantees’implementation activities (e.g., training of MCH providers) and the intermediate results of those activites (e.g., screening/evaluation of underserved children by trained fellows or faculty), which are expected to lead to the long term goal of early screening, early diagnosis and early treatment of individuals with ASD.

The study is designed to measure the grantees’ progress in meeting the five objectives of MCHB’s Combating Autism Act Initiative, which coincide with the goals outlined in the Combating Autism Act legislation. These goals are:

“to increase awareness, reduce barriers to screening and diagnosis, promote evidence-based interventions for individuals with autism spectrum disorder or other developmental disabilities, and train professionals to utilize valid and reliable screening tools to diagnose or rule out and provide evidence-based interventions for children with autism spectrum disorder and other developmental disabilities.”¹

Table B-1 provides a crosswalk of each of the legislative objectives with the contents of the semi-structured interview instrument for each grant program.

TABLE B-1 CROSSWALK OF CAA OBJECTIVES AND EVALUATION MEASURES
How effective are the Combating Autism Act Initiative (CAAI) grant programs in increasing public/provider awareness of ASD and other DD?
Grant Program	Data Source	Data Indicators
LEND	Interviews	What activities has your LEND program undertaken to raise awareness about ASD/DD among providers and the broader community? (If necessary, use the following prompts to probe further. Refer to individual activities for each LEND identified from grantee continuation applications.): ASD/DD General Information What populations have been targeted? What sorts of activities have been implemented? ASD/DD Screening What professional fields have been targeted in these awareness-building efforts (e.g., practicing pediatricians aiming to get the latest information on screening, rural health care providers, social workers, educators)? What topics/content areas did these efforts focus on (e.g., early signs and symptoms, evidence based screening tools)? ASD/DD Early Diagnosis What professional fields have been targeted for increasing awareness of early assessment for ASD/DD? What topics/content areas did these efforts focus on? ASD/DD Treatment & Referral process What professional fields have been targeted for increasing awareness about treatment and referral processes for ASD/DD? What topics/content areas did these efforts focus on? Have you seen any changes in practices or attitudes as a result of increased awareness efforts? (If necessary, use the following prompts to probe further): How has your LEND program worked with State Title V or other agencies to raise awareness about ASD/DD? Please describe any ways in which these partnerships or collaborations have contributed to the goal of increasing awareness of ASD and other developmental disabilities among health professionals, paraprofessionals, family members, advocates, etc). Potential collaboration partners: State Title V Block Grant and State Children with Special Health Care Needs Program, The National Medical Home Autism Initiative (NMHAI), Family Professional Partnerships, Cultural Competence Program, State Implementation Grants for CSHCN, Health Insurance and Financing, Early and Continuous Screening, Healthy and Ready to Work, other.
DBP		One of the objectives of the CAAI is to increase awareness of ASD and other DD among professionals, parents, and others. Can you tell me a little bit about your program’s efforts in this area? How has the CAAI funding impacted these activities? What types of awareness-building activities has your program undertaken? What types of audiences have been targeted in these awareness building efforts (e.g., practicing pediatricians aiming to get the latest information on screening, rural health care providers, social workers, educators)? Approximately how many professionals have been reached? What topics/content areas did these efforts focus on (e.g., early signs and symptoms, evidence based screening tools, DSM 5 changes)? Any efforts developed and targeted to underserved populations? What are some of the challenges to increasing awareness among health and other professionals? What strategies have been devised to overcome these challenges? In general, where do you think your awareness-building efforts have had the most impact? What more remains to be done in order to promote more widespread understanding of ASD and the importance of early detection, evaluation and intervention? Probe: With families? With the underserved?
Research Programs	Interviews	We understand that your Network has also focused on dissemination of information for health professionals and families impacted by ASD and other DD. This will help to increase awareness about improving health and well-being of children and adolescents with ASD and their families. Tell me a little about this work. What are the challenges that the Network has faced with respect to dissemination? [If not already discussed]: What kind of materials have been developed? Who are the intended audiences? How have the materials been disseminated? While not a grant requirement, some grantees are also working on raising awareness with other interested parties, such as parents. To what extent has the Network engaged in raising public awareness? Probes: What kinds of training, information sharing, and other sessions have you held geared towards health professionals? What have been the contents of those sessions? Format of that training or information-sharing (e.g. monthly teleconference?) What other sorts of community outreach and educational sessions have you held? Content of those sessions? Frequency? Audience? [For AIR-P]: [If not already discussed], tell me about the monthly webinar activities. How successful were these efforts?
States	Interviews	Based on a review of your last noncompeting continuation application, I understand that your grant has been working to build awareness of ASD among (identify target populations) by (identify strategy, e.g., customizing and distributing LTSAE materials). Is that accurate? Please describe any collaborations that enabled you to extend the reach of your awareness building efforts. Did you use or build on existing materials, for example, LTSAE materials? Did you work with other entities, like the Family to Family Health Information Centers? Can you tell me a bit about the communication strategies used (e.g., dissemination of print materials, webinars, PSAs)? Have any of your awareness building activities focused specifically on reducing disparities in screening rates, diagnostic referrals or early intervention services? Please explain. Have any of your efforts focused on raising awareness about family-centered medical homes for children with ASD/DD? If so, what audiences have you targeted? Have any of your efforts focused on raising awareness about transition services – either where families can find such services, or the need to develop more comprehensive transition services for young adults with ASD/DD? Please describe any efforts to measure changes in awareness among your target population(s). To what extent would you say your grant has successfully achieved what it set out to accomplish with respect to raising awareness of ASD and other DD? Were there specific challenges that hindered your progress? What work remains to be done?
How effective are the CAAI grant programs in reducing barriers to screening and diagnosis?
LEND	Interviews	What are the most significant barriers to screening and diagnosis in the communities you serve? What specific activities has your LEND program undertaken to address these barriers? Which activities do you think have been most effective so far? Have you seen any changes in children’s access to screening and diagnostic services? How are changes in access being tracked? What is your LEND program doing to reduce the continuing racial and ethnic disparities in identification of ASD and referral for intervention services? Was this a continuation of previous efforts, or a new activity? Have you seen any indication that more underserved children are receiving the services they need?
DBP	Interviews	What are the most significant barriers to screening, diagnosis and intervention in your area or region? Which of these barriers, if any, has your DBP program been working to address since receiving the CAAI grant? Have there been any efforts to track changes in access to screening and diagnostic services? To what extent has your DBP program reached children from underserved populations with respect to (a) screening, (b) diagnostic, and (c) intervention services (or referral to intervention)? Have you developed or carried out any specific plans to reduce barriers to early intervention among underserved children? Probe: Any preliminary results you may have observed from these activities?
States		What are the most significant service barriers in your State? Which of those are being addressed through your grant? What strategies are you using to address them? Probe on following topics if grantee reported any of the following: Medical home initiatives Learning collaboratives Establishing patient registries Insurance coverage for services To what extent have your efforts in this area led to improved access to early screening, diagnostic and intervention services for ASD/other DD? What efforts, if any, are in place to track screening and referral rates? Is there any evidence of improved access to early screening and diagnostic services? What efforts have been made, or what efforts do you think are needed to ensure that any advances achieved during the grant period are sustained?
How effective are the CAAI grant programs in supporting research on evidence-based interventions for children and adolescents with ASD and other DD?
LEND	Interviews	Are any of your faculty members or trainees currently involved in planning or conducting any research studies relating to ASD and related developmental disabilities? If yes, please describe the study or studies. What is being investigated? Who is doing the research? How do you anticipate the results might be used (e.g., will they provide evidence about an intervention’s effectiveness? What is the funding source? Does CAAI funding support these projects? [If not discussed above.] Are trainees involved in research? In what capacity? What sort of training do they receive in conducting research? With whom do you collaborate on research projects? For example, other departments in the university? Local hospitals? Schools?
DBP	Interviews	Are any of your faculty members or trainees currently involved in planning or conducting any research studies relating to ASD/DD? If yes, please describe the study or studies. What is the anticipated impact of the results/how might they be used (e.g., will they provide evidence about an intervention’s effectiveness)? How might they contribute to evidence-based practices or guidelines? How is the research being funded? Any leveraging of funds? [If not already discussed:] Are trainees involved in research? In what capacity? What sort of training do they receive in the research process? With whom do you collaborate on research projects? (Probe: Other departments in the university? Local hospitals? Schools?) We’re also interested in activities related to translating research into practice. [If not already discussed:] Are faculty or trainees in your program engaged in any translational research related to ASD/DD? How has the CAAI funding impacted these activities? To what extent have the results of these research activities been disseminated? Any partnerships or collaborations that contributed to these activities? Funding source? Any leveraged funding? To what extent has this research been translated into policy or practice at the local, regional, or federal level? Has your DBP program undertaken or planned to undertake any other research activities focused on translating research into practice, such as research guidelines or case studies?
Research Programs	Interviews	We understand that your grant has focused on research that addresses the [physical/behavioral/developmental] health needs of children and adolescents with ASD. Tell me a little bit about the network’s processes. What strategies do you use to develop and fund or obtain funding for research concepts and proposals? Probes: How successful have these strategies been in conducting your research? What worked best? What would you change? [look for Successes/positive aspects – shortcomings and negative aspects] Have you had any challenges in publishing the research? Thinking about the development of the research products, what are some of the challenges you have faced? Can you describe the ways in which parents and families are involved in your research? What contributions did they make? What, if anything, did you change as a result of family input? You have noted that you are working on [INSERT NUMBER] studies. Is our understanding accurate? [ASK ANY REMAINING QUESTIONS ABOUT THE PURPOSE OF THE STUDIES] Are there any other studies you are working on that are funded as part of your CAAI grant? [INSERT NUMBER] of these are studies that have continued from the last grant period, correct? [INSERT NUMBER] of these use leveraged funds from other sources? Which sources? Can you describe whether and how your research studies target underserved populations? Which populations? What steps, if any, did you take/are you taking to ensure the participation of underserved and minority populations, in the design and execution of study protocols?
How effective are the CAAI grant programs in promoting the development of evidence based guidelines for interventions?
Research Programs	Interviews	You have noted that you are working on [INSERT NUMBER] guidelines. Is our understanding accurate? [ASK ANY REMAINING QUESTIONS ABOUT THE PURPOSE OF THE GUIDELINES] [INSERT NUMBER] of these are guidelines that have continued from the last grant period, correct? [INSERT NUMBER] of these have been released, is that correct?
How effective are the CAAI grant programs in training professionals to utilize valid screening/diagnostic tools and provide evidence based interventions?
LEND	Interviews	For continuing grantees: As a continuing LEND program grantee, we are most interested in your program’s activities and outcomes since the fall of 2011. How has the continued CAAI funding influenced your program’s training capacity? For example, has the funding enabled you to support a larger number of trainees? Has it opened up new clinical opportunities for trainees? What specific enhancements have been implemented since 2011? Potential follow-up question: Have you seen changes in the fields or disciplines the trainees come from? How has the curriculum changed for LEND trainees since fall 2011? Has the didactic curriculum for long or medium term trainees been modified to include any new content, especially content specific to ASD? Has there been any change in the kinds of clinical training curriculum or opportunities that are available to trainees? For new grantees (AK, AZ, GA, MUSC, NV, TX): How has the grant influenced the training of professionals at your institution? For example, what specific changes or enhancements were implemented with the CAAI funding? For all grantees: What training activities have been particularly effective in building trainees’ expertise in diagnosing/ruling out ASD and other DD? Thinking about how the ASD-related training has evolved over time, can you describe any training activities that have been discontinued, either because they were ineffective in preparing professionals or because they were based on outdated information? Thinking about the training offered through your LEND program, and more specifically, the training related to identification of ASD in children, what would you say is the program’s most important contribution to reaching the goal of early identification and intervention for ASD? Overall, have you seen any other changes with respect to the impact of the LEND program on the training of professionals to screen, diagnose, and/or treat ASD/DD? Potential follow up questions on training in: Cultural competence Medical homes Incorporation of family members into training Transition to adulthood
DBP	Interviews	For continuing grantees: You are a continuing CAAI DBP grantee. For the purposes of the current evaluation, we are interested in how the grants have influenced your program since 2011. How has the continued CAAI funding influenced the training offered through your program? For example, has the funding enabled you to support a larger number of trainees, or has it opened up new clinical opportunities for trainees? What specific enhancements have been implemented since 2011? To what extent do you think you could have implemented these changes or enhancements if you did not have CAAI funding? Has the number of long, medium, or short term trainees changed? Who are the trainees and what disciplines do they come from? Has the didactic curriculum changed? Has it been modified to include any new content, especially content specific to ASD? Probe: Valid and reliable screening and diagnostic tools? Interventions? Transition to adulthood? Have you incorporated any new training opportunities that focus specifically on screening tools, diagnostic instruments, or interventions, particularly for ASD? Probe: Instruments covered? Type of training? Audience? [If not already discussed]: Have the clinical opportunities for trainees changed as a result of the CAAI funding? Any new clinical opportunities, either in hospital-based settings, schools, or other community settings? When describing some of the important elements of the DBP training program, previous grantees discussed the program’s emphasis on training professionals to provide advocacy, counseling and coordination of care. To what extent does your program emphasize these aspects of training? How does the program prepare trainees to provide advocacy, counseling, and coordination of care? Probe: How do trainees learn about this aspect of DBP? How have the CAAI grants impacted this component of training? To what extent has your DBP program engaged family members in the care process? Have any assessments been made with respect to the effectiveness of those efforts? What training activities have been particularly effective in building trainees’ expertise in screening and diagnosis of ASD/DD? Thinking about the training offered through your DBP program, and more specifically, the training related to identification of ASD in children, what would you say is the program’s most important contribution to reaching the goal of early identification and intervention for ASD? What role has the CAAI funding played in those changes?
States		From what I understand, your grant has focused on training (identify target audiences) to (identify goal of training initiatives). Would you say these were the primary training-related activities, or were there other activities I haven’t mentioned? How did you identify this population as one in need of training? What was the intended outcome or long term goal of your grant’s training efforts? To what extent would you say the training initiatives have brought the State closer to that goal? What metrics are you using to assess progress toward the goal? [If not already discussed]: How have you engaged families in your training efforts? If there is a LEND in the State or in a neighboring State: Can you describe any efforts to collaborate with the LEND program in your (State/area) on training-related initiatives? How did your training efforts complement the training offered through LEND or other professional training programs? Can you describe any aspects of the training that focused on: Cultural competency? Coordination of care? Family-centered care?

B.3. Methods to Maximize Response Rates and the Issue of Non-Response

All grantess of the HRSA grant funded programs involved in the CAAI are required, as a condition of their grants, to provide the requested information to HRSA, so nonresponse will not be an issue. HRSA will provide that information to the Insight Policy Research (the contractor).

B.4. Tests of Procedures

The semi-structured interview protocols were not pretested, but all procedures were developed with input from the HRSA project officers and HRSA grantees. The protocols were designed with this input in mind, and follow a standard approach to ensure that high quality data are obtained across all grant programs.

B.5. Consultants

No additional consultants are envisioned for data collection or data analysis.

1 Public Law 109-416§42 U.S.C.§1