Supporting Statement B- MCHB- PMHCA- New- 04302024

Supporting Statement B

Health Resources and Services Administration Maternal and Child Health Bureau Pediatric Mental Health Care Access Program National Impact Study

OMB Control No. 0906-XXXX

2. Collection of Information Employing Statistical Methods

This section presents information about the collection of data for the Health Resources and Services (HRSA) Maternal and Child Health Bureau (MCHB) Pediatric Mental Health Care Access (PMHCA) Impact Study (Impact Study). As noted in Supporting Statement A, the Impact Study's purpose is to guide future program decisions regarding (1) the impact of HRSA’s PMHCA program on changes in children/adolescents’ and families/caregivers’ access to behavioral health care; their subsequent receipt and utilization of behavioral health care services, including culturally and linguistically appropriate care; and related behavioral health impacts, and (2) monetary and societal PMHCA program costs and benefits. A comprehensive, multimethod data collection effort is proposed to address the unavailability of (1) secondary sources of quantitative data essential to understanding PMHCA enrolled/participating health professionals' (HPs) experiences with screening, diagnosing, treating, and referring children/adolescents with behavioral health conditions and (2) qualitative data essential to understanding the family/caregiver perspectives regarding their experiences with behavioral health care services for their children/adolescents.

1. Respondent Universe and Sampling Methods

Exhibit 1 describes the potential respondent universe for each data collection tool. The respondent universe for the HP Impact Survey will comprise identified enrolled and/or participating HPs from all 24 2021 PMHCA awardees, 5 2022 PMHCA awardees, and 25 2023 PMHCA awardees. The reason we elected to collect data from the respondent universe rather than using sampling is that, in actual practice, some sampling bias occurs in almost all studies to some extent; given the variability in how states, political subdivision of a state, freely associated states, territories, tribal organizations, or counties are implementing their programs, a sample may not accurately represent enrolled/participating HPs. Selecting the universe will provide the largest number of respondents and be more appropriate for drawing comparisons to regional- and national-level data.

Specifically, for the HP Impact Survey, obtaining responses from all participating HPs is preferred over sampling to allow for examination within a state, political subdivision of a state, freely associated state, territory, tribal organization, or county and/or across PMHCA programs regarding (1) screening, diagnosing, treating, and referring children and adolescents with behavioral health conditions among enrolled/participating HPs; (2) perception of behavioral health impact; and (3) first and last names and practice ZIP Code(s) to link their data with other data sources (i.e., Medicaid claims data) for the purpose of identifying impacts of the PMHCA program on access to behavioral health care.

Sampling methods will be used for the Family/Caregiver FGD. Each of the 54 PMHCA awardees will identify and select a minimum of 2 and a maximum of 5 families/caregivers to participate. Impact Study staff will call all identified families/caregivers and ask them a brief background questionnaire (Family/Caregiver Demographic Questionnaire) that will collect demographic information (e.g., child age, participant’s ZIP Code), which will then be used to inform the Family/Caregiver FGD sampling groups (e.g., regional, child’s age, rural/non-rural). Based on the identified sampling groups, families/caregivers will be randomly invited to participate within their sampling group.

See Section B.3 for information regarding expected response rates for the data collection activities. Data collection tools are included in Attachments B1–B3 and consent forms for each tool are included in Attachments B4-B5.

Exhibit 1: Potential Respondent Universe

Form Name	Number of Entities in the Universe
HP Impact Survey	Based on awardee estimates of HPs to be enrolled/participating in the PMHCA program; 21,070 HPs are eligible to be surveyed.
Family/Caregiver FGD	42 individuals (up to 6 FGDs with between 5 and 7 participants)
Family/Caregiver Demographic Questionnaire	270 individuals (up to 54 PMHCA awardees identifying a maximum of 5 families/caregivers to participate in the Family/Caregiver FGD)

2. Procedures for the Collection of Information

To conduct the evaluation, data will be collected using three data collection methods (i.e., surveys, FGDs, questionnaires). Each data collection method supports the Impact Study’s goals and objectives. The data collection process will follow a systematic, mixed methods data collection approach to gathering high-quality data from each 2021, 2022, and 2023 PMHCA cooperative agreement-funded program’s participating/enrolled HPs and families/caregivers who have sought and/or received behavioral health care for their children/adolescents.

We will use surveys and FGDs to collect data from the target population. A general description of key data collection procedures is provided below. Emails referenced in the sections below are included as attachments in Supporting Statement A.

HP Impact Survey. The HP Impact Survey will be administered annually to HPs enrolled/participating in 2021, 2022, and 2023 PMHCA programs, with anticipated data collection in 2024 and 2025. The survey will take 10 minutes or less to complete, be administered via a web-based platform (i.e., Alchemer), and sent directly to all HPs via an email with a link to the online survey. At each time point, the survey link will be available for a minimum of 31 days. To facilitate completion, the web-based platform will allow participants to complete the survey in more than one sitting, if needed.

At the time of each survey administration, JBS International, Inc. (JBS) or the awardees (depending on decisions made while coordinating survey administration logistics with each awardee) will send an email to the HPs that contains an introduction to the Impact Study and data collection, the link to complete the survey, and an email address that HPs can use to submit any questions or concerns about the survey. If HPs have difficulties accessing the web-based survey (e.g., due to firewall issues or other technical problems), we will email a fillable and printable PDF and include instructions for completing and returning it.

Family/Caregiver FGD and Demographic Questionnaire. FGDs and questionnaires will be conducted one time in 2025. JBS will email PMHCA program staff to obtain contact information for their program care coordinator or person/organization filling the role, if applicable to the program. Once identified, we will request that each program identify a minimum of two and a maximum of five families/caregivers to participate in the FGDs. Care coordinators (or designees) may contact families/caregivers with whom they have directly worked or for whom they provided information through a primary care health professional engaged in the PMHCA program.

JBS will provide individuals assisting with recruitment messaging to use, including the importance of the data collection to strengthening access to behavioral health care. If interested in participating, families/caregivers will be asked to provide verbal consent, which will allow the Impact Study team to contact them by phone. Once the Impact Study team receives the contact information from all identified family members/caregivers, we will reach out to them to (1) provide additional information about participating, (2) answer any questions about participating, (3) conduct the Family/Caregiver Demographic Questionnaire, and (4) provide instructions for giving informed consent to participate in the FGD and to be recorded. JBS will obtain informed consent from participants in advance of FGD participation via DocuSign.

Using the Family/Caregiver Demographic Questionnaire, JBS will create a crosswalk to map the identified families/caregivers to characteristics of interest (e.g., child age, region, rural/non-rural) that will inform the FGD sampling groups and scheduling of the FGDs. Based on the identified sampling groups, families/caregivers will be randomly invited to participate within their sampling group and provided at least 3 dates and times during a 2-week period to facilitate scheduling the FGD.

If we do not hear back on the initial request, we will follow up by email or phone. If the family/caregiver does not respond to the second email/phone call, JBS will inform HRSA and reach out to the next randomly selected family/caregiver within their sampling group to share participation details and to facilitate scheduling. We will employ methods to reduce the burden on participating families/caregivers (e.g., giving advance notice of timing of the FGD).

Two-person teams will conduct the FGDs in English, which will be recorded for notetaking and transcription purposes. Impact Study team members with experience conducting FGDs will facilitate them, presenting the purpose of the FGD and guiding the discussion to gather information on the topics of interest. The moderator will use the written Family/Caregiver FGD guide to conduct the discussion, and a note-taker from the Impact Study team will take detailed notes.

Monetary incentives will be offered to the families/caregivers who participate (see Supporting Statement A for more information).

The Family/Caregiver FGD will take about 1 hour to complete and be conducted and recorded via a web-based platform (e.g., Microsoft Teams, Zoom). JBS will transcribe the recorded FGDs to facilitate qualitative content analysis. Recordings will be destroyed after the transcripts are developed and cleaned.

3. Methods to Maximize Response Rates and Deal with Nonresponse

The ability to gain the cooperation of potential respondents is important to the success of the PMHCA Impact Study. For the HP Impact Survey, participating providers are informed of evaluation activities upon participation/enrollment in PMHCA programs. For the Family/Caregiver FGDs, awardees will identify and select participants who are eligible and interested in participating in data collection. JBS will provide awardees with materials (e.g., graphics, PowerPoint slides, newsletter blurbs) to promote both engagement with their programs and data collection. A discussion of methods to maximize response rates for the different data collection methods is presented below.

HP Impact Survey. It is anticipated that the survey will be completed by 20%-30% of HPs asked to participate; this estimate is based on previous experience from the HRSA MCHB Evaluation Project (also conducted by JBS). In the evaluation, a survey of HPs enrolled/participating in the PMHCA program, which collects different data than those proposed in HP Impact Survey, was administered yearly from 2020 to 2023 and had response rates ranging from 18.6% to 24.28%, with an average of 20.5%. To address response rates, following the participation email to HPs for each time point, we will access the web-based system and track surveys each week to assess the number of HPs who have initiated and completed the survey. After sending the initial participation email, JBS or the awardees will send a reminder email to HPs who have not completed the web-based survey and continue to send them at specified intervals to those who have not responded; JBS estimates between four and six reminder emails, depending on completions and on coordination with HRSA. If an awardee has decided to communicate directly with its enrolled/participating HPs during the survey process, JBS will provide the cooperative agreement-funded program with draft email content for all necessary communications discussed above. We will provide HRSA with updates on the response rates during the standing, biweekly contract calls.

Family/Caregiver FGD. It is anticipated that 42 individuals will participate in the FGDs (100% response rate). Based on the identified sampling groups (detailed above), families/caregivers will be randomly invited to participate within their sampling group and provided at least 3 dates and times during a 2-week period to facilitate scheduling the FGD. If we do not hear back on the initial request, we will follow up by email or phone. If the family/caregiver does not respond to the second email/phone call, JBS will inform HRSA and reach out to the next randomly selected family/caregiver within their sampling group to share participation details and to facilitate scheduling. We will employ methods to reduce the burden on participating families/caregivers (e.g., giving advance notice of timing of the FGD). Planning and preparation in advance of the FGDs are crucial to these protocols, which will include proper timing and scheduling of the FGDs to accommodate family members/caregivers. Additionally, as detailed in Supporting Statement A, family members/caregivers who participate in the FGDs will be offered a $25 payment.

Family/Caregiver Demographic Questionnaire. It is anticipated that the demographic questionnaire will be completed by all family members/caregivers who talk to the Impact Study team. We estimate that all individuals will participate because the questionnaire offers the response option “Prefer not to disclose”, allowing individuals to disclose only the information they are comfortable with. Additionally, we have elected to include OMB’s SPD-15 race and ethnicity question with minimum categories only (i.e., American Indian or Alaska Native, Asian, Black or African American, Hispanic or Latino, Middle Eastern or North African, Native Hawaiian or Pacific Islander, White) to reduce the burden on participants as the demographic data collected will primarily be used for FGD sampling.

To ensure each identified family/caregiver has an equal opportunity to be included in a sampling group, if a family member/caregiver does not respond to the initial phone call, we will attempt to contact them one additional time (two total contacts). Additionally, if an email address is provided, we will attempt to email them to schedule a date and time to conduct the initial phone call.

4. Tests of Procedures or Methods to be Undertaken

JBS conducted pilot tests of the data collection tools (i.e., survey and FGD) to be used in the Impact Study with:

• Representative subsamples of the target populations (i.e., HPs, families/caregivers)

• Representatives with experience in health, public health, and/or behavioral health research and evaluation, including study design; instrument development; and data collection, analysis, and reporting

• Medical advisors (e.g., physicians specializing in family, preventive, and/or addiction medicine) with current or previous practice in clinical practice sites (e.g., private practice, non-academic hospital-based practice)

• Individuals with positions in academic settings

All pilot tests were conducted with nine or fewer individuals.

HP Impact Survey. Pilot test participants were asked to complete the survey questions via the same online platform that will be used for survey administration and to provide feedback on the (1) time needed to complete the survey, (2) clarity and format of the survey questions, and (3) any technology difficulties experienced taking the survey.

Family/Caregiver FGD. Participants were asked to review the questions and to provide feedback on the (1) clarity and appropriateness of the questions and procedures for the intended audience, (2) estimated amount of time to complete the FGD, and (3) any foreseen challenges with conducting virtual FGDs.

Attachments B6–B7 provide a summary of pilot test feedback for each data collection tool tested and outline the changes made to the data collection tools based on this feedback.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

As noted in Section A.8, consultations on the Impact Study’s key research questions, design, data collection instruments (i.e., HP Impact Survey, Family/Caregiver FGD) and protocols, data management, data quality plan, and analysis occurred throughout Year 1 of the project, in 2023. These consultations have provided, and will continue to provide, the opportunity to ensure the technical quality and appropriateness of the overall Impact Study design and data analysis plans, obtain advice and recommendations concerning the data collection instruments, and structure the Impact Study and instruments to minimize overall and individual response burden. Consultations occurred with the members of an External Partner Group (EPG) established for the Impact Study, as well as The RAND Corporation (subcontractors to JBS for the Impact Study). See Exhibit 2 below.

Exhibit 2: Consultants

Name and Title	Affiliation	Contact Information	Years and Areas of Consultation
Non-Federal EPG Members
Amie Bettencourt, PhD, MS, Assistant Professor, Clinician (EPG Chair)	Johns Hopkins Medicine, Department of Psychiatry and Behavioral Sciences; Maryland PMHCA; National Network of Child Psychiatry Access Programs (NNCPAP) member	abetten3@jhu.edu	Consultation Year: 2023 Reviewed/Consulted on: Formative research (e.g., literature review, existing data sources) Key research questions Study design Data quality plan Primary data collection instruments
Shannon Dial, LMFT, Executive Officer	Integrated Service Division; Chickasaw Nation PMHCA	Shannon.Dial@chickasaw.net	Consultation Year: 2023 Reviewed/Consulted on: Formative research (e.g., literature review, existing data sources) Key research questions Study design Data quality plan Primary data collection instruments
Julie Gorzkowski, MSW, LSW, Director	American Academy of Pediatrics	JGorzkowski@aap.org	Consultation Year: 2023 Reviewed/Consulted on: Formative research (e.g., literature review, existing data sources) Key research questions Study design Data quality plan Primary data collection instruments
Joy Hogge, PhD, MS, Executive Director	Families as Allies	jhogge@faams.org	Consultation Year: 2023 Reviewed/Consulted on: Key research questions Study design Data quality plan Primary data collection instruments
Laura Hurwitz, LCSW	Consultant in school health, mental health, and social and emotional learning	Laura@laurahurwitz.net	Consultation Year: 2023 Reviewed/Consulted on: Formative research (e.g., literature review, existing data sources) Key research questions Study design Data quality plan Primary data collection instruments
David Lynch, MA, Senior Research Analyst	University of Texas (TX) at Austin; TX Institute for Excellence in Mental Health	david.lynch@austin.utexas.edu	Consultation Year: 2023 Reviewed/Consulted on: Formative research (e.g., literature review, existing data sources) Key research questions Data quality plan Primary data collection instruments
Catherine Maclean, PhD, Health and Labor Economist, Associate Professor	George Mason University; National Bureau of Economic Research; Institute for Labor Economics	jmaclea@gmu.edu	Consultation Year: 2023 Reviewed/Consulted on: Formative research (e.g., literature review, existing data sources) Key research questions
John Straus, MD, Founding Director	Massachusetts PMHCA, NNCPAP	John.Straus@beaconhealthoptions.com	Consultation Year: 2023 Reviewed/Consulted on: Formative research (e.g., literature review, existing data sources) Key research questions Study design Data quality plan Primary data collection instruments
Federal EPG Members
Moushumi Beltangady, MPP, MSW, Senior Policy Advisor (SPA), Director	Tribal Early Childhood; Tribal Home Visiting Program; Office of Early Childhood Development; Administration for Children and Families	moushumi.beltangady@acf.hhs.gov	Consultation Year: 2023 Reviewed/Consulted on: Formative research (e.g., literature review, existing data sources) Key research questions
Julie Donney, PhD, MPH, Public Health Analyst (PHA)	HRSA MCHB Office of Epidemiology and Research (OER)	JDonney@hrsa.gov	Consultation Year: 2023 Reviewed/Consulted on: Formative research (e.g., literature review, existing data sources) Key research questions Study design Data quality plan Primary data collection instruments
Chris Dykton, MA, Deputy Director	HRSA MCHB Division of State and Community Health	CDykton@hrsa.gov	Consultation Year: 2023 Reviewed/Consulted on: Formative research (e.g., literature review, existing data sources) Key research questions
Kate Ginnis, MSW, MPH, SPA	Centers for Medicaid & Medicare Services; CHIP Services Office of the Center Director	katherine.ginnis@cms.hhs.gov	Consultation Year: 2023 Reviewed/Consulted on: Formative research (e.g., literature review, existing data sources) Key research questions
Rui Li, PhD, Director	HRSA MCHB OER	rli@hrsa.gov	Consultation Year: 2023 Reviewed/Consulted on: Formative research (e.g., literature review, existing data sources) Key research questions Study design Data quality plan Primary data collection instruments
Dana Mason, PhD, MS, Senior PHA	HRSA MCHB Office of the Associate Administrator	DMason@hrsa.gov	Consultation Year: 2023 Reviewed/Consulted on: Key research questions Study design Data quality plan Primary data collection instruments
Nicole Pascua, MPH, LICSW, Lieutenant Commander	Substance Abuse and Mental Health Services Administration; National Center of Excellence for Integrated Health Solutions	Nicole.Pascua@samhsa.hhs.gov	Consultation Year: 2023 Reviewed/Consulted on: Formative research (e.g., literature review, existing data sources) Key research questions
RAND
Stephanie Rennane, PhD	RAND	srennane@rand.org	Consultation Year: 2023 Reviewed/Consulted on: Formative research (e.g., literature review, existing data sources) Key research questions Study design Data quality plan Primary data collection instruments
Mark Sorbero, MS	RAND	sorbero@rand.org	Consultation Year: 2023 Reviewed/Consulted on: Formative research (e.g., literature review, existing data sources) Key research questions Study design Data quality plan Primary data collection instruments
Bradley Stein, PhD	RAND	stein@rand.org	Consultation Year: 2023 Reviewed/Consulted on: Formative research (e.g., literature review, existing data sources) Key research questions Study design Data quality plan Primary data collection instruments

JBS staff designed the data collection instruments and will lead the data collection and analysis efforts. The EPG members and RAND provided consultation on the data collection design, and RAND will provide support on the data collection analysis.

5