Supporting Statement B - AMCS Virtual Site Visits OPRE Pilot with OIRA_Sept 2020 Extension

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Assessing Models of Coordinated Services for Low-Income Children and Their Families (AMCS) Site Visits

OMB Information Collection Request

0970-0535

Supporting Statement

Part B

SEPTEMBER 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Laura Hoard and Ivelisse Martinez-Beck

Part B

B1. Objectives

Study Objectives

The purpose of the AMCS project is to understand how states and local communities are coordinating services related to early care and education (ECE), family economic security, and/or other health and human services across sectors to most efficiently and effectively serve low-income children and their families. As noted in section A2, information gathered through “virtual site visits” will inform ACF leadership and the field on the day-to-day operation of coordinated services approaches, in addition to informing technical assistance for Preschool Development Grant Birth to Five (PDG B-5) grantees. Coordinated services approaches are of keen interest to ACF leadership, as well as the broader field of human services. As such, this study seeks to provide timely descriptive information on state and local coordinated services approaches that are currently operating in the United States. “Virtual site visit” interviews and focus groups will address the research questions listed in Table 1 of section A2.

Generalizability of Results

This data collection is intended to present an internally valid description of coordinated services approaches at the state and local level in chosen sites. Data on program effectiveness will not be collected, and the information will not enable statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

The goal of this data collection is to produce accurate summaries of current coordinated services approaches. The descriptive information collected under this approved Information Collection (IC) will provide timely information about current coordinated services approaches for ACF and its grantees. Qualitative data serve the purposes of the study well, because this is an initial effort to understand coordinated services approaches and identify where states and localities are in their efforts to coordinate.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and Design

Target Population

The target populations for this IC include leadership and staff at, and families who receive services from coordinated services approaches operating at the state and local levels that are coordinating early care and education services with family economic security and/or other health and human services.

We anticipate conducting interviews with up to 30 staff and focus groups with up to 12 parents/guardians from each individual state and local coordinated services approaches.

Sampling and Site Selection

We will identify sites to “virtually visit” based on information compiled through a national scan of publicly available information (not subject to PRA), as well as model profile completion and confirmation, and telephone interviews (activities approved under the overarching generic: Formative Data Collections for ACF Research (0970-0356; 201904-0970-003)).

The coordinated services approaches identified through the scan of publicly available information met the following criteria:

• Currently operates in the United States;

• Has a public website or public documents available for review;

• Serves at least some low-income families;

• Directly provides early care and education services for children age 5 and younger;

• Provides family economic security and/or family-focused health and human services in addition to early care and education; and

• Intentionally coordinates multiple health and human services.

From the universe of state and local coordinated services approaches identified using the six criteria listed above, purposive samples were identified for two activities that are covered under the project’s information collections under the overarching generic: Formative Data Collections for ACF Research:

• State and local model profile confirmation and completion. The team created state and local model profiles for a purposively selected set of state and local coordinated services approaches using publicly available data. We are inviting coordinated services approaches to confirm and complete the state or local model profile, as relevant, for their coordinated services approach.

• State and local telephone interviews. We will selected a purposive sample of 20 coordinated services approaches to invite to participate in a telephone interview based on what we learned through the publicly obtained data in the national scan, as well as during the model profile confirmation and completion. For each coordinated services approach, the telephone interview involved a group of staff with different roles in administering the coordinated services approach.

“Virtual site visits” will be conducted with six coordinated services approaches at the state or local level.

• “Virtual site visits”, including semi-structured interviews with coordinated services approach staff and parent focus groups. From the set of state and local coordinated services approaches included in the telephone interviews, we will select a purposive sample of six (6) coordinated services approaches from which to invite to participate in “virtual site visits”. These six (6) state and local coordinated services approaches will be selected based on what we learn through the preceding data collection activities. The AMCS team will aim to maximize variation in features of the coordinated services approaches, while also considering what coordinated services approaches might be best able to provide information to answer the study’s research questions (see section B1). We will use the following criteria to help determine the sample of six (6) coordinated services approaches:

• States with:

  • A state coordinated services approach;

  • A local coordinated services approach affiliated with the state approach (local implementation site); and,

  • A local coordinated services approach not affiliated with the state approach.

• State and local coordinated services approaches coordinating three or more different types of services, especially services at the child-level;

• State and local coordinated services approaches that include features of particular interest to ACF and its grantees (e.g. inclusion of early intervention services for children, use of hub model); and,

• Geographic diversity.

At each of the six coordinated services approaches, the AMCS study team will communicate with administrative staff to identify the appropriate staff to participate. Interview respondents will be selected purposively using organizational charts and information on each staff person’s role in the coordinated services approach. Depending on the preferences of the coordinated services approach staff and the schedules/availability of staff, interviews may be one-on-one or conducted with up to 3 staff at a time. We expect to interview up to 30 staff per coordinated services approach (180 staff across six approaches). Three types of staff will be included in interviews:

• Coordinated services approach leaders: includes staff who are executive directors, program directors, CEOs, or staff in similar roles who oversee the entire coordinated services approach.

• Directors of services within the site: includes staff who may lead particular service components (e.g., a director of early childhood services) that are within the overall coordinated services approach.

• Frontline staff: includes staff who work directly with clients to provide services (including service navigators or coordinators).

The number of staff interviewed across each respondent type will vary based on the composition of each individual site, but we will aim for even distribution across the respondent types.

The AMCS study team will conduct 60-minute focus groups with up to 10 parents/guardians at each of the six coordinated services approaches. We will allow up to 12 participants to enroll in the focus group to account for no-shows. At each site, prior to the “virtual visit”, the study team will work with staff to identify the best way to recruit parents/guardians to participate in focus groups. To be eligible to participate in the focus group, parents/guardians must 1) have a child aged five years or younger who has participated in the child-focused services and 2) have participated in the coordinated services approach’s services for at least six months.

To account for the possibility that not all six (6) state and local coordinated services approaches identified will be willing to participate, we will have alternate sites selected that will serve as replacements. If possible, we will choose replacements with similar characteristics to the state or local coordinated services approach that they are replacing.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

The AMCS study team designed the state and local master site visit interview protocol (Instrument 1) and parent focus group protocol (Instrument 2) to build upon the previously developed AMCS instruments (i.e. the state and local model profile templates, and the state and local telephone interview protocols). Those prior protocols are designed to collect broad data from a larger set of coordinated services approaches. The state and local model profile templates collect data at the broadest level, with the telephone interviews gathering more in depth information about the same overall categories of information.

For the current IC, the master site visit interview protocol is designed to gather even more in-depth information about a small set of coordinated services approaches (six). The master site visit interview protocol includes categories of information that appear on the prior protocols, but the questions included in those categories are more detailed. The master site visit interview protocol also expands topics that are only briefly addressed in prior instruments. For example, there is a section focused exclusively on early care and education services and a section about best practices, barriers, challenges, and lessons learned that goes beyond what was included in prior instruments. To avoid duplication, the AMCS study team will draw on information gathered through the state and local model profile templates and state and local telephone interviews to tailor the master site visit interview protocol and only ask questions that have not previously been answered.

The parent focus group protocol focuses on collecting qualitative data from a group of respondents not addressed in other study instruments. The questions are designed to elicit a parent/service recipient perspective on the coordination topics that staff are asked about through the other AMCS study instruments.

B4. Collection of Data and Quality Control

“Virtual site visitors” will spend the equivalent of 1.5 days at each coordinated services approach (i.e. 6-8 interviews), conducting semi-structured interviews with staff (45 minutes to 2 hours) and a focus group with parents (60 minutes).

Inviting sites to participate

After coordinated services approaches have been selected for “virtual site visits”, the AMCS study team will send the director (or primary contact identified through the telephone interviews) an email inviting them to participate. Outreach will begin by contacting the state coordinated services approach. The email will be used establish the coordinated services approach’s interest in the “virtual site visit” and to schedule a phone call to discuss the “virtual site visit” in more detail. During the telephone call, a member of the AMCS study team will confirm the coordinated services approach’s interest in participating, describe the purpose of the “virtual site visit”, the types of staff that the team would like to talk to, and begin developing a schedule for the “virtual visit”.

Preparing for the site visit

Experienced and trained interviewers, who are members of the AMCS research team, will conduct the “virtual site visits”. “Virtual visits” will be conducted in pairs, with one senior and one junior member of the team. The senior team member will be the primary interview lead, though the junior member may also conduct some interviews under the supervision of the senior member. Prior to the “virtual site visit”, the pair who will be conducting the interviews will review the model profile and telephone interview data for the coordinated services approach, as well as the master site visit interview protocol. Drawing upon these sources, the interviewers will annotate the master site visit interview protocol, noting what information has already been obtained, what information will require deeper investigation, and what information has not yet been obtained. To tailor the protocol, we will consider what sections are most appropriate for the type of respondent and then select questions within those sections for each individual, using the information described above to guide our selection. We will also take into account any information we have about the specific staff we will interview and their primary area of knowledge. Interviewer pairs will then work with the project director and task lead to tailor the protocol to the interviewees. Where possible, protocols can be tailored for named individuals. However, typically, protocols will be tailored for a particular type of staff rather than a specific person). The table below provides an example of how the protocol sections might be distributed across types of staff. Individual questions within each of the sections may also be further tailored to the specific interviewee.

Master site visit interview protocol respondents by protocol section

Protocol section	Coordinated services approach leaders	Service directors	Frontline staff
Respondent roles and responsibilities	X	X	X
Target population		X
Services		X	X
Program overview	X
Eligibility and enrollment		X	X
Staffing and organizational structure	X	X
Partnership building	X	X
Data and data systems	X	X
Funding	X
Coordination and alignment	X	X
Early care and education		X	X
Best practices, barriers, challenges, and lessons learned	X	X	X

At each site, prior to the “virtual visit”, the study team will work with staff to identify the best way to recruit parents/guardians to participate in focus groups. A flyer will be provided to staff to distribute to parents/guardians and/or to post at the approach (APPENDIX 3). We will also rely on coordinated services approach staff to directly talk to parents/guardians about the focus groups.

“Virtual site visit” data collection

Semi-structured interviews with coordinated services approach staff will last between 45 minutes and 2 hours, depending on the number of interviewees and the topics covered. With permission from respondents, interviewers will audio record interviews to support their written notes. Respondents will be provided assurances about the privacy of the interviews as described in the introduction to each interview. The master site visit interview protocol (Instrument 1) will not be administered in its entirety in each interview. Rather, for each interview, the “virtual site visit” team will select and ask questions that are relevant to each respondent and coordinated services approach (as described above).

Parent focus groups will last approximately 60 minutes. With permission from respondents, interviewers will audio record interviews to support their written notes. Respondents will be provided assurances about the privacy of the focus group as described in the introduction to the focus group. The focus group protocol (Instrument 2) will not be administered in its entirety in each focus group. Rather, interviewers will select questions most relevant to the focus group and coordinated services approach, based on the documents reviewed ahead of the “virtual site visit” (see above) and the experiences and conversation among the participants in the group.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

“Virtual site visit” interview

The six coordinated services approaches we contact to participate in “virtual site visits” will be purposively selected from the group of 20 coordinated services approaches who participated in the telephone interviews (generic IC). It is important to note that we are using purposive sampling that is not intended to be representative and we will not be making statistical inferences from the data. We will pursue coordinated services approaches using the methods described in section B4 to try to maximize response rates. If coordinated services approaches refuse to participate in the “virtual site visit” we will contact coordinated services approaches from the replacement group (as described in B4).

Parent focus groups

Parents/guardians enrolled in services at participating sites will be identified through a convenience sample with the help of coordinated services approach staff. The AMCS study team will work with staff prior to the “virtual site visit” to identify the best way to recruit parents/guardians to participate in focus groups (posted flyers, word of mouth). Parents/guardians will be provided with a $25 gift card as an incentive for participation. Parent focus groups are not designed to be representative.

NonResponse

The AMCS study team will use a mix of strategies to ensure high response rates for data collection. A mix of email and telephone outreach will provide different ways for state and local coordinated services approaches to respond to requests to participate in and schedule “virtual site visits”. AMCS study team members will explain the goals of AMCS and our interest in learning more about the potential participants’ coordinated services approach.

Several other specific strategies include:

• The scheduling of interviews will be flexible, to account for the availability of key state and local coordinated services approach staff.

• The scheduling of parent focus groups will be flexible to account for parent/guardian schedules.

During the “virtual site visit” interviews themselves (based on our experience asking similar questions with similar populations) the AMCS study team does not anticipate significant item nonresponse to interview or focus group questions. Interview questions will be tailored to coordinated services approaches operating at the state or local levels. Parent focus group questions have been tailored to parents/guardians receiving services from a state or local coordinated services approach. Interviewers will be trained to use appropriate probes and prompts to encourage responses to all questions. Additionally, we do not expect questions to be sensitive for participants; thus, we expect they will feel comfortable answering all questions.

B6. Production of Estimates and Projections

No estimates or projections will be produced based on this work.

B7. Data Handling and Analysis

Data Handling

“Virtual site visit” interviews and parent focus groups will be audio recorded and these recordings will be used as needed to supplement, correct, or clarify interviewers typed notes. Typed notes will then be reviewed by senior project staff to identify any missing information or areas lacking clarity. After this review, notes will be imported into a qualitative coding software (NVivo) for analysis.

Data Analysis

Once interview and focus group notes have been organized, we will do a thematic coding of responses based on the AMCS research questions. Trained AMCS team members will review the information gathered during the interviews and identify common themes or features across coordinated services approaches. Themes will be determined through meetings with the research team. Themes will focus on coordinated services approach implementation and might include topics such as efficiencies realized through coordination, barriers to coordination, or ways in which staffing and service delivery operate within a coordinated services approach. The AMCS study team will code information from the state and local “virtual site visit” interviews and focus groups using a qualitative coding software. During coding, a senior member of the team will review the coding to ensure consistency in how the codes are applied to the data. We will provide descriptive information about the number of times interview participants describe particular coordinated services approach features and then summarize the qualitative information, organizing the summary by topic. We will summarize both the primary themes for each state or local coordinated services approaches’ interview and focus group, and themes across interviews and focus groups.

Data Use

The information collected will be used to produce individual “virtual site visit” descriptions, as well as a summary report with themes from across the “virtual site visits”. We will also combine information from the “virtual site visit interviews” with information collected previously through a scan of publicly available information about coordinated services approaches and information collected through telephone interviews. The combined information will be included in a final project report that provides information to answer the research questions and describes implications and proposed steps for future data collection efforts ACF or others might want to undertake. Findings from this study will help to improve federal, state, and local policymakers’, practitioners’, and other stakeholders’ knowledge and understanding of approaches to coordinating services to support low-income children and their families. And, will be used to provide technical assistance for Preschool Development Grant Birth to Five (PDG B-5) grantees, who are currently developing, updating and implementing coordinated services approaches.

All printed and electronic publications from this information collection will clearly state that findings are from a descriptive study, and should not be used to make conclusions about the effectiveness of the described services.

Please see Section A.16 for information on the schedule for tabulation and reporting based on these findings.

B8. Contact Person(s)

OPRE

Laura Hoard, Ph.D.

laura.hoard@acf.hhs.gov

Business Strategy Consultants

Tracy Carter Clopet, Ph.D.

tracy.clopet@acf.hhs.gov

Mathematica Policy Resesarch

Elizabeth W. Cavadel, Ph.D. (Project Director)
ecavadel@mathematica-mpr.com

Attachments

INSTRUMENT 1: Master Site Visit Interview Protocol COVID-19 Additions

INSTRUMENT 2: Parent Focus Group Protocol COVID-19 Additions

APPENDIX 1: 60 Day Federal Register Notice

APPENDIX 2: AMCS Public Comments

APPENDIX 3: Recruitment Flyer COVID-19 Additions

APPENDIX 4: 60 Day Federal Register Notice (Extension)

APPENDIX 5: AMCS Public Comment (Extension)

9