Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
The Experiences, Needs, and Voices of Workers in Low-Income Households During the COVID-19 Public Health Emergency
Formative Data Collections for ACF Research
0970 – 0356
Supporting Statement
Part B
July 2020
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officers: Hilary Bruck
B3. Design of Data Collection Instruments…………………………………………………………………………………..…5
B4. Collection of Data and Quality Control 5
B5. Response Rates and Potential Nonresponse Bias ….………………………………………………………………...6
B6. Production of Estimates and Projections 6
B7. Data Handling and Analysis 6
B8. Contact Person(s).……………………………………………………………………………………………………………………7
Study Objectives
The study will serve two key purposes. First, the findings will inform the conduct of ACF’s Evaluation of Employment Coaching for TANF and Related Populations (Evaluation of Employment Coaching, OMB #0970-0506) and Next Generation of Enhanced Employment Strategies Project (NextGen Project, OMB #0970-0545) to ensure they are relevant and account for current conditions under the 2019 Novel Coronavirus (COVID-19) public health emergency (PHE), to fully understand the implementation of these programs, and to inform the provision of technical assistance during the evaluation period to these programs to strengthen their service delivery and better equip the programs to respond to the ongoing and evolving PHE. This will help ensure the programs being evaluated provide a strong test of effectiveness for the evaluations. Second, the findings will inform ACF’s research agenda more broadly by providing timely and instrumental insights and lessons about how to meet low-wage workers’ needs during an ongoing or future PHE.
Generalizability of Results
This study is intended to present an internally-valid description of the experiences of program participants during the COVID-19 PHE in selected programs, not to promote statistical generalization to other programs or service populations.
Appropriateness of Study Design and Methods for Planned Uses
Data from this study are intended to offer rich, nuanced narratives of participants’ experiences and needs during the COVID-19 PHE and their perspectives on whether and how a select set of the NextGen Project programs and Evaluation of Employment Coaching programs responded to meet their needs. The study team will collect data through in-depth, semi-structured discussions with program participants. As described in Guest et al. (2013), an in-depth discussion or interview is “a conversation designed to elicit depth on a topic of interest. It is versatile across a range of study topics, adaptable to challenging field conditions, and excellent for not just providing information but for generating understanding as well.”1 Data collectors will use an in-depth discussion guide (Instrument 1. In-depth discussion guide) that consists of scripted open-ended questions and suggested probes to ensure consistency across the discussions and coverage of the topics of interest while allowing flexibility in the order and depth of topics covered. This format will enable data collectors to follow the participant’s lead and to delve more deeply into issues that arise during the discussion.
The data are not intended to be representative and should not be used to assess participant outcomes. Key limitations of the study will be included in written products associated with the study.
As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.
Target Population
The target population for this study is low-wage workers whose employment status or work experience has been affected by the COVID-19 pandemic. Respondents, who will be recruited from four selected programs as described below, must meet the following study eligibility requirements: be between 18 and 59 years old (i.e., working age) and were working in either February or March 2020 when the COVID-19 pandemic hit their area or were laid off, furloughed, had their hours reduced, or quit as a result of COVID-19.
Respondent Recruitment and Site Selection
Respondents will be recruited from four programs that are being evaluated in one of two ongoing studies of employment programs for people with low income that Mathematica is leading under contract to OPRE—(1) the Evaluation of Employment Coaching and (2) the NextGen Project. The four programs are:
Families Achieving Success Today (FAST). FAST provides employment services to a subset of participants in the Minnesota Family Investment Program (that is, Minnesota’s Temporary Assistance for Needy Families, or TANF, program) in Ramsey County, Minnesota (St. Paul). Respondents will be recruited from among FAST program participants. FAST is included in the NextGen Project.
Bridges from School to Work (Bridges). Bridges is an employer-driven program that provides job readiness instruction, placement, and post-placement support for young adults (generally ages 17-22, up to age 24) with disabilities. The program operates in 12 sites: Atlanta, Baltimore, Boston, Chicago, Dallas, Ft. Worth, Los Angeles, New York City, Oakland, Philadelphia, San Francisco, and Washington D.C. Respondents will be recruited from the parents and guardians of program youth participants. Bridges is included in the NextGen Project.
LIFT. LIFT is a non-profit organization that provides career and financial coaching to parents and caregivers of young children. LIFT operates in Chicago, Los Angeles, New York City, and Washington, DC. Respondents will be recruited from among LIFT program participants across all locations. LIFT is included in the Evaluation of Employment Coaching.
Family Development and Self-Sufficiency (FaDSS). Seventeen local human services agencies in Iowa use grants from the state Department of Human Rights to provide TANF participants with coaching during home visits to address their challenges to employment and job retention. Respondents will be recruited from rural communities in the state. FaDSS is included in the Evaluation of Employment Coaching.
The two NextGen programs represent the only two evaluation sites that have thus far been selected. The two Evaluation of Employment Coaching sites were chosen because one program, FaDSS, provides services to rural areas, which is important for capturing commonalities and variations in participant experiences across locations with varying levels of urbanicity. The other program, LIFT, operates in four sites and provides the opportunity to understand programmatic changes and challenges vis-à-vis different client experiences in geographic areas that have experienced high rates of COVID-19 infection and related social and economic disruptions.
Because participants will be purposively selected, they will not be representative of the population of participants in the selected programs or of participants in other programs designed to support low-wage workers.
Development of Data Collection Instrument(s)
Content experts at Mathematica developed the in-depth discussion guide by adapting open-ended questions and probes that have been used successfully during in-depth interviews with program participants in other OPRE evaluations with similar populations. For example, questions were adapted from two-hour in-depth interviews with program participants with low income about their experiences with employment coaching for the Evaluation of Employment Coaching (OMB #0970-0506), and from two-hour in-depth interviews with fathers with low income about their experiences as financial providers and parents for the Parents and Children Together (PACT) evaluation (OMB #0970-0403). Forty-four interviews were conducted in the Evaluation of Employment Coaching and 87, 59 and 24 interviews, respectively, were conducted over three waves of data collection for the PACT evaluation. Only questions that capture essential data for the study’s main research questions are included in the in-depth discussion guide for this proposed information collection. The instrument was closely examined to confirm that it was streamlined and does not collect duplicative data within the same interview.
The study team will ask program staff to identify participants who meet the study eligibility requirements and call them to ask if they wish to participate in the study. Program staff will use the flyer (Appendix A. Study flyer) to describe the study and, if possible, will email the flyer to the participants. If a program participant is interested in participating in the study, the program staff member will ask the program participant to call a study-specific toll-free telephone number.
Depending upon the time at which the call is initiated and staff availability, callers to the toll-free number may be directly connected to a trained member of the study team who can provide them information about the study, answer their questions and schedule a time for the interview. If a scheduler is not available to answer the incoming call, the caller will hear a pre-recorded message thanking them for their interest in the study and asking them to leave contact information so that a member of the study team can call to schedule a time for the discussion.
The study team will request that program staff in each of the four programs identify 15 program participants who meet study eligibility requirements and indicate to program staff that they are interested in participating in the study and willing to call the toll-free number to schedule an interview. Because not everyone who indicates to program staff that they are interested in participating will actually call the toll-free number and complete the scheduled interview, it is assumed that only 11 of the 15 interviews per program will be completed, for a total of 44 interviews.
A member of the study team will call participants at the scheduled time to conduct a discussion by telephone or video conference, according to the participant’s preference. Each discussion will last about 2 hours (120 minutes). Study team members will use an in-depth discussion guide (Instrument 1. In-depth discussion guide) to facilitate the discussion. Before reaching out to any study subjects, all study team members will participate in a training to ensure quality and consistency of the data. Training will cover the study objectives, discussion topics, discussion facilitation techniques, and culturally-responsive evaluation.
After conducting the first few discussions, project leadership will review the discussion transcripts and provide feedback to monitor quality and ensure consistency across discussions, and the study team will meet to debrief and troubleshoot. Through regular team meetings project leadership will continue to answer questions and provide ongoing guidance to data collectors.
Response Rates
The in-depth discussions are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.
NonResponse
As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent demographics will be documented and reported in written materials associated with the data collection.
The data will not be used to generate population estimates, either for internal use or dissemination. We do not intend for this information to be used as the principal basis for public policy decisions.
Data Handling
To facilitate analysis, with consent from participants, discussions will be audio recorded and transcribed.
All transcripts will be coded with the assistance of a qualitative analysis software package (specifically, NVivo). The study team will take an open coding approach to preparing the data for analysis. Open coding is an iterative process whereby data are analyzed and categorized, and these categories are confirmed or adapted as additional data are added (Strauss and Corbin 1990).2
The team will apply an initial set of codes to the discussion guide before the start of data collection. The coding scheme will become more detailed and layered over the course of the data collection phases, but this core set of codes will remain consistent. The data collectors will be trained to code the data. For quality assurance purposes, a member of the study team will oversee the coding process and independently code 10 percent of the coded transcripts. The data collectors, coding task lead, and project leadership will have ongoing meetings to discuss emerging themes. As new themes or concepts emerge, the study team will create new codes that will be applied to all transcripts. Project leadership will review all coded transcripts before they are final.
Data Analysis
When the data are coded, the team will be able to retrieve and sort data linked to specific research questions and constructs. The team will analyze the data by code, comparing and contrasting responses across discussion transcripts.
In addition to enriching understanding about how participants view, experience, and navigate changes and challenges resulting from COVID-19, the information collected will also be analyzed in combination with program-focused data that will be collected under other components of the ongoing NextGen Project (OMB #0970-0545) and the ongoing Evaluation of Employment Coaching (OMB #0970-0506). The program-focused data will provide information, from the perspectives of program administrators and staff, about the challenges and changes that were made in response to COVID-19 and background data on program characteristics, such as program target population, organization and service delivery arrangements, and services. For the NextGen Project, approval for the collection of this program-focused data will be sought under future information collection requests (OMB #0970-0545). For the Evaluation of Employment Coaching, interviews with program staff were conducted in Spring 2019 (as approved under OMB #0970-0506) and, pending OMB approval of a forthcoming non-substantive change request to this previous information collection request, additional interviews will be conducted in Fall 2020 to learn about how the programs responded to the PHE. Additionally, for both ongoing studies, publicly available information about the health and socio-economic area conditions in which these programs operate (for example, rate of COVID-19 diagnoses and deaths, local PHE restrictions, unemployment rates, and other publicly available socioeconomic characteristics) will also be analyzed in order to provide rich contextual information about the conditions facing both participants and the programs that serve them.
Data Use
This proposed formative data collection will result in a report and an accompanying brief about how COVID-19 has impacted the circumstances and needs of program participants. The findings will provide a window into understanding how participants understand and navigate the significance and implications of COVID-19 in their lives. These findings will, in turn, inform the conduct of the ongoing Evaluation of Employment Coaching and the ongoing NextGen Project (OMB #0970-0545 and #0970-0506, respectively) to ensure they are relevant and account for current conditions under the PHE, to fully understand the implementation of the programs included in these studies, and to inform the provision of technical assistance to these programs during the evaluation period in order to strengthen their service delivery and better equip them to respond to the ongoing and evolving PHE. Ultimately, this effort will support ACF’s rigorous and relevant research agenda by helping ensure the programs being evaluated provide a strong test of effectiveness for the evaluations, and by providing timely and useful insights and lessons about how to meet low-wage workers’ needs during an ongoing or future PHE and how to support the well-being of families particularly vulnerable to the economic effects of COVID-19.
In addition to a stand-alone report and a brief that describe the findings from the in-depth interviews with participants, relevant findings from the integrated analyses of program and participant interview data will be presented in descriptive evaluation reports produced by the NextGen Project and Evaluation of Employment Coaching. In addition, for the NextGen Project, the findings will be used to inform the evaluation design and the provision of evaluation technical assistance.
All publications will acknowledge limitations in the study methods and findings, and clearly state that the findings are not representative or generalizable to other programs or populations (see B1. Generalizability of Results).
Principal Investigator:
Sheena McConnell, Senior Fellow
Address:
Mathematica
1100 First Street, NE
Washington, DC 20002
Email: SMcConnell@mathematica-mpr.com
Other Key Staff:
Pamela Holcomb, Senior Researcher
Jacqueline Kauff, Senior Researcher
Attachments
Instruments
Instrument 1. In-depth discussion guide
Appendices
Appendix A. Study flyer
1 Guest, Greg, Emily E. Namey and Marilyn L. Mitchell. Collecting Qualitative Data: A Field Manual for Applied Research. Newbury Park, CA: Sage, 2013.
2Strauss, A.L., and J.M. Corbin. Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Newbury Park, CA: Sage, 1990.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Naomi Goldstein |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |