Attachment P - Nonsub change request_Coaching Evaluation - October 2020

MEMORANDUM

To: Jordan Cohen, Office of Information and Regulatory Affairs, Office of Management and Budget (OMB)

From: Hilary Bruck and Victoria Kabak, Office of Planning, Research, and Evaluation, Administration for Children and Families (ACF)

Date: 10/5/2020

Subject: Request for non-substantive changes to implementation and impact study data collection for the Evaluation of Employment Coaching for TANF and Related Populations (OMB #0970-0506)

BACKGROUND

Type of Request: Non-substantive change to implementation study data collection instruments to: (1) reframe some questions and add some questions to the guides for the semi-structured management, staff, and supervisor interviews and in-depth participant interviews to systematically capture implementation information related to COVID-19; (2) slightly increase the incentive amount for the additional in-depth participant interviews; and (3) slightly increase the estimated burden based upon conduct of additional semi-structured management, staff, and supervisor interviews and in-depth participant interviews. In addition, non-substantive change to the incentive structure and amount for two sites for the impact evaluation’s first and second follow-up surveys, and minor revisions to the survey instruments and notifications to reflect the changes.

Study Features Salient to Request: The purpose of the Evaluation of Employment Coaching for TANF and Related Populations (OMB #0970-0506) is to describe select employment coaching interventions for low-income populations and estimate their effectiveness. ACF will use information from the evaluation to inform policymakers and practitioners interested in funding, designing, or implementing interventions to improve employment outcomes of low-income populations.

The implementation study involves the following data collection activities across the six sites participating in the evaluation: (1) semi-structured management, staff, and supervisor interviews; (2) a staff survey; (3) in-depth participant interviews; (4) staff reports of participant service receipt; and (5) video recordings of coaching sessions.

The impact evaluation involves a randomized controlled trial in the six sites. During study enrollment, people eligible for employment coaching were randomized into a program group who were offered employment coaching and a control group who were not offered employment coaching. Two follow-up surveys are collecting data on the outcomes of members of the program and control groups. The first follow-up survey is administered between 6 and 12 months after random assignment; the second follow-up survey is administered between 21 and 24 months after random assignment. As the timing of study enrollment varied by site, the timeline for survey data collection is specific to each site.

Time Sensitivity: This request has high time sensitivity. Collecting the information on how programs’ services have changed as a result of COVID-19 is critical to ensure that we accurately document decisions, changes, successes, challenges, and lessons learned to responding to COVID-19 while the information is fresh in respondents’ minds. It is important to collect this information now so findings can be incorporated into our implementation reports, which the study team is currently drafting, and our impact analyses and reporting, which the study team will begin conducting next year. Further, to be relevant to policymakers and practitioners, it is important that we report this information quickly so it can inform other programs’ decisions and broader policy.

The request to increase the follow-up survey incentive for two sites—MyGoals Baltimore and MyGoals Houston—is time sensitive as well. Response rates for the follow-up surveys are at risk of being much lower than anticipated because in-person location has stopped due to COVID-19. No timetable for resumption of field locating has been established at this time but it will not be until January 2021 at the earliest, contingent on improvement in conditions related to the pandemic. Previously, OMB approved an increase in the survey incentive amount for four of the six sites in the evaluation; we are now requesting the same increase for the other two sites in order to help increase the response rates and decrease the program-control group response rate differential, to avoid bias in the estimates of the programs’ effectiveness. It is critical to increase response rates before the first follow-up survey is closed, which can be no later than the start of the second follow-up survey.

REQUEST DETAILS: IMPLEMENTATION STUDY

Progress to Date

The study team conducted semi-structured interviews with program managers, staff, and supervisors; in-depth participant interviews; and video recordings of coaching sessions in MyGoals in Baltimore, MyGoals in Houston, Family Development and Self-Sufficiency (FaDSS), Jefferson County Colorado Works, and LIFT in spring 2019 and in Work Success in February 2020. The staff survey was completed in Work Success in January 2020 and in the other sites in fall 2018. The study team has continued to collect participant service receipt data from staff reports in all six sites since June 2018.

As a result of the 2019 novel coronavirus disease (COVID-19) pandemic, the coaching programs have had to alter their policies and operations to respond to social distancing requirements and changing participant needs. Employment coaching sessions that normally occur in person are now happening over the phone or via video. Coaches and supervisors are working remotely and are unable to provide in-person support to one another as they address difficult participant situations. Families are facing new or worsened challenges, including health concerns, furloughs, layoffs, and lack of childcare, that have implications for their financial stability and family well-being. Because the implementation study’s data collection was conducted before these changes occurred, we have not been able to systematically capture how the programs are now serving program participants. ACF is therefore requesting approval to conduct additional semi-structured management, staff, and supervisor interviews and additional in-depth participant interviews in five of the six sites. Work Success has not continued serving participants during COVID-19, so we are not requesting approval to conduct additional interviews in that site.

Previous Terms of Clearance

The two data collection efforts pertinent to this request were approved by OMB in March 2018 (OMB #0970-0506):

• Attachment D. Semi-structured management, staff, and supervisor interviews

• Attachment F. In-depth participant interviews

For the management, staff, and supervisor interviews, OMB approved an annual burden of 66 hours (with 44 respondents annually, 132 respondents total). For the in-depth participant interviews, OMB approved an annual burden of 40 hours (with 16 respondents annually, 48 respondents total). Additionally, for the in-depth participant interviews, OMB approved providing each interviewee a $50 gift card as an incentive.

Proposed Revisions for OMB Approval

We have revised the semi-structured management, staff, and supervisor interviews discussion guide and the in-depth participant interviews discussion guide to gather implementation information relevant to COVID-19. The requested changes are included in Attachment D. Semi-structured management, staff, and supervisor interviews_rev and Attachment F. In-depth participant interviews_rev (both clean and tracked changes versions provided). The study team proposes to conduct 1.5-hour long interviews with a total of about 37 staff respondents (between five and 10 per site) and 2.5-hour long interviews with about 10 study participants (two per site). All interviews will be conducted either by video or by phone, according to each respondent’s preference.

For the in-depth participant interviews, we propose providing each study participant a $60 gift card as a thank you for their participation in the interview. We propose to offer $10 more for these interviews than those conducted earlier. We believe a slight increase in the incentive will help ensure sufficient numbers of people among the pool eligible agree to be interviewed. It will be more difficult to recruit study participants who are still engaged in the coaching programs for these interviews now because fewer people are still actively participating in the programs. OMB recently approved a $60 incentive for in-depth participant interviews for the Next Generation of Enhanced Employment Strategies Project (OMB #0970-0545).

To conduct these additional interviews, ACF proposes increasing the overall annual burden estimate by 9.5 hours. As noted above, OMB previously approved interviewing a total of 132 staff respondents and 48 participant respondents. Due to remaining burden for the previously approved associated instruments, conducting these additional interviews will only require adding burden for eight additional staff respondents (which increases the annual burden by 4.5 hours) and five additional participant respondents (which increases the annual burden by 5 hours).

Justification

This evaluation includes descriptive and impact studies of six coaching programs. Study participants are still being served by five of the programs, and the evaluation is in the midst of data collection to assess impacts (via participant follow-up surveys and administrative data collection). It is therefore critical to understand how programs changed as a result of COVID-19 and what study participants’ experiences with the pandemic have been, to inform the interpretation of study findings. Collecting this descriptive information is important to understand the services that program group participants received during this time, to fully contextualize our findings, and to account for the pandemic in our analysis. Additionally, understanding the changes made by and the lessons learned from these programs will help inform other programs’ policies and implementation as the country continues to respond to the pandemic and other future public health emergencies.

REQUEST DETAILS: IMPACT EVALUATION

Progress to Date

The impact evaluation’s two follow-up surveys are collecting data on the outcomes of members of the program and control groups. In March 2018, OMB approved a two-tiered incentive structure with an “early bird” incentive that provides survey respondents $35 if they complete the survey within four weeks of the initial notification, and $25 if they complete it after four weeks (OMB #0970-0506). The study team employed this incentive structure for participants in all six programs during the administration of both the first and second follow-up surveys until early spring 2020. In March 2020, OMB approved a non-substantive change request proposing that the two-tiered incentive structure continue only among study participants in the two MyGoals sites in Baltimore and Houston, and that participants from the other four sites (FaDSS, LIFT, Jefferson County Colorado Works, and Work Success) be offered a $50 incentive for completing each survey, irrespective of whether the participants complete the survey within the four-week “early bird” period. We proposed this change due to patterns of survey response for those four sites showing a risk that our analysis would result in biased estimates of program impacts and would underrepresent participants in key analytic groups. The study team implemented this change in incentives in mid-March 2020. As discussed below, the incentive increase has been effective in increasing the rate of response by web and telephone for cohorts released after the increase.

As a result of COVID-19, in-person data collection operations for the follow-up surveys ceased in March 2020. There is no firm date for in-person data collection operations to resume but it will not be until January 2021 at the earliest, contingent on improvement in conditions related to the pandemic. The lack of in-person field location has depressed response rates across all sites and, we anticipate, will continue to do so for the foreseeable future. Table 1 provides current response rates for the first follow-up survey by site. Cases are released for data collection in monthly cohorts, depending on when they were enrolled in the study. Table 1 shows the response rates for cases that have been released for data collection for at least six months. While the response rates overall are about on target for the two MyGoals (Baltimore and Houston) sites, this reflects that the majority of MyGoals respondents were released for data collection prior to the COVID-related shutdown of in-person locating. MyGoals respondents released since March 2020 have participated at much lower rates. The first six monthly cohorts affected by COVID are averaging a 41 percent response rate in Baltimore and 48 percent in Houston.

Table 1. Current response rates for the first follow-up survey^a

Site	Total number releasedb	Overall response rate	Response rate in program group	Response rate in control group
FaDSS	478	60%	62%	58%
Jefferson County	354	55%	60%	50%
LIFT	368	70%	74%	68%
Work Success	201	61%	60%	62%
MyGoals Baltimore	475	82%	83%	81%
MyGoals Houston	605	80%	82%	78%

^a The survey began in May 2018 for MyGoals Baltimore and Houston, March 2019 in FaDSS, July 2019 in Jefferson County, June 2019 in LIFT, and November 2019 in Work Success.

^b Includes sample members released for at least six months.

We also anticipate lower than targeted final response rates for the second follow-up survey. The inability to conduct in-person locating because of COVID-19 will also affect the second follow-up survey. Moreover, we anticipate that lower than anticipated response rates for the first follow-up survey will decrease our ability to locate the study participants for the second follow-up survey.

Progress related to approved change in incentives:

Since March 2020, we have found markedly increased web or phone response rates for the first follow-up survey in the four sites in which an increased incentive was offered, but not in the MyGoals sites in which an increased incentive was not offered (Figure 1). We compare the response rates for the April and May 2020 cohorts (which were offered the increased incentive in four sites) with the response rates for the January 2019 to February 2020 cohorts (cohorts that were not affected by the increased incentive). To compare “like with like,” Figure 1 shows the response rates in the two months after the cohort was released.

Figure 1 shows that the response rate increased significantly around March 2020 in all four sites in which there was an increase in incentive. The increase ranged from 4 percentage points for Jefferson County to 16 percentage points for LIFT. During this period, differential response rates remained low, decreasing modestly at the four sites after implementation of the increased incentive (Figure 2). The decrease in differential response ranged from less than one percentage point in FaDSS to two percentage points in Work Success.

Figure 1. First follow-up survey web or phone response rates before and after incentive change, by site*

*Response rate in the two months after the cohort was released.

Figure 2. Difference in program and control group first follow-up survey web or phone response rates before and after the incentive change, by site*

*Response rate in the two months after the cohort was released.

To shed light on whether it was the increased incentive that increased the response rate, or other factors that occurred at about the same time, such as COVID-19, we compared response rates for the four sites that experienced an increase in incentives with the two MyGoals sites, which were not offered the higher incentive approved in March 2020. If the changes in the responses we saw in the four increased-incentive sites was because of COVID-19, we would expect to see the same pattern of response rates in the MyGoals sites.

The pattern of response rates differed markedly between the four increased-incentive sites and the two MyGoals sites. Instead of the response rates in the MyGoals sites increasing after March 2020, the response rates fell from 29 percent before March 2020 to 25 percent after March 2020. The web or phone response rate decreased more for the program group (from 38 to 30 percent) than for the control group (from 21 to 19 percent). As a result, the differential web or phone response rates in the MyGoals sites decreased from 17 percentage points for the earlier cohorts to 10 percentage points for the later cohorts in the MyGoals sites (Figure 2).

These findings are strongly suggestive that the increased incentive approved in March 2020 increased response rates. For this reason, we propose raising the incentive to $50 for the MyGoals sites that were not part of the March 2020 increase. However, the findings do not offer definitive proof of a causal effect because we cannot rule out the possibility that differences in the change in response rates are due to differences in the sites that received the increased incentive and those that did not. 

Other Mitigation Efforts to Date

From the beginning of survey fielding, ACF has actively monitored survey production and response rates. To date, the study team has taken the following steps to improve respondent location and encourage survey completion:

Initial Design

• The follow-up surveys are offered to respondents in two modes, web and telephone, allowing study participants to complete the surveys how and when it is convenient for them.

• Survey outreach uses multiple channels and multiple contact attempts. Prior to attempted telephone contact, all participants are notified about the data collection through an advance letter. Participants who provided an email address at the time of study enrollment are sent email messages providing login information for the survey web form. Participants with an email address and/or assent to text are sent two email messages and/or two text messages prior to telephone contact attempts.

• Nonresponding participants receive multiple telephone calls over several weeks on different days and at different times asking them to complete the survey.

• Non-working telephone numbers or those that do not result in contact are subject to locating through contact with individuals for whom the participant provided contact information at the time of study enrollment.

• A specialized letter is sent to participants who make non-adamant refusals, followed by contact from a specially-trained and experienced telephone interviewer.

• Before March 2020, if participants could not be reached by telephone, field locators would make in-person contact with them and provide them with a cell phone to complete the survey with a telephone interviewer.

• As of mid-March 2020, field locating operations have ceased due to COVID-19. No timetable for resumption of field locating has been established at this time but it will not be until January 2021 at the earliest, contingent on improvement in conditions related to the pandemic.
  
  

Additional Mitigation

As it became apparent that survey production would likely be insufficient, the study team took additional steps to address non-response:

• The survey fielding period was extended to provide additional time to locate participants and complete surveys.

• Site staff began providing updated participant contact information for nonresponding sample members.

• The study team debriefed staff at the FaDSS and Jefferson County sites to diagnose why participants were not completing the follow-up surveys. Staff reported that housing costs in some areas increased levels of mobility and homelessness among study participants.

• The study team debriefed interviewers and found that the stated 60-minute length of the survey discouraged some participants from starting the survey. Survey administration data to date showed that the average length of the interviews was actually 45 minutes. Therefore, survey notifications were amended to indicate the survey is 45 minutes since a more accurate estimate of the survey length could increase the likelihood that sample members agree to complete a survey. OMB approved this change as part of the non-substantive change request in March 2020.

• ACF requested a change in the incentive structure for four of the sites in spring 2020, which was approved by OMB and implemented in March 2020.

Previous Terms of Clearance

As noted above, in March 2018, OMB approved a two-tiered incentive structure with an “early bird” incentive that provides survey respondents $35 if they complete the survey within four weeks of the initial notification, and $25 if they complete after four weeks, across all six sites participating in the evaluation (OMB #0970-0506).

In March 2020, OMB approved a non-substantive change to the first and second follow-up survey incentive structure and amount in four sites—FaDSS, Jefferson County Colorado Works, LIFT, and Work Success (OMB #0970-0506); the survey incentive structure and amount was not changed for MyGoals Baltimore or MyGoals Houston:

• Participants receive $50 gift card for completing the first and/or second follow-up survey (irrespective of when the participants complete the survey)

The survey instruments and notifications reference the incentive amount and, in the case of the MyGoals sites, structure:

• Attachment C. First follow-up survey

• Attachment I. Notifications

• Attachment N. Second follow-up survey

Proposed Revisions for OMB Approval

ACF requests approval to increase the incentive for the MyGoals respondents to a $50 gift card for completing either the first or second follow-up survey (irrespective of when the participants complete the survey). Approval of this change will mean respondents in all six sites will receive the same $50 gift card incentive.

This proposed change to the incentives is aimed at increasing the likelihood that MyGoals sample members who are successfully contacted respond to the survey. The further incentive increase is also intended to increase the likelihood that friends and relatives of these respondents who are contacted by the survey team will tell respondents about the opportunity to complete the survey. The study team will also continue the increased mitigation efforts described above.

ACF requests minor revisions to the survey instruments and notifications to reflect the increase in the incentive amount; these revisions are reflected in the following attachments:

• Attachment C. First follow-up survey_rev

• Attachment I. Notifications_rev

• Attachment N. Second follow-up survey_rev

Expected Benefits and Proposed Assessment

We hypothesize that increasing the incentives in these two sites will increase response rates conditional on successful contact. In the four sites offered the earlier approved increased incentive, we observed increased response rates without a worsening of differential response rates. If the same increase induces a similar response among MyGoals respondents, it would reduce the risk of bias in our experimental impact estimates.

This proposal was designed to maximize improvements in data quality given the results of efforts to date and the constraints of remaining time and budget for the data collection. ACF is interested in using this opportunity to contribute to the body of evidence on the role of incentives in mitigating non-response bias, as well as the ability to conduct data collection among similar populations without the benefit of using in-person locating techniques. We intend to calculate pre-post response rates for program and control group members and the study participants as a whole throughout the period in which the incentives were changed. In addition, we will examine the demographic characteristics of pre- and post-$50 incentive respondents and compare them with the full baseline study sample. We will use this analysis as a measure of the impact of our changes to fielding protocol on observable non-response bias.

ACF anticipates that the information resulting from our survey administration efforts, including the proposed change in incentive amount, will be of sufficient quality to meaningfully contribute to ongoing learning about strategies for improving the quality, efficiency, and effectiveness of our survey data collections.

Justification

The response rates within the MyGoals sites for the follow-up surveys are at risk of being lower than anticipated because in-person location has stopped due to COVID-19. No timetable for resumption of field locating has been established at this time but it will not be until January 2021 at the earliest, contingent on improvement in conditions related to the pandemic. The response rates are unlikely to hit our original target of 80 percent. We saw significant increases in response rates—with no major increase in the program-control response rate differential—in the four sites for which incentives were increased in March 2020 (as presented in Figures 1 and 2 above). We anticipate that increasing the incentive to $50 for MyGoals sample members will increase response rates and help ameliorate the negative impact of the inability to conduct in-person locating.

We are still exploring ways to increase the representativeness of the estimates we can report from the survey data. We anticipate that the proposed increase in incentives would increase response rates further and reduce the chance that our analysis is subject to attrition bias.