Memo - NonSub Change Incentive Structure and Questions

DATE: August 23, 2016

TO: Steph Tatham

Office of Information and Regulatory Affairs (OIRA)

Office of Management and Budget (OMB)

FROM: Erica Zielewski and Carli Wulff

Office of Planning, Research and Evaluation (OPRE)

Administration for Children and Families (ACF)

SUBJECT: Request for Non-Substantive Change to Job Search Assistance Strategies Evaluation (OMB Control Number 0970-0440)

The Office of Planning, Research and Evaluation (OPRE) in the Administration for Children and Families (ACF) at the U.S Department of Health and Human Services has funded the Job Search Assistance (JSA) Strategies Evaluation (hereafter, JSA Evaluation). This memorandum requests approval for a revision to the Participant Six-Month Follow-up Survey instrument and a minor modification to the appreciation payment structure for the follow-up survey. The survey instrument was previously approved under OMB Control Number 0970-0440, with an expiration date of 02/28/2018. The OMB Clearance also included approval of a monthly interim tracking survey administered via cell phone text messaging.

Relative to that approval the non-substantive changes requested in the current memo would alter the research plan as follows:

• Because early response to the interim tracking survey is lower than anticipated, ACF and Abt Associates would like to modify the Participant Six-Month Follow-up Survey to add three to five questions seeking respondent viewpoints concerning the monthly text contacts they have received. Interim tracking via texts is an innovative data collection technique that can help inform researchers about the effectiveness of texting as part of future cost-effective data collection strategies. We would like to take the opportunity of this early experience to learn from text recipients how to improve this approach for the JSA study as well as in future applications.

• Because a portion of potential study participants are declining to consent to participate in the study in part due to reluctance to complete the baseline information form, ACF and Abt would like to request a shift in structure of the survey appreciation payments to include $5 at the time of study enrollment in appreciation for filling out the baseline information form and $20 at the time of the six-month follow-up survey interview (instead of no payment at enrollment and $25 at the time of the six-month survey). This would not change the total amount of the existing token of appreciation.

The remainder of this memo provides background on the JSA evaluation and more detail on these requested revisions.

Background on the JSA Evaluation

The JSA Evaluation will randomly assign individuals receiving cash assistance to different approaches for providing job search assistance and then compare their employment and earnings to determine the relative effectiveness of the these approaches in moving individuals to work and increasing their earnings. At the time of study enrollment, when individuals are applying for cash assistance under the Temporary Assistance for Needy Families (TANF) program and/or being assigned to work support services, they are asked to sign an informed consent form agreeing to be part of the study and to complete a baseline information form providing identifiers, demographic information, and contact information. Those who consent are randomly assigned into the two research groups and receive job search services appropriate to their assignments.

The evaluation includes one follow-up survey of all study sample members, conducted by phone six months after random assignment. Achieving a high response rate (of 80 percent) on the six-month follow-up survey is contingent in large part upon strong respondent connection to the study and the availability of accurate contact information. These activities are particularly important given that cash assistance applicants and recipients have historically been a hard-to-reach population in other similar survey efforts. To facilitate these outcomes, the study includes a series of monthly contacts via text messages—interim tracking surveys—with study participants that begin shortly after random assignment and continue until just before the six-month follow-up survey. The text messages are sent only to those study participants who consent to be texted as part of the evaluation—roughly 80 percent of all enrolled study participants. In addition, an appreciation payment of $25 is provided to those who complete the survey

The interim tracking surveys provide an opportunity to remind study participants that they are part of the evaluation and to obtain up-to-date contact information to enhance the ability of interviewers to both locate the participants and gain their cooperation for the six-month follow-up survey. The interim surveys are an important part of the participant tracking efforts because each contact helps to strengthen the participant’s connection to the study and enhance the likelihood of participation in future data collection activities. The interim surveys allow respondents to update or confirm their personal contact information as well as the contact information for up to three alternative contacts that will know how to reach them. Because there is no in-person component to the survey administration effort, obtaining up-to-date phone numbers and knowledge of other individuals who may know how to reach the respondent is critical to the eventual six-month response rate. Finally, the appreciation payment is an important element in encouraging completion of the survey.

Non-Substantive Change Request

Additional Questions to the Six-Month Follow-up Survey

ACF and Abt Associates request to add three to five questions (depending on the skip pattern) to the six-month follow-up survey to collect respondent feedback on their interim survey experiences. The additional questions would add only one minute or less to the average interview.

The series of monthly interim surveys is the primary method for maintaining contact with study participants and updating their contact information over the six-month follow-up period. The interim surveys are conducted via short text messages every month over five months, each accompanied by instructions on how to respond. Non-responders in any month receive a follow-up text after five days to remind them to complete the survey. The response to the interim surveys to-date has been very low (less than 10 percent).

The proposed non-substantive change to the six-month telephone survey would add three to five very short, closed-ended questions (the principal questions and two potential sub-questions based on previous responses) at the end of the interview and have a minimal impact on respondent burden, adding approximately one minute or less to the average interview. By placement at the end of the interview, the questions will not affect collection of other information needed for the evaluation’s analyses. Like any survey questions, respondents can choose not to answer the added questions.

The specific questions we propose are the following:

1) Do you recall getting a text message asking you to participate in a monthly mini survey for the job search study in which you are a part?

2) Did you respond to any of the text messages and complete the mini survey it offered?

IF NO: Ask Q3, IF YES: Ask Q4

3) IF Q2=NO: Why did you choose not to respond? [INDICATE ALL THAT APPLY]

1. Time

2. Costs associated with texting

3. Did not understand what the text was about

4. Did not think the text was legitimate

5. Did not know how to respond

3a) Would you have responded if the appreciation payment had been higher than $2?

4) IF Q2=YES: What influenced you to want to respond to the text survey? [INDICATE ALL THAT APPLY]

1. It seemed like it would be quick

2. It seemed like the right thing to do

3. The $2 for completing each one was appreciated

4. I was curious about the process

5. Other please specify_________________________

The responses to these questions will enable ACF and Abt Associates to make changes to the interim texting survey in its application for the remainder of the JSA Evaluation (sites are coming into the study on a staggered basis and three of potentially five sites have not started random assignment or data collection activities). By addressing respondent feedback on question wording and text presentation, the additional information will help us to determine other potential adjustments such as whether or not to change the texts’ structure and wording, whether to conduct fewer interim surveys by text and instead collect updated contact information by sending out a second contact update form, and/or whether to alter the appreciation payment. It is worth noting that any and all adjustments deemed worth implementing based on respondent feedback obtained from the proposed questions would be submitted as a non-substantive change. Lastly, since text surveys are a relatively new mode of data collection—particularly for inter-wave participant tracking—responses to the added questions will also be important to inform the field for other future evaluation surveys through better understanding of participant experiences with this new methodological-approach.

Change to the Follow-up survey Incentive Structure for Participants

ACF and Abt Associates request a modification to the structure of the appreciation payment for the six-month follow-up survey in an effort to help increase the rate at which individuals consent to be in the study. Currently, approximately one-third of potential study members do not consent to participate in the study because of the time required to complete the baseline information form¹, a situation that is having a negative effect on the study’s potential sample sizes. The current OMB approval allows for the provision of a $25 appreciation payment upon completion of the six month follow-up survey. The team would like to split that $25 payment into two installments. The first installment would be a $5 gift card, distributed after completion of the baseline information form and random assignment. The remaining $20 would be provided upon completion of the six-month follow-up survey, following already approved procedures.

The requested changes to the survey payment structure offer two potential benefits. First, providing part of the survey incentive at the time when consent is provided is likely to increase the study consent rate. A second benefit is that providing an incentive at the time of enrollment will help the participants connect the program and the evaluation. This relatively minor change will both encourage individuals to complete the baseline information form for the study while maintaining a substantive appreciation payment of the six-month survey.

1 The baseline information form for the JSA evaluation contains basic identification, demographic, education, employment and income, and contact information for study participants. Each item is necessary for the analysis and will enable the research team to: (1) describe the characteristics of study participants at the time they are randomly assigned to one of the experimental groups; (2) ensure that random assignment was conducted properly; (3) create subgroups for the analysis; (4) provide contact information to locate individuals for potential follow-up surveys; and (5) improve the precision of the impact estimates. Specifically, identifying and locating information is necessary for tracking and locating sample members for the follow-up survey and for ensuring that we can obtain and accurately match administrative records on sample members. Accurate locating information is also crucial to achieving high response rates on the follow-up survey. And, demographic information and data on education and employment history is essential to describe the study sample and to document differences in the populations served across the study sites, as well as to understand factors that could affect their ability to find and keep jobs. This information will also allow us to conduct subsequent analyses of subgroups.

3