Memo - NonSub Change: Wording changes based on Pretesting

DATE: February 26, 2018

TO: Steph Tatham

Office of Information and Regulatory Affairs (OIRA)

Office of Management and Budget (OMB)

FROM: Nicole Constance and Amelia Popham

Office of Planning, Research, and Evaluation (OPRE)

Administration for Children and Families (ACF)

SUBJECT: Request for Non-Substantive Change to Pathways for Advancing Careers and Education (PACE) (OMB Control Numbers 0970-0397)

This memo requests approval for revisions to the 72-month Follow-up Survey for the PACE study (OMB Control Number 0970-0397, approved on 9/18/2017). These changes are based on findings from the pretest. The changes have been reviewed and approved by the Abt Associates Institutional Review Board. The remainder of this memo provides information about the revisions made to the materials.

The 72-month follow-up survey captures data on the following outcomes of interest to the PACE study, six years after random assignment:

• Employment success and promotions;

• Current/most recent job conditions, job quality, benefits, on the job training;

• Education and credentials;

• Adult well-being, life challenges, social networks, perceived stress, and physical health;

• Household composition, family formation and marital stability;

• Income and economic well-being, student debt, financial resilience;

• Time out of home/child supervision;

• Child education-related goals and support;

• Child outcomes;

• Transition to adulthood; and

• Contact information.

In late December the study team completed a pretest of the survey with seven PACE participants. The instrument was programmed for administration via computer assisted personal interviewing (CAPI). Overall, the questionnaire worked well in CAPI with both respondents and the interviewer understanding most of the questions. The interviewer confirmed that the instrument flows well.

However, the study team identified a number of small, non-substantive changes that will improve the clarity of the questions—making it easier for interviewers to administer and for participants to respond. We are seeking OMB approval to make these minor modifications. These changes do not change the estimated respondent burden.

Reason for the Non-Substantive Change Request

This change request includes changes to the survey instrument for five primary purposes:

• Improving the question text by adding fill logic for non-standard cases (e.g., the words filled in when someone does not name their employer);

• Adding introductory or clarifying text to explain the purpose of question;

• Modifications to some of the response options to ensure they are clear to respondents;

• Adding logic checks to the programming to improve data quality; and

• Correcting the skip logic (i.e., which question to ask next based on the respondent’s previous responses).

We summarize the requested changes by type of change in this memo. The revised survey instrument is attached: Pathways for Advancing Careers and Education (PACE) 72-Month Follow-up Survey.

Improved Fill Logic for Non-Standard Cases

• B3, B4, B5, B7, B8, B9, and B10 fill logic correction. These questions have the name of the employer (from B2) filled in to make sure the respondent keeps track of which job is being asked about. We updated the fill logic for these questions so that they flow more smoothly when the respondent does not know the name of their employer in question B2. This change was made to the instrument for the pretest. For example,

  • B3 is: “Are you still work at [name of employer from B2]?

  • B3 for respondents that don’t report name of employer will be updated to say “Are you still working at that job?”
    
    
    

• B24 fill logic addition. In the current instrument, B24 referred to “promotions,” however in B22 (“How many promotions have you received since [MYLI]?”) we determine the number of promotions the respondent has received. We are updating the B24 to read “promotion” (if 1 promotion in B22) or “promotions” (if more than one promotion in B22).

• C6, C6a, C7, C8, C9, and C10 fill logic correction. We will update the fill logic for these questions so that they flow more smoothly when the respondent does not know or refuses the name of their employer in question B16. For example:

  • C6 is: “Did you usually work a regular daytime schedule or some other schedule at [employer from B16]?”

  • C6 for respondents that do not report name employer will be updated to say: “Did you usually work a regular daytime schedule or some other schedule at that job?”

• D3a fill logic addition. Update the question text to read “What type of place were you taking classes at?” when the respondent indicates they are not currently enrolled in classes, but have been enrolled since date of last interview (D2A = yes) and keep the wording as is for those currently enrolled (D2=yes) as “What type of place are you currently taking classes at?”

• D3e fill logic correction. Update the wording of this question when the respondent does not know or refuses to provide the name of the place they are taking classes. In this situation, D3e should read “I am going to read you a list of funding sources that you might have used to pay these school or living expenses while attending classes at that institution.”

Adding introductory or clarifying text to a question

• D4. D5, and D6 introduction clarification. For participants that responded to the 36-month survey, we will read back the list of credentials they told us about then to ensure that they only report new credentials since that time. We added clarifying language to introductory text to these three questions. We also added an option for interviewers to note if the respondent expressed concerns about their prior responses and gave the interviewers a script to read back.

• D8a word clarification. Change the word from “have” to “receive” so this question is changed from “…did you have instruction or training from a co-worker or supervisor?” to “…did you receive instruction or training from a co-worker or supervisor?”

• D9 wording clarification. Some respondents were confused by this question, specifically the word “ones” as in “Other than the ones you just told me about…” We are changing “ones” to “jobs” in all places: so that the full question now reads “Think about the jobs you’ve had over the past three years, since [MYLI]. Other than the jobs you just told me about, have you participated in a formal training program offered by an employer or a union that helped you to learn or improve the skills needed to do your job?”

• G16 introduction clarification. G16 (“Think about all other debt you have...?”) confused some respondents because it is not clear what is considered “other” debt. We are adding clarifying text similar to what was provided in the 36-month instrument to help guide the respondent on what to include and exclude in their response.

“The next question is about the total amount of debt that you have:

Please do not include any amount YOU owe for your own school loans or for your home mortgage, or car loans. Do include all outstanding balances on any other loans, credit cards, and medical bills and any overdue child support payments. 

Please also include any amount [IF F1A=1: your spouse/ IF F1B=1: your romantic partner] currently owes in school loans.

Which of the following categories best describes the amount you [IF F1A=1: and your spouse/ IF F1B=1: and your romantic partner] currently owe IN TOTAL for those things. Would you say your total other debt was… ”

Modification of response options to improve quality

• D1 typo correction. Response option 6 reads “1 or more years of college, no degree” – we are updating this to “1 or more years of college credit, no degree” to make it consistent with response option 5 (“Some college, but less than 1 year of college credit”) and consistent with its original source.

• D3 changing response options. Respondents weren’t always sure how to determine full or part time enrollment status. To improve the quality of this data we changed the response options to capture number of hours spent in training per week rather than whether they were enrolled full time, part time or an equal mix of both.

• F2 fixing response categories. As the instrument was programmed, the list of response options (a year or less, 2-3 years, 4-6 years, more than 6 years) did not account for respondents who have been married/living together for over a year, but less than two years. We are changing the categories to “Less than a year, 1 to 3 years, 4 to 6 years, or more than 6 years.”

Adding quality checks

• F5 Add quality check. Add logic to F5 (how many of these children are you primary caregiver for?) so that the answer provided cannot be greater than the answer provided at F4 (how many children live with you?).

• F9 Add quality check. Add logic to F9 (how many of these children live with you more than half the time?) so that the answer provided cannot be greater than the answer provided at F8 (how many children have you had/fathered since RA date?).

Correcting the skip logic

• D2 skip logic fix. In the current version, if the respondent answered don’t know or refused to D2 (currently enrolled in classes), they were skipped out of D2a (ever enrolled since last interview). This will be corrected so that respondents that answer don’t know or refused to D2 get asked D2a.

• G22 skip logic fix. G22 “Do you take health insurance through [Employer]…” is only to be asked of respondents who are currently employed. We added skip logic before this question to remove people not currently employed from this question.

• H3b skip logic fix. The logic before this question is incorrect in the current version of the instrument (H2 = not enrolled in school or H3a = Child is too young to go to kindergarten). The logic is intending to have H3b asked of children who are not currently enrolled in school, but are old enough for kindergarten. We are updating the logic to read “H2 = no or H3a does not equal Child is too young to go to kindergarten.”

• H3d skip logic fix. We updated the logic that determines if this question is asked of the respondent to also include “ASK IF H3A = 14, 15, 16, or 17” for the pretest. The logic wording was correct, but the responses to implement the skip logic were incorrect.

• H3d skip logic fix. In the current instrument, if the focal child does not live with the respondent at least half the time, regardless of the child’s age, the remainder of section H is skipped and the case is taken to section I. However, if the focal child is over 18 years of age and does not live with the respondent half the time (which will be common in this group), we will instead ask H25 through H29a (Qs asking whether working, going to school, married, or have children).

• H22 skip logic fix. In the current instrument, the skip logic for each response took the case to H25. However, H25 is only to be asked of children who are 18+. The logic will be updated to go to the skip logic before the H23 series of questions. This change will ensure the interview follows the correct path.

Expected Benefits

We expect these requested changes to improve the overall quality of the instrument design and flow, without any impact on respondent burden. These changes are expected to improve the survey in several ways. First, they will improve the instrument usability. Clearly worded questions improve the interviewer’s ability to administer the questions and the respondent’s ability to respond to them. Second, improved wording and response options improve the quality of the data collected. For example, by quantifying the hours spent in training in D3, as opposed to capturing a more subjective response on full time or part time status, offers more precise measure of the total time participants spent in training. Finally improving the skip logic ensures that questions are not asked of a participant unnecessarily.

4