Alternative Supporting Statement Instructions for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Success Sequence Interviews
OMB Information Collection Request
Supporting Statement
Part A
January 2022
Submitted by:
Office of Planning, Research and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officer:
Caryn Blitz
Part A
Executive Summary
Type of Request: This information collection request is for a new information collection. We are requesting one year of approval.
Progress to Date: To ensure the interview questions are clear, uniformly understood, and not overly sensitive, we conducted a pre-test with 40 participants total in English and Spanish across the United States, under the umbrella generic, Pre-testing of Evaluation Data Collection Activities (OMB #0970-0355). The pre-test participants had varying income levels, educational attainment, employment status, and family life experiences.
Description of Request: This request is to collect interview data through virtual asynchronous interviews with adults to learn about their various pathways taken toward a set of life milestones – referred to as the success sequence – in the areas of high school graduation, employment, marriage, and childbearing. We will use the data to help understand the success sequence and the factors in the lives of young adults that may be associated with achieving economic self-sufficiency. The ultimate goal of the data collection is to guide Family and Youth Services Bureau Sexual Risk Avoidance Education programming and research efforts. We do not intend to use the information as the principal basis of public policy decisions. Data are not meant to be generalizable and will not be used as the principal basis for creating policy or making policy decisions.
Study Background
As part of the federal government’s ongoing efforts to support youth in making healthy decisions about their relationships and behavior, Congress updated Title V, Section 510 of the Social Security Act in February 2018 to authorize the Sexual Risk Avoidance Education (SRAE) grant program. SRAE—which is administered by the Family and Youth Services Bureau (FYSB) of the Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS)—funds programs that teach adolescents to refrain from sexual activity. One topic SRAE-funded programs are required to cover is the “success sequence.”1 The success sequence is a set of life milestones that, if followed in the prescribed order, are theoretically associated with economic self-sufficiency in young adulthood (Haskins and Sawhill 2009).2 The milestones listed in the prescribed order are high school graduation, employment, marriage, and childbearing.
To expand available evidence on the success sequence, ACF contracted with Mathematica to conduct a literature review followed by economic data analyses of longitudinal data sets to understand the success sequence more comprehensively. The purpose of the literature review was to summarize the current state of research related to the success sequence, including how researchers define the success sequence, success sequence milestones and the order in which they are achieved, and the relationship between the ordering of the milestones (pathways) and economic outcomes (Goesling et al. 2020).3 The literature review demonstrated a need for more research on the success sequence—in particular, how different pathways shape later outcomes.
A subsequent secondary economic data analyses (Inanc et al. 2021)4 used data from the National Longitudinal Survey of Youth’s 1997 Cohort (NLSY97) to identify the pathways young adults followed most commonly, variations by demographic characteristics, and associations of those sequences with self-sufficiency as a young adult. Findings revealed that, for some individuals, completing the success sequence milestones in the prescribed order was associated with economic self-sufficiency in young adulthood. Further, the results showed other sequences of milestones youth completed were associated with levels of economic success similar to those associated with youth who completed the success sequence in the prescribed order. The economic analyses also pointed to other pathways associated with economic success. Adolescents’ trajectories to adulthood are diverse and complex, signaling a need for more research in this area. Ultimately, the findings raised questions about the underlying factors associated with economic self-sufficiency. Specifically, the analyses demonstrated that economic self-sufficiency could be achieved even if youth did not follow the milestones as prescribed.
With this information collection request (ICR), ACF proposes to conduct interviews with adults, ages 30 through 35. The data collected from the interviews will help fill the gaps in knowledge identified through the literature review and economic analyses. It will help ACF and the broader research field understand adults’ perspectives and experiences related to the milestones, and can provide FYSB’s SRAE grant program with greater insight into the program content and strategies related to the success sequence milestones and their ordering that could best resonate with youth.
ACF has contracted with Mathematica to conduct the Success Sequence Interviews study.
Legal or Administrative Requirements that Necessitate the Collection
There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.
Purpose and Use
The qualitative data we plan to collect will add important contextual information about the specific factors influencing the order of the milestones (high school graduation, employment, marriage, and childbearing), respondents’ perspectives of economic self-sufficiency, and circumstances and specific factors that influenced decisions during their youth and young adulthood related to each milestone, adding to needed research in this developing area, as noted in Section A1.
The information collected will be used to provide ACF with a deeper understanding of the factors that influence the order in which respondents complete the milestones and pathways and how that relates to economic self-sufficiency. It will also help provide FYSB with greater insight into current program content and strategies related to the success sequence that could best resonate with SRAE program participants, informing future training and technical assistance needs of grantees. The study results will be provided to ACF in an internal memorandum, followed by an external public-facing brief.
The collected information will contribute to the body of knowledge on ACF programs. However, the collected information is not intended to be used by federal decision makers as the principal basis for creating policy or making policy decisions. Moreover, we do not expect the information to meet the threshold of influential or highly influential scientific information.
Research Questions or Test
ACF proposes to examine the following guiding research questions for this study:
What are the decisions, circumstances, and other factors associated with participants’ achievement of success sequence milestones, including high school graduation, postsecondary schooling, qualification for licenses and certifications, partnership stability, cohabitation, marriage, and childbearing?
What, if any, factors other than income should be included in the definition of “economic self-sufficiency” as used in success sequence models?
What are the decisions, circumstances, and other factors associated with participants who followed the prescribed success sequence milestones in order, but have not achieved economic self-sufficiency?
What are the decisions, circumstances, and other factors associated with participants who did not follow the prescribed success sequence milestones, or followed them out of order, but have not achieved economic self-sufficiency?
What were participants’ plans related to achieving success sequence milestones, and how do those plans relate to their lived experiences? What circumstances or other critical milestones, if any, affected and/or influenced their paths, and how?
Study Design
The research team at Mathematica will conduct up to 225 virtual asynchronous interviews in English and Spanish to collect data for this study. As discussed in Section A1, secondary analyses conducted using the NLSY97 data5 pointed to the complexity of the success sequence model, with young adults taking a multitude of different pathways, 64 of a possible 65, related to the key success sequence variables of interest: high school graduation, employment, marriage, and childbearing. For this proposed study, we use the findings from the NLSY97 analysis to guide the sample size determinations. Table C.1, in Appendix C, shows the percentage of individuals in the NLSY97 analysis by the order in which they have completed the key success sequence model milestones. Most individuals, over 65 percent, fell into 10 different pathways. For example, 19 percent of the sample reported first completing high school with full-time employment following, but had not yet married or had children; 14 percent of the sample had obtained all four key milestones of interest, and had done so in the order prescribed by the success sequence model, first completing high school, then obtaining full-time employment, marrying, followed by childbearing.
We propose using an interview sample proportional to that of the NLSY97: for example, aiming to conduct at least one interview in the 11 least common pathways taken by the survey participants (indicated in Table C.1 as less than 0.05 percent in the percentage observed in the NLSY97 data column), and up to 55 interviews in the two most common pathways indicated on the first two rows of the table, with all other pathways covered through the interviews as noted in the target sample size column.
To ensure efficient and widespread geographic variation across the United States, Mathematica will collaborate with market research vendors which maintain large standing panels of research participants. With oversight from Mathematica, the vendors will recruit up to 225 interview participants from their panel members (Appendix A: Success Sequence Recruitment Materials). All interested panel members will complete a five-minute telephone screener (Instrument 1) to collect data on demographics, household income, geographic location, and the milestones achieved by each prospective participant. Mathematica will use the information to select participants who reflect a diverse interview sample. Participants will include English- and Spanish-speaking people from a range of demographic groups and geographic areas and will represent people with achievement of, and differences across, various milestones. Supporting Statement Part B, Section B2 provides details about the study methodology and the sample.
The Mathematica research team will use the Interview Protocol (Instrument 2: Success Sequence Interview Protocol), to ask participants to walk through parts of their life including high school graduation, employment, marriage, and childbearing while offering a chronology of how the sequence of events unfolded (Paulson 2011).6 The virtual asynchronous interviews will use an online chat board platform that permits interviewees to log in to the chat board at any time during a three-day period and respond to the interview questions via chat boards moderated asynchronously by the research team’s interviewers. This method enables multiple interviews to take place in parallel, but participants cannot see other participants’ responses. Such a methodology has proven convenient for participants and offers an anonymous environment for answering sensitive questions (Section A11), thereby encouraging respondents to be honest in their responses and potentially increase data quality.7
In Table A.1, we summarize the study design, including the two data collection activities, the data collection instruments and their content, respondent type, and the mode and duration of each data collection activity. Supporting Statement Part B, Section B2 describes further the methods, design, and sample of the study.
Table A.1. Study design summary
Data collection activity |
Instruments |
Participant, content, purpose of collection |
Mode and duration |
||||
Success Sequence Interview Screening |
Instrument 1: Success Sequence Screener |
Respondents: Adults ages 30–35 who speak English and Spanish Content: Focuses on demographics, life milestones, and income Purpose: To screen interview participants for the interview, ensuring participants’ diversity of milestone experiences and demographic and geographic variation |
Mode: Telephone Duration: 5 minutes |
|
|||
Administration of Success Sequence Interview |
Instrument 2: Success Sequence Interview Protocol |
Respondents: English- and Spanish-speaking adults ages 30–35 Content: Focuses on education, employment, cohabitation, marriage, childbearing, economic self-sufficiency, and rationale for decision making Purpose: To gain understanding of the complex decisions and circumstances of youth transitions to adulthood and to explore the complexities around achieving the success sequence milestones, thereby providing ACF and the broader field with information to complement the quantitative analyses of the Success Sequence model |
Mode: Individual chat board interviews Duration: 45 minutes |
|
|||
Potential limitations of the study design include lack of visual or verbal clues from respondents during the interview and relatively less control of the interview by the interviewer compared to in-person interviews (Reisner et al. 2018).8 However the research team’s recent experience conducting 40 pre-test interviews9 found no indication that the mode of interview negatively affected the quality of the data (see Supporting Statement Part B, Section B3). For this proposed study, the research team will again closely monitor and vigorously moderate each chat board during the interview period to ensure timely probing and a close-to-real-time response.
Even though we will attempt to reach a diverse sample, we will likely miss some populations, including those incarcerated, unhoused, without reliable Internet access, or with disabilities that could prevent them from accessing a research panel. Materials resulting from the data collection will note the limitations.
Other Data Sources and Uses of Information
The additional secondary data analyses conducted on the NLSY97 data set (discussed in Sections A1 and A2) have provided a richer understanding of the underlying factors associated with completing specific milestones and their timing, and their link to eventual economic self-sufficiency. The study team will use that information in concert with the information collected as a part of this request to inform future training and technical assistance needs of grantees.
A3. Use of Information Technology to Reduce Burden
Mathematica will collect qualitative information from the interviews via QualBoard, a chat board platform that imposes a smaller burden than other interview methods.10 The online message board permits respondents to maintain anonymity when discussing their life histories with the interviewer and provides flexibility in scheduling the interview with participants since participants can answer interview questions and interviewees can respond and probe asynchronously at any time.
Participants type their responses into the board as messages, which are stored and used as the interview transcript. The transcript can be exported, coded, and analyzed. Boards will be open for responses 24 hours a day. The research team will monitor the boards during both day and evening hours so participants can respond when it is convenient for them.
A4. Use of Existing Data: Efforts to Reduce Duplication, Minimize Burden, and Increase Utility and Government Efficiency
The Success Sequence Interviews study is the first effort to collect qualitative data on the success sequence model. ACF has carefully reviewed the information collection requirements, and no other federal or nonfederal studies are collecting the same type of data.
A5. Impact on Small Businesses
This information collection will not involve any small businesses.
A6. Consequences of Less Frequent Collection
This is a one-time data collection.
A7. Now subsumed under 2(b) above and 10 (below)
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on November 9, 2021, Volume 86, Number 214, page 62178 and provided a 60-day period for public comment. During the notice and comment period, no comments were received.
We will not be seeking consultation from experts outside of the study.
A9. Tokens of Appreciation
To recruit participants and conduct the interview activities efficiently, we plan to offer a $40 Visa gift card. This is the standard incentive offered by research firms who recruit respondents for studies such as this one; these firms will not help to support data collection if the proposed incentive is lower than what they can offer for other studies. Also, these tokens of appreciation will help ensure participants reflect the diversity of backgrounds and characteristics of the U.S. population and that participants have varying income levels, including those with low incomes. We expect the proposed data collection design, which includes using a virtual asynchronous method, to offset some of these costs by giving participants flexibility in when they can respond, but still anticipate some incidental costs related to participation.
A10. Privacy: Procedures to Protect Privacy of Information, While Maximizing Data Sharing
Personally Identifiable Information
The research team for the Success Sequence Interviews study will not collect personally identifiable information (PII). As part of recruitment, the market research panel vendors will record each participant’s name and email address and use this contact information to send the participants invitation emails. The vendors serve as a protective intermediary for PII and responses to interview questions. Neither Mathematica nor ACF will have access to the PII; the vendors will provide Mathematica with unique identifying numbers for recruited interviewees and their responses to the screener questions (Instrument 1: Success Sequence Screener). Using Instrument 1, the vendors will also collect demographic information, such as age, race, and ethnicity to screen a diverse set of participants. When participants log in to the virtual chat board, they will receive instructions to not share their full name, but rather to use their first name, initial, or an alias.
Assurances of Privacy
The study team will inform respondents of all planned uses of data, that their participation is voluntary, and that the team will keep their information private to the extent permitted by law. As the contract specifies, the contractor will protect respondents’ privacy to the extent permitted by law and will comply with all federal and departmental regulations for private information. Appendix B provides a study consent form (Appendix B: Consent Form) containing assurances of privacy; all participants must read and acknowledge the form before participating in the data collection.
Data Security and Monitoring
The contractor has developed a data safety and monitoring plan that assesses all protections of respondents’ information. The contractor will ensure all of its employees, subcontractors (at all tiers), and employees of each subcontractor who perform work under this contract and subcontract receive training on data privacy issues and comply with the above requirements. All research team staff involved in the study will receive training in (1) limitations of disclosure; (2) safeguarding the physical work environment; and (3) storing, transmitting, and destroying data securely. All Mathematica staff must sign the Mathematica Confidentiality Agreement, complete online security awareness training when they are hired, and participate in a refresher training annually.
As specified in the evaluator’s contract, the contractor will use encryption compliant with the Federal Information Processing Standard (Security Requirements for Cryptographic Module, as amended) to protect all sensitive information during storage and transmission. The contractor will securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Information Processing Standard. The contractor will incorporate this standard into its property management and control system and establish a procedure to account for all laptop and desktop computers and other mobile devices and portable media that store or process sensitive information. The contractor will secure any data stored electronically in accordance with the most current National Institute of Standards and Technology requirements and other applicable federal and departmental regulations. In addition, the contractor’s data safety and monitoring plan includes strategies for minimizing to the extent possible including sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive information that ensures secure storage and limits on access.
Only users who are added to each chat board as moderators (Mathematica staff) or participants will have access to the board. Participants will gain access by logging on to the secure website with a user name and password. QualBoard will fully mask individual participants’ responses to interview questions and moderator probes, meaning that only the participant and the moderators can view the responses.
A11. Sensitive Information
Participants could perceive some of the interview questions included in the interview protocol as sensitive, such as those pertaining to relationship history and childbearing. These questions are necessary for the Success Sequence Interview study to address ACF’s informational needs: namely, to improve SRAE program content related to the success sequence (Section A2).
Before the start of the interviews, participants will read the consent form (Appendix B), which describes the purpose of the study, indicates their rights as research participants, and acknowledges their consent before starting the interview. The Interviewers will inform all participants they do not have to respond to any questions that make them uncomfortable and their participation is voluntary.
An institutional review board has approved the protocol and all related materials, such as the consent form.
Table A.2 lists sensitive questions that will be in the interview and the justification for asking them.
Table A.2. Justification for sensitive questions in the screener and interview protocol
Topics and interview questions |
Justification |
Income and economic support
|
A primary goal of success sequence research is to determine whether adults who follow the sequence are economically self-sufficient. These questions are the primary outcome variables and provide context for the variations in income and support as they relate to the sequence of life milestones. |
Relationship history and childbearing
|
To understand participants’ rationale for completing or not completing the success sequence milestones in the prescribed order, it is necessary to understand the circumstances that led to participants’ decisions about relationships, cohabitation, and marriage and childbearing. |
Gender and sexual orientation
|
Gender and sexual orientation may impact the life milestones and the ordering in which a participant completes them, for example whether a pregnancy has occurred or the impact of sexual orientation on marriage and childbearing. |
Mental or physical health barriers or other barriers
|
The attainment and ordering of the life milestones may be impacted by mental or physical health barriers, or by other barriers such as engagement with the criminal justice system. These questions are necessary to gain a more complete picture of barriers that may have impacted the success sequence. |
A12. Burden
Explanation of Burden Estimates
In Table A.3, we summarize the estimated reporting burden and costs for each instrument.
Screener. As described in Section A2, to ensure interviews are conducted proportional to the number of individuals reporting on their milestone order completion from the NLSY97 economic analysis, we propose conducting 225 completed interviews. We estimate it will be necessary to complete an eligibility screen with about 675 respondents to obtain 225 completed interviews. Each potential participant will be screened on the key success sequence milestones. To conduct interviews in proportionality to the NLSY97 success sequence milestones findings as described in Appendix C, Table C.1, we anticipate screener participants will mirror the milestones achieved in the NLSY97 data. Therefore, we anticipate that of the 675 screener participants, 438 will fall into the top 10 most common milestone pathways and 328 will screen out of the study simply to achieve the proposed target interview sample sizes for those categories. We aim to screen an additional 237 participants to obtain about 115 interview participants for the remaining less common pathways found in the NLSY97 data. The overall goal is to obtain 110 interview participants in the 10 most common pathways and 115 participants in the less common pathways. The eligibility screener will take about 5 minutes (0.083 hours), for a total estimated annual burden of 56 hours.
Interview. We estimate up to 225 of the eligible online research panel participants selected to participate in the study will complete the interviews. The number of completes is necessary to systematically test the success sequence pathways found in the economic analysis with a purposively selected diverse sample. As described further in Supporting Statement Part B, Section B2, the target sample size of 225 is necessary to proportionally match the interview sample to the number of cases by milestone combinations found in the economic analysis. We estimate the interview will take 45 minutes for completion via an online chat board, for an estimated annual burden of 169 hours.
Estimated Annualized Cost to Respondents
We estimate the average hourly wage for the participants of this study is the average hourly wage of “all occupations” taken from the May 2020 National Occupation Employment and Wage Estimates, U.S. Bureau of Labor Statistics ($27.07).11 The estimated burden results appear in Table A.3.
Table A.3. Total burden requested under this information collection
Instrument |
No. of participants (total over request period) |
No. of responses per participant (total over request period) |
Avg. burden per response (hours) |
Total/annual burden (hours) |
Average hourly wage rate |
Total annual participant cost |
||||||
Instrument 1. Success Sequence Screener |
675 |
1 |
0.083 |
56 |
$27.07 |
$1,515.92 |
||||||
Instrument 2. Success Sequence Interview Protocol |
225 |
1 |
0.75 |
169 |
$27.07 |
$4,574.83 |
||||||
Estimated total annual burden |
225 |
|
$6,090.75 |
|||||||||
A13. Costs
There are no additional costs to respondents.
A14. Estimated Annualized Costs to the Federal Government
The total cost to the federal government for this study is $173, 659 (Table A.4). This includes the costs incurred for designing and administering all collection instruments, processing and analyzing the data, and preparing reports.
Table A.4. Estimated total cost by category
Cost category |
Estimated costs |
|
|
|
|
Field work |
$123,659 |
|
Summary brief |
$50,000 |
|
Total/annual costs over the request period |
$173, 659 |
|
A15. Reasons for changes in burden
This is a new information collection request.
A16. Timeline
Recruitment for interviews should begin immediately after OMB approval, during late winter or early spring 2022. We expect to conduct the interviews over a six-week period. Following data collection, Mathematica will include the results in an internal memorandum to ACF. The memorandum will summarize key themes and describe the study methods and the limitations regarding generalizability and use for making policy decisions. Mathematica will develop the first draft of the memorandum during late spring 2022, and deliver the final memorandum and an external brief to ACF in summer 2022.
A17. Exceptions
No exceptions are necessary for this information collection.
Attachments
Appendices
Appendix A: Success Sequence Recruitment Materials (English and Spanish)
Appendix B: Consent Form (English and Spanish)
Appendix C: Success Sequence Interview Study Target Sample
Appendix D: QualBoard Invitation Email Sample Screen Shot (English and Spanish)
Instruments
Instrument 1: Success Sequence Screener (English and Spanish)
Instrument 2: Success Sequence Interview Protocol (English and Spanish)
1 Family and Youth Services Bureau. “State Personal Responsibility Education Program Fact Sheet.” Washington, DC: Family and Youth Services Bureau, 2020. Available at https://www.acf.hhs.gov/fysb/fact-sheet/title-v-state-sexual-risk-avoidance-education-fact-sheet.
2 Haskins, R., and I.V. Sawhill. Creating an Opportunity Society. Washington, DC: Brookings Institution Press, 2009.
3 Goesling, B., H. Inanc, and A. Rachidi. “Success Sequence: A Synthesis of the Literature.” OPRE Report #2020-41. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2020.
4 Inanc, H., A. Spitzer, and B. Goesling. “Assessing the Benefits of the Success Sequence for Economic Self-Sufficiency and Family Stability.” OPRE Report #2021-148. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
5 Inanc, H., A. Spitzer, and B. Goesling. “Assessing the Benefits of the Success Sequence for Economic Self-Sufficiency and Family Stability.” OPRE Report #2021-148. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
6 Paulson, S. “The Use of Ethnography and Narrative Interviews in a Study of ‘Cultures of Dance.’” Journal of Health Psychology, vol. 16, no. 1, 2011, pp. 148–157.
7 Reisner, S.L., R.K. Randazzo, J.M. White Hughto, S. Peitzmeier, L.Z. DuBois, D.J. Pardee, E. Marrow, S. McLean, and J. Potter. “Sensitive Health Topics with Underserved Patient Populations: Methodological Considerations for Online Focus Group Discussions.” Qualitative Health Research, vol. 28, no. 10, 2018, pp. 1658–1673.
8 Reisner, S.L., R.K. Randazzo, J.M. White Hughto, S. Peitzmeier, L.Z. DuBois, D.J. Pardee, E. Marrow, S. McLean, and J. Potter. “Sensitive Health Topics with Underserved Patient Populations: Methodological Considerations for Online Focus Group Discussions.” Qualitative Health Research, vol. 28, no. 10, 2018, pp. 1658–1673.
9 Pre-test conducted under generic OMB clearance package #0970-0355.
10 Opdenakker, R. “Advantages and Disadvantages of Four Interview Techniques in Qualitative Research.” Qualitative Research in Ibero America, vol. 7, no. 4, 2006, p. 7.
11 U.S. Bureau of Labor Statistics. “May 2020 National Occupational Employment and Wage Estimates.” Available at https://www.bls.gov/oes/current/oes_nat.htm.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Carol Soble |
| File Modified | 0000-00-00 |
| File Created | 2022-01-13 |