Supporting Statement for OMB Clearance Request
Part A: Justification
Innovative Strategies for Increasing Self-Sufficiency (ISIS) – Follow-up Data Collection
OMB No. 0970-0397
July 2013
Submitted by:
Brendan Kelly
Office of Planning,
Research
and Evaluation
Administration for
Children
and Families
U.S. Department of Health and Human Services
Supporting Statement for OMB Clearance Request – Part A: Justification
Table of Contents
A.1 Necessity for the Data Collection 1
A.1.1 Legal or Administrative Requirements that Necessitate the Collection 1
A.1.5 Data Collection in the ISIS Evaluation 3
A.2 Purpose of Survey and Data Collection Procedures 5
A.2.1 Overview of Purpose and Approach 5
A.2.2 Data Collection Process 6
A.2.3 Who Will Use the Information 6
A.2.4 Instrument Item-by-Item Justification 7
A.3 Improved Information Technology to Reduce Burden 11
A.4 Efforts to Identify Duplication 11
A.5 Involvement of Small Organizations 12
A.6 Consequences of Less Frequent Data Collection 12
A.8 Federal Register Notice and Consultation 13
A.10 Confidentiality of Respondents 15
A.10.1 Data Confidentiality Protections 15
A.12 Estimation of Information Collection Burden 16
A.13 Cost Burden to Respondents or Record Keepers 19
A.14 Estimate of Cost to the Federal Government 19
A.16 Plan and Time Schedule for Information Collection, Tabulation and Publication 19
A.16.2 Time Schedule and Publications 20
A.17 Reasons Not to Display OMB Expiration Date 20
A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 20
In this document, we provide justification for the next set of data collection activities for the Innovative Strategies for Increasing Self-Sufficiency (ISIS) evaluation sponsored by the Office of Planning, Research and Evaluation (OPRE) in the Administration for Children and Families (ACF) in the U.S. Department of Health and Human Services (HHS).
OMB approval was received in the fall of 2011 for ISIS baseline data collection (OMB No. 0970-0397). Instruments approved in that submission included the Basic Information Form (BIF), the Self-Administered Questionnaire (SAQ), and the implementation interview guides for the first round of site visits to each of the nine career pathways programs included in the study.
This submission seeks OMB approval for several follow-up data collection instruments. Specifically this request includes the following:
Modification to the currently approved Basic Information Form (BIF) requesting some basic information about all of the study participants’ children (if applicable);
Follow-up survey that will be administered to all study participants approximately 15 months following enrollment in the study;
Interview guides for the in-person visits to the ISIS sites to structure discussions with program staff and key stakeholders, including: program leadership/managers, instructional staff, case managers/advisors, and partners;
Online surveys for case managers/advisors, managers/supervisors, and instructional staff that supplement the above interviews; and
In-depth interview guide for a small sample of study participants.
Subsequent OMB submissions will seek clearance for any future follow-up data collection activities.
OPRE seeks approval for the follow-up data collection activities described in this request in order to support a study conducted for it by Abt Associates, Inc. (Abt). The ISIS evaluation will assess a range of promising post-secondary career pathways programs that promote the improvement of education, employment and self-sufficiency outcomes for economically disadvantaged adults. The major goal of ISIS is to assess the effectiveness of a group of these programs in increasing 1) the receipt of educational credentials, 2) employment and earnings, and 3) self-sufficiency and other measures of well-being. ACF believes that development of rigorous evidence on these matters will be of great use to both policymakers and program administrators.
There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection as part of its ongoing effort to improve the economic well-being of the low-income population.
ACF conceived of the ISIS project as a test of promising interventions for improving the economic prospects of low-income individuals and families. After extensive outreach to the program and policy community (conducted under OMB clearance No. 0907-0343), ACF determined that the focus of ISIS would be programs that fit into the career pathways framework. Appendix A is an exhibit of the career pathways framework and the theory of change logic model used to guide the ISIS evaluation.
ISIS study sites target low-income adults who are interested in occupational skills training. The sites conduct random assignment of individuals to one of two groups: a treatment group that is offered the innovative career pathways interventions, or a control group that is able to access a set of “business-as-usual” or any other services except the ISIS services. The target sample size in eight of the nine sites is roughly 1,000—with about 500 in each of the two research groups (in one of these sites, the sample will come from three sub-sites). The ninth site has an estimated sample of 2,700 across eight sub-sites, with 1,800 in the treatment group and 900 in the control group. Appendix B provides summaries of the nine ISIS programs.
Research questions the ISIS evaluation will address include:
Implementation—What services are provided under each intervention? What are the characteristics of the populations served? How are services implemented? What are the issues and challenges associated with implementing and operating the service packages and policy approaches studied? How do services available to r the treatment group compare to the services available to the control group? How does the take-up and utilization of services by the treatment group compare to the take-up and utilization of the control group?
The implementation study will use the interview guides included in this clearance package to answer the first set of questions. Researchers will use interviews with program staff and partners to collect information that will provide a fuller understanding of the conditions surrounding these career pathway programs and the contexts in which they operate. This information also will allow researchers to assess the quality of the implementation of these programs—assessments that will be important to the interpretation of program impact results. The implementation study will also use an online survey of instructors, case managers/advisors and managers/supervisors to supplement information collected during the staff interviews. The implementation study will also rely on the 15-month follow-up survey to address the question of the participation patterns of treatment group members, as well as to estimate the difference in service receipt between those in the treatment and control groups. Finally, the implementation study will use in-depth interviews with a sample of treatment and control group members to gain a more comprehensive understanding of their experiences with services. Additional implementation research instruments, developed for another ACF-funded research study, will also be administered to the ISIS sites. These instruments will be submitted to OMB under a different clearance request (see future data collection).
Impact—What are the net impacts of career pathway programs on educational outcomes (program completion, attainment of credentials and degrees) and economic outcomes (earnings, employment levels, and wage progression)?
The impact study will use baseline data (BIF and SAQ) and 15-month follow-up survey data to address impact study research questions. Additionally, data on study participants’ wages will be collected from the National Directory of New Hires (NDNH). This is expected to reduce the burden on study participants by negating the need to ask detailed earnings question on the follow-up survey. As well, administrative records are not subject to recall error or non-response. Additionally, the research team plans to use the National Student Clearinghouse (NSC) to gather information about college persistence and degree completion for study participants. NSC is currently the only data source which tracks postsecondary student enrollment across states.
Data collected for ISIS will provide a rich body of information from which to answer these key research questions. For example, the research team will be able to say with confidence whether a program improved credential receipt at the time of the first follow-up. The research team will analyze and report on each ISIS program as a separate study. In addition, the research team will conduct limited cross-site impact analyses. This package seeks clearance for the following impact study information collection items: revised BIF (addition of questions on the study participants’ children), the 15-month follow-up survey, and the tracking letter with contact update form and advance letter, which will help to ensure the research team can locate the study participants for the follow-up data collection activities.
Cost effectiveness—What are the costs of career pathway programs in the study? Do the estimated benefits of providing services outweigh the costs of these programs?
The project will address the cost-effectiveness of programs through comparison of net economic benefits with net program costs in the cost-benefit study. Although the bulk of cost data will come from programs’ existing administrative records, ISIS researchers will augment their understanding of program costs through interviews with program staff.
In 2011, the ISIS project obtained OMB clearance for the baseline data collection (OMB No. 0970-0397). The following instruments were approved under this clearance:
Basic Information Form (BIF) for participants that collects general demographic and contact information. The BIF is administered during intake prior to random assignment by an intake staff person or self-administered on a paper form.
Self-Administered Questionnaire (SAQ) for participants that collects more sensitive and personal information, including several psycho-social items designed and validated by the testing firm ACT, Inc. The SAQ is also completed during intake prior to random assignment and is self-administered on a paper form.
Interview Guides for interviews with program staff that is used to collect information from site staff personnel. The project team is interviewing program administrators and staff at ISIS sites and other organizations that partner with ISIS sites to deliver services. These interviews take place on site visits during the pilot and early full implementation stages.
The goal of the baseline data collection is to document the demographic and other characteristics of individuals at the time they enroll in the ISIS study. In addition, the study team is conducting an initial round of visits to each site to conduct interviews with key informants to document the program structure and service delivery process, as well as the implementation of study procedures in each of the nine ISIS programs.
This current submission seeks clearance for several follow-up data collection instruments developed for the ISIS evaluation. A copy each instrument is in the Appendix. The instruments include:
Instrument #1: Basic Information Form Modification. This small but very important modification to the BIF will allow the study team to collect information at baseline regarding the sample members’ children (as applicable). The newly added questions will help to establish a sampling frame for future follow-up activities that estimate the effects of the programs on the children of those in the study who are parents. (Appendix D)
Instrument #2: 15-month Follow-up Survey. The follow-up survey will collect information from study participants 15 months following random assignment. The follow-up survey will be administered by telephone using specially trained interviewers and will capture information on outcome measures for treatment and control group members in several domains including education and training, employment and income, and life circumstances. Field follow-up will be used to contact participants who could not be reached after multiple phone attempts. (Appendix H)
Instruments #3-6: Staff Interview Topic Guide for Program Leadership/Managers (#3), Instructional Staff (#4), Case Managers/Advisors (#5), and Partners (#6). Interview topic guides for the implementation study will be used during a second round of site visits to each program to collect information from ISIS program staff and other organizations involved in the delivery of services. The interview guides will collect data to describe the programs as implemented, including core components, management and staffing and contextual factors. In addition to describing the interventions, this information will help the research team interpret impact results. (Appendices I through L)
Instruments #7-9: Online Surveys of Case Managers/Advisors (#7), Managers/Supervisors (#8) and Instructional Staff (#9). Online staff surveys will be administered at each of the nine ISIS programs. The case manager/advisor survey will focus on the issues covered with students (personal, academic, career planning, employment, financial), the amount of time spent with students, staff development activities, and how closely student progress and completion is monitored. The managers/supervisor survey will inquire about staff background, the nature of assistance provided to program participants and staff development and morale. The instructor survey will elicit quantitative data about class size, the extent to which basic skills are integrated with training instruction, the use of, and time spent on, different instructional modes, instructor backgrounds, staff development activities, staff autonomy and morale. (Appendices M through O)
Instrument #10: In-depth Study Participant Interviews. In-depth interview guides will be used to collect information from a sample of study participants from each site at two points in time, as well as for a brief interim telephone check in. In 7 programs that are single sites (i.e., no sub-sites), the team will interview 10 treatment and 5 control group members. In programs with sub-sites, the team will interview 10 treatment and 5 control group members per sub-site in one program and half in another. This information will be used to gain a more comprehensive understanding of treatment and control members’ experiences with the services. (Appendix P)
ACF will seek OMB approval at a later point for additional information collections as part of the ISIS evaluation:
Data sources that will need OMB approval:
Employer and stakeholder/network surveys. ACF is funding development of these surveys as part of the Health Profession Opportunity Grants National Implementation Evaluation (HPOG-NIE). The surveys will be submitted to OMB for approval as part of a future clearance request. Burden associated with administering these surveys at ISIS sites will be included in that request.
Follow-up surveys: ACF may fund additional follow-up surveys in the future (the next, if funded, is tentatively scheduled to take place 36 months after random assignment). Additional survey instruments will be submitted to OMB for clearance.
Other Data sources:
Government administrative records: These records will include Unemployment Insurance (UI) and federal wage records. The project has established an agreement with ACF's Office of Child Support Enforcement to utilize the UI and wage records from the National Directory of New Hires (NDNH).
National Student Clearinghouse: NSC includes 3,641 participating public and private institutions that collectively represent approximately 93 percent of higher education enrollments nationwide. The project will use NSC data for information on college persistence and degree completion.
Program records: The project team will collect data on outcomes from the programs for the implementation study and for the impact study in sites where program records are available for both treatment and control group members. Illustrative outcomes include measures of basic academic skills, services received, and credits and credentials earned. For some outcomes – such as educational attainment at post-secondary institutions – it is likely that data will be gathered from centralized state databases. The information included in these records will differ from site to site based on the information collected by each site's management information system(s). Where the site is a community college, the study team may be able to get reasonably comparable data on both treatment and control group members, but for the most part, the team will be limited to data on the former. This will not impose burden on programs because they currently collect this data for their own use.
The ISIS project is an evaluation of promising programs and policies for improving employment and self-sufficiency outcomes for low-income, low-skilled adults. The ISIS study is utilizing an experimental design in nine programs to assess the impact of promising interventions on credential attainment, employment, earnings, and general well-being and will also include an implementation study and cost-benefit study.
The purposes of collecting baseline data with the BIF and SAQ instruments are several. First, the contact information collected at baseline is necessary to enhance researchers’ ability to locate respondents for follow-up surveys that will measure intervention outcomes. A second purpose is to create a rich dataset for future researchers to explore and test hypotheses. Other analytic purposes of the baseline data include characterizing the ISIS study sample, adjusting for chance differences in observable characteristics and thereby increasing precision of impact estimates, identifying subgroups of interest, checking the integrity of random assignment, and adjusting for non-random survey sample attrition. The purpose of adding the child roster questions to the existing BIF is to create a sampling frame for follow-up surveys that collect data about child outcomes. The study team seeks approval for this modification to the existing BIF, which was approved in the previous OMB submission. The modified version is included as Appendix D.
The follow-up survey will measure outcomes and impacts approximately 15 months after random assignment on service receipt, educational attainment, employment, and other life circumstances. This data will be used for the ISIS impact study, as well as to understand treatment and control differentials for the implementation study. The instrument can be found in Appendix H.
The implementation study interview guides and staff surveys included in this clearance package will allow the ISIS evaluation to collect information that will be used to provide a detailed description of programs in the study as actually implemented, including program components, the contexts in which they operate, and changes made to program structure, management or staffing over the course of the study. This information will also help researchers interpret program impact results. The interview topic guides can be found in Appendix I; the online case manager/advisor, manager/supervisor and instructor surveys can be found in Appendix J. The study participant in-depth interview guides will be used to collect information about educational and training experiences from a sample of study participants from each site at two points in time. The guides can be found in Appendix K.
The follow-up survey data collection will take place approximately 15 months following random assignment, which began in the first program in fall of 2011. Therefore, the follow-up data collection will start in 2013 upon OMB approval. The last program will end random assignment in September 2015. Thus, follow-up survey data collection will conclude in December 2016.
ISIS evaluators will conduct the second round of implementation research site visits during the second year of program implementation and will document more mature program operations and any changes that occurred in the sites since the initial visit.
The primary beneficiaries of this planned data collection effort will be ACF, other federal agencies, program operators, and low-income individuals themselves. ACF will use the information to assess the effects of the ISIS programs for low-income individuals. These data will begin to answer ACF's questions about impacts of the post-secondary career pathways programs in all study domains: for example, education and credential achievement, employment and earnings, and income. The implementation research will help identify promising practices in career pathway program structure and implementation, and potentially will guide decisions to implement post-secondary career pathways programs nationwide. Similarly, the Departments of Labor and Education have expressed a strong interest in the ISIS study in particular and career pathways program effectiveness in general. The results of the ISIS study could inform programmatic and funding decisions for all three agencies. Organizations (e.g., community colleges, workforce development agencies, community-based organizations) that are operating or creating career pathways programs will use the study information to refine or design programs for their target populations. Finally, low-income individuals will benefit
from this information to the extent that it demonstrates the cost-effectiveness of career pathways programming and contributes to a body of evidence to inform program and policy design and investments.
Secondary beneficiaries of this data collection will be those in the public policy and social science research community who are interested in further understanding initiatives to promote economic self-sufficiency of individuals and families through comprehensive career pathways programs. At the conclusion of the ISIS study, the research team will provide ACF with a restricted-use data set containing individual level data stripped of all personally identifying information. The restricted-use data will be made available to researchers for approved secondary uses.
Ultimately, these data will benefit researchers, policy analysts, and policy makers in a wide range of program areas. The effects of post-secondary career pathways programs on the well-being of low-income individuals and families could manifest themselves in many dimensions and could be relevant to an array of other public programs. This project offers the first opportunity to obtain reliable measures of these effects. The long-term indirect benefits of this research are therefore likely to be substantial.
Exhibit A-2 describes the target respondents, content, and reason for inclusion for each data collection activity that involves study participants and staff from participating agencies. Copies of the data collection instruments are provided as Appendices.
Exhibit A-2. Item-by-Item Justification of Data Collection Instruments
Data Collection Activity |
Data Collection Instrument(s) |
Respondents, Content, and Reason for Inclusion |
Additional Intake Information on Potential Study Participants |
Instrument 1: Modification to the Basic Information Form (BIF)
(Appendix D) |
Respondents: Study participants with children, estimated to be 8,700 ISIS sample members from nine ISIS programs.
Content:
Reason: Roster will be used to create a sampling frame for future follow-up surveys that will collect data about child outcomes. See Appendix E, which presents a logic model illustrating how career pathways programs may generate changes in child outcomes and which summarizes relevant literature. |
Study Participant Follow-up Survey |
Instrument 2: 15-Month Follow-up Survey
(Appendix H) |
Respondents: Overall expected sample of 8,560 (80% of 10,700 sample members from nine ISIS programs).
Content:
Reason: In most ISIS programs, key activities and services will be provided within the first 12 months following random assignment. This follow-up period of 15 months will help to ensure that respondents can accurately report information about service receipt and experiences, and early employment and additional education outcomes. |
Interviews with ISIS Program Leadership |
Instrument 3: Interview Topic Guide for Program Leadership
(Appendix I) |
Respondents: Program leadership from each of the nine ISIS programs.
Content:
Reason: To document any changes in program operations or the provision of services, as well as implementation challenges. |
Interviews with ISIS Program Instructional Staff |
Instrument 4: Interview Topic Guide for Instructional Staff
(Appendix J) |
Respondents: Instructors in or affiliated with each of the nine ISIS programs.
Content:
Reason: To document any changes in program operations and instructional practices, as well as implementation challenges. |
Interviews with ISIS Program Case Managers/Advisors |
Instrument 5: Interview Topic Guide for Case Managers/Advisors
(Appendix K) |
Respondents: Case managers/advisors from each of the nine ISIS programs.
Content:
Reason: To document any changes in program operations or the provision of services, as well as implementation challenges. |
Interviews with ISIS Program Partner Staff |
Instrument 6: Interview Topic Guide for Partner Staff
(Appendix L) |
Respondents: Staff ISIS program referral or service provider partners.
Content:
Reason: To document role in the program and any changes over time in the provision of services, as well as implementation challenges. |
Online Survey for ISIS Case Managers and Advisors |
Instrument 7: Case Managers/Advisors Online Survey
(Appendix M) |
Respondents: All case managers and advisor staff from each ISIS program. The average number of case managers/advisors in single site programs is 5; in programs with sub-sites it is 5 per sub-site. There are 7 single site-programs and 2 programs with sub-sites.
Content:
Reason: This survey will supplement the information collected during the implementation study interview. Specifically, the data will be used to develop quantitative measures of key features of career pathway program design and implementation – particularly the four core program inputs (assessment, curriculum, supports, and employment connections). In addition, the survey will collect systematic information on the processes programs use in managing services for participants. The survey will also collect information about other factors that could affect service delivery including staff background, training and professional development, staff philosophy, and staff perceptions of success. Because quantitative measures of key program dimensions will be developed, the survey data will also facilitate comparisons across sites. |
Online Survey for ISIS Managers/ Supervisors |
Instrument 8: Manager/Supervisor Online Survey
(Appendix N) |
Respondents: All supervisors from each ISIS program. The average number of supervisors in a single site programs is 3; in programs with sub-sites it is 2 per sub-site. There are 7 single site-programs and 2 programs with sub-sites.
Content:
Reason: This survey will supplement the information collected during the implementation study interview. Specifically, the data will be used to develop quantitative measures of key features of career pathway program design and implementation – particularly the four core program inputs (assessment, curriculum, supports, and employment connections). In addition, the survey will collect systematic information on the processes programs use in managing services for participants. The survey will also collect information about other factors that could affect service delivery including staff background, training and professional development, staff philosophy, and staff perceptions of success. Because quantitative measures of key program dimensions will be developed, the survey data will also facilitate comparisons across sites. |
Online Survey for ISIS Instructional Staff |
Instrument 9: Instructional Staff Online Survey
|
Respondents: All instructors that work for or are affiliated with programs. The average number of instructors in a single site is 10; the average number of instructors in programs with sub-sites is 6 per sub-site. There are 7 single site-programs and 2 programs with sub-sites.
Content:
Reason: This survey will supplement the information collected during the implementation study interview. Specifically, the data will be used to develop quantitative measures of key features of career pathway program design and implementation – particularly the four core program inputs (assessment, curriculum, supports, and employment connections). In addition, the survey will collect systematic information on the processes programs use in managing services for participants. The survey will also collect information about other factors that could affect service delivery including staff background, training and professional development, staff philosophy, and staff perceptions of success. Because quantitative measures of key program dimensions will be developed, the survey data will also facilitate comparisons across sites. |
Study Participant Implementation Study Interviews |
Instrument 10: Study Participant In-depth Interview Guides
|
Respondents: A sample of study participants (treatment and control group members) from each of the nine ISIS programs at three points in time (interviews at Time 1 and 2 and an interim brief check in call). Seven of the nine programs are single site; in each the team will interview 10 treatment and 5 control group members. One program has 3 sub sites. The study will interview 10 treatment and 5 control group members at each sub site. The final site has eight sub sites. The team will interview 10 treatment and 5 control group members at each of 4 sub sites.
Content:
Reason: To gain a more comprehensive understanding of treatment and control members’ experiences with services. The study will not use these small numbers of interviews to make treatment-control comparisons. However, they may generate ideas about causal paths, which can then inform subsequent survey content that can lead to actual impact estimates. |
The ISIS evaluation will generate a substantial amount of data and will use a combination of data collection methods. For each data collection activity, the study team has selected the form of technology that enables the collection of valid and reliable information in an efficient way while minimizing burden. This evaluation will use improved technology to facilitate the collection of the survey data in standardized and accurate ways that also ensures the protection of the data collected.
The follow-up survey will be administered using CATI (computer-assisted telephone interviewing) technology for telephone interviews and CAPI (computer assisted personal interviewing) for in-person interviewing when the individual cannot be located for a telephone survey. CATI and CAPI technology reduces respondent burden, as interviewers can proceed more quickly and accurately through the survey instruments, minimizing the interview length. Computerized questionnaires ensure that the skip patterns work properly, minimizing respondent burden by not asking inappropriate or non-applicable questions. For example, respondents who did not participate in post-secondary training will be routed past questions only relevant to those who did. Computer-assisted interviewing can build in checkpoints, which allow the interviewer or respondent to confirm responses thereby minimizing data entry errors. Finally, automated survey administration can incorporate hard edits to check for allowable ranges for quantity and range value questions, minimizing out of range or unallowable values. The staff surveys for instructors, case managers/advisors and managers/supervisors will be programmed online for easy access by respondents via the internet.
The purpose of the follow-up survey for the ISIS evaluation is to obtain current information on the status and wellbeing of individuals in the ISIS evaluation study sample. Information about these respondents' educational achievement, economic well-being, and job skills development are not available through any other source. The evaluation will utilize administrative data (e.g., wage records) in conjunction with survey data to avoid duplication of reporting.
The research team will also avoid duplication in this study by use of the centrally maintained data system, which links all the data collected at baseline and follow-up (and during the subsequent active and passive tracking efforts) with subsequent information gathered from administrative sources. This eliminates the need to ask about personal characteristics or background factors for known household members on follow-up surveys.
Of the nine sites included in ISIS, three are programs that received Health Profession Opportunity Grants (HPOG) administered by ACF and a fourth is a sub-grantee to an HPOG-funded program. ACF is funding implementation and impact evaluations of the HPOG program and the ISIS and HPOG research teams worked closely to coordinate data collection in the four programs that are part of both studies. Areas of coordination include:
Baseline data. Data collected on the ISIS BIF for program group members is “streamed” to the HPOG Performance Reporting System (PRS) so that HPOG staff will not have to double-enter this information and the participant will not have to report it twice. Additionally, the ISIS research team shared the BIF with the HPOG team in order to attain feedback and incorporate questions relevant to the HPOG study, and to ensure the alignment of a core group of questions.
Development of implementation research instruments and the 15 month follow-up survey included in this clearance request. The timing of the studies necessitates clearance for ISIS implementation and survey instruments in advance of those for HPOG. The ISIS research team shared all evaluation instruments with the HPOG team in order to attain feedback and incorporate questions relevant to the HPOG study, and to ensure the alignment of a core group of questions.
Implementation site visits. ISIS and HPOG teams will conduct site visits jointly to the HPOG programs in ISIS so as to reduce burden on site staff.
Data sharing. All data collected for the three HPOG sites in ISIS (from surveys, administrative data and implementation visits) will be shared with the HPOG research team for inclusion in the HPOG implementation and impact studies.
Administration of HPOG team-developed instruments. The HPOG National Implementation Evaluation is developing employer surveys and stakeholder/network surveys that will be administered to all ISIS sites. The instruments and the associated burden will be included in a future OMB clearance request.
The primary organizations involved in this study are community colleges, workforce development agencies, and community-based organizations that operate occupational training programs. Burden is minimized for these entities by requesting the minimum information required to achieve the study’s objectives. On-site interviews with program staff will cover topics on which the study team is unable to collect sufficient information by other means.
The data collection effort described in this document is essential to the ISIS evaluation. If data is collected less frequently, it would jeopardize ACF’s ability to conduct the impact and implementation analyses. Delays in the administration of the follow-up survey run an inherent risk that the respondent will have trouble recalling the details about the services received and potentially lead to missing the achievement of key milestone events as study participants move through training and education.
The implementation study data, particularly the key informant interviews, are critical to understanding both the individual programs as well as differences in program implementation across sites. During this follow-up period, the implementation team will make a visit to each program approximately 15 months after random assignment begins—when the programs should be mature. Any changes in operations or the provision of services, implementation challenges, and programs and services available to the control group services will be documented, as well as efforts to scale up for the study and sustain their program activities beyond the ISIS study period. The research team will use this information to understand how the intervention may have contributed to program impacts.
The proposed data collection activities are consistent with the guidelines set forth in 5 CFR 1320.6 (Controlling Paperwork Burden on the Public, General Information Collection Guidelines). There are no circumstances that require deviation from these guidelines.
In accordance with the Paperwork Reduction Act of 1995, the Administration for Children and Families (ACF) at the Department of Health and Human Services published a notice in the Federal Register on July 20, 2012, page 42743. The document number is FR Doc. 2012–17641. A copy of the notice is shown in Appendix L. No comments were received on the instruments or the proposed data collection.
For the evaluation to be most successful, the study team determined that monetary gifts should be provided to the study participants in appreciation of the time they spend participating in the data collection activities. These tokens of appreciation are a powerful tool for maintaining low attrition rates in longitudinal studies, especially for participants in the control group because these sample members are not receiving any (other) program benefits or services. The use of monetary gifts for the ISIS follow-up surveys can help ensure a high response rate, which increases confidence in producing unbiased impact estimates. Low response rates increase the danger of differential response rates between the treatment and control groups, leading to possible non-comparability between the two groups and potentially biased impact estimates.
Three factors helped to determine the amounts for the follow-up survey:
Respondent burden, both at the time of the interview and over the life of the evaluation;
Costs associated with participating in the interview at that time; and
Other studies of comparable populations and burden.
Previous research has shown that sample members with low incomes and/or low educational attainment have proven responsive to incentives, as have minority group members. These characteristics are expected to be heavily represented in the ISIS study population.1
The amounts offered for the ISIS follow-up survey and tracking responses are as follows:
Follow-up Survey (all study participants): $30
In-depth Interviews with Study Participants (subsample of study participants): $40
Respondents to the follow-up survey will receive $30 to complete a 50 minute questionnaire that will be administered via telephone or in person if the individuals are not reachable by phone. The proposed amount is based on similar surveys the research team has conducted with this population. Tokens of appreciation help to secure the cooperation of the individual over the duration of the study period and reduce the potential for individuals to fail to complete them.
For the implementation study, the research team will offer $40 to sample members who participate in the face-to-face in-depth interviews. This will cover any costs incurred traveling to and from the interview site (which is not an issue for the 15-month survey).
Many surveys are designed to offer incentives of varying types with the goal of increasing survey response. Monetary incentives at one or more phases of data collection have become fairly common, including some federally sponsored surveys. Examples include the National Survey on Drug Use and Health (NSDUH, Substance Abuse and Mental Health Services Administration), the National Survey of Family Growth (NSFG, National Center for Health Statistics), the National Health and Nutrition Examination Survey (NHANES, National Center for Health Statistics), the National Survey of Child and Adolescent Well-Being (NSCAW, Administration for Children and Families), and the Early Childhood Longitudinal Study-Birth Cohort (ECLS-B, U.S. Department of Education).
There has been extensive publication about the relative efficacy of different monetary incentives. The U.S. Census Bureau has experimented with and begun offering monetary incentives for several of its longitudinal panel surveys, including the Survey of Income and Program Participation (SIPP). SIPP has conducted several multi-wave incentive studies, most recently with their 2008 panel, comparing results of $10, $20, and $40 incentive amounts to those of a $0 control group. They examined response rate outcomes in various subgroups of interest (e.g., the poverty stratum), use of targeted incentives for non-interview cases, and the impact of base wave incentives on later participation. Overall, $20 incentives increased response rates and improved the conversion rate for non-interview cases (Creighton et al, 2007). The National Survey on Drug Use and Health (NSDUH, Substance Abuse and Mental Health Services Administration) conducted an experiment in which the cost per interview in the $20 incentive group was five percent lower than the control group, whereas the $40 incentive group cost was four percent lower than the control, due to reduced effort needed in gaining cooperation (Kennet et al., 2005). The NSDUH adopted an intermediate incentive of $30 because the greatest increase in response rate was found in the $20 incentive condition, and the $40 condition obtained a higher variation in per-interview costs. A similar incentive experiment conducted for the National Survey of Family Growth (NSFG, National Center for Health Statistics) Cycle 5 Pretest examined $0, $20, and $40 incentive amounts. The additional incentive costs were more than offset by savings in interviewer labor and travel costs (Duffer et al, 1994).
The information collected under this data collection will be kept private to the fullest extent provided by law. The information requested under this collection will be private in a manner consistent with 42 U.S.C. 1306, 20 CFR 401 and 402, and OMB Circular No. A-130.
The study team has established rigorous data security and privacy provisions. First, all data users will be aware of and trained on their responsibilities to protect participants’ personal information, including the limitations on uses and disclosures of data. The research databases will be designed to limit access to authorized users with levels of access commensurate with each person’s role on the project. The web server hosting the database will be maintained in a secure facility with power back up, network redundancy and system monitoring. In addition, daily back up of the server will be maintained at the data center and an off-site location. The database and website will be password protected, and access will be provided only after user authentication.
The ISIS Participation Agreement (see Appendix C) ensures a commitment to keeping personal information private. This assurance will also be made to all respondents as part of the introduction to the follow-up survey. For both survey data and corresponding administrative data on sample members, computer security will be maintained by passwords known only to a limited number of project staff, members who require access to these files.
The follow-up survey includes questions about physical and emotional health, sexual behavior (for young adults), substance use, and history of domestic violence, items that some respondents may consider sensitive. The literature provides ample support for including these items as barriers to education and employment. Including these items is necessary to describe the study population and evaluate mediating effects on program impacts. Program staff will remind study members during the interviewing process that they may refuse to answer individual items. Study members will also be reminded that their responses will be kept private to encourage their candid responses.
The newly added questions to the revised BIF include a child roster, which requests basic information needed to design a sampling strategy for future data collection: the first name and birth date for each of the respondent’s minor children and respondents’ relationships to the children (e.g., biological parent, foster parent).
The research team included this roster because studies of related programs suggest that programs such as ISIS may have impacts on children, and that these impacts may vary with age. Understanding impacts on both children as well as ISIS participants is important to ACF given the agency’s dual focus on the well-being of low-income children and families. Appendix E presents two logic models that illustrate the effects that career pathways programs may generate for preschool-aged children and for older children. It also includes a narrative that explains the logic models and that cites research supporting them.
Generally, the research team believes it is important to include the child roster because, as the logic models and supporting literature illustrate, career pathways programs such as ISIS, have the potential to generate both positive and negative outcomes for children. Impacts for children ages 0-5 and for older children may be quite different because older children may be more likely than preschoolers to engage in negative behaviors as a consequence of decreased supervision that results from parental engagement in education and/or employment. Research on related welfare reform programs illustrates the potential for ISIS programs to generate these kinds of outcomes for youth, but that research does not specifically assess outcomes associated with career pathways programs.
Including the child roster in the baseline data collection allows the study the potential to sample a focal child (or children) for inclusion in a follow-up survey; directly, via questions posed to parents, or by accessing publicly available data. Including children’s birth date supports a sampling plan that would allow the research team to target age-appropriate follow-up survey questions to particular sample members. Including questions about children’s relationships to respondents is important because child/respondent relationship may be a moderating factor. Future follow-up surveys will collect data about participants’ children’s developmental and behavioral outcomes, so that impacts on children can be estimated.
Although the child roster does not ask many questions specifically about children (only their first names, dates of birth, and relationship to study participants), participants may be reticent to offer any such information. Prior to enrolling in the study, participants will be given a participant agreement form that states that participation in the study is voluntary and that they may refuse to answer any question (see Appendix C). Also, any future data collection efforts involving children would require additional permission from parents.
Baseline Data Collection Already Approved
The total burden for the instruments already approved (Basic Information Form, and Self-Administered Questionnaire) was estimated to be 8,100. Burden remaining as of the estimated time of OMB clearance is 4,611 hours. Annualized remaining burden is 1,537 over three years.
Current Information Collection Request
Follow-up data collection for the ISIS evaluation will occur for each individual approximately 15 months following random assignment. Exhibit A-12 shows the estimated burden on sample members participating in the additional Basic Information Form questions, follow-up survey and in-depth interviews for the ISIS evaluation, and for the program staff and directors participating in key informant interviews and online surveys for the ISIS implementation study. It shows the average time, in hours, that study participants and program staff and partners are estimated to spend completing each data collection instrument.
These estimates do not include the burden associated with any potential future follow-up surveys completed by program participants. (Future follow-up surveys will be included in a separate submission, requesting clearance of these instruments.)
The average hourly wage was calculated for each respondent group based on information from the Bureau of Labor Statistics2 or the federal minimum wage. The average hourly rate3 for each respondent group was calculated as follows:
Study participant: the minimum hourly wage ($7.25) plus a 40 percent adjustment to account for benefits, or $10.15 per hour.
Case manager: Community and Social Service Occupations (SOC code 21-0000) wage rate of $21.07 plus a 40 percent adjustment for benefits, or $29.49.
Instructional staff: Education, Training, and Library Occupations (SOC 25-0000) wage rate of $24.46, plus a 40 percent adjustment for benefits, or $34.24.
Program leadership/Manager: Social and Community Service Manager Occupations (SOC 11-951) wage rate of $30.43, plus a 40 percent adjustment for benefits, or $42.60.
Instrument |
Total Number of Respondents |
Total Number of Responses per Respondent |
Average Burden Hours per Response |
Total Burden Hours |
Average Annual Burden Hours |
Average Hourly Wage |
Total Annual Cost |
Estimated Annual Burden Remaining |
|||||||
Basic Information Form |
7,950 |
1 |
0.25 |
1,988 |
663 |
$10.15 |
$6,729 |
Self-Administered Questionnaire |
7,950 |
1 |
0.33 |
2,624 |
875 |
$10.15 |
$8,881 |
Total Burden Hours Remaining |
|
|
|
4,612 |
1,538 |
|
$15,610 |
Estimated Annual Burden New Collection |
|||||||
#1 Basic Information Form Modification |
5,645 |
1 |
0.05 |
282 |
94 |
$10.15 |
$954 |
#2 15 Month Follow-up Survey, no child roster |
8,560 |
1 |
0.833 |
7,130 |
2377 |
$10.15 |
$24,127 |
#2 15 Month Follow-up Survey child roster addition |
1,562 |
1 |
0.05 |
78 |
26 |
$10.15 |
$264 |
#2 15 Month Follow-Up Survey HPOG Questions Addition |
2,974 |
1 |
0.083 |
247 |
82 |
$10.15 |
$832 |
#3 Program Leadership/Managers/Supervisors Interview Guide |
46 |
1 |
2 |
92 |
32 |
$42.60 |
$1,363 |
#4 Instructional Staff Interview Guide |
58 |
1 |
2 |
116 |
40 |
$34.24 |
$1,370 |
#5 Case Managers/Advisor Interview Guide |
50 |
1 |
2 |
100 |
34 |
$29.49 |
$1,003 |
#6 Partners Interview Guide |
54 |
1 |
2 |
108 |
36 |
$29.49 |
$1,062 |
#7 Case Managers/Advisors Online Survey |
90 |
1 |
0.5 |
45 |
15 |
$29.49 |
$442 |
#8 Manager/Supervisor Online Survey |
43 |
1 |
0.5 |
22 |
7 |
$42.60 |
$298 |
#9 Instructional Staff Online Survey |
136 |
1 |
0.5 |
68 |
23 |
$34.24 |
$788 |
#10 Study Participant Interview Guide |
210 |
2 |
1 |
420 |
140 |
$10.15 |
$1,421 |
#11 Study Participant Check-in Call |
210 |
1 |
0.16 |
34 |
11 |
$10.15 |
$112 |
Total Burden Hours: New Collection |
|
|
|
8,742 |
2,917 |
|
$34,036 |
TOTAL Burden Hours (Remaining + New) |
|
|
|
13,354 |
4,455* |
|
$49,646 |
^Assume 71 percent of respondents have children (based on baseline data collected to date)
* Rounding for annual burden based on ICRAS/ROCIS calculations, which annualize at the respondent level.
Total Burden Hour Request
Total burden is displayed in Exhibit A-12. The total burden for already approved information collection and the new request is 13,354 hours, or 4,4554 hours per year over three years.
This data collection effort involves no recordkeeping or reporting costs for respondents other than those described in item A.12 above.
The annual cost for all information collection under this OMB number is $1,999,668. This includes the cost of collecting baseline information (site staff time to administer the BIF and SAQ), the development of data collection instruments and tools, the administration of the follow-up survey, the collection of implementation research data, and the analysis and reporting of data.
This evaluation involves new data collection that increases the public reporting burden under this OMB number. Section A.12 details the burden figures.
The evaluation contractor, Abt Associates Inc., and its subcontractors will collect, analyze, tabulate and report the data collected for the ISIS evaluation to HHS.
The ISIS data collection activities will support the following major deliverables:
Site-specific implementation reports. The site-specific implementation reports will describe program design, including the underlying logic; how the program was implemented and changes that were made during implementation; the treatment group's participation patterns and experiences with program services; the differences in service receipt between treatment and control group members; and the extent to which the program scaled up. For eight sites, reports will be drafted by September 2015; for the ninth, by July 2016.
Cross-site implementation report. The report will describe and compare the similarities and differences across the ISIS sites. It will include a description of participant demographic information, income, education, and job status at baseline and 15 months post random assignment; program services and components, including the relative emphasis placed on each career pathway component; services received by program participants; treatment-control difference in service receipt and completion, and implementation challenges; and prospects for sustainability and further scaling. The report will be drafted by September 2016.
Site-specific 15-month impact studies. Each interim report will describe the program impact on key indicators including career pathways-relevant training, earnings and career-track employment, and family well-being. Inasmuch as programs use different strategies and target varying outcomes, the site-specific impact studies will also focus on whether impacts vary by subgroups. The site-specific interim reports, using data from the 15-month survey, will also include cost-benefit information. Interim impact reports will be drafted by December 2016 in eight sites and September 2017 in the ninth.
In addition to these primary analyses, data from this information collection will be important to conducting a number of secondary analyses. For example, the ISIS impact evaluation will consider the extent to which component parts of a career pathway program contribute to program impacts and compare existing career pathway resources with programming developed under ISIS, assuming the available data supports such a comparison. This question is of primary concern for the HPOG impact evaluation and will form part of the secondary analysis for the ISIS impact evaluation.
Exhibit A-16 presents an overview of the project schedule for information collection. It also identifies deliverables associated with each major data collection activity.
Exhibit A-16. Overview of Project Data Collection Schedule
Data Collection Activity |
Timing |
Associated Publications |
|
Currently operating under OMB # 0970-0397 |
Site-specific implementation reports, interim impact reports |
|
Beginning in 2013 upon OMB approval |
Site-specific implementation reports, interim impact reports |
|
Beginning in 2013 upon OMB approval |
Site-specific implementation reports, interim impact reports, cost-benefit report |
|
Beginning in 2013 upon OMB approval |
Site-specific implementation reports |
|
Beginning in 2013 upon OMB approval |
Site-specific implementation reports |
|
Beginning in 2013 upon OMB approval |
Site-specific implementation reports |
All data collection instruments created for the ISIS evaluation will display the OMB approval number and expiration date.
This submission describing data collection requests no exceptions to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).
1 See among the sources documenting this recommendation: Allen P. Duffer et al., "Effects of Incentive Payments on Response Rates and Field Costs in a Pretest of a National CAPI Survey" (Research Triangle Institute, May 1994), passim; see also "National Adult Literacy Survey Addendum to Clearance Package, Volume II: Analyses of the NALS Field Test" (Educational Testing Service, September 1991), pp. 2-3.
2 http://www.bls.gov/oes/current/oes_nat.htm
3 Assuming 2080 FTE hours worked.
4 Rounding for annual burden based on ICRAS/ROCIS calculations, which annualize at the respondent level.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Abt Single-Sided Body Template |
| Author | Missy Robinson |
| File Modified | 0000-00-00 |
| File Created | 2021-01-29 |