ISIS OMB Package A 11-01-11_clean

In this document, we request OMB clearance for the next in a series of data collection activities for the Innovative Strategies for Increasing Self-Sufficiency (ISIS) project.. This request is for a new collection. This submission seeks to amend the project and get OMB approval for three data collection instruments we plan to administer at baseline (i.e., intake to the programs studied):

a Basic Information Form (BIF) for participants;
a Self-Administered Questionnaire (SAQ) for participants; and
an Interview Guide for interviews with program staff.

Subsequent OMB submissions will seek clearance for follow-up data collection activities.

This section provides supporting statements for each of the eighteen points outlined in Part A of the OMB guidelines for the collection of baseline information in selected study sites in the Innovative Strategies for Increasing Self-Sufficiency (ISIS) evaluation.

A.1 Circumstances Necessitating Data Collection

The Administration for Children and Families (ACF) in the U.S. Department of Health and Human Services (HHS) seeks approval for the baseline data collection activities described in this request in order to support a study conducted for ACF by Abt Associates. The ISIS study will evaluate post-secondary career pathway programs that target economically disadvantaged families and individuals.

A.1.1 Legal or Administrative Requirements that Necessitate the Collection

There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection as part of its ongoing effort to improve the well-being of the low-income population.

A.1.2 Study Background

ACF conceived of the ISIS project as a test of promising interventions for improving the economic prospects of low-income families. The agency purposefully left broad the range of interventions that might be included in the evaluation at the outset of the project. Therefore, an early project activity was an intensive outreach effort to solicit the views of stakeholders in the program and policy communities on what ISIS might test. ACF undertook this intensive outreach effort under the auspices of the previously approved data collection under OMB control No. 0970–0343. The data collected under this request revealed that there was a strong consensus across stakeholder groups that ISIS should study interventions for low-income families broadly, rather than limiting attention to those receiving Temporary Assistance for Needy Families (TANF) cash assistance. Informants recommended testing an array of interventions designed to engage and support individuals in jobs and work activities and improve employment skills. In the course of subsequent engagement with stakeholders while recruiting programs into the study, the focus of ISIS has tightened on post-secondary career pathway programs.

A.1.3 Study Design

ISIS study sites will target low-income adults who are interested in occupational skills training. The sites will conduct random assignment of individuals to one of two groups: a treatment group that will be offered the innovative career pathway interventions such as those described briefly above, or to a control group that will be able to access a set of “business-as-usual” or other services in the community only. The target sample size for each site is roughly 1,200—with about 600 in each of the two research groups.

A.1.4 Research Questions

Research questions the ISIS evaluation will address include:

Implementation—What services are provided under each intervention? What are the characteristics of the populations served? How are services for the target population implemented? How do services for the treatment group compare to the services available to the control group? What are the issues and challenges associated with implementing and operating the service packages and policy approaches studied?
Impact— What are the net impacts of career pathway programs on educational outcomes (program completion, attainment of credentials and degrees) and economic outcomes (earnings, employment levels, and wage progression)? What are the net impacts for hypothesized mediators of these primary impacts in domains such as academic skills, psycho-social skills, career awareness, family resources, and other personal and family challenges? What are the longer-run impacts on indicators of family well-being?
Subgroups—How do career pathway program impacts vary by subgroup?
Cost effectiveness—What are the costs of career pathway programs in the study? Do the estimated benefits of providing services outweigh the costs of these programs?

The Interview Guide for implementation research included in this clearance package will allow the ISIS evaluation to answer the first question. Researchers will use interviews with program staff and key stakeholders to collect information that will provide a fuller understanding of the initial conditions surrounding these career pathway programs and the contexts in which they operate. This information will also allow researchers to assess the quality of early implementation of these programs—assessments that will be important to the interpretation of program impact results.

Baseline data collected with the BIF and SAQ instruments will help inform the ISIS Impact Studies, addressing the last three research questions. The contact information collected at baseline is necessary to enhance researchers’ ability to locate respondents for follow-up surveys that will measure intervention outcomes in both the treatment and control groups. ISIS researchers also will utilize information collected at baseline to identify subgroups of interest and to compare impacts across subgroups. Other analytic purposes of the baseline data include describing the ISIS study sample, adjusting for chance differences in observable characteristics and increasing precision of impact estimates, and checking the integrity of random assignment.

Finally, the project will address the cost-effectiveness of programs through comparison of any economic benefits with net program costs in the Benefit-Cost Study. Although the bulk of cost data will come from programs’ existing administrative records, ISIS researchers will augment their understanding of program costs through interviews with program staff.

A.1.5 Planning, Pilot, and Full Implementation Stages of ISIS Sites

After a site formally agrees to participate in the ISIS study, the site moves through three distinct stages of implementation. During the planning stage, the site and the ISIS team plan the details of the random assignment test, including recruitment strategies and random assignment procedures. During the pilot stage, the site tests recruitment and random assignment procedures with participants. When the site and the ISIS team are jointly satisfied that these procedures are working well, then the site moves into the full implementation stage. Participants who enroll during the pilot may be included in the Impact Study sample if the procedures run smoothly during the pilot.

A.1.6 Data Collection in the ISIS Evaluation

In 2008, the project obtained OMB clearance for an initial data collection. In that collection, the ISIS team conducted interviews with a diverse group of stakeholders to identify the most promising self-sufficiency strategies to test and promising program sites to participate in the study.

This submission seeks clearance for three baseline data collection instruments in the ISIS evaluation. These three instruments are:

A Basic Information Form (BIF) that will collect general demographic and contact information. The BIF will be administered during intake prior to random assignment by an intake staff person or self-administered on a paper form.
A Self-Administered Questionnaire (SAQ) that will collect more sensitive and personal information from participants, including several psycho-social items designed and validated by the testing firm ACT, Inc. The SAQ will also be completed during intake prior to random assignment and will be self-administered either on a paper form.
An Interview Guide for implementation research that will be used to collect information from site staff personnel. The project team will interview program administrators and staff at ISIS sites and other organizations that partner with ISIS sites to deliver services. These interviews will take place on site visits during the pilot and early full implementation stages.

In the future, the ISIS evaluation will be collecting additional data, including:

Follow-up surveys: The project will administer at least one follow-up survey to the entire ISIS sample. The first follow-up survey is tentatively scheduled to take place 12 months after random assignment. Additional surveys may be funded in the future (the next, if funded, is tentatively scheduled to take place 36 months after random assignment). ISIS will submit each follow-up survey to OMB for clearance. ISIS may collaborate with ACT, Inc. to develop items on follow-up surveys designed to measure changes in psycho-social outcomes.
Government administrative records: These records will include unemployment insurance (UI) wage records and may also include TANF records, Supplemental Nutrition Assistance Program (SNAP) records, and Child Support Enforcement (CSE) records. The project anticipates being able to utilize the wage records from the National Directory of New Hires (NDNH), a database maintained by ACF's Office of Child Support Enforcement.
Site program records: The project team will collect data on direct program outcomes from the sites for the Implementation Study and for the Impact Study where program records are available for both treatment and control group members. Illustrative outcomes include measures of basic academic skills, services received, and credits and credentials earned. For some outcomes– such as educational attainment at post-secondary institutions – it is likely that data will be gathered from centralized state databases. The information included in these records will differ from site to site based on the information collected by each site's management information system.
Staff interviews and participant focus groups: The project team will be interviewing program administrators and staff at ISIS sites and other organizations that partner with them to deliver services in a second round of site visits. The team may also convene focus groups of program participants to obtain their personal perspectives on the programs. The data collected in these interviews and focus groups will be used for the Implementation Study. ACF will submit the discussion guides for second visit interviews and focus groups in a future OMB clearance package.
Other innovative data collection: The project is interested in collecting types of data that have not typically been collected in previous "welfare-to-work" studies. Examples of types of data that could be collected in ISIS include: i) direct child assessments, ii) daily time diaries (for which there are a variety of potential methods), and iii) instructor or work supervisor ratings of the skills of study participants.

Of the nine sites included in ISIS, we expect that some will be programs that have received grants through the Health Profession Opportunity Grants (HPOG) program administered by ACF. The HPOG program is itself the subject of a separate evaluation. ACF has encouraged the ISIS and HPOG evaluation teams to maximize data collection efficiencies and minimize burden across the two projects.

A.2 How, by Whom, and for What Purpose Are Data to be Used

A.2.1 Overview of the Role of Baseline Data Collection for ISIS

The purposes of collecting baseline data with the BIF and SAQ instruments are several. First, the contact information collected at baseline is necessary to enhance researchers’ ability to locate respondents for follow-up surveys that will measure intervention outcomes. A second purpose for collecting baseline data with the BIF and SAQ instruments is to create a rich dataset for future researchers to explore and test hypotheses. Other analytic purposes of the baseline data include characterizing the ISIS study sample, adjusting for chance differences in observable characteristics and increasing precision of impact estimates, identifying subgroups of interest, and checking the integrity of random assignment.

The Interview Guide for implementation research included in this clearance package will allow the ISIS evaluation to collect information that will provide a fuller understanding of the initial conditions surrounding these career pathway programs and the contexts in which they operate. This information will also allow researchers to assess the quality of early implementation of these programs—assessments that will be important to the interpretation of program impact results.

A.2.2 Data Collection Process

Baseline data collection will extend for the entire period random assignment is conducted—from the pilot phase beginning in fall 2011 through at least fall 2013. Site staff will administer the informed consent agreement, the BIF, and the SAQ at the time of enrollment in the program. During enrollment, site staff will first describe the program and administer the informed consent form. Potential study members must sign the paper informed consent form to be part of the study. Next, site staff will administer the BIF and then the SAQ. Depending on site-specific circumstances, sites will choose to administer the BIF and the SAQ in either a group or individual setting with a paper format. After individuals complete the informed consent form, the BIF, and the SAQ, a secure, web-based software program will randomly assign them into either the treatment or control group. A more detailed description of data collection is outlined in Section B.2.

ISIS evaluators will conduct two rounds of implementation research site visits. The first will occur during the pilot stage and will document the planned intervention and the context in which each program operates, including the counterfactual condition. The information collected at this point will inform the decision to move to the full implementation stage. The second set of visits will occur during the second year of the full implementation stage and will document “steady state” program operations and any changes that occurred in the sites since the pilot visit. The Interview Guide for the second set of visits will be submitted to OMB in a future clearance package.

A.2.3 Who Will Use the Information

Researchers on the ISIS team will have access to and use of the data collected as part of the ISIS evaluation. Some of the data will also be available to researchers studying the HPOG program where such data increases what can be learned in the HPOG study. At the conclusion of the ISIS study, Abt Associates will provide ACF with a restricted-use data set containing individual level data that is stripped of all personally identifying information. The restricted-use data will be made available to researchers for approved secondary uses.

A.2.4 Construct Justification

Baseline is a crucial point to obtain measures for key inputs in the theory of change for interventions evaluated. In varying ways, the programs ISIS is testing incorporate innovative strategies to influence intermediate outcomes as part of each step in a targeted career pathway. A strong measurement strategy is needed to establish a baseline profile of where participants start with respect to each of the major constructs that may be influential. Given the wide set of objectives of career pathway programs, we plan to collect a broad set of measures at baseline in order to measure progress toward these objectives. This section provides general justification for constructs included on the BIF and SAQ. An item-by-item justification can be found for the BIF in Appendix F and for the SAQ in Appendix G.

The BIF asks respondents questions on a range of constructs, including age, race, family composition, education, employment history, and income. Significant evidence shows that each construct is correlated with success in adult education and job training programs. Discrimination on the basis of race or gender may moderate the potential for ISIS programs to increase women’s and minorities’ employment and earnings outcomes. Previous educational success could indicate a student has the skills needed to succeed in future training programs. Likewise, those who have a stable job history have demonstrated their ability to obtain and keep jobs in the past, potentially making them more likely to secure better jobs at the conclusion of a training course. Family composition and income could be signs of barriers that will make it difficult for some individuals to participate in or complete ISIS programs. The more family pressures adult face, the less time they have to devote to education and training. Many adults also struggle to get by financially if they are unable to work as much. Each of these characteristics could moderate program impacts. It is important to measure these constructs at baseline to see which are the most important for benefitting from career pathway programs.

The career pathway theory of change posits that fostering successful employment outcomes requires effectively addressing the variety of skills, personal challenges, social supports, and other resources needed for low-income adults to succeed in post-secondary education and training. In addition, there has been growing recognition and sophisticated responses to addressing psycho-social skills needed to sustain motivation, emotional balance, and social engagement. Finally, a widely held assumption is that non-traditional student populations often have insufficient knowledge and skills to formulate and pursue a career plan or navigate the institutional complexities associated with researching and deciding on occupations and training programs, applying for admission and financial aid, planning a program of study, and obtaining advice and support when needed.

A.3 Use of Information Technology to Collect the Information

Site staff will enter all information from the BIF into an electronic database system designed for ISIS by Relyon Media. Using technology enhances data quality in several ways. First, computerized questionnaires ensure that the skip patterns work properly. Including efficient skip patterns minimizes respondent burden by not asking inappropriate or non-applicable questions. Second, computer-assisted interviewing can build in checkpoints which allow the interviewer or respondent to confirm responses thereby minimizing data entry errors. Finally, automated survey administration can incorporate hard edits to check for allowable ranges for quantity and range value questions, minimizing out of range or unallowable values. In situations where sites are unable to use electronic systems, staff will administer the BIF and/or the SAQ with paper forms.

A.4 Why We Cannot Use Duplicate Information

The ISIS BIF and SAQ will collect data about respondents, including information on demographic characteristics, educational attainment, economic well being, psycho-social characteristics, mental and physical health, domestic violence, and use of alcohol or drugs. These data are essential for describing the baseline characteristics of the sample, providing more precise impact estimates, and identifying subgroups that may have stronger or weaker effects. Some of the demographic information is available through administrative records kept by partner organizations. However, many organizations do not use standard or comparable definitions or categories, even for characteristics as simple as race and ethnicity, and the currency of items for changeable characteristics may vary across sites and within sites. A high quality evaluation requires consistency in baseline measures across and within sites. Compared to the effort required for sites to extract specific items, the BIF and SAQ provide an efficient approach to obtaining high quality data. Consistent data on respondents’ psycho-social characteristics, mental and physical health, and experience with domestic violence and substance abuse are not available from other sources.

Of the nine sites included in ISIS, up to three may be programs that have received grants through the Health Profession Opportunity Grants (HPOG) program administered by ACF. As part of a separate evaluation of the HPOG program, a performance management system that collects data on services received is being developed. Data collected by this system in HPOG sites included in ISIS will also be used in the ISIS Implementation Study.

A.5 Minimizing Burden on Small Respondents

The primary organizations involved in this study are community colleges, workforce development agencies and community-based organizations that operate occupational training programs. Burden is minimized for these entities by requesting the minimum information required to achieve the study’s objectives. On-site interviews with program staff will cover topics on which the study team is unable to collect sufficient information by other means.

For the HPOG sites that are included in the ISIS evaluation, responses to ISIS items will be used to minimize burden during the HPOG intake data collection. Potential study members will first go through the ISIS informed consent process and fill out the ISIS BIF and ISIS SAQ. After random assignment, those individuals who are assigned to the treatment group will have the data elements that are in common to the ISIS BIF and the HPOG intake form automatically uploaded to the HPOG Performance Reporting System (PRS). Thus, treatment group members will not have to respond again to the same questions. Instead, they will only have to respond to the HPOG items that are not included in the ISIS data collection. The common data elements are listed below.

The informed consent process at the HPOG sites that are included in the ISIS evaluation will inform potential study members of the uses of their data in both the ISIS and the HPOG studies. These HPOG sites will administer a special combined ISIS-HPOG informed consent form which states these uses (Appendix B). The single informed consent process will also serve to minimize burden during the HPOG data collection.

Proposed HPOG Data Elements	Collected by ISIS in Proposed ISIS Basic Information Form?
Individual Characteristics at Enrollment
I1. Date of Birth	Yes
I2. Gender	Yes
I3. Race/Ethnicity	Yes
I4. Highest school grade completed	Yes
I5. Marital status	Yes
I6. Number and ages of dependent children	No
I7. Parent/guardian status (single, non-custodial, legal guardian)	Yes
I8. Veteran	No
I9. Tribal affiliation	No
I10. Individual with a disability	No
I11. Foster care youth	No
I12. Individual with limited English proficiency	No
I13. Homeless individual	No
I14. Offender	No
I15. Employed at enrollment If yes: Hourly wage in last full week Hours worked in last full week	Yes
I16. Unemployment insurance status	No
I17. Ever worked or trained in health profession prior to participation	No
I18. Public assistance status (TANF, other, none)	Yes
Contact Information
C1. Social Security Number	Yes
C2. Street Address, City, State, ZIP	Yes
C3. Home, work, cell phone #	Yes
C4. Alternative contacts (name, address, relationship, phone #) – up to 3	Yes
C5. Referral source	No

A.6 Consequence of Not Collecting Information or Collecting it Less Frequently

Earlier we identified a number of critical uses of baseline data. Here, we note briefly how several of these uses would be affected were baseline data not available.

A.6.1 Description of Sample at Random Assignment

Obtaining baseline data allows the ISIS team to find out whether there are any systematic differences between the treatment and control groups. Without baseline data, we would lack an important basic validity check that random assignment produced very similar treatment and control groups. Furthermore, we would not be able to describe in more detail the population these innovative programs are serving.

A.6.2 More Precise Estimates of Program Impacts

Without baseline values of measures correlated with outcomes our estimates for the impact analysis would be less precise. The more precise the estimates are, the more confident evaluators can be in determining which strategies are truly improving self-sufficiency for low-income families.

A.6.3 Analysis of Sub-Populations

Lack of baseline data would mean the ISIS team would be unable to make separate impact estimates for different subgroups of interest. These data allow researchers to show whether the programs are more effective for some groups than for others. Subgroup analyses could help programs target the participants they can help the most in the future.

A.7 Special Circumstances

The proposed data collection activities are consistent with the guidelines set forth in 5 CFR 1320.6 (Controlling Paperwork Burden on the Public, General Information Collection Guidelines). There are no circumstances that require deviation from these guidelines.

A.8 Solicitation of Public Comment and Other Consultations with the Public

In accordance with the Paperwork Reduction Act of 1995, the Administration for Children and Families (ACF) at the Department of Health and Human Services published a notice in the Federal Register on September 28, 2010, page 59726. The document number is FR Doc. 2010–24122. The notice provided a 60-day period for public comments, and comments were due by November 27, 2010. A copy of the notice is shown in Appendix G. ACF published the Federal Register Notice and received a number of requests for copies of the instruments but did not receive any substantive comments on the instruments or the proposed data collection.

A.9 Payment to Respondents

There will be no payment to respondents.

A.10 Assurances of Confidentiality

The information collected under this data collection will be kept private to the fullest extent provided by law. The agency does not believe that data collected require an assurance of confidentiality and does not intend to seek a certificate of confidentially. The information requested under this collection will be private in a manner consistent with 42 U.S.C. 1306, 20 CFR 401 and 402, 5 U.S.C.552 (Freedom of Information Act), 5 U.S.C. 552a (Privacy Act of 1974) and OMB Circular No. A-130.

A.11 Justification for Sensitive Questions

The SAQ includes questions about physical and emotional health, substance use, and history of domestic violence, items that some respondents may consider sensitive. The literature provides ample support for including these items as barriers to education and employment as outlined in Appendix G. Including these items is necessary to describe the study population and evaluate mediating effects on program impacts. Program staff will remind potential study members during the enrollment process that they may refuse to answer individual items. Potential study members will also be reminded that their responses will be kept confidential, to encourage their candid responses.

A.12 Estimates of Public Reporting Burden

Baseline data collection for the ISIS evaluation will run from August 2011 to September 2013. Exhibit A-1 shows the estimated annual respondent burden for the baseline data collection. It shows the average time, in hours, that we estimate completing intake surveys, and key informant interviews. It also shows the minimum and maximum estimates for the length of the surveys, depending on the individual circumstances of the respondents. The estimates of burden assume that ISIS will recruit nine partnering sites to participate in the evaluation. We expect to recruit an average of 600 program participants per year from each site for an annual sample size of 5,400 individuals. Each program participant will fill out the BIF and SAQ at intake. We estimate that the entire process will take on approximately 40 minutes to complete.

Exhibit A-1 also shows the estimated annual burden on program staff and directors participating in key informant interviews for the ISIS Implementation Study. ISIS researchers will interview up to 12 staff members at each site during the first year of the project. Each interview will take approximately 1.5 hours. The total burden of data collection from these staff is 162 hours.

Exhibit A-1

Estimated Respondent Burden Hours and Costs

Instrument	Annual Number of Respondents	Number of Responses per Respondent	Average Burden Hours per Response	Total Annual Burden Hours
Basic Interview Form	5,400	1	0.25	1,350
Self-Administered Questionnaire	5,400	1	0.33	1,800
Key Informant Interviews	108	1	1.5	162
TOTAL Burden Hours				3,312

The annual burden for all components of baseline data collection is 3,312 hours. There is no separate cost burden associated with collecting the baseline data. In return for participating in the evaluation, the study is entering into planning contracts with each site to reimburse costs associated with implementing the ISIS evaluation.

A.13 Annual Cost to the Respondents (Other)

This data collection effort involves no recordkeeping or reporting costs for respondents other than those described in item A.12 above.

A.14 Annual Cost to Federal Government

ACF is funding the costs of the study. The annual costs to the Federal Government for ISIS baseline data collection will be $35,840 in 2010, $468,072 in 2011, $726,131 in 2012, and $678,944 in 2013 (total across years is $1,908,986). These costs include the cost of site staff time to administer the BIF and SAQ, the development of data collection instruments and tools, and ISIS team costs to collect implementation research data.

A.15 Program Changes or Adjustments to the Information Collection Request

This evaluation involves new data collection that increases the public reporting burden. Section A.12 details the burden figures.

A.16 Plans for Publication of Information Collection Results

The overall project schedule for analysis and reporting includes three major deliverables, as follows:

Implementation/Process Study

Data Collection October 2011 – March 2013

Data Analysis January 2012 – June 2014

Report Issued June 2014

12-Month Impact Study

Data Collection August 2011 – September 2014

Data Analysis October 2013 – June 2015

Report Issued June 2015

Final Report (Cost-Benefit Analysis and 36-Month Impact Study)

Data Collection October 2014 – September 2016

Data Analysis October 2016 – June 2017

Report Issued June 2017

The objectives of the Implementation Study will be to: assess the degree to which sites implement their interventions as planned; document the operations, structure, program and interventions in the evaluation; document relevant contextual factors that affect implementation; and help interpret findings of the impact studies. Researchers will use baseline data collected via the BIF and the SAQ to describe the sample and to conduct impact analyses and write reports. The 12-Month and 36-Month Impact Studies will provide estimates of program impacts on key participant outcomes for each site. The Final Report will consist of the 36-month Impact Study and the Cost-Benefit Analysis. We are likely also to report analyses of baseline data in site profile reports and research briefs issued by the study.

A.17 Displaying the OMB Approval Expiration Date

All data collection instruments created for the ISIS evaluation will display the OMB approval number and expiration date.

A.18 Exceptions to Certification Statement

This submission describing data collection requests no exceptions to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Basic Information Form
Author	NicholsonJ
File Modified	0000-00-00
File Created	2021-02-01