HPOG 1 Impact 72 Mo Follow-up OMB Part A -06092017

This section provides supporting statements for the collection of information for the first round of the Health Profession Opportunity Grants (HPOG) program funded by the U.S. Department of Health and Human Services (HHS), Administration for Children and Families (ACF). This information collection will provide data for the Impact Study of the Health Profession Opportunity Grants (HPOG-Impact) study. HPOG funds programs that provide Temporary Assistance for Needy Families (TANF) recipients, other low-income individuals, and members of Native American tribes with training and support needed to find and keep employment in healthcare occupations and fill the growing demand for skilled healthcare workers. ACF awarded thirty-two grants in September 2010 to government agencies, community-based organizations, post-secondary educational institutions, and tribal-affiliated organizations to conduct these activities. The grants were for a 5-year period. Of the 32 HPOG grants, 27 were awarded to agencies serving TANF recipients and other low-income individuals and are relevant to this request.

Twenty of the 27 grantees are participating in HPOG-Impact and all 27 are participating in the HPOG National Implementation Evaluation (NIE), a complimentary study covered by the same OMB number, but with no current or upcoming data collection requiring OMB approval. Abt Associates is conducting the evaluation under contract to ACF. Four data collections related to this current submission were previously approved by the Office of Management and Budget (OMB) under OMB Clearance number 0970-0394:

The Implementation, Systems and Outcome Evaluation of the Health Profession Opportunity Grants to Serve TANF Recipients and Other Low-Income Individuals, for the Performance Reporting System (PRS) (clearance received September 2011).
The HPOG-Impact Study’s baseline data collection instruments (clearance received October 2012).
HPOG-Impact Study’s 15-month follow-up survey and the National Implementation Evaluation’s data collection through surveys and interviews of HPOG grantees, management/staff, stakeholders, and employers (clearance received August 2013).
HPOG-Impact Study’s 36-month follow-up survey and the National Implementation Evaluation’s data collection through surveys and interviews of HPOG grantees, management/staff, stakeholders, and employers (clearance received December 2014).

This submission seeks OMB approval for an additional follow-up data collection instrument:

A 72-Month Follow-up Survey for the HPOG-Impact study

HPOG-Impact is one project within the broader portfolio of research that OPRE is utilizing to assess the success of career pathways programs and models. This strategy includes a multi-pronged research and evaluation approach for the HPOG program (including this impact study and the national implementation evaluation), as well as the Pathways for Advancing Careers and Education (PACE) study, to better understand and assess the activities conducted and their results.¹ In order to maximize learning across the portfolio, survey development for the HPOG and PACE baseline and follow up surveys was coordinated, and the majority of the data elements collected in these surveys are the same. Three data collection efforts for PACE (three of nine programs in PACE are HPOG grantees) have been approved under OMB clearance number 0970-0397 and a fourth (new) request is being submitted at the same time as this request. The HPOG-Impact and PACE research teams coordinated on development of the 72-month survey.

Other HPOG-related research and evaluation activities for the first round of HPOG programs include a separate evaluation of the Tribal HPOG grants conducted by NORC at the University of Chicago (OMB clearance number 0970-0395).

ACF and its contractors are engaged in many efforts to coordinate research activities so that each study capitalizes on related work conducted in other projects. Coordinating research efforts minimizes burden for grantees and for study participants. In addition, comparable data from different, related studies may be combined to enhance the cumulative development of knowledge useful to government policy makers, program operators, and the public.

A.1 Necessity for the Data Collection

ACF seeks approval for a 72-month follow-up data collection for HPOG-Impact.

A.1.1 Study Background

As part of the Affordable Care Act (ACA) of 2010, Congress authorized funds for the HPOG program “to conduct demonstration projects that provide eligible individuals with the opportunity to obtain education and training for occupations in the healthcare field that pay well” (Grant Announcement HHS-2010-ACF-OFA-FX-0126). ² These demonstration projects address two pervasive and growing problems: the increasing shortfall in the supply of qualified healthcare professionals in the face of expanding demand, and the increasing requirement for a post-secondary education to secure a job with a living wage for families.

The HPOG evaluations collected data to document and demonstrate how effectively grantees implemented the HPOG program and will continue to collect data to assess how variations in services offered at each program affect program outcomes and participant impacts. As such, the studies fill a void in the sectoral training and career pathways literature both about program effectiveness and about which types of programs or program components are most effective. Few large-scale impact studies of career pathways efforts exist, and none that show the impact of specific program components and models (Werner, Dun Rappaport, et al., 2011).³

A.1.2 Legal or Administrative Requirements that Necessitate the Collection

H.R. 3590, the ACA requires an evaluation of the HPOG demonstration projects (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a) (3) (B)). The Act further indicates that the evaluation will be used to inform the final report to Congress (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a) (3) (C)). The Act calls for evaluation activities to assess the success of HPOG in “creating opportunities for developing and sustaining, particularly with respect to low-income individuals and other entry-level workers, a health professions workforce that has accessible entry points, that meets high standards for education, training, certification, and professional development, and that provides increased wages and affordable benefits, including healthcare coverage, that are responsive to the workforce’s needs” (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(B)). HPOG was extended by the Protecting Access to Medicare Act (PAMA) of 2014 (H.R. 4302; Public Law. 113–93, April 1, 2014, Title I Medical Extenders, Section 208, “Extension of Health Workforce Demonstration Project for Low-Income Individuals,” Section 2008(c)(1) of the Social Security Act (42 U.S.C. 1397g(c)(1)) is amended by striking “2014” and inserting “2015.”

A.2 Purpose of the Survey and Data Collection Procedures

A.2.1 Overview of Purpose and Approach

The HPOG-Impact study, in conjunction with the related studies under the HPOG research umbrella, will increase the knowledge base about the effectiveness of HPOG programs in providing TANF recipients and other low‐income individuals with opportunities for education and training that lead to employment and advancement in the healthcare workforce. The rest of this section describes the data that was and will be collected for HPOG-Impact.

For HPOG-Impact, baseline data were collected through the HPOG PRS, including the supplemental baseline questions that were previously approved under this OMB number. The purposes of these data are several. First, the contact information collected at baseline is necessary to enhance researchers’ ability to locate respondents for follow-up surveys that will measure intervention outcomes. A second purpose is to create a rich dataset for researchers to explore and test hypotheses, including those about the impact of HPOG programs and the relative effectiveness of various components and implementation features of those programs. Other analytic purposes of the baseline data include characterizing the HPOG-Impact Study sample, adjusting for chance differences in observable characteristics and thereby increasing precision of impact estimates, identifying subgroups of interest (including program-related subgroups), checking the integrity of random assignment, and adjusting for non-random survey sample attrition. The child roster questions were used to create a sampling frame for 36-and 72-month follow-up surveys that collect data about child outcomes.

The 15-Month Participant Follow-Up Survey approved under this OMB Clearance Number collected data on outcomes, including HPOG services received, participation in non-HPOG trainings or services, receipt of degrees or certifications, and employment and earnings outcomes. These data are used to understand treatment and control differentials in the experiences and outcomes of study participants.

The 36-Month Participant Follow-Up survey, approved under this OMB Clearance number and currently underway, will allow for an understanding of the experiences and intermediate outcomes of study participants in both the treatment and control groups. Using experimental impact analysis and these data, the research team will estimate the extent to which HPOG programs lead to differential mean individual outcomes between the treatment and control groups. In assessing the relative impacts of specific program components, the team is combining prospective systematic variation of program models within selected HPOG grantees with natural variation in program models across many HPOG grantees.

The 72-month Participant Follow-Up survey—the focus of this request for clearance—will allow for understanding how the experiences of study participants in the treatment group (those who participated in the HPOG program) differ from those in the control group over a longer period of time. Using experimental impact analysis and these data, the research team will estimate the extent to which HPOG programs lead to differential mean individual outcomes between the treatment and control groups. In assessing the relative impacts of specific program components, the team is combining prospective systematic variation of program models within selected HPOG grantees with natural variation in program models across many HPOG grantees.

The primary beneficiaries of this planned data collection effort will be ACF, other federal agencies, program operators, other policy makers and researchers, and the healthcare community. ACF will use the information to assess the effects of the HPOG programs on low-income individuals and on the healthcare community. These data will begin to answer ACF's and other policy makers’ questions about the implementation and impacts of the career pathways programs focused on training staff for the healthcare industry. It will help identify which program components and features appear to result in impacts related to education and credential achievement, employment and earnings, income, adult and child well-being and will provide information on the systems change that occurs as a result of these programs.

Secondary beneficiaries of this data collection will be those in the public policy and social science research community who are interested in further understanding initiatives to promote economic self-sufficiency of individuals and families through career pathways programs, particularly as they relate to the healthcare industry. At the conclusion of the HPOG studies, the research team will provide ACF with a restricted-use data set containing individual-level data stripped of all personally identifying information. The restricted-use data will be made available to researchers for approved secondary uses. Ultimately, these data will benefit researchers, policy analysts, and policy makers in a wide range of program areas.

A.2.2 Research Questions

Overall, HPOG-Impact will address the following research questions:

1. What impacts do HPOG programs, as a group, have on the outcomes of participants and their families?

2. To what extent do these impacts vary by subgroups of interest?

3. Which locally-adopted program components influence average impacts?

4. To what extent does participation in a particular HPOG component (or components) change the impact experienced by individual trainees?

Questions specific to the ongoing data collection and the new data collection we are seeking approval for under this package focus on interim and long-term (36-month and 72-month) participant impacts. These questions include:

What are the longer-term effects of the HPOG program on its populations of interest?
How do effects of career pathways programs vary over time, across outcomes or domains, by occupational sector, by program model and by participant characteristics?
Do different HPOG models, strategies or components, (e.g., a particular curricular model, such as I-BEST, particular recruitment strategies or support services, etc.) lead to different impacts for participants?
How can career pathways models be adjusted to promote longer-term outcomes for participants?

A.2.3 Study Designs

The HPOG studies are guided by the career pathways framework, as shown in the HPOG logic model (Appendix A). The framework puts into practice the assertion that “post-secondary training should be organized as a series of manageable and well-articulated steps accompanied by strong supports and connections to employment” (Fein et al., 2012). These articulated steps provide opportunities for students to advance through successively higher levels of education and training, exiting into employment at multiple possible points. The framework also incorporates customization, supports and employer connections.

Guided by the framework, the goal of HPOG-Impact is to evaluate the effectiveness of approaches used by 20 of the HPOG grantees with regard to improving HPOG participants’ attainment of education, training, employment, and advancement within the healthcare field. HPOG-Impact also intends to assess whether HPOG programs affect the outcomes of participants’ families by including questions on the 36-month and 72-month survey that focus on their children. Data from the three HPOG/PACE grantees (or dual sites) will be used in some of the analyses.

HPOG-Impact intends to evaluate variation in participant impact that may be attributable to different HPOG program components and models. Twenty of the 27 HPOG grantees serving TANF and low-income individuals will be included in the HPOG-Impact analysis.⁴

The HPOG-Impact design includes randomizing program-eligible participants to treatment and control status in all sites. Ten grantees (19 sites total) randomized applicants into two treatment arms (a basic and an enhanced version of the intervention) and a control group. This third experimental arm creates planned variation in some sites that is consistent with natural variation of these program components in other sites, thereby creating an opportunity to learn more both about the effects of program components but also about the methods we use to estimate those effects.

Those in the third arm participated in one of three enhanced HPOG services, in which the basic HPOG programs are augmented by an additional program component. In this subset of grantees, program applicants are randomly assigned to (1) the “basic” HPOG program, (2) an “enhanced” HPOG program (i.e., the HPOG program plus an enhancement) or (3) a control group that is not offered the opportunity to enroll in HPOG.⁵

Control group members have access to whatever other programs and services are available in the local community.

In a parallel activity, as mentioned above, the PACE project has submitted an OMB clearance request (revision of OMB # 0970-0397) for a 72-month follow-up survey of PACE study participants, including the three HPOG grantees that are participating in PACE. PACE is also conducting a randomized experiment that will assess similar program impacts. Data from the PACE programs that are HPOG grantees will be included in the HPOG impact analysis.

Additionally, the research team will match participant data collected through HPOG-Impact for both the treatment and control groups to long-term employment and earnings data from ACF’s National Directory of New Hires (NDNH). An agreement with the Office of Child Support Enforcement (OCSE) is in place. The research team will also use the National Student Clearinghouse (NSC) to gather information about college persistence and degree completion for study participants. NSC is currently the only data source that tracks postsecondary student enrollment in all states.

A.2.4 Universe of Data Collection Efforts

To address these research questions, the studies will use a number of data collection sources. Instruments in the current clearance request include the following:

The HPOG-Impact third follow-up survey (at 72-months post-random assignment) of both treatment and control group members.

These data are not available through any current source.

Study instruments approved by OMB in prior information collection requests include the following:

HPOG Performance Reporting System (PRS), a management information system for documenting program activities and accomplishments against program goals and to assist with program management (approved September 2011 under this OMB No. 0970-0394)
Supplemental Baseline Questions to the PRS, to be used at participant intake for random assignment into the Impact Study and for analysis for both HPOG-NIE and HPOG-Impact(approved October 2012 under this OMB No. 0970-0394)
PACE 15-month Follow-Up survey for the three HPOG grantees that are participating in the PACE project (approved August 2013 under OMB No. 0970-0397)
The HPOG-NIE Sampling Questionnaire for the HPOG surveys (approved August 2013 under this OMB No. 0970-0394)
The HPOG-NIE Follow-Up phone call protocol for the Stakeholder/Network survey (approved August 2013 under this OMB No. 0970-0394)
The HPOG-NIE Grantee survey (approved August 2013 under this OMB No. 0970-0394)
HPOG-Impact Implementation interview guide for partnering employers (approved August 2013 under this OMB No. 0970-0394)
HPOG-Impact Implementation interview guide for instructors (approved August 2013 under this OMB No. 0970-0394)
HPOG-Impact Implementation interview guide for HPOG program management (approved August 2013 under this OMB No. 0970-0394)
HPOG-Impact Implementation interview guide for HPOG program staff (approved August 2013 under this OMB No. 0970-0394)
The HPOG-NIE Management and Staff survey (approved August 2013 under this OMB No. 0970-0394)
The HPOG-NIE Stakeholder/Network survey (approved August 2013 under this OMB No. 0970-0394)
The HPOG-NIE Employer survey (approved August 2013 under OMB No. 0970-0394)
The HPOG-Impact and HPOG-NIE 15-month Participant Follow-Up survey (approved August 2013 under this OMB No. 0970-0394)
The HPOG-Impact15-month Control Group Member Follow-Up survey (approved August 2013 under this OMB No. 0970-0394)
The HPOG NIE Screening Questionnaire (approved December 2014 under this OMB No. 0970-0394)
The HPOG NIE Semi-structured Interview (approved December 2014 under this OMB No. 0970-0394)
The HPOG-Impact and HPOG-NIE 36-month Participant Follow-Up survey (approved December 2014 under this OMB No. 0970-0394) (See Appendices L1 and L2)
PACE 36-month Follow-Up survey for the three HPOG grantees that are participating in the PACE project (approved December 2014 under OMB No. 0970-0397)

Other extant data sources will be used for the HPOG studies. These include the following:

National Directory of New Hires (NDNH). These data will provide information about employment and earnings of HPOG participants.
National Student Clearinghouse: NSC includes 3,600 participating public and private institutions that collectively represent approximately 98 percent of higher education enrollments nationwide. The project will use NSC data for information on terms enrolled (college persistence) and two-and four-year degree information.

A.2.5 Data Collection Process

The 36-month follow-up survey began in March 2016 and will continue through April 2018. The 72-month follow-up survey data collection for HPOG study participants will take place approximately 72 months following random assignment, which begins in late fall of 2017, pending OMB approval. Local interviewers will attempt to complete interviewers first by telephone and then in-person for those respondents who cannot be reached by telephone (Appendix B: HPOG-Impact 72-month Follow-Up Survey). Many of the questions to be asked at 72 months were approved for the 36-month survey and most other items have been asked in other OMB-approved studies. A summary of the survey item sources is provided in Appendix C.

A.2.6 Instrument Item-by-Item Justification

Exhibit A-1 describes the target respondents, content, and reason for collection (i.e., which analyses will use the information) for each new data collection activity submitted with this current request. For more information about previously approved instruments, see previous information collection request (# 0970-0394) approved December 2014. The new 72-month survey is Appendix B and a summary of the survey questions and sources for the new 72-month instrument is in Appendix C. All other survey support materials are provided in Appendices E-M.

Exhibit A-1: Item-by-Item Justification of New Data Collection Instruments

Data Collection Activity	Data Collection Instrument(s)	Respondents, Content, and Reason for Collection
Study Participant Follow-up Survey	Instrument 1:72-Month Follow-up Survey (See Appendix B)	Respondents: Overall expected sample of 2,960 (74% of sample members selected for 72-month follow-up survey data). Content: Employment success and promotions Current/most recent job conditions, job quality, benefits, on the job training Education and Credentials Adult Well-Being-psycho social skills, life challenges, social networks, perceived stress, and physical health Household composition, family formation and marital stability Income and economic well-being, student debt, financial resilience Time out of home/child supervision Child education-related goals and support Child outcomes Transition to adulthood Contact information Reason: This follow-up period of 72 months will provide a longer-term look at the sustainability of early and interim impacts on education, employment, and overall adult well-being, and an opportunity to measure child outcomes after they have aged six years.

Data Collection Activity

Data Collection Instrument(s)

Respondents, Content, and Reason for Collection

Study Participant Follow-up Survey

Instrument 1:72-Month Follow-up Survey

(See Appendix B)

Respondents: Overall expected sample of 2,960 (74% of sample members selected for 72-month follow-up survey data).

Content:

Employment success and promotions
Current/most recent job conditions, job quality, benefits, on the job training
Education and Credentials
Adult Well-Being-psycho social skills, life challenges, social networks, perceived stress, and physical health
Household composition, family formation and marital stability
Income and economic well-being, student debt, financial resilience
Time out of home/child supervision
Child education-related goals and support
Child outcomes
Transition to adulthood
Contact information

Reason: This follow-up period of 72 months will provide a longer-term look at the sustainability of early and interim impacts on education, employment, and overall adult well-being, and an opportunity to measure child outcomes after they have aged six years.

A.3 Improved Information Technology to Reduce Burden

The HPOG studies will generate a substantial amount of data and will use a combination of data collection methods. For each data collection activity, the study team has selected the form of technology that enables the collection of valid and reliable information in an efficient way while minimizing burden. This evaluation will use improved technology to facilitate the collection of the survey data in standardized and accurate ways that also ensures the protection of the data collected.

As with the previously approved 15-month and 36-month Participant Follow-Up surveys, the 72-month Participant Follow-Up survey administration will use automated technology. Local interviewers will administer the survey electronically in the field, using CAPI (computer-assisted personal interviewing) technology. CAPI reduces respondent burden, as interviewers can proceed more quickly and accurately through the survey instruments, minimizing the interview length. Computerized questionnaires ensure that the skip patterns work properly, minimizing respondent burden by not asking inappropriate or non-applicable questions. For example, respondents who did not work during the follow-up period are routed past questions only relevant to those who did. Computer-assisted interviewing can build in checkpoints, which allow the interviewer or respondent to confirm responses thereby minimizing data entry errors. Finally, automated survey administration can incorporate hard edits to check for allowable ranges for quantity and range value questions, minimizing out of range or unallowable values.

A.4 Efforts to Identify Duplication

A.4.1 Surveys

The HPOG-Impact 72-Month Participant Follow-Up survey will collect information that is not available from any other sources. This includes information on the control group experiences post-random assignment, for which there is no other information, as well as information on HPOG participants’ (treatment members) in the 20 HPOG-Impact sites post-HPOG experiences.

At 72 months post random assignment, we expect that program participants will have had time to finish their short-term training objectives and transition into more formal educational attainment and stable employment situations. The long-term follow-up will focus on measuring the impacts of programs on long-term outcomes including educational progress, employment success, economic well-being, and adult and child well-being.

In addition, this study will use administrative information on wages and employment from the NDNH linked to PRS data that will eliminate the need for gathering as complete an employment history in the survey as might otherwise be necessary. However, these administrative data do not have information on hourly wages, benefits, or other aspects of the job that we will collect in the follow-up survey.

A.4.2 Coordination and Streamlining of Study Efforts

The HPOG and PACE research teams have and will continue to work closely to coordinate data collection across both studies. Areas of coordination include:

Previously approved data collection efforts have coordinated closely, with HPOG-Impact using HPOG-NIE Grantee and Management and Staff surveys to measure program features, and HPOG-Impact site visits collecting information for use in the HPOG-NIE Descriptive Implementation Study and Systems Change Analysis. PACE and HPOG teams have conducted site visits jointly to reduce the burden for site staff. PACE has shared data collected from the three HPOG sites in the PACE study with the HPOG research team for inclusion in both the NIE and Impact studies.
Questions and constructs for the 72-Month Participant Follow-Up survey included in this clearance request are based on the 36-month follow-up evaluation instruments, to the extent feasible, in order to ensure the alignment of a core group of questions.

A.5 Involvement of Small Organizations

The primary organizations involved in previously approved data collection efforts for this study were community colleges, workforce development agencies, employers, and community-based organizations that operate occupational training programs and provide related services. The research team minimized burden for these entities, including those that could be considered small organizations, by requesting only the information required to achieve the study’s objectives and offering them the use of on-line data collection tools so that they can respond to the information request at their convenience. In addition, at the time of grant award, ACF informed all grantees of the congressionally mandated evaluation and the reporting requirements, and adequate resources have been provided to coordinate the data collection and reporting. There was no adverse impact for any grantees participating in the study. There are no small organizations involved in the data collection efforts under this information clearance request.

A.6 Consequences of Less Frequent Data Collection

The data collection effort described in this document is essential to the HPOG-Impact study. Less frequent data collection would jeopardize ACF’s ability to conduct these congressionally mandated studies in time to provide relevant and timely results to shape policy. Collecting data identified in the current request will allow measurement of longer-term impacts for individuals for the HPOG-Impact study.

Data collected through the HPOG-Impact study are critical to ACF’s comprehensive strategy to evaluate the HPOG demonstration grants. HPOG is a significant policy initiative aimed at training and placing TANF recipients and other low-income individuals in stable healthcare industry occupations with a career path. Many of the HPOG grantees adopted cutting-edge education and training technologies developed over the past decade to meet the needs of older, non-traditional students with little or no post-secondary educational experience. These relatively new approaches, which generally align with the career pathways framework, are largely untested by strong evaluation designs. Together, these studies will develop knowledge about the effectiveness of the new training modules and what works best, for whom.

A.7 Special Circumstances

There are no special circumstances for the proposed data collection.

A.8 Federal Register Notice and Efforts to Consult Outside the Agency

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13 and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995)), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on Thursday, January 12, 2017, Volume 82, Number 8, page 3797-3798, and provided a 60-day period for public comment. A copy of this notice is included as Appendix D. During the notice and comment period, the government received no requests for information or substantive comments.

To ensure the length of the instrument is within the burden estimate, we took efforts to pretest with fewer than 10 people and edit the instruments to keep burden to a minimum. During internal pretesting, all instruments were closely examined to eliminate unnecessary respondent burden and questions deemed unnecessary were eliminated.

A.9 Incentives for Respondents

Monetary incentives show study participants that the study team respects and appreciates the time they spend participating in the study's information collection activities. Although published evidence of the effectiveness of incentives in reducing nonresponse bias appears to be nearly nonexistent, it is well-established that incentives strongly reduce attrition in panel studies.⁶In accordance with OMB guidelines, the team took several factors into consideration when determining whether or not to use incentives.⁷ Specifically, the team took into account data quality issues, efforts to reduce non-response bias, the complexity of the study design and panel retention over a 72-month period, and prior use of incentives for this study population.

In longitudinal studies such as HPOG, panel retention during the follow-up period is critical to minimizing the risk of nonresponse bias and to achieving sufficient sample for analysis. Although low response rates do not necessarily lead to nonresponse bias, and it is at least theoretically possible to worsen nonresponse bias by employing some techniques to boost response rates (Groves, 2006), most statisticians and econometricians involved in the design and analysis of randomized field trials of social programs agree that it is generally desirable to obtain a response rate as close to 80 percent as possible in all arms of the trial (Deke and Chiang, 2016). The work of Deke and Chiang underlies the influential guidelines of the What Works Clearinghouse (WWC). Under those guidelines, the evidential quality rating of an evaluation is downgraded if the differential exceeds a certain tolerance (e.g., 4 percentage points when the overall response rate is 80 percent).

Mindful of these risks and the solid empirical base of research demonstrating that incentives do increase response rates, OPRE and OMB authorized incentives for the prior rounds of data collection at 15 and 36 months (OMB control number 0970-0397). In an effort to maximize response rates, the team periodically requests that participants update their contact information, through a contact update form (See Supporting Statement Part B, for more information on the contact update process and Appendix E for the previously approved contact update form and Appendices F1 and F2 for the previously approved contact letter and the new 72-month contact letter respectively). OMB authorized incentives for both completion of the survey and for the contact updates. At 15 months, the incentive was $30 for completing the follow-up survey interview and $5 for providing updated contact information in advance of the scheduled interview time. With these incentives, HPOG achieved response rates for the treatment groups varying from 65.2 percent to 92.3 percent (treatment enhanced) across all sites. Response rates on the control group were generally lower varying from 59.8 percent to 80.5 percent. Overall, HPOG-Impact achieved a 75.6 percent response rate and experienced a differential response rate of 6.9 percentage points for the treatment and control groups.

Given these response rates and gaps at 36 months, the conditional incentive for completing the main interview at 36-months was increased to $40, the $5 incentive for updating contact information was changed to a $2 prepayment included in the request for a contact update, and a prepayment of $5 was added to the advance letter package to remind them of the study and take note that a legitimate interviewer would be calling them shortly to learn about their experiences since study enrollment.

In most longitudinal studies, response rates decline over follow-up rounds. The team has tried to minimize this expected decline and ensure a high response rate with a low treatment-control differential through the continued use of the participant contact update forms and the provisions of tokens of appreciation. Through these tools, the team hopes to address three goals:

Overcome participant mobility—over a long follow-up period, many study participants relocate multiple times, making it difficult to find them and update their contact information or complete a follow-up interview;
Reduce survey data collection costs— the quicker interviewers can locate the respondent and complete an interview, the lower the costs per completed survey, and
Maintain participant engagement in a complex panel study—the ability to keep participants engaged in the research study six years after enrollment is crucial to understanding long-term outcomes.

All study participants received periodic requests to update their contact information using the previously approved contact update form, in the time between the 15- and 36-month follow-up surveys. Participants receive tokens of appreciation for providing the updated contact information leading up to the 36-month follow-up survey. The study team will continue to send these update letters to all participants who have not reached the 36th month after random assignment yet. The team will use the same contact update form, with revised letters, prior to the 72-month survey and, pending approval from OMB the team will again provide tokens of appreciation to participants that complete. The purpose of these forms is to ensure we have up-to-date contact information so that we can ensure all participants have equal likelihood of being reached when it is time to conduct their interviews.

The participant contact update form does not collect any data for analytic use, but these updates are crucial to ensuring that the contact information in the sample database is as up to date as possible during the follow-up period. Although the team is not aware of any true experiments on the effects of requests for contact updates, the team does have strong circumstantial evidence for their effectiveness in raising response rates in the follow-up survey. This evidence arises in particular from the first follow-up survey for the PACE study, where no requests were sent to early cohorts, but requests were sent to subsequent cohorts. While the final response rate differential did not differ substantially between the earlier and later cohorts, the earlier cohorts had to be worked much longer to achieve completion targets. Those early cohorts were worked about 10 to 12 months to completion, about 4 to 6 months longer than the later cohorts. The research team knows from experience on the 15-month and 36-month follow-up surveys that this sample is difficult to locate. The contact update forms help to ensure that the contact information in our records is accurate. Accurate contact information allows field interviewers to complete more cases by telephone, which reduces the average hours per complete and costs associated with in-person locating and interviewing efforts. Keeping the hours per complete low for most of the sample, allows additional time and resources to put toward efforts to interview the hardest to locate participants. Updated contact information also helps to keep the overall field period short—allowing cases to be interviewed closer to the 72-month random assignment anniversary date and ensuring the utility of the survey to explore factors influencing progression for outcomes such as cumulative credits earned and attainment of credentials, which are very sensitive to the lag between randomization and interview.

Abt Associates is currently about mid-way through data collection for the 36-month follow-up. Perhaps due to the increased incentives, among other efforts (such as the periodic contact updates and an established rapport with interviewers from the 15-month data collection), currently, the average response rate is only three points lower than it was for the 15-month follow-up.

The 72-month follow-up survey data collection will begin three years later, so the team developed a protocol to help retain the panel and keep sample members engaged in the study. The team will use the contact update form (Appendix F) again. The form will be supplemented with a participant newsletter (Appendix H) and used as part of contact update check-in call (Appendix M). The participant newsletter and the check-in call address both the challenge of the participant mobility and participant engagement. The first contact update request will be sent in the traditional way, accompanied by a letter to participants 12 months prior to the start of data collection.⁸

The team will also send the contact update form to participants eight months prior to the start of survey data collection. This time it will be sent with a participant newsletter. This newsletter intends to help show participants the importance of their continued study participation. The team will look to strengthen participant engagement through a brief contact update call. The contact update check-in call will be completed four months prior to the start of survey data collection. During the check-in call interviewers will call study participants to collect updated contact information. The check-in call also reminds participants about their role in the study, alerts them of upcoming data collection efforts, and allows them a chance to ask any questions.

Since two and a half to three years have elapsed since the research team last contacted the participants, the use of incentives will also aid in the re-engagement effort as our prior experience with this study population shows that it does respond positively to incentive payments.

Three factors helped to determine the incentive amounts for each survey:

Respondent burden, both at the time of the interview and over the life of the study;
Costs associated with participating in the interview at that time; and
Other studies of comparable populations and burden.

The HPOG 72-month follow-up study survey sample is small (up to 4,000 participants or roughly 40 percent of the total HPOG sample across all sites). The small sample size further emphasizes the importance of minimizing attrition. Study participants were contacted every three to four months over the first three years of the follow-up period and asked to participate in one of the routine contact update efforts or one of the prior follow-up surveys. Under this information collection request, they will also be asked to participate in the re-engagement efforts leading up to the 72-month follow-up study. This is a long period for participants to remain engaged in the research.

To fully utilize the data collected through the 72-month follow-up survey, we believe it is necessary to take every possible step to minimize panel attrition over the study follow-up period. This minimal attrition rate is the core justification for an incentive system for the follow-up interview. The need to maintain the panel is further complicated by the high mobility rates experienced in prior rounds of data collection with this study population. Tokens of appreciation help to secure the cooperation of the individual over the duration of the study period and reduce the potential for individuals to fail to complete the survey. This is particularly important for this round of data collection, as only a subsample of participants will be selected for interview.

Despite these re-engagement efforts, given a target response rate of 74 percent for the 72-month follow-up, and based on the incentive amounts approved for previous rounds of data collection with the potential respondents, we feel it would be wise to increase incentives, as well as to slightly restructure them.

The incentive amounts proposed (subject to OMB approval) for the 72-month survey and contact update responses are as follows:

token of appreciation valued at $5 for responding to the contact update letters;
token of appreciation valued at $10 for the completing the contact update call;
token of appreciation valued at $45 for completing the survey.

These tokens of appreciation are provided to help offset any potential expenses incurred by the participant such as cell phone minutes for those completed by telephone, and childcare, or transportation costs for those completed in-person. The proposed amounts take into consideration the incentive structure approved leading up to the 15- and 36-month survey efforts as well as changes to the overall sample retention approach. The amount requested for completion of the 72-month survey is a modest increase for a survey completed about three years after our last contact with participants. The contact update call is new to this study population, while it captures similar information to the contact update form, it does require additional burden on the participants to complete this by telephone. For that reason, we propose increasing the amount provided to $10 rather than $5. We propose returning to the previously approved $5 conditional incentive for those who return their contact update form. These incentive rates and proposed increases are comparable to what was offered and previously approved under the prior rounds of data collection under this OMB Clearance Number. These rates are also similar to what was previously approved under the prior rounds of PACE data collection (0970-0398) and is comparable to what is proposed for the 72-month follow-up effort for that study.

A.10 Privacy of Respondents

The 72-month Participant Follow-Up survey itself does not involve collecting individual identification data. The HPOG-Impact study collected individual identification data through previously approved information collections that have concluded. All HPOG-Impact study participants completed both the PRS and the Supplemental Baseline Questions that were added to PRS for the impact study.

A.10.1 Data Privacy Protections

ACF recognizes that HPOG grantees serve vulnerable populations (per the authorizing legislation), and that grantees must protect those populations from any risks of harm from the research and evaluation activities. Accordingly, earlier phases of the study obtained informed consent (see Appendix G) from all participants who entered the HPOG study. This informed consent ensured that participants understood the nature of the research and evaluation activities being conducted. The introductory text in the 36-month and 72-month Follow-Up Surveys (Appendices L1 and L2, and B respectively) reiterates the voluntary nature and privacy of the survey.

For both survey data and corresponding administrative data on sample members, computer security will be maintained by individual passwords and folder permissions that limit access to files to only those project staff members who require access to these files. Individuals provided their (1) last and first name; (2) Social Security number; and (3) date of birth at the time of enrollment. The last and first names and Social Security Numbers of each participant are needed by evaluators to obtain accurate administrative data on individuals’ quarterly earnings and receipt of cash and noncash public benefits. Those data are necessary to measure key impacts. They also help with the administrative data matching to educational records. Administrative data matches will only be done for those participants who provided informed consent. Individuals will be told how the data would be used and how they would be securely stored.

A.11 Sensitive Questions

The 72-month follow-up survey includes one question each about overall physical health (E5), emotional health (E4), and whether substance use is a barrier to work or family responsibilities (E3c), items that some respondents may consider sensitive. The 36-month survey includes two of those questions: emotional health (K4), and whether substance use is a barrier to work or family responsibilities (K7c). The literature provides ample support for including these items as barriers to education and employment. These items will help to describe the study population and evaluate mediating effects on program impacts. Interviewers will remind study members during the interviewing process that they may refuse to answer individual items. Study members will also be reminded that their responses will be kept private to encourage their candid responses.

A.12 Estimation of Information Collection Burden

A.12.1 Data Collection Already Approved

Previously Approved and Completed Data Collection Burden

The total burden for the instruments already approved (the PRS and Supplemental Baseline Questions and 15 month follow-up) was estimated to be 10,140 hours total. Baseline and 15-month data collection is complete.

The previously approved burden for the HPOG-NIE screening questionnaire and semi-structured discussion guides was 59 hours total. The HPOG-NIE data collection is complete.

Interim Outcome (36-month Follow-up Survey) Data Collection Already Approved

The total burden for the HPOG-Impact 36-month Follow-Up Survey interim instrument already approved was estimated to be 8,706 hours, or 2,902 hours annually. Administration of these previously approved interviews continues and the total remaining burden is 6,073 hours, or 2,234 hours annually over three years.

A.12.2 Current Information Collection Request

Under this information collection request, interviewers will complete the 72-month Follow-Up Survey with up to 2,960 of the HPOG-Impact study participants selected for interviewing. Exhibit A-2 shows the estimated burden for the new instrument (Appendix B: 72-month Follow-Up Survey) and instruments which have previously been approved but have burden remaining (Appendix L1 and L2: 36-Month Follow-Up Survey). It shows the average time, in hours, that study participants are estimated to spend completing each data collection instrument.

We calculated the average hourly wage for survey respondents based on the minimum wage with an adjustment as follows:

The minimum hourly wage ($7.25) plus a 40 percent adjustment to account for employment fringe benefits, or $10.15 per hour.

Exhibit A-2: Annual Information Collection Activities and Cost

Instrument	Total Number of Respondents	Annual Number of Respondents	Number of responses per respondent	Ave. burden hours per response	Annual burden hours	Ave. Hourly Wage	Total Annual Cost
Previously Approved Instruments Remaining Burden
36-Month Follow-up Survey	6,703	2,234	1	1	2,234	$10.15	$22,675
Current Request for Approval
72-Month Follow-up Survey	2,960	987	1	.75	740	$10.15	$7,511
Total
					2,974		$30,186

A.12.3 Annual Burden Hour Request

Exhibit A-2 displays annual burden. The annual burden to continue use of already approved information collection in addition to the new request is 2,974 hours per year over three years.

A.13 Cost Burden to Respondents or Record Keepers

This data collection effort involves no costs for respondents other than those described in Exhibit A-2 above.

A.14 Estimate of Cost to the Federal Government

The total cost for the data collection activities (and all related analyses and reports) under this current request will be $6,560,627. This amount includes costs for new data collection activities under this request and the remaining costs from previously approved collections still in progress. Annual costs to the Federal government will be $2,186,876 for the proposed data collection under this OMB clearance number (0970-0394).

A.15 Change in Burden

This evaluation involves new and continuing data collection that increases the public reporting burden under this OMB number. Additionally, some data collection under this OMB number has been completed. The revised burden estimates are presented in Section A.12.

A.16 Plan and Time Schedule for Information Collection, Tabulation and Publication

A.16.1 Analysis Plan

HPOG-Impact and HPOG-NIE have complementary analysis plans. HPOG-NIE reports focus on the structure of the programs designed and implemented by the grantees as well as on immediate outputs of the programs in simple terms such as numbers of graduates. HPOG-Impact reports will focus on evaluating the overall effectiveness of the grant program, as well as evaluating the relative efficacy of different program designs and studying linkages between specific program features and impacts.

Reports already published based on data collection approved under this OMB number (0970-0394) and planned future reports include the following:

An interim NIE report on baseline descriptive outcomes. This report, published in August 2014, profiled participants in terms of data collected in the Performance Reporting System (PRS).
An NIE report on grantees’ use of performance measurement information. This report, published in December 2015, provides information on grantees use of performance measurement information to track their individual program’s performance and to modify programs and practices.
An NIE report on descriptive implementation and outcomes. This report, published in April 2016, contains two major sections. The first section is a comprehensive description of the HPOG program as designed and implemented across 27 grantees and their sites. The section covered program context, including local labor market characteristics, program operations, resources, and costs; and individual level outcomes or HPOG participants. The report also includes updates to the interim report on baseline descriptive outcomes using additional available data from the PRS.
An NIE systems change and network analysis report. This report, published in May 2016, discussed changes to the service delivery system associated with program implementation. In addition, the Systems Change Analysis described and analyzed the institutional and stakeholder network in which the HPOG program operates.
An NIE final report. This report, expected by September 2017, will use the 15-month Participant Follow-Up survey data from participants to give a more complete understanding of the conditions of employment. Some of the outcomes that can be studied with participant survey data (but could not be studied with data from the PRS or NDNH) include: post-program employment and earnings in a health job, wages, benefits, further career training and career advancement.

Impact evaluation final report on short-term (15-month) outcomes. This report, expected by September 2017, will focus on how average outputs (including education and training experiences) and outcomes (including, credential/ certificate/degree attainment, employment, earnings/wages, job benefits and other characteristics) differ between the randomized groups, differences that—when statistically significant—are attributable to the HPOG program since no other systematic differences exist.

Impact evaluation final report on intermediate (36-month) outcomes. This report, expected in March 2019, will include similar analyses of employment, education, and other outcomes for treatment and control group members at approximately 36 months after random assignment.
Impact evaluation final report on long-term (72-month) outcomes. This report, expected in September 2021 will include similar analyses of employment, education, and other outcomes for treatment and control group members at approximately 72 months after random assignment.

Upon completion, each final report undergoes ACF’s thorough review process. As part of the review process, ACF will ensure each report is 508 compliant for dissemination on their website. All published reports are on the Career Pathways website:

http://www.career-pathways.org/recently-published/ or https://www.acf.hhs.gov/opre/research/project/evaluation-portfolio-for-the-health-profession-opportunity-grants-hpog.

A.16.2 Time Schedule and Publications

Exhibit A-3 presents an overview of the project schedule for information collection. It also identifies publications associated with each major data collection activity.

Exhibit A-3: Overview of Project Data Collection Schedule

Data Collection Activity	Timing	Associated Publications
Previously approved data collection efforts
Baseline data collection for HPOG-Impact	March 2013–November 2014 (COMPLETE)	Baseline Data Collection Report (February 2015)
Survey sample frames	October – December 2013 (COMPLETE)	NA
Surveys of HPOG grantees, management/staff, stakeholders, and employers	November 2013 – May 2014 (COMPLETE)	Descriptive Implementation and Outcome Report (May 2015) Systems Change and Network Analysis Report (May 2015)
15-month Participant Follow-Up survey	March 2014–November 2015 (COMPLETE)	Impact Evaluation Final Report on Short-term Outcomes (June 2016) Final NIE Report (September 2017)
36-month Participant Follow-Up survey	March 2016 to December 2017 Currently Underway under 0970-0394	Impact Evaluation Report on Intermediate Outcomes (March 2019)
Screening questionnaire	Winter 2014/2015 (COMPLETE)	HPOG Grantees’ Use of Performance Measurement Information (May 2015)
Semi-structured Interviews about Use of Performance Measures	Winter/Spring 2015 (COMPLETE)	HPOG Grantees’ Use of Performance Measurement Information (May 2015)
New data collection requests
72-month Participant Follow-Up survey	Beginning in November 2017 pending OMB approval	Impact Evaluation Report on Long Term Outcomes

A.17 Reasons not to Display OMB Expiration Date

All instruments created for HPOG-Impact will display the OMB approval number and the expiration date for OMB approval.

A.18 Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

References

Fein, D., Judkins, D., Martinson, K., Rolston, H., Gardiner, K., Bell, S., and Barnow, B. (2012). Innovative Strategies for Increasing Self-Sufficiency Evaluation Design Report. Unpublished draft manuscript. Bethesda, MD: Abt Associates Inc.

Hendra, R., Dillman, K., Hamilton, G., Lundquist, E., Martinson, K., and Wavelet, M. (2010). The Employment Retention and Advancement Project: How Effective Are Different Approaches Aiming to Increase Employment Retention and Advancement? Final Impacts for Twelve Models. New York: MDRC.

Maguire, S., Freely, J., Clymer, C., Conway, M., and Schwartz, D. (2010, July). Tuning In to Local Labor Markets: Findings from the Sectoral Employment Impact Study. Philadelphia: Public/Private Ventures.

Roder, A., and Elliott, M. (2011, April). A promising start: Year Up’s initial impacts on low-income young adults’ careers. New York, NY: Economic Mobility Corporation.

Werner, A., Dun Rappaport, C., and Lewis, J. (2011). Implementation, Systems and Outcome Evaluation of the Health Profession Opportunity Grants to Serve TANF Recipients and Other Low-Income Individuals, Draft literature review: Career Pathway Programs. Cambridge, MA: Abt Associates Inc.

1 The PACE study was originally called the Innovative Strategies for Improving Self-Sufficiency (ISIS) study. The study name was later changed to PACE.

2 HPOG was authorized by the Affordable Care Act (ACA), Public Law 111-148, 124 Stat. 119, March 23, 2010, sect. 5507(a), “Demonstration Projects to Provide Low-Income Individuals with Opportunities for Education, Training, and Career Advancement to Address Health Professions Workforce Needs,” adding sect. 2008(a) to the Social Security Act, 42 U.S.C. 1397g(a) and extended by the Protecting Access to Medicare Act (PAMA) of 2014 (H.R. 4302; Public Law. 113–93, April 1, 2014, Title I Medical Extenders, Section 208, “Extension of Health Workforce Demonstration Project for Low-Income Individuals,” Section 2008(c)(1) of the Social Security Act (42 U.S.C. 1397g(c)(1)) is amended by striking “2014” and inserting “2015.”

3 Public Private Ventures’ Sectoral Employment Impact Study (Maguire et al., 2010) is an impact evaluation of sectoral employment programs; A Promising Start: Year Up’s Initial Impacts on Low-Income Young Adults’ Careers (Roder and Elliott, 2011) is a small-scale random assignment impact study of a sectoral employment effort that does not target healthcare. The impact evaluation of the national Employment Retention and Advancement (ERA) Project (Hendra et al, 2010) is another recent impact study of a workforce development program, but it is not specifically focused on career pathways or healthcare.

4 Three HPOG grantees being evaluated under PACE (though data collection is being coordinated, and data from the HPOG/PACE grantees may be used in the HPOG Impact Study) and four HPOG grantees who are engaged in independent research projects with a University partner are not included in the HPOG Impact Study.

5Participants assigned to the HPOG enhanced group receive regular HPOG program services and enhancements such as peer support, emergency assistance, or non-cash incentives.

6 See Chapter 12 of Lynn (2009), in particular, section 12.5 that reviews the effects of incentives in several prominent longitudinal studies.

7 See page 69, questions 75 and 76, https://obamawhitehouse.archives.gov/sites/default/files/omb/assets/omb/inforeg/pmc_survey_guidance_2006.pdf

8 Given the timing of OMB review and approval and the planned start of data collection, the early cohorts will have a compressed contact update request process.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Debi McInnis
File Modified	0000-00-00
File Created	2021-01-22