Supporting Statement Part B SCSEP Impact Study Revised 12 4 24_Edited

PART B: COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

OMB Control Number. XXX

omb expiration date

SUPPORTING STATEMENT FOR

THE OLDER WORKERS STUDY IMPACT EVALUATION

OMB CONTROL NO. XXXX-XXXX

In this document, the Department of Labor (DOL) requests clearance from the Office of Management and Budget (OMB) under the Paperwork Reduction Act (PRA) for a new collection associated with the Older Workers Study.

PART B: COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

The Senior Community Service Employment Program (SCSEP), administered by the U.S. Department of Labor (DOL), aims to help low-income, seniors ages 55 and older obtain private-sector, unsubsidized employment through job training and placement activities. To learn more about SCSEP and to inform continuous improvement of the program, the DOL Chief Evaluation Office (CEO), in collaboration with the DOL Employment and Training Administration (ETA) funded Urban Institute and its partner, Capital Research Corporation, to conduct the Older Workers Study. The study includes the following components:

1. A review of existing knowledge and data to inform evaluation activities,

2. An implementation evaluation design,

3. An early implementation study and in-depth implementation study of programs receiving the 2020 DOL SCSEP grants,

4. An impact evaluation that identifies an intervention for a pilot and rigorously evaluates the impact of the intervention on older workers’ employment outcomes (primarily placement outcomes), and

5. An evaluability assessment and potential future research options that would address important gaps in the evidence base related to employment services for older workers.

The next sections discuss the Baseline Information Form that is part of the impact study (component #4).

B.1. Respondent Universe and Sampling

This section describes the respondent universe and sampling for each instrument. Part “a” discusses participant selection and part “b” describes the expected response rate.

1. Selection of participants

The Older Workers Impact Study will rigorously evaluate the effectiveness of an intervention that provides occupational training, on-the-job learning, and job search assistance to interested SCSEP participants on employment outcomes. All grantees that receive Sector-Based Training for Low-Income Older Adult Workers Demonstration Grants will participate in the impact study. The evaluation team intends to use a randomized controlled trial (RCT) design. Grantees will inform new SCSEP enrollees and already enrolled participants about the Demonstration services. Once informed about the demonstration, SCSEP participants will have the option of not participating or agreeing to take part in the study and thus be randomly assigned to the treatment or control group. Those who consent to be in the study will then sign a form and fill out the form we are seeking approval, the “Impact Evaluation of the Senior Community Service Employment Program Sector-based Training Grants Demonstration for Low-income Older Workers Baseline Information Form (referred to as Impact Study Baseline Information Form (BIF)).”

The Demonstration Grants on Sector-Based Training for Low-Income Older Adult Workers Funding Opportunity Announcement (FOA) states that grant applicants must recruit and enroll a minimum of 400 study participants (200 each in the treatment and control groups). Applicants that recruit and enroll at least 800 study participants (400 in each group) will receive the maximum amount of funding. Given these parameters, the evaluation team estimates a pooled minimum sample size of 2,400 (1,200 treatment and 1,200 control) and maximum sample size of 4,800 (2,400 in each group). For purposes of burden estimates, we use the maximum sample size. The universe and sample size estimates for the BIF is provided in Table B.1.

Table B.1. Summary of universe and sample counts

Evaluation component	Universe Description	Estimated Size of Universe	Expected Sample Size 1	Sampling method
Impact Study Baseline Information Form	All new and ongoing SCSEP enrollees that consent to be in the impact study	4,800	4,800	Universe of all SCSEP participants who consent to be in the study

2. Response rates

Potential study participants will only be enrolled in the study and randomly assigned if they consent to be in the study and complete the BIF as part of the study intake process. The project team therefore anticipates that the response rate to the BIF will be100 percent of study participants. However, potential study participants may choose not to respond to particular BIF questions. In these situations, the project team will still include study participants in the analyses. The project will impute any missing variables separately for the treated and control groups so that the information can be used in pre-program data without excluding individuals from the sample.

B.2. Procedures for the collection of information

The BIF will be a one-time data collection. Data collection will begin in January 2025. Grantee staff will recruit potential study participants interested in the sectoral training demonstration, either from new enrollees or already enrolled participants with at least 12 months of remaining eligibility. Using talking points developed by the evaluation team, staff will describe the Demonstration using varying methods, depending on if the potential participant is a new or current SCSEP participant. Staff will stress the following points: (1) access to Demonstration services will be determined by a lottery; (2) to enter the lottery, participants must consent to be in the study; and (3) those who do not consent to be in the study will receive standard SCSEP services and will not participate in data collection. Participants who sign an informed consent form will complete the BIF and then be randomly assigned to the treatment group or the control group.

a. Estimation procedures

The data collected through the BIF will supplement information gathered through the DOL Grants Performance Management Data System (GPMS), which houses SCSEP participant records beginning at the time of application. Specifically, the BIF will gather information not available through GPMS, such as details about their longest pre-SCSEP job, reasons they are interested in the Demonstration services, and quality of life. The BIF will also collect participants’ Social Security Number, which is required for the evaluation team to access participant employment and earnings records through the National Directory of New Hires, a national database of quarterly employment and earnings data maintained by the Office of Child Support Services in U.S. Department of Health and Human Services Administration for Children and Families.

The BIF data will be used to profile the study population, test for systematic differences between the treatment and control groups, define subgroups, and improve the precision of impact estimates. The evaluation team will conduct t-tests on each baseline measure in isolation to examine differences between the research groups due to random sampling. The evaluation team will also conduct a joint F-test to assess the joint significance of the baseline differences. The impact analyses will control for baseline characteristics, correlated with the outcomes, to improve the precision of the estimates.

To impute missing responses on the BIF the project will use a chained stochastic regression approach to impute variables using information from other variables.¹ The chained equation method runs a series of regression models that temporarily fill in missing values of variables when predicting other ones. This updating process continues until the change to the newly predicted values are below a pre-specified stopping criterion. The project will then use predictive mean matching to impute missing observations. This method works, for example, by filling in a person’s missing education level by (1) identifying a group of individuals with similar predicted education values, based on a combination of other background characteristics, to those of the person with the missing value and (2) using the actual education level of a randomly selected person in that group as the imputed value. This method is valid under the assumption that data are missing at random, conditional on the variables included in the imputation model.

b. Statistical power

This section documents the expected sample sizes for the study and the minimum detectable effects (MDE) for the primary impact study outcome: unsubsidized employment in the fifth quarter post random assignment. Table B.2 shows the estimated MDE based on the minimum (2,400) and maximum (4,800) projected sample sizes. The true sample size will likely be somewhere in between.

Rigorous studies of job training programs over the last decade generally have found effects on employment rates of about 5 percentage points.² These programs broadly treated an adult population. Adults ages 25 to 54 have about an 80 percent labor force participation rate; adults 55 to 64 and 65 and older have lower participation rates (65 and 27 percent, respectively). The impact of a sector training program on older workers—and particularly SCSEP participants, who face additional multiple barriers to obtaining subsidized employment—is unknown.³

The percentage of SCSEP participants exiting the program into unsubsidized employment hovered around 15 percent between 2010 and 2016.⁴ Because SCSEP participants have not been screened for interest, there is presently no good measure of this employment rate, so the evaluation team used two assumptions, 15 and 25 percent.

The first MDE column of Table B.2 shows the study could detect a 4.1 percentage point difference in employment assuming the minimum sample size (2,400) and base employment rate for control group (15 percent). The second column shows the MDE with a larger sample size. In this scenario, the MDE is 3.2 percentage points. The third and fourth columns show the MDE if the base employment rate increases 25 percent. As shown, the MDE increases to 5.0 percentage points for the smaller sample and 3.5 percentage points for the larger one. The study is likely to be able detect between a 5.0 and 3.2 percentage point or more increase in employment.

Table B.2. Minimum Detectable Effects for Employment, by Baseline Assumptions

	Minimum Detectable Effects
Employment outcome Q5	4.1 pp	3.2 pp	5.0 pp	3.5 pp
Sample Size	2,400	4,800	2,400	4,800
Control group employment assumptions	15 percent	15 percent	25 percent	25 percent

Notes: Q5 refers to the fifth quarter after randomization. The MDEs are calculated based on the assumptions that grantee enrollment and control group size are the same size, a .05 significance level, and 80 percent power. pp stands for percentage points.

c. Statistical methodology for sample selection

All SCSEP participants who consent to be part of the study will be subject to random assignment. Random assignment will be conducted online using a system with pre-specified random assignment strings, developed separately for each sample intake location. Strata will be formed using information from the BIF and GPMS to ensure the research groups are balanced along key dimensions such as age, sex, race, highest level of education, and SCSEP priority of service factors (e.g., Veteran, disability, limited English proficiency).

B.3. Methods to maximize response rates and minimize nonresponse

The study team expects a 100 percent response rate on the BIF. Grantee staff will be trained to describe the study, including random assignment and data collected, how the data will be used, and to answer any questions from potential participants. Data from completed BIFs will be reviewed throughout the fielding period for accuracy and consistency.

B.4. Tests of procedures or methods to be undertaken

The data collection procedures and instrument included in this request to be used in the evaluation have been reviewed by content and methodological experts to ensure clarity and optimal ordering of the questions. The questions are based closely on prior surveys that have been extensively tested to evaluate the clarity of the questions to be asked. The procedures used to collect the data will be based closely on the procedures used successfully for similar studies, which ensures that they can be used effectively to conduct the data collection for this study.

B.5. Individuals consulted on statistical aspects of design and on collecting and/or analyzing data

Staff responsible for overseeing the collection and analysis of data are listed in Table B.3 and individuals consulting on the impact study data collection efforts are listed in Table B.4.

Table B.3 Individuals overseeing the collection and analysis of data for the Older Workers Impact Study

The Urban Institute	Barbara Butrica, Co-Principal Investigator Pamela Loprest, Project Director Karen Gardiner, Impact Study Lead
Capital Research Corporation	John Trutko, Researcher

Table B.4 Individuals consulting on the collection and analysis of data for the Older Workers Impact Study

The Urban Institute	Barbara Butrica, Co-Principal Investigator Pam Loprest, Project Director Karen Gardiner, Impact Study Lead
Capital Research Corporation	John Trutko, Researcher
Technical Work Group Members	Jacqueline L. Angel, Wilbur J. Cohen Professor in Health and Social Policy, LBJ School of Public Affairs, The University of Texas at Austin Cal J. Halvorsen, Assistant Professor, Boston College School of Social Work and affiliate of the Center for Aging & Work at Boston College Maria Heidkamp, Chief Innovation and Policy Officer · New Jersey Council of County Colleges Susan Houseman, Vice-President and Director of Research, Upjohn Institute for Employment Research David Judkins, Principal Associate, Abt Associates

1 Rubin, D.B. 1987. Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons Inc., New York.
http://dx.doi.org/10.1002/9780470316696

Raghunathan T.W., Lepkowksi J.M., Van Hoewyk J., Solenbeger P. 2001. A multivariate technique for multiply imputing missing values using a sequence of regression models. Survey Methodology, 27, 85–95. [Google Scholar]

2 Katz, L.F., J. Roth, R. Hendra, and K. Schaberg. 2020. “Why Do Sectoral Employment Programs Work? Lessons from WorkAdvance.” NBER Working Paper 28248. Cambridge, MA: National Bureau of Economic Research. http://doi.org/10.3386/w28248;

Barnow, B.S., and J. Smith. 2016. “Employment and Training Programs.” In Economics of Means-Tested Transfer Programs in the United States, Volume II, edited by Robert A. Moffitt, 127-234. Chicago: University of Chicago Press;

Maguire, S., Freely, J., Clymer, C., Conway, M. & Schwartz, D. 2010. Tuning in to local labor markets: Findings from the Sectoral Employment impact study. Philadelphia: Public/Private Ventures. [Wisconsin Regional Training Partnership]

3 Butrica, Barbara A. 2022. “Senior Community Service Employment Program: Background to Inform the Older Workers Implementation and Descriptive Study.” Washington, DC: Urban Institute.

4 Butrica 2022