Green Jobs Supporting Statement - Part B - (updated 3-9-12)

Extension of OMB

Approval for the Baseline Data Collection for the ARRA-funded Grants; Job Training Evaluation (OMB 1205-0481)

Supporting Statement Part B

Part B: Collection of Information Involving Statistical Methods

The Employment and Training Administration (ETA) of the U.S. Department of Labor (DOL) is undertaking the Green Jobs and Health Care Impact Evaluation of the Pathways Out of Poverty (POP – Green Jobs) and Health Care and High Growth Training (HCHGT) grant initiatives. The overall aim of this evaluation is to determine the extent to which enrollees achieve increases in employment, earnings, and career advancement as a result of their participation in the training provided by (POP – Green Jobs) and (HCHGT) grantees and to identify promising best practices and strategies for replication. ETA is requesting an extension from the Office of Management and Budget (OMB) for a previously approved information collection for data for Green Jobs and Health Care Impact Evaluation (GJ-HC).

The evaluation uses a random assignment research design to measure the educational and economic impacts of programs operated by selected grantees and will include a process study to examine program implementation and operations. The ETA has contracted with Abt Associates and Mathematica Policy Research to conduct this evaluation. The full evaluation involves four data collection efforts requiring OMB approval:

• Baseline data collection using a web-based Participant Tracking System (PTS) (focus of this request for routine extension)

• Process study site visits

• Follow-up interview 18 months after baseline collection

• Follow-up interview 36 months after baseline collection

This submission requests extension of the approval for baseline data collection.

1. Respondent Universe and Sampling

DOL is selecting four sites from the universe of (POP – Green Jobs) and (HCHGT) grantees to participate in the evaluation based on their likely numbers of applicants, quality of implementation, early placement information, program service strategies, targeted industries, and appropriateness of implementing a random assignment design. These four sites will be the universe for the impact study; results will not be generalized to all (POP – Green Jobs) and (HCHGT) grantees.

All individuals who consent to participate in the study will be included in the data collection, and no sampling will be used. Sites will use their existing eligibility criteria to identify people who qualify to receive program services. No attempt will be made to draw inferences to any population other than the set of units that responded to the data collection effort.

Table B.1. Sample Size Requirements

Requirement		Value
Number of participating sites	4
Number of participating individuals	4,024 (average of 1,000 per site)
Ratio of treatment to control group members	Varied by site (e.g., 1:1, 2:1)
Anticipated response rates	100 percent (baseline data collection)
Anticipated number of respondents	4,024 (baseline data collection)

2. Analysis Methods and Degree of Accuracy

Statistical methods will not be used to select the sample of sites. Sites will be selected based on their likely numbers of applicants, quality of implementation, early placement information, program service strategies, targeted industries, and appropriateness of implementing a random assignment design. The universe of individuals admitted by the selected sites into their grant programs during the study intake period will be included in the study. Statistical analysis of the baseline data will consist solely of descriptive tabulations (to profile the population participating in the grant programs examined). Because there will be no sampling variation in the data for this purpose, the data analysis will have a high degree of accuracy.

3. Methods to Maximize Response Rates and Data Reliability

This package is requesting extension of the approval for the use of the informed consent form and the BIF to be administered to all eligible individuals at the selected sites and entered into the PTS as they go through an intake process.

Response Rates. The methods to maximize response for the consent form and BIF will be based on approaches that have been used successfully in many other random assignment studies to ensure that the study is clearly explained to both study participants and staff and that the forms are easy to understand and complete. Care has been taken to explain the study accurately and simply to potential participants. The approaches taken will be fully reviewed and approved by the IRB of Abt Associates (the lead research firm). The forms and procedures should minimize refusal rates and maximize voluntary participation in the program. Staff will be thoroughly trained on how to address study participants’ questions about the forms and their questions. Grantee staff will also be provided with a site-specific operational procedures manual prepared by the research team, contact information for members of the research team, and detailed information about the study.

Furthermore, the forms are designed to be easy to complete. They are written in clear and straightforward language, at the sixth-grade reading level, with closed response categories. The time required for participants to complete both forms is estimated to be 15 minutes, on average. This estimate was confirmed in the small pretest conducted on the informed consent form and the BIF. In addition, the forms will be available in Spanish to accommodate Spanish-speaking participants. Grantee staff will administer the forms orally to participants with low literacy.

Data Reliability. Both forms required at intake are unique to the current evaluation and will be used across all program sites. Using the same forms across all sites will ensure consistency in the collected data. The forms will have been reviewed extensively by project staff and staff at ETA and have been thoroughly tested in a pretest involving nine or fewer individuals from nonparticipating sites. Staff will receive training covering each item on the BIF to ensure staff understand each item and record the information accurately. In addition, each participating site will be provided with access to a web-based system, the PTS, for entering the information from the BIF. To ensure complete and accurate data capture, this platform will flag missing data or data outside a valid range.

4. Tests of Procedures or Methods

To assess the clarity of content and wording of the surveys, organization and formatting of the questionnaire, ease of administration, respondents’ burden time, and potential sources of error, Mathematica conducted a small pretest of the informed consent form and the BIF during the week of May 30-June 2, 2011. A total of 8 customers from a private nonprofit “Workforce Development Center” local to Mathematica’s Washington DC offices were each given $20 cash for participating in the 45-minute in-person interview. After the forms were completed by each pretest participant, project staff debriefed each participant using a standard debriefing protocol to determine if any words or questions were difficult to understand and answer. No major problems were uncovered in the pilot test. However, some minor formatting and wording changes were made as a result of the test. A memo detailing the pilot test results is included as Appendix E.

Completion of all forms took an average of 15 minutes.

5. Individuals Consulted on Statistical Methods

Consultations on the statistical methods used in this study have been undertaken to ensure the technical soundness of the research. The following individuals were consulted in preparing this submission to OMB:

Abt Associates

Dr. Stephen Bell (301) 634-1700

Ms. Karin Martinson (301) 634-1700

Mr. Jacob Klerman (617) 520-2613

Mathematica Policy Research

Ms. Anne Ciemnecki (609) 275-2323

Dr. Karen Needels (541) 753-0201

Dawn V. Nelson (202) 250-3532

As noted earlier, consultations on the research design, sample design, and data collection procedures are part of the study design phase of the evaluation. The purposes of these consultations are to ensure the technical soundness of the study and the relevance of its findings and to verify the importance, relevance, and accessibility of the information sought in the study. The following individuals were consulted in preparing this submission to OMB:

Peer Review Panel Members

1. Maureen Conway, maureen.conway@aspeninstitute.org

2. Harry J. Holzer, hjh4@georgetown.edu

3. Robert J. LaLonde, r-lalonde@uchicago.edu

4. Larry Orr, Larry.Orr.Consulting@gmail.com

5. Burt S. Barnow, barnow@gwu.edu

6. Mindy Feldbaum, mfeldbaum@aed.org

References

Bureau of Labor Statistics. An Occupational Analysis of Industries with Employment Gains: Occupational Employment Statistics (OES) Highlights. Washington: BLS, 2010.

Dohm, Arlene, and Lynn Shniper. “Occupational Employment Projections to 2016.” Monthly Labor Review, vol. 130, no. 11, 2007.