Green Jobs Site Visits Supporting Statement - Part B - 7-13- 2012

Part B: Collection of Information Involving Statistical Methods

The Employment and Training Administration (ETA) of the U.S. Department of Labor (DOL) is undertaking the evaluations of four Solicitations for Grant Applications (SGAs) that were issued under the American Recovery and Reinvestment Act (ARRA): Pathways Out of Poverty (POP), Energy Training Partnership (ETP), State Energy Sector Partnership (SESP), and Health Care and Other High Growth and Emerging Industries Training.

The overall aim of this evaluation is to determine the extent to which enrollees achieve increases in employment, earnings, and career advancement as a result of their participation in the training provided by Green Jobs and Health Care grantees and to identify promising best practices and strategies for replication. While the full evaluation involves several data collection efforts including surveys, this extension request is only for site visit data collection.

• Process Study Site Visits.

Impact Evaluation:

This research activity involves conducting two rounds of site visits to the grantees in the impact evaluation for the purpose of documenting the program environment, participant flow through random assignment and program services, the nature and content of the training provided, the control group environment and grantee perspectives on implementation challenges and intervention effects.

During the visits, site teams will interview key administrators and staff (including program partners and employers) using a semi-structured interview guide and focus groups held with participants (first round only).

1. Respondent Universe and Sampling

Impact study site visits

DOL is selecting four sites from the universe of POP-Green Jobs and HCHGT grantees to participate in the evaluation based on their likely numbers of applicants, quality of implementation, early placement information, program service strategies, targeted industries, and appropriateness of implementing a random assignment design.

The site visits will involve semi-structured interviews with administrators and staff, key program partners, and employers. All individuals who serve these roles at each site, are able and willing, and consent to be interviewed will be included in the qualitative data collection; therefore, no sampling will be used. For the focus groups, site staff assist us in identifying approximately eight students who would be available to participate in a focus group; this will be a convenience sample and will not be intended to represent the broader group of participants.

Table B.1. Sample Size Requirements for Impact Study Visits

Sample Size Requirements: Process Study Visits
Number of participating sites	4
Number of participating individuals	84 (average of 21 per site)
Anticipated response rates	100 percent
Anticipated number of respondents	84

2.Analysis Methods and Degree of Accuracy

Statistical methods will not be used to select the sample of sites. Sites will be selected based on their likely numbers of applicants, quality of implementation, early placement information, program service strategies, targeted industries, and appropriateness of implementing a random assignment design.

The universe of individuals admitted by the selected sites into their grant programs during the study intake period will be included in the study. Statistical analysis of the baseline data will consist solely of descriptive tabulations (to profile the population participating in the grant programs examined). Because there will be no sampling variation in the data for this purpose, the data analysis will have a high degree of accuracy.

No statistical analysis will be performed on the qualitative data collected during the site visits.

3. Methods to Maximize Response Rates and Data Reliability

The methods to maximize response rates and data reliability are discussed below for the site visit protocols.

Response rates .The strategy to collect qualitative data during site visits will ensure that response rates are high and that the data are reliable. Site visit team members will begin working with site staff well in advance of each visit to ensure that the timing of the visit is convenient. Because the visits will involve several interviews and activities each day, there will be flexibility in the scheduling of specific interviews and activities to accommodate the particular needs of respondents and site operations.

Data Reliability. Strategies to ensure that the data are reliable and as nearly complete as possible include flexibility in scheduling of visits and the assurance given to respondents of privacy of the information that they provide. Furthermore, the neutral tone of the questions in the data collection protocols and the absence of sensitive questions, along with the training of the site visitors, will facilitate a high degree of accuracy in the data. In addition, shortly after each site visit, the site visit team members will synthesize the data from each interview, observation, and group discussion to the requirements of a structured write-up guide. Because most questions will be asked of more than one respondent during a visit, the analysis will allow for the triangulation of the data so that discrepancies among different respondents can be interpreted.

4. Tests of Procedures or Methods

To ensure that the site visit protocols are used effectively as field guides and that they yield comprehensive and comparable data across the study sites, senior project team members will conduct internal pilot tests of the interview guides with other research team members. The purposes of these tests are to ensure that the site visit protocols, which will guide researchers as they collect data on site, include appropriate probes that assist researchers in delving deeply into topics of interest and that the protocols do not omit relevant topics of inquiry. Furthermore, testing of the protocols can enable the staff leading this task to assess that the site visit agenda that the site visit team develops—including how data collection activities should generally be structured during each site visit—is practical given the amount of data that is to be collected and the amount of time allotted for each data collection activity. Adjustments to the site visit guides will be made as necessary.

5. Individuals Consulted on Statistical Methods

As noted earlier, consultations on the research design, sample design, and data collection procedures are part of the impact study, design phase of the evaluation. The purposes of these consultations are to ensure the technical soundness of the study and the relevance of its findings and to verify the importance, relevance, and accessibility of the information sought in the study.

Peer Review Panel Members of the Impact Study

1. Maureen Conway, maureen.conway@aspeninstitute.org

2. Harry J. Holzer, hjh4@georgetown.edu

3. Robert J. LaLonde, r-lalonde@uchicago.edu

4. Larry Orr, Larry.Orr.Consulting@gmail.com

5. Burt S. Barnow, barnow@gwu.edu

6. Mindy Feldbaum, mfeldbaum@aed.org

References

Bureau of Labor Statistics. An Occupational Analysis of Industries with Employment Gains: Occupational Employment Statistics (OES) Highlights.Washington: BLS, 2010.

Dohm, Arlene, and Lynn Shniper. “Occupational Employment Projections to 2016.”Monthly Labor Review, vol. 130, no. 11

pg. 1