GTW_PRA Part B_Supporting Statement_04-16-15 clean

Supporting Statement for the

Paperwork Reduction Act of 1995

Part B. Collections of

Information Employing Statistical Methods

Evaluation of Getting to Work: A Training Curriculum for HIV/AIDS Service Providers and Housing Providers

U.S. Department of Labor

200 Constitution Ave., NW

Washington, DC 20210

TABLE OF CONTENTS

Page

Part B. Collections of 1

Information Employing Statistical Methods 1

TABLE OF CONTENTS i

Page i

PART B. SUBMISSION FOR COLLECTIONS OF 1

INFORMATION EMPLOYING STATISTICAL METHODS 1

1. Respondent Universe and Sampling Methods 1

1.1 Web-based Survey 1

1.2 Focus Groups 1

2. Information Collection Procedures 3

2.1 Statistical Methodology for Stratification and 3

2.2 Sample Selection 3

2.5 Minimal Effect-Sizes Required 4

2.6 Software Packages or Formulas to Be Used 4

2.7 Unusual Problems Requiring Specialized Sampling Procedures 4

2.8 Use of Periodic Data Collection Cycles to Reduce Burden 4

3. Maximizing Response Rates and Addressing Nonresponse 4

3.1 Plans to Maximize Response Rates and Address Nonresponse 5

3.2 Elimination of Response Bias 5

3.3 Accuracy and Reliability of Information 5

3.4 Justification for Data Not Systematically Collected for Entire Respondent Population 6

4. Tests of Procedures or Methods 6

5. Statistical Consultants 6

PART B. SUBMISSION FOR COLLECTIONS OF

INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

The U.S. Department of Labor (DOL) requests clearance for IMPAQ International, LLC (IMPAQ) to conduct two principal research activities: 1) a survey of managers and counselors who work for housing providers funded by the U.S. Department of Housing and Urban Development (HUD), and 2) focus groups with staff at HUD-funded housing providers. This data collection will be used to evaluate Getting to Work: a Training Curriculum for HIV/AIDS Service Providers and Housing Providers (Getting to Work or GTW). Getting to Work is a 3-module online training produced by DOL’s Office of Disability Employment Policy (ODEP) and HUD. IMPAQ is conducting the evaluation of the training on behalf of DOL.

The purpose of the evaluation is twofold: 1) Build a “demographic profile” of GTW participants. 2) Determine whether, post-training, individuals report changes in thinking or practice. Specifically, have there been changes in housing providers’ thinking or practice related to employment and training for people living with or at risk of HIV/AIDS?

1.1 Web-based Survey

The web-based survey will be sent to the universe of individuals who complete GTW. At this time, based on HUD plans to require the training for the next round of Housing Opportunities for Persons with AIDS (HOPWA) grantees, we anticipate that this group will include 1,000 individuals¹. No sampling will be involved. We expect an 80 percent response rate for the survey, based on pre-survey and follow-up activities coordinated with the U.S. Department of Labor, as well as the IMPAQ team’s experience on related efforts. Examples of similar efforts that yielded such a response rate include the Job Corps National Survey Data Collection Project and Project GATE, both of which were conducted for the DOL’s Employment and Training Administration.

1.2 Focus Groups

The IMPAQ team will conduct focus groups with staff at up to eight housing provider organizations. The purpose of the focus group is to determine whether there is evidence of organizational level change related to GTW. The selection of focus group sites will be purposive, based on the following criteria:

• Organizations must have completed the training at least four months prior to the focus group.

• Organizations must have staff members in both management and counselor positions who completed the training.

• A variety of organizations is preferable. Variations include the following:

  • Organization size in terms of number of customers/clients served

  • Urbanicity of population served

  • Services offered by organizations (i.e., organizations that offer employment services versus those that do not)

• Organizations are in the vicinity of the researchers (mid-Atlantic)

One possible breakdown of focus groups is illustrated in Exhibit 2. It is expected that approximately nine individuals will participate in any focus group. Informed consent for focus group participants can be found in Appendix B.

Exhibit 2: Possible Focus Group Sample

# HUD-Funded, Mid-Atlantic grantees where both counselor and management staff completed training	STRATA			Total Focus Groups (# Persons)
	# Focus Groups / Organization Size*	# Focus Groups per Urban / Rural Area	# Focus Groups per Services Offered
50	4 Large	2 Urban	1 w/ Employment Services	8 (72 persons)
			1 w/o Employment Services
		2 Rural	1 w/ Employment Services
			1 w/o Employment Services
	4 Small	2 Urban	1 w/ Employment Services
			1 w/o Employment Services
		2 Rural	1 w/ Employment Services
			1 w/o Employment Services

* Definitions for large versus small organization will be relative and determined based on the parameters of organizations that complete the training.

2. Information Collection Procedures

2.1 Statistical Methodology for Stratification and

2.2 Sample Selection

The web-based survey is being administered to the universe of individuals who complete the training; i.e., there is no sampling. Within this frame, strata for focus group sample selection were determined and prioritized to ensure that the focus groups provide information from providers with a range of characteristics.

GTW is expected to be completed by approximately 1,000 individuals when the survey is disseminated. All training participants will be surveyed; there is no sampling. The outcomes of GTW are not intended to be generalized to a wider population, but rather to inform ongoing work with the respondent universe (specifically, ongoing training on related topics).

All data will be based on a 100 percent sample of the inference population. In all reports and other publications and statements resulting from this work, no attempt will be made to draw inferences to any population other than the set of units that responded to the data collection effort.

2.3 Procedures for Estimating Variance

N/A

2.4 Analytical Approach

Researchers will use descriptive statistics to characterize the survey respondents’ demographics, and the range of responses. Researchers will also compare the responses of different subgroups of participants (such as experienced versus inexperienced counselors) to see whether they report change differently.

Demographic Profile/Review

IMPAQ will document the demographic variables of users who complete the training and generate a demographic profile. The demographic profile will include the following information:

• Provider type (e.g., nonprofit pubic, nonprofit private, etc.)

• Provider funding sources

• Service recipient characteristics

• Number of clients served annually

• Respondent role in organization

• Respondent years’ experience working with people living with or at risk of HIV/AIDS

• Whether or not provider offers employment/training services

• Types of provider’s employment/training partnerships (if any)

Response Comparison

Cross-tabs will allow researchers to compare the outcomes (item responses) of different training participant subgroups. For example, the survey presents respondents with a list of activities and asks them about the frequency with which they conduct these activities now, compared to before completing GTW. The response options are “More than Before,” “About the Same,” or “Less than Before.” Researchers will use a Chi-square analysis to compare the response distributions of housing counselors versus housing managers. The same analysis procedure will be applied to other comparisons such as experienced versus inexperienced staff, and staff from small versus large organizations.

Qualitative Analyses

Researchers will use standard content analysis to analyze open-ended questions, distilling common themes and identifying areas where variance or disagreement might exist. IMPAQ will use NVivo software to manage and assist in the analysis of both qualitative survey data and focus group data.

2.5 Minimal Effect-Sizes Required

N/A

2.6 Software Packages or Formulas to Be Used

In addition to using NVivo software for the qualitative analyses, IMPAQ researchers will use SAS or Stata software for the quantitative analyses.

2.7 Unusual Problems Requiring Specialized Sampling Procedures

There are no unusual problems requiring specialized sampling procedures.

2.8 Use of Periodic Data Collection Cycles to Reduce Burden

The data collection efforts for this research occur at a single point-in-time.

3. Maximizing Response Rates and Addressing Nonresponse

3.1 Plans to Maximize Response Rates and Address Nonresponse

IMPAQ researchers expect an 80 percent response rate to the web-based survey. This rate is based on their experience conducting the Job Corps National Survey Data Collection Project, as well as the Growing America through Entrepreneurship (GATE) research for DOL’s Employment and Training Administration. Additionally, extensive efforts have been planned to achieve an 80 percent response rate by sending advance notices and follow-up reminders via e-mail.

Specifically, each training participant will receive an advance notice one-week before he or she receives a survey. Surveys will be automatically sent to participants three-four months after they complete the training (exact timeframe will be determined prior to any data collection). After participants receive the survey, they will receive weekly reminders if they do not it a completed survey within a four week period. Examples of the advance notice and follow-up reminders are provided in Appendix C.

Researchers are surveying the universe of individuals who complete GTW. Still, there will be value in understanding whether or not there are systematic differences in the characteristics of those who completed the training and the follow-up survey, versus those who completed the training, but did not complete the follow-up survey.

Registration in GTW asks participants to provide basic information about themselves and their organization. Because the follow-up survey will be linked to training information, researchers will be able to compare information of survey respondents and non-respondents. For example, is there any significant difference between these groups in terms of gender? Experience? Size of organization? These determinations will be made using Chi-square tests for independence. If a significant difference is found, researchers can better contextualize survey findings, and can suggest areas/populations for follow-up by training developers. Focus group respondents will be provided with a $30.00 gift card as an incentive for participation.

3.2 Elimination of Response Bias

While the agency anticipates a high level of participation (based on the previously cited work), the value of a non-response bias analysis may be limited for this particular project. As previously indicated, all data will be based on a 100 percent sample of the inference population. In all reports and other publications and statements resulting from this work, no attempt will be made to draw inferences to any population other than the set of units that responded to the data collection effort.

3.3 Accuracy and Reliability of Information

Since basic descriptive information on individual respondents and their organizations will be available in the training registration data, researchers will crosswalk survey data with registration data to validate respondent characteristics. Otherwise, the data collected is perception data that will be taken at face value.

3.4 Justification for Data Not Systematically Collected for Entire Respondent Population

N/A

4. Tests of Procedures or Methods

Research staff and programmers will thoroughly test the web-based survey. A testing protocol will be developed along with various responses to ensure that the instrument is performing correctly for all types of respondents. Testing results will be used to evaluate whether question wording and response choices are accurate when translated from paper to web-based administration, whether instructions are clear, and whether skip patterns are functioning properly. Using a convenience sample of nine HOPWA grantees, testing will ensure that any errors are corrected prior to full survey administration. The test survey will be administered by IMPAQ over the Internet. The nine respondents will be instructed to log in to a specific website and complete the survey. After each respondent has completed the survey, a researcher will conduct a telephone interview with him or her. The telephone interview will identify questions that are poorly understood, terms that are ambiguous in meaning, and difficult transitions between topics.

5. Statistical Consultants

To ensure that the best decisions were made regarding the statistical aspects of the design, the following individuals outside of ODEP contributed to, reviewed, and/or approved the design:

Name	Organization	Phone Number	E-mail Address
Futoshi Yumoto	IMPAQ	443.718.4355	fyumoto@impaqint.com
Anne Chamberlain		443.539.1391	epyamagata@impaqint.com
Alex Tenaglio		443.283.1645	atenaglio@impaqint.com

1 The Department of Housing and Urban Development’s HOPWA Program will require its 137 formula grantees and 780 project sponsors to take the curriculum, and 137 competitive grantees and 141 project sponsors will have the option of taking the training as well. This will amount to a minimum of 1,000 users.