Part A PTC 3_3_17

Supporting Statement for the

Paperwork Reduction Act of 1995

Part A. Justification

ICR # 201609-1230-002

Demonstration and Evaluation of Community College Interventions for Youth and Young Adults with Disabilities

December 22, 2016

U.S. Department of Labor

200 Constitution Ave., NW

Washington, DC 20210

Table of Contents

1. JUSTIFICATION

A.1 Circumstances Necessitating the Information Collection 1

A.2 Purpose and Use of the Information 2

A.3 Use of Information Technology 2

A.4 Identification of Duplication of Information Collection Efforts 4

A.5 Impacts on Small Businesses or Other Small Entities 4

A.6 Consequence to Federal Program or Policy if Collection is not Conducted 4

A.7 Special Data Collection Circumstances 4

A.8 Federal Register Notice 5

A.9 Payments/Gifts to Respondents 5

A.10 Assurance of Privacy 6

A.11 Justification of Questions of a Sensitive Nature 7

A.12 Estimate of Annualized Burden Hours and Costs 7

A.13 Estimates of Annualized Respondent Capital and Maintenance Costs 8

A.14 Estimates of Annualized Cost to the Government 8

A.15 Changes in Hour Burden 9

A.16 Plans for Tabulation and Publication 9

A.17 Approval to Not Display the Expiration Date 10

A.18 Exceptions to the Certification Statement 11

OMB SUPPORTING STATEMENT PRA PART A

The U.S. Department of Labor (DOL) is requesting clearance for information collection to conduct (1) in-depth interviews with grantee staff, other community college administrators and staff, students, and grantee partner organizations, (2) focus groups with faculty, and (3) surveys of community college students. These data collections are essential elements of the evaluation of the Pathways to Careers: Community Colleges for Youth and Young Adults with Disabilities Demonstration Project.

1. Circumstances Necessitating the Information Collection

In June 2014, the Office of Disability Employment Policy (ODEP) announced the availability of funds for two cooperative agreements, to conduct pilot projects to research, develop, test and, in coordination with DOL, evaluate innovative systems models for providing inclusive integrated education and career development services to youth and young adults with disabilities. In September of 2014, ODEP competitively selected Onondaga Community College (OCC) and Pellissippi State Community College (PSCC) for the Pathways to Careers (PTC) Grant Program. The program models are designed to (1) increase credential and job attainment of students with disabilities, (2) increase their job placement, and (3) decrease the wage earning differential between students with and without disabilities, and between students with different types of disabilities. Appendix A-1 presents a logic model for the PTC program and a theory of change diagram for the two grantee programs.

The grantees are expected to design approaches that work to shift practice and policy across the institution. This involves transforming the entire college’s approach for providing services, as opposed to a single division, and enlisting support from and engagement of administrators, deans, department chairs, faculty, student services, and other divisions that have a role in ensuring students’ success. It is expected that grantees will leverage their partnerships and relationships with national affiliates, association members or business organizations, and a variety of other entities including the public workforce system. Grantees are also required to capture and use data to assess and manage their program performance, and to participate in an independent evaluation to be conducted by the Department’s Chief Evaluation Office (CEO) and ODEP. Division G, Title I, Section 107 of Public Law 114-113, the “Consolidated Appropriations Act, 2016” authorizes the Secretary of Labor to reserve not more than 0.75 percent from specific budget accounts for transfer to and use by the Office of the Chief Evaluation Officer for departmental program evaluation.

The evaluation consists of an implementation study and a descriptive outcomes study. The implementation study will document the institutional change at the two colleges; assess the fidelity of the implemented programs to the intended program model; assess the models for replicability and scalability; and determine the extent to which the grantees used ODEP-recommended principles (e.g., Universal Design for Learning (UDL) principles, Guideposts for Success) in the development and operation of their programs. The outcomes study will document PTC participant outcomes and examine the extent to which the grantees meet target goals. DOL is requesting clearance to collect the information needed to inform the evaluation.

2. Purpose and Use of the Information

The information collection is needed to determine whether academic and employment outcomes for students with disabilities can be improved through enhanced community college programming. To do so, the evaluation will document additions or changes to programming that were made and the degree to which it was implemented. Two sets of research questions are organized under two corresponding analytical strands: program implementation and student outcomes.

The implementation strand focuses on assessing the progress of the grantees towards full operation and institutionalization of their respective PTC program. The outcomes strand focuses on assessing the extent to which the grantees accomplished the goals and objectives of their respective PTC program. Each of these research strands are further organized into three analytical dimensions that focus on specific facets of the PTC program. The three dimensions of the implementation study are fidelity, incorporation, and operation. The three dimensions of the outcomes study are satisfaction, academic, and employment.

The data collection involves in-depth interviews of community college administrators, staff, PTC student participants, and partner organizations; focus groups of faculty in the PTC program; web-based surveys with telephone follow-up for student participants, and analysis of participant-level college administrative data. The interviews with college administrators and staff will provide information on how the program interventions were implemented, challenges to implementation, and promising practices. Interviews will focus on program design, strengths and weaknesses of infrastructure and logistics, clarity of communication, effectiveness of enrollment, intake, and orientation, impressions of program delivery, and access to support services. The administrative data will provide information as to when participation occurred, the nature and frequency of services and supports, coursework, and grades.

The implementation analysis will document challenges to implement grant requirements, document program components and system-level characteristics, track changes in program and system-level characteristics that may affect outcomes, and conduct separate case studies of each grantee, conduct a cross-grantee analysis.

The outcome analysis is not intended to, nor could it determine program impact because of the small number of participants and lack of an acceptable comparison group. Rather, the outcome analysis is intended to document the achievements of PTC participants and provide information that can inform a feasibility assessment for a future demonstration of the PTC program model. The analysis will document PTC participant outcomes, Specifically the short-term outcomes include:

• Rates of persistence,

• Credential attainment,

• Graduation, and employment and

• Average earnings using the administrative data and participant interview and survey data.

Long-term outcomes include: To the extent possible, the evaluator will also examine the trend in persistence of students with disabilities at each grantee college from about five years prior to PTC through the years with PTC to determine if the trend changes after the introduction of the PTC program. A bump up in the trend would be consistent with a potential effect, informing an evaluability assessment of a potential change in persistence rates.

3. Use of Information Technology

The evaluation contractor will develop and conduct an electronic survey of PTC participants using web-based technology. DOL anticipates that the online survey will be less burdensome, as it will offer easy access and submission, while also allowing participants to complete the survey at a time convenient to them and at their own pace. An online survey has the additional advantages of reducing the potential for errors by checking for logical consistency across answers, accepting only valid responses and enforcing automated skip patterns. PTC participants will be provided the URL and a unique PIN to access the survey. To increase the response rates, automated weekly email reminders will be sent to all non-respondents. After sending out two reminders, to further increase response rates, interviewers will contact all non-respondents and invite them to complete the survey by telephone.

The interviews with grant staff, students, community college personnel, and partner organizations will be semi-structured in-person interviews at the community college. Similarly, the faculty focus groups will be conducted in-person at the college as well. The contractor will take notes during the interviews and focus groups and record them, subject to approval of each respondent. The recording of interviews reduces the burden of needing to contact respondents after the interview to review accuracy of the notes. Some interviews will be conducted by telephone (in October 2017, and 2018) to minimize data collection costs associated with travel to the colleges.

4. Identification of Duplication of Information Collection Efforts

The proposed data collection is designed to meet the needs of the PTC evaluation at the two community colleges only. DOL is not aware of any previous or planned effort to collect similar information concerning PTC program implementation at the colleges or concerning participant outcomes. The data collection is needed to gather the information necessary to address the research questions of the evaluation. The contractor is working with the grantees (through data sharing agreements) to access community college administrative records and data about the PTC program and participants.

While these administrative data are important and useful data sources, they do not provide information about the implementation of the program and detailed outcomes for the PTC participants. Conducting the surveys, focus groups, and interviews will capture this information.

5. Impacts on Small Businesses or Other Small Entities

It is possible that a few employers (less than 10) will be identified for in-depth interviews because they are partners to the grantees on the PTC program. It is also possible that a partner employer may be classified as small. Because the number of small employers is expected to be very small and the length of the interviews is expected to be no more than 60 minutes, the impact on small business is expected to be minimal.

6. Consequence to Federal program or policy activity if collection is not conducted

If the interviews and surveys are not conducted, detailed information needed to conduct the evaluation and answer the research questions otherwise would not be available.

7. Special Data Collection Circumstances

There are no special circumstances that would require this information collection to be conducted in any manner other than that described above. The data collection will be performed in a manner consistent with federal guidelines.

8. Federal Register Notice

The 60-day notice to solicit public comments was published in the Federal Register on October 6, 2015 (80 FR 60407). No public comments were submitted during the notice period.

9. Payments/Gifts to Respondents

Both grantees will enroll four cohorts of student participants: Cohort 1 enrolled in fall 2015, Cohort 2 enrolled in spring 2016, Cohort 3 enrolled in fall 2016, and Cohort 4 enrolled in spring 2017. All participants will be invited to complete the longitudinal survey every 6 months, starting in the spring of 2017 through the fall of 2018. Hence, up to four waves of survey data will be collected for cohorts 1, 2, and 3 and three waves for cohort 4. The average administration time to complete each longitudinal survey is 30 minutes.

Table A-1 presents estimated numbers of completed surveys, assuming a 90 percent response rate at each wave. OCC enrolled 38 participants PSCC enrolled 74 participants into the PTC program in fall 2015 (and more participants in the spring and fall of 2016 as indicated in the table). The number of participants starting in spring 2017 at each college is an estimate. The total number of participants is expected to be 213 and the estimated total number of completed surveys is 647.

Table A-1. Estimated number of completed surveys by cohort

Cohort		Number of new PTC participants at OCC	Number of new PTC participants at PSCC	Total number of new PTC participants	Survey dates
					Spring 2017	Fall 2017	Spring 2018	Fall 2018	Total
1	Fall 2015	38	74	112	101	91	82	74	348
2	Spring 2016	6	4	10	9	8	7	6	30
3	Fall 2016	38	38	76	68	61	55	49	233
4	Spring 2017	10	5	15	--	13	12	11	36
Total		92	121	213	178	173	156	140	647

Note: Estimated number of completed interviews assumes 90% response at each wave.

Participants will receive an incentive of $25 for each completed survey. The incentive is to reduce non-response bias due to participant attrition. In addition, an “early bird” bonus of $5 will be used to encourage participants to complete the survey within the first three weeks of administration. The total amount of incentives is estimated to be $16,175. This amount does not include the “early bird” bonus for responding within the first three weeks of a survey.

PTC participants will be invited to participate in cross-sectional or longitudinal in-depth qualitative interviews. A cross-section of PTC participants from each of first year participants (cohorts 1 and 2) and from the second year participants (cohorts 3 and 4) at each of the grantee colleges in years 2, 3 and 4 of the evaluation will be invited. Similarly, first-year PTC participants and second-year PTC participants at each of the grantee colleges will be invited for longitudinal interviews (to see how their experiences change over time). Table A-2 presents the estimated number of PTC participants to be interviewed by type of interview and year as well as by college and cohorts. A total of 45 participants will complete the cross-sectional interviews and a separate group of 28 participants will complete the longitudinal interviews.

An incentive of $40 will be provided to each participant who completes an interview. The total amount of interview incentives is estimated to be $5,240.

Table A-2. PTC Participant Interviews

College	Cohort	Cross-sectional sample						Longitudinal sample*						TOTAL
		Spring 2016	Spring 2017	Fall 2017	Spring 2018	Fall 2018	Total	Spring 2016	Spring 2017	Fall 2017	Spring 2018	Fall 2018	Total
OCC	C1 & C2	3	5	--	5	--	13	4	3	2	--	--	9	22
	C3 & C4		--	5	--	5	10		10	9	8	7	34	44
PSCC	C1 & C2	2	5	--	5	--	12	4	3	2	--	--	9	21
	C3 & C4		--	5	--	5	10		10	9	8	7	34	44
TOTAL		5	10	10	10	10	45	8	26	22	15	15	86	131

^* Estimated number of completed interviews assumes attrition of one participant per year for each sample.

10. Assurance of Privacy

There are no assurances of confidentiality to respondents.

Respondents will be informed that their responses will be kept private to the extent permitted by law. All participants will be informed that the information collected will be reported in aggregate form only and no reported information will identify any individuals. Terms of the DOL contract authorizing data collection require the contractor to maintain the privacy of all information collected, unless written permission is provided by the respondent. The evaluation contractor will protect personal information in accordance with Federal and state laws and contractual requirements.

The security procedures implemented by the evaluation team covers all aspects of data handling for hard and electronic data. When not in use, all completed hardcopy documents will be stored in locked file cabinets or locked storage rooms. Unless otherwise required by DOL, these documents will be destroyed when no longer needed for the project. Evaluation team members working with the collected data will have previously undergone background checks that may include filling out an SF-85 or SF-85P form, authorizing credit checks, or being fingerprinted.

A two-person team will conduct the interviews, with one person conducting the interview and the other taking notes. Recordings will also be made of the interviews, subject to respondent approval. Interviewers will ensure a private meeting space. Written materials and analyses from the interviews to be used as part of study reports will be prepared in such a way as to protect the identity of individuals. Only the site visit study team staff present at the interviews, the principal investigator, project director, and selected staff helping transcribe the recordings will have access to the notes. Notes will be securely stored in protected electronic files or locked cabinets. Only the staff members present at the interviews or transcribing the recordings will have access to the recordings. All site visit staff, project leadership, and transcribing staff will be asked to sign privacy agreements before the interviews are conducted or before working with the data.

Access to the online surveys will require a unique PIN provided by the contractor to the respondent. Survey data collection will use secure sockets layer encryption technology to ensure that information is secure and protected.

11. Justification of Questions of a Sensitive Nature

There are no questions in the interviews or surveys that are of a sensitive nature.

12. Estimate of Annualized Burden Hours and Costs

The total participant burden for information collected is shown in table A-3. The frequency of collection varies with the respondent types. Table A-4 presents the annualized values for burden hours and costs.

The total burden cost of collecting this information is $17,031 for 3 years or approx. $5,677 annually. This cost represents the sum across the interviews, surveys, and focus groups when the average hourly rate for each respondent category is multiplied by the corresponding number of hours. The average hourly wage rates were obtained using the latest Occupational Employment Statistics data on wages, adjusted for inflation.¹

Table A-3 Total Estimated Respondent Hour and Cost Burden over 36 months

Data collection activity			Total Number of Respondents	No. of Responses per Respondent	Total Number of Responses	Average Burden per Response (in hrs.)	Total Burden Hours	Average Hourly Wage Rate	Total Burden Costs
Participant surveys	C1		112	4	448	0.5	224	$10.15	$2,274
	C2		10	4	40	0.5	20	$10.15	$203
	C3		76	4	304	0.5	152	$10.15	$1,543
	C4		15	3	45	0.5	23	$10.15	$228
In-depth student interviews	Cross-sectional sample	C1 & C2	25	3	75	1	75	$10.15	$761
		C3 & C4	20	3	60	1	60	$10.15	$609
	Longitudinal sample	C1 & C2	8	4	32	1	32	$10.15	$325
		C3 & C4	20	4	80	1	80	$10.15	$812
In-depth interviews	Grant leaders		2	6	12	2	24	$48.99	$1,176
	Grant partners		12	6	72	1	72	$45.06	$3,244
	Other college staff		10	6	60	1	60	$48.99	$2,939
Pre-site visit checklist			2	6	12	0.5	6	$48.99	$294
Faculty focus groups			16	3	48	1.5	72	$36.43	$2,623
TOTAL			328	--	1,288	--	900	--	$17,031

Table A-4 Estimated Annualized Respondent Hour and Cost Burden

Data collection activity			Number of Respondents	No. of Responses per Respondent	Total Number of Responses	Average Burden per Response (in hrs.)	Total Annual Burden Hours	Average Hourly Wage Rate	Total Annual Burden Costs
Participant surveys	C1		112	1.333	149	0.5	75	$10.15	$758
	C2		10	1.333	13	0.5	7	$10.15	$68
	C3		76	1.333	101	0.5	51	$10.15	$514
	C4		15	1	15	0.5	8	$10.15	$76
In-depth student interviews	Cross-sectional sample	C1 & C2	25	1	25	1	25	$10.15	$254
		C3 & C4	20	1	20	1	20	$10.15	$203
	Longitudinal sample	C1 & C2	8	1.333	11	1	11	$10.15	$108
		C3 & C4	20	1.333	27	1	27	$10.15	$271
In-depth interviews	Grant leaders		2	2	4	2	8	$48.99	$392
	Grant partners		12	2	24	1	24	$45.06	$1,081
	Other college staff		10	2	20	1	20	$48.99	$980
Pre-site visit checklist			2	2	4	0.5	2	$48.99	$98
Faculty focus groups			16	1	16	1.5	24	$36.43	$874
TOTAL			328	--	429	--	300		$5,677

13. Estimates of Annualized Respondent Capital and Maintenance Costs

There will be no start-up or ongoing financial costs incurred by respondents and record-keepers for this information collection. All administrative data will be captured within existing infrastructures and systems.

14. Estimates of Annualized Cost to the Government

The estimated cost to the Federal Government for this data collection is covered under the contract to conduct the demonstration and evaluation of the PTC program.

The period of performance for the contract supporting the study runs from August 2014 through August 2019. The total estimated Federal cost of the PTC evaluation is $1,200,800 and the annualized estimated Federal cost, averaged over the five years of the study, is $240,160.

This estimate of Federal costs is a combination of (1) direct costs for information collection and (2) Salary associated with Federal oversight and project management.

1. The current budget and contract anticipates costs of $1,200,800 for all costs associated with the evaluation of the PTC grant program. The estimated cost for evaluation design and development data collection instruments is $308,808. The estimated cost for conducting up to four waves of surveys with PTC participants is $238,325. The estimated cost for site visits, focus groups, interviews, and the gathering of participant and program data from the grantees is $323,710. The estimated cost for analysis and reporting is $330,037.

2. Federal activities resulting from approval of information collection would include time spent on communication and correspondence with the contractor, oversight activities for administering the underlying contract, documenting contract activities and monitoring the resulting data collection and deliverables. This work is expected to be done by a GS-15 Senior Evaluation specialist, and project management tasks are expected to consume 1/10^th of the specialist’s annual hours.

15. Changes in Hour Burden

This is a new data collection.

16. Plans for Tabulation and Publication

We will calculate rates of persistence of PTC participants semester to semester and year to year, and rates of credential attainment, including certifications and degrees and entering employment after credential attainment. Rates would be calculated for each semester to support a time-trend analysis to demonstrate the progress made over time. The analysis will also present estimates of the probability of PTC participants persisting, achieving one or more credentials, and obtaining employment, controlling for student characteristics, program of study, and services received.

In addition, we will calculate the rates of employment, continued employment, enrollment in four-year post-secondary education, and completion of additional education or training. We will also report on average wage rates for those employed and compare them to the state average wage rate and the state minimum wage to determine if, on average, PTC participants earn competitive wages. In addition, these analyses will examine how outcomes differ across PTC participants, considering such factors as the mix of interventions received, prior educational attainment, characteristics of the individuals, and prior employment experience.

Notes from the interviews will be analyzed to discern patterns in the responses and produce summary tables. The analysis may be performed using a qualitative software package to more efficiently discern patterns and common themes.² Information from similar types of respondents will be aggregated by topic area.

The contractor will prepare a comprehensive report on the findings in an interim report in 2017 and in a final report in 2019, based on analysis of data from the interviews, surveys and administrative data to address the research questions. The reports will contain a detailed description of the methodology and summary statistics on the data collection. Findings may be used in the report or briefing to Congress and the final report will be published at the DOL website. The contractor also will produce a public-use data set with documentation. All directly identifying information will be removed. All indirect identifying information will be recoded and combined to de-identify the dataset. See Table A-5 for a timeline for the demonstration, evaluation and data collection activities.

Table A-5 Timeline for demonstration, evaluation and data collection activities

Activity	Estimated Date
Submit PRA Clearance Package	October 8, 2015
Intake of first PTC cohort	Fall 2015, Spring 2016
First cohort signs consent forms	Fall 2015, Spring 2016
Colleges begin compiling data on cohorts	Fall 2015
Site visit: observation only	October, December 2015
Colleges transfer first cohort records to Westat	January 2016, 2017, 2018, 2019 May 2016, 2017, 2018, 2019
Obtain OMB approval	January 2017
Site visit: interviews, focus group, observation	Spring 2016, 2017, 2018
Conduct student surveys	Spring and Fall 2017 and 2018,
Participant interviews	Spring 2016, Spring and Fall 2017 and 2018
Intake of second PTC cohort	Fall 2016, Spring 2017
Second cohort signs consent forms	Fall 2016, Spring 2017
Telephone interviews (replace site visit)	Fall 2017 and 2018
Colleges transfer second cohort records to Westat	January 2017, 2018, 2019 June 2017, 2018
Draft interim report	September 2017
Final interim report	October 2017
Draft final report	May 2019
Final report	July 2019

17. Approval to Not Display the Expiration Date

The collection of interview and survey data will show the OMB expiration date on any written instrumentation. The following Public Burden Statement will appear:

A Federal agency may not conduct or sponsor, and a person is not required to respond to this collection of information, unless it displays a currently valid OMB control number. Your obligation to reply to this survey is voluntary. The public burden for this survey is estimated to be XX minutes per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the necessary data, and completing and reviewing the collection of information. Send comments concerning this burden estimate or any other aspect of this collection of information to the U.S. Department of Labor, Chief Evaluation Office, Room 2218, Constitution Ave., Washington, DC 20210.

18. Exceptions to the Certification Statement

There are no exceptions to the certification statement for the interviews, focus groups or surveys.

APPENDIX A-1 Logic Model and Theory of Change

Figure 1 – Logic model for the Pathways to Careers Programs

Figure 2 Theory of change for Pathways to Careers Programs

1 Bureau of Labor Statistics, “Occupational Employment Statistics—May 2014 National Occupational Employment and Wage Estimates” http://www.bls.gov/oes/current/oes_nat.htm#00-0000

2 The analysis of the qualitative data will be informed by principles of Grounded Theory modified to fit the scope and purpose of the study (Glaser & Strauss, 1994). Given the interview questions are focused on particular issues of program features, operations and implementation, and as confirmed by the pre-test, the themes are focused and easy to discern.

1