The Social Security Administration (SSA) is requesting re-clearance for the collection of data needed to implement and evaluate the Youth Transition Demonstration (YTD) projects. YTD projects help young people with disabilities make the transition from school to work. By waiving certain disability program rules (SSA YTD Waivers) and offering services to either youth who are receiving disability benefits or at risk of receiving them, these projects encourage youth to work and continue their education. At the start of this project in 2007, SSA established a random assignment evaluation of YTD projects in six sites across the country. The evaluation produces empirical evidence on the impacts of the waivers and project services on not only educational attainment, employment, earnings, and receipt of benefits by youth with disabilities but also on the Social Security Trust Fund and federal income tax revenues.
Given the importance of estimating YTD impacts as accurately as possible, the evaluation uses rigorous analytic methods based on the random assignment of youth to a treatment or control group. SSA concluded several data collection efforts for this program, including: (1) baseline interviews and informed consent with youth and their parents or guardians prior to random assignment; (2) follow-up interviews at 12 months after random assignment with youth or their parents or guardians; (3) interviews and roundtable discussions with local program administrators, program supervisors, and service delivery staff; (4) focus groups of youth, their parents, and service providers; and (5) in-depth interviews with youth and their parents or guardians within three months of completing the 12 month follow-up interview. At this time, SSA is conducting 36-month follow-up interviews with youth or their parents or guardians, which will help us complete our study. We plan to close this information collection request upon completion of this study in FY 2014.
SSA sponsors the YTD projects, and the related evaluation of those projects, to examine the effectiveness of providing services to youth with disabilities during their transition to adulthood. In addition to the multitude of issues facing all transition-age youth, those with disabilities have special issues related to health, social isolation, multiple service needs, and lack of access to supports. This set of challenges complicates their planning for future education and work and often leads to poor educational and employment outcomes, and high risk of dependence. SSA is investing considerable resources in developing and evaluating strategies to maximize the economic self-sufficiency of youth with disabilities, focusing on youth at the ages of 14 to 25 years, as they transition from school to work. Hallmark features of the YTD evaluation include carefully designed and targeted demonstration projects that are policy-relevant and can operate at a scale required by the evaluation and a rigorous random assignment evaluation design.
Since 1980, Congress has required SSA to conduct demonstration projects to test the effectiveness of possible program changes that could encourage individuals to work and decrease their dependence on disability benefits. In fostering work efforts, we intend these demonstrations and the program changes they test to produce savings in the trust funds or improve program administration.
To achieve these objectives, SSA’s demonstration authority contains several key features that provide SSA with a potentially valuable tool for assessing the effectiveness of policy alternatives. One of these features is SSA’s authority to waive certain disability insurance and Medicare program rules. For example, when conducting demonstrations, SSA may exempt certain beneficiaries from requirements that workers with disabilities earn below a certain amount to remain eligible for benefits. Another key aspect of SSA’s demonstration authority is the requirement that demonstration projects be of sufficient scope and conducted on a wide enough scale to ensure a thorough evaluation and results that are applicable to the program as a whole.
In addition, the legislation authorizes SSA to use trust fund monies to pay for the demonstrations and requires SSA to periodically report to Congress on its demonstration activities, providing, when appropriate, recommendations for legislative or administrative changes.
Sections 234 and 1110 of the Social Security Act (Act), Appendix A direct the Commissioner of SSA to carry out experiments and demonstration projects to determine the relative advantages and disadvantages of the following:
Various alternative methods of treating the work activity of individuals receiving benefits, including such methods as a reduction in benefits based on earnings designed to encourage these beneficiaries to return to work
Altering other limitations and conditions, such as lengthening the trial work period or altering the 24 month waiting period for Medicare
Implementing a sliding scale benefit offset
The Act requires that we design these demonstration projects to show that savings will accrue to the trust funds, or will otherwise promote or facilitate the administration of the program. Section 234 also provides that we conduct these projects in a manner that will allow SSA to evaluate the appropriateness of implementing such a program on a national scale.
To overcome the barriers to employment for beneficiaries, YTD provides individualized work-based experiences, youth empowerment, family supports, system linkages, social and health services, and benefits counseling. We randomly assigned enrollees in the demonstration to either the treatment or the control group. Enrollees in the control group have access to the traditional services available in the community and existing work incentives available, while enrollees in the treatment group receive the demonstration’s enhanced services as well as waivers of certain disability insurance rules to strengthen work incentives. The evaluation will assess the impact of these services and waivers on educational attainment, employment, earnings, and reduced use of disability benefits. The demonstration and planned evaluation meet SSA’s legislative and congressional mandates.
The purpose of this collection is evaluation of the Youth Transition Demonstration. Mathematica Policy Research and MDRC, private contractors SSA employes, collect the data. Information MPR collects will answer three key questions central to assessing the effectiveness of YTD projects:
How Are the YTD Projects Implemented and Operated?
What are the important issues and challenges in designing, implementing, and operating YTD projects, and what lessons can we draw from the experience? What approaches do we take to provide services to promote self-sufficiency among youth with disabilities? What are the characteristics of the interventions and the context of their provision? Who participates in the YTD projects, for how long, and what services do they get? Who provides these services? How do those services differ from those received by members of the control group? To what extent do the youth use the SSA waivers? How does participation in YTD differ for population subgroups?
What Are the Short Term and Longer Term Impacts of the Projects?
How effective are the projects in increasing employment and earnings and reducing dependence on disability benefits? Do the projects affect educational attainment or other intermediate outcomes, such as work attitudes or work experience? Do they improve social-psychological well-being? Do they increase the likelihood that disabled youth will be able to live independently as adults? Do these impacts differ across subgroups of the population of youth with disabilities?
What Are the Costs of Operating the Projects, and Do the Benefits Outweigh the Costs?
What are the projects’ operating costs? What other costs are incurred because of the YTD projects? To what extent do the projects lead to net changes in disability benefit receipt? Are there any induced entry effects? How do the projects affect income and payroll tax receipts, benefit outlays, and the status of the Social Security Trust Funds? From the perspectives of key stakeholders, do the benefits of the projects exceed their costs?
To address these three sets of questions, the evaluation includes process, impact, and benefit-cost analyses. This supporting statement requests clearance only for the 36-month follow-up data collection, which we will need to complete the impact and benefit-cost analyses. We will use the data we collected through the other, concluded, evaluation tools to complete the process analysis.
Process Analysis. The process analysis documents the delivery of the intervention services, including information provided to participating youth on SSA waivers and the extent of utilization of those waivers. It identifies implementation successes, issues, and challenges and examines program costs. It also provides details on the nature of each YTD intervention, and the ways in which the projects achieved the observed results. Data for this analysis comes primarily from site visits, project records and documents, and the projects’ management information systems. Site visits included discussions with staff of the YTD projects and partner organizations, SSA field office staff, and other youth service providers; focus group discussions with participating youth and their families; case reviews; and program observations. We use baseline survey data to describe the youth enrolled in the study. We use MIS data to describe and analyze service receipt and utilization among treatment group members. We use data from the first follow-up survey to examine participant experiences and satisfaction with YTD services.
Impact Analysis. We are using a rigorous random assignment design to determine the differences these YTD projects make in educational attainment, employment, earnings, and reduced use of disability benefits, as well as such outcomes as living arrangements, quality of life, and other measures of well-being among the transition-age youth enrolled in the study. Under this design, we randomly assigned youth eligible for YTD services to a treatment group (offered YTD waivers and services), or to a control group (not offered YTD waivers or services but using existing SSA work incentives and services available in the community). Outcomes for the two groups will be compared using data collected in follow-up interviews, conducted 12 and 36 months after youth enter the demonstrations, as well as data obtained from SSA program files, administrative files of state and local agencies, and possibly SSA summary earnings records (SER). Based on these comparisons, we will assess the net effects of the YTD intervention approaches for the youth enrolled in the study and the differential effectiveness of YTD services for members of certain subgroups. We will use administrative data to address impacts on SSA disability benefits receipt and the use of SSA waivers, earnings, and other public assistance. We will use the more comprehensive data from the follow-up surveys to examine employment and other outcomes such as education, income, health, and measures of life quality and well-being.
Benefit-Cost Analysis. A key element of the YTD evaluation is to measure the costs necessary to operate the YTD projects. For policy action, cost information is essential, because legislators and administrators will not be able to proceed with a program or policy unless they have a good idea of the program costs. The evaluation will conduct a comprehensive benefit-cost analysis of the YTD projects. We will start with a comprehensive cost analysis of each project; the goal is to construct an estimate of overall project costs as well as estimates of average unit costs, such as the cost per participant and cost per program component. Drawing on data reports from project records and on information from program staff interviews; we will build up an estimate of the cost of each project. Information from in-depth interviews with youth or their guardians about service utilization will provide information needed for the service cost analysis. In addition to its usefulness as an adjunct to the process study’s description of program operations, the cost analysis will provide important input for the benefit-cost analysis. For purposes of this analysis, key costs include operating and administrative costs. Benefits include, but are not limited to, net increased earnings and tax payments, net reduced disability benefits, and net reductions in the receipt of public assistance. The benefit-cost analysis also will examine net changes in services used because of the YTD projects. The benefit-cost analysis will examine the extent to which the projects lead to net increases or reductions in SSI benefit receipt (and, hence, the cost or savings to SSA). The analysis will examine the costs and benefits of the projects from the perspectives of a variety of stakeholders—including SSA, other government agencies, the YTD participants, and society as a whole—and will be produced in a format consistent with the requirements of SSA’s actuaries.
Under this design, we randomly assigned youth eligible for YTD services to a treatment group (offered YTD waivers and services) or to a control group (not offered YTD waivers or services but using existing SSA work incentives and services available in the community). Outcomes for the two groups will be compared using data collected in follow-up interviews, conducted 12 and 36 months after youth enter the demonstrations, as well as data obtained from SSA program files, administrative files of state and local agencies, and SSA summary earnings records (SER). Participation in YTD and the interviews is voluntary for respondents. Based on these comparisons, we will assess the net effects of the YTD intervention approaches for the youth enrolled in the study and the differential effectiveness of YTD services for members of certain subgroups. We will use administrative data to address impacts on SSA disability benefits receipt and the use of SSA waivers, earnings, and other public assistance. We will use the more comprehensive data from the follow-up surveys to examine employment and other outcomes such as education, income, health, and measures of life quality and well-being.
If effective, the most immediate impacts of the interventions should reflect in increased employment-focused services and work-related experiences for those in the treatment group. We also expect more paid employment, greater income resulting from increased employment and more generous work incentives offered by the waivers, attitudes that are more positive and expectations about the future, and continued progress in education for projects that emphasize education.
In the intermediate and longer terms, we expect treatment group youth in the projects to increase their employment and earnings, have greater income, reduce risky behaviors, and have greater self-determination and self-efficacy and move toward independent living. Furthermore, we expect that, in the considerably longer term, particularly after the waivers are no longer in effect, the projects will reduce youths’ dependence on disability programs.
We have grouped the key domains for which we will examine YTD impacts in the longer run into five areas, as summarized in Table A.1.
Data from the 36-month follow-up survey will supplement administrative files that SSA maintains, and will provide data on outcomes not available from administrative sources such as attitudes and expectations.
Table A.1
Outcomes for Which WE Anticipate Impacts in the
Longer Term
Outcome Measure |
Description of Measure |
Employment and Earnings |
|
Primary outcomes |
Fraction of time employed in past year, earnings in past year |
Supplementary outcomes |
Ever employed during follow-up period, earnings patterns, hours worked, full-/part-time work, number of jobs held, wage rates, benefits, accommodations, impacts from administrative data (over the three-year period following random assignment) |
Youth Income |
|
Primary outcomes |
Total income from earnings and benefits during prior 12 months or entire follow-up period |
Supplementary outcomes |
Type and amount of earnings and benefits received, amount of SSA benefits, use of SSA work incentives and Individual Development Accounts (IDAs) |
Engagement in Gainful Activity |
|
Primary outcomes |
Fraction of youth either employed or participating in an education or training program |
Supplementary outcomes |
Time spent engaged in gainful activities, educational attainment |
Reduction in Criminal Justice System Contact |
|
Primary outcomes |
Contact with criminal justice system (arrests, incarcerations, other involvement with the criminal/juvenile system) |
Supplementary outcomes |
Types of criminal activity, drug and alcohol treatment |
Self-Determination and Self-Efficacy |
|
Primary outcomes |
Self-determination and self-efficacy scale, independence (such as traveling, having a bank account, living arrangements) |
Supplementary outcomes |
Items comprising the scales, other measures of independence, attitudes and expectations, self-esteem, social interactions |
Other Exploratory Analyses |
|
Medicaid utilization |
Number and total amount of Medicaid paid claims during a calendar year |
Health status |
SF-12 health scale; self-reported health status: excellent, very good, good, fair, poor; self-perceptions of health conditions and disabilities |
Quality of life |
Selected components of the Instrumental Activities of Daily Living (IADL) that are relevant to youth; limitations in mental, emotional, and social functioning |
For the YTD evaluation, MPR and its partner, Social Solutions, implemented a management information system (MIS), the Efforts-to-Outcomes (ETO) database, to facilitate the real-time exchange of data between MPR’s survey division and the YTD projects. The ETO database draws information from several data sources including SSA administrative data, respondent survey data, claims and utilization data, and data entered by YTD project staff. We designed the ETO database to make data from all sources linkable so that it fully supports drawing extracts and generating reports and summaries to facilitate administering, monitoring, and evaluating the study.
We use computer-assisted interviewing to collect data for the baseline and follow-up surveys. We administered the baseline and 12-month follow-up surveys as both computer-assisted telephone (CATI) and face-to-face interviews. These applications incorporated standard checkpoints to assess each respondent’s level of fatigue and to provide the respondent with an opportunity to take a break, if necessary. Both the baseline and 12-month follow-up interviews used Computer Assisted Telephone Interviewing (CATI) software. We will use computer-assisted interviewing to collect data for the 36-month follow-up survey as well, and we programmed the questionnaire into the software application. The software is able to customize the flow of the questionnaire based on the answers provided, as well as information already known about the sample member such as their gender, treatment or control status, or state of residence. Interviewers read questions that appear on their computer screen and enter the respondents’ answers. In this sense, it is similar to SSA’s MCS and MSSICS systems.
Telephones equipped with amplifiers are available for use as needed to accommodate sample members who are hearing impaired. In addition, we use TTY and Relay technologies to facilitate participation in the telephone survey. A TTY is a special device that lets people who are deaf, hard of hearing, or speech-impaired use the telephone to communicate by allowing them to type messages back and forth to one another instead of talking and listening. A TTY is required at both ends of the conversation in order to communicate. MPR’s telephone operations center is equipped with TTY technology. We use the Telecommunications Relay Service (TRS) for sample members who are deaf, hard of hearing, or speech-impaired, but who do not have a TTY. With TRS, a special operator types whatever the interviewer says so the people called can read the interviewer’s words on their telephone display. The caller then types back a response, which the TRS operator will read aloud for the interviewer to hear over the phone. Both methods, TTY and TRS, increase survey administration times but enable us to conduct interviews with sample members who, without the help of these technologies, would not be able to participate. Forms are not available electronically because they are not self-administered.
The surveys only ask respondents about information that is not available in SSA’s administrative records. We reviewed administrative records in detail to limit repetition. Some information about treatment group members that we collect through the surveys may be redundant with data that could also be available from the ETO database. However, this duplication is necessary to collect comparable data from sample members in the control group.
The Youth Transition Process Demonstration Evaluation does not involve small entities.
The baseline survey was a one-time collection and was necessary to conduct a credible evaluation. We needed the baseline survey to identify and select sample members into the study groups, assure the treatment and control groups were comparable, and obtain important covariates for subsequent analyses. The data collected during the baseline interview were not available from other sources.
We conducted the first follow-up survey 12 months after random assignment, during which we collected information on short-term outcomes regarding education, earnings, employment, living arrangements, health, and quality of life. We planned the second follow-up survey for 36- months after random assignment. It collects information about longer-term outcomes in the same domains as the 12-month follow-up survey. The questions focus on the past 12 or 24 months so as not to overlap with the 12-month survey recall period. The follow-up survey collects a richer set of information than we can gather from administrative records. For example, administrative records might have data on earnings from jobs but would not have details about the jobs such as rates of pay, hours worked, or if the job was competitive or supported employment. We cannot conduct the impact analysis without the 36-month follow-up survey.
There are no special circumstances related to the collection of information required to carry out the evaluation of YTD.
The 60-day advance Federal Register Notice published on February 14, 2012, at 77 FR 8322, and we received no public comments. SSA published the second Notice on June 14, 2012, at 77 FR 35739. If we receive comments in response to the 30-day Notice, we will forward them to OMB. We did not consult with the public in the revision of this form.
We need an interdisciplinary project team of economists, disability policy researchers, survey researchers, and information systems professionals to carry out the design and implementation of the evaluation. MPR is the prime contractor with overall responsibility for implementing and evaluating the demonstration. However, staff members from three other organizations are integral members of the study team. The participating organizations include the following:
Mathematica Policy Research, Inc.
600 Maryland Ave., SW
Suite 550
Washington, DC 20024-2512
(202) 484-4698
600 Alexander Park
Princeton, NJ 08540
(609) 799-3535
MDRC
19th Floor
16 East 34 Street
New York, NY 10016-4326
(212) 532-3200
Social Solutions, Inc.
2400 Boston St.
Suite 360
Baltimore, MD 21224
(410) 732-3560
TransCen, Inc.
451 Hungerford Drive
Suite 700
Rockville, MD 20850
(301) 424-2002
We have listed key staff from these organizations, their roles, and contact information below:
Tom Fraker
MPR-DC
Project Director
tfraker@mathematica-mpr.com
(202) 484-4698
David Butler
MDRC-NY
Leader of task force on program development
david.butler@mdrc.org
(212) 340-8621
Karen CyBulski
MPR-NJ
Survey Director
kcybulski@mathematica-mpr.com
(609) 936-2797
Anu Rangarajan
MPR-NJ
Leader of task force on evaluation
arangarajan@mathematica-mpr.com
(609) 936-2765
John Martinez
MDRC-NY
Leader of task force on pilot sites
john.martinez@mdrc.org
(212) 340-8690
Anne Ciemnecki
MPR-NJ
Senior Methodologist
aciemnecki@mathematica-mpr.com
(609) 275-2323
Richard Luecking
TransCen
President, Programmatic TA
rluecking@transcen.org
(301) 424-2002 x230
Matt Schubert
Social Solutions
CEO, ETO
matt@socialsolutionsonline.com
(410) 732-3560
Kelli Crane
TransCen
Director, programmatic TA
kcrane@transcen.org
(240) 418-2684
In addition, SSA has convened a technical working group with these members:
Harold Beebout
Independent Consultant
(540) 987-3320
Michael Callahan
Marc Gold & Associates/Employment for All
(228) 497-6999
(228) 497-6966
Nancye Campbell
DHHS/ACF
(202) 401-5760
Elizabeth McGuire
DHHS/HRSA
emcguire@hrsa.gov
(301) 443-9290
Alexandra Kielty
USDOL/ETA
(202) 693-3730
(202) 693-3818
K. Charlie Lakin
University of Minnesota
(612) 624-5005
(612) 625-6619
Rebecca Maynard
University of Pennsylvania
(215) 898-3558
Howard Rolston
Abt Associates
703-532-4195
Betsy Valnes
Black Hills Special Services
(605) 224-5336
Mary Wagner
SRI International
(650) 859-2867
Beneficiaries and their parents participated in the pretest of the follow-up survey.
For the baseline, we offered beneficiaries a $10 gift for returning a completed consent form which varied by location. In New York City, for example, we offered a $10 MetroCard. In other places, we offered a $10 Target or Wal-Mart gift card. We offered another $10 gift to beneficiaries completing the 12-month follow-up interview and we will offer the same for the 36-month follow-up interview. Through our experience, we know this incentive will increase interview response rates and reduce sample attrition between the 12-month and 36-month interviews.
We offered all focus group participants $40 to cover their time, transportation, or other costs of participating. We did not offer program staff remuneration for completing interviews because they do this as part of their job responsibilities.
The information provided for this project is protected and held confidential in accordance with 42 U.S.C. 1306, 20 CFR 401 and 402, 5 U.S.C. 552 (Freedom of Information Act), 5 U.S.C. 552a (Privacy Act of 1974), and OMB Circular No. A-130. We will treat data in a confidential manner unless otherwise compelled by law.
The study team takes seriously the ethical and legal obligations associated with the collection of confidential data. We ensure the secure handling of confidential data via several mechanisms. These include obtaining suitability determinations for designated staff, training staff to recognize and handle sensitive data, protecting computer systems from access by staff without favorable suitability determinations, limiting access to secure data on a “need to know” basis and only for staff with favorable suitability determinations, and creating data extract files from which identifying information has been removed.
We take several steps to assure sample members that we will treat the information they provided confidentially and use it for research purposes only. We make clear the assurances and limits of confidentiality and make sure the Paperwork Reduction and Privacy Act statements appear on all materials sent to participants. We also restated these assurances at the beginning of each interview session.
Subcontractors, consultants, and vendors are required to establish confidential information safeguards that meet prime contract security requirements. The project director or task leader takes action to ensure that any confidential information provided to or generated by a subcontractor, consultant, or vendor is properly disposed of at the completion of the agreement between the parties.
The purpose of the study is to test the effects of waivers of SSA program rules and an innovative array of enhanced employment and educational services for youth with disabilities. Therefore, obtaining information about potentially sensitive topics, such as the health status and the disabling condition of sample members, is central to the intervention. The surveys do not collect data that we can obtain directly from other sources (for example, we can obtain information about receipt of disability benefits directly from SSA administrative records).
The survey includes questions about the following topics that one may consider potentially sensitive:
Health status, including disability information and severity of disabling condition
Assistance needed with Instrumental Activities of Daily Living (IADLs)
Mental health status
Participation in drug and alcohol treatment programs
Involvement with the criminal justice system
Many of the questions were adapted without modification from other national surveys of similar populations, such as the National Longitudinal Transition Survey (NLTS), the National Beneficiary Survey (NBS), and the Evaluation of Job Corps (JC). The instrument also contains items from the Short Form 12 (SF-12).
Table A.2 shows the annualized number of expected participants in the data collection, the number of interviews, hours per response, and the total associated response burden for the 36-month follow-up questionnaire in 2012. We determined the burden though experience administering similar items on the 12-month follow-up interview and in pretest-based estimates of the 36-month follow-up questionnaire. All eligible youth who consented to participate in the YTD demonstration are eligible to participate in the 36‑month follow-up survey. We will administer 949 36-month follow-up surveys in 2012. We estimate 50 minutes (0.83 hour) per completed interview, for 788 total burden hours.
TABLE A.2
ANNUALIZED BURDEN
Data Collection |
Collection |
Number of Respondents |
Responses Per Respondent |
Average Burden |
Total Response
Burden |
2012 |
36 month follow-up |
949 |
1 |
0.83 |
788 |
Table A.3 shows the total burden for the entire Youth Transition Process Demonstration Evaluation Collection from 2007 through 2013. As mentioned in #1 above, we intend to finish the data collection elements in 2013, and we will spend 2014 evaluating the data. While the chart below shows burden for all the data collection instruments we used during the course of this seven-year study, in 2012 we will only collect information using the 36-month study (table A.2 above). We expect to obtain responses from 3,962 total respondents for the 36-month follow-up interview from 2007 – 2013. Respondents will incur no monetary costs for completing the interview
TABLE A.3
COMBINED BURDEN DATA FOR YEARS 2007 - 2013
|
Collection |
Number of Respondents |
Responses Per Respondent |
Average Burden |
Total Response
Burden |
Grand Total |
Baseline |
5,651 |
1 |
0.55 |
3,108 |
|
Informed Consent |
5,651 |
1 |
.083 |
469 |
|
12 month follow-up |
4,752 |
1 |
0.83 |
3,944 |
|
In-depth interviews |
240 |
1 |
.42 |
101 |
|
Focus group |
440 |
1 |
1.5 |
660 |
|
Program staff/service provider |
192 |
1 |
1 |
192 |
|
36 month follow-up |
3,962 |
1 |
0.83 |
3,288 |
Grand Total |
|
|
|
|
11,762 |
There are no direct costs to respondents other than their time to participate in the study, as described above. We do not ask beneficiaries to maintain any new records. The evaluation contractor collects and maintains all survey data. Costs for data collection, storage, processing, and other functions related to these data are borne solely by the contractor. We summarize these costs in #14 below and consider costs to the federal government, paid through SSA contracts.
Table A.4 shows the annualized costs by year for the next three years (2012 – 2014). We budgeted labor costs by estimating the number of hours of required staff at the various wage levels, multiplying by the applicable wage rates, and multiplying the resulting subtotals by factors to cover fringe benefits and burden expense. The basis for estimating other direct costs varies with the type of cost estimated. For example, we based the estimates of survey telephone expense and computer expense for CATI on the estimated hours of interviewer time, while we based reproduction expense on the number of pages of material reproduced. Finally, we summed and multiplied by a factor to cover general and administrative expenses, and to obtain the total of labor costs, other direct costs and the added fee.
Table A.5 shows the total cost to SSA of $46,829,767 for conducting the YTD evaluation from 2007 through our evaluation in 2014.
TABLE A.4
ANNUAL COSTS TO THE FEDERAL GOVERNMENT
Year |
Cost |
2012 2013 2014 |
$1,620,474 $1,312,455 $1,472,168 |
Total |
$4,405,097 |
TABLE A.5
COMBINED ANNUAL COSTS TO THE FEDERAL GOVERNMENT
2007-2014
Grand Total |
$46,829,767 |
The burden changes to this collection stem from the fact that we are no longer collecting information for surveys other than the 36-month Follow-Up.
Baseline data collection began in July 2006 and continued through 2010. The 12-month follow-up data collection began in November 2007 and continued through 2011. Likewise, the 36-month data collection, for which we are requesting clearance, began in August 2009 and will continue through 2013.
We planned a series of reports throughout the life of the demonstration. We produced project-specific early assessment reports 8 months after the demonstrations enrolled their first youth, beginning in the spring of 2007. We randomly assigned process and implementation, and created early impact reports 18 months after the last youth at a project site, beginning in September 2009. We will produce the final report and public use data files by October 2014. We may produce up to three reports on special topics over the life of the demonstration by October 2014.
The process and implementation reports document and describe how we planned and implemented the demonstration, explain program processes, document beneficiary experiences with the demonstration and describe outcomes or results. We addressed the following distinct components of program implementation: (a) outreach, recruitment, and participation; (b) the intervention, including whether each component was implemented as planned, differences in implementation across subgroups, existing service systems, and the use of services; (c) organizational arrangements, communication, and coordination; (d) coordination with SSA field offices; and (e) experiences and satisfaction of beneficiaries and other stakeholders. We explicitly document implementation issues we encountered as well as how we addressed them. We also document how major features of the program change over the course of the evaluation, the reasons for the changes, and the implications for program outcomes measured in the evaluation.
The impact reports investigate the demonstration’s effects on a wide array of education, earnings, and self-determination outcomes; the amount of benefits the beneficiary receives from SSA; and the beneficiary’s quality of life, both overall and for meaningful subgroups. Our methodological approach combines a random assignment design with regression adjustment to improve the precision of our estimates. Because we randomly assigned individuals to the control group and to the treatment group, the impact analysis focuses on differences in the outcomes of beneficiaries between these two groups using a regression framework to control for other explanatory variables. We use regression-adjusted comparison of randomly assigned treatment group to control group for the full sample to address the impact of the intervention on beneficiaries’ education, labor market, and other outcomes. We use a regression-adjusted comparison of randomly assigned treatment group to control group for subgroups defined by pre-randomization values of age, race, gender, and type of disability.
The exact statistical technique we use to estimate regression-adjusted impacts depends on the nature of the dependent variable and the type of issues we addressed. For example, if the dependent variable is continuous, then ordinary least squares regression produces estimates of impacts that are unbiased. For binary outcome variables (such as whether or not the beneficiary is employed), logistic regression models generate estimates that are consistent and efficient if the parametric assumptions underlying those models are correct. If the dependent variable is a count variable then we use an ordered logit model. If the dependent variable is ordinal, we first reduce the measure to binary outcomes and then estimate a logit model. To account for the fact that we observe sample members for different lengths of time, we also consider using event-history or hazard models for binary outcome measures. These models provide unbiased estimates of program effects on binary outcomes when participants’ data are truncated.
The purpose of the benefit-cost analysis is to determine whether the program impacts of the YTD demonstration are sufficiently large to justify the costs of providing program services. The results of this analysis will play an integral part in the decision to expand the demonstration to the larger population. We will base the analysis on an accounting framework that summarizes the intervention’s effects and resource use from the perspective of SSA and other key stakeholder groups, including society as a whole.
To ensure that the benefit-cost findings are as helpful as possible to SSA, we plan to present the information in a way that has proven useful for communicating this type of information to the SSA Office of the Actuary and to OMB. First, we will summarize all of the information based directly on data collected during the demonstration period. The second set of estimates will present the size of future effects (if any) that would be required for the program to generate benefits that exceed costs along with an analysis of how likely it seems that future effects of that size will occur. In this way, SSA actuaries will be able to see the net value generated during the observation period and then use the more speculative analysis of possible future benefits and costs to draw conclusions about whether the YTD projects would ultimately pay for themselves. In addition to using this general presentation format, we will work with the actuaries during the evaluation to ensure that the other assumptions used in the analysis—the discount rate, correction for inflation, and projections about potential productivity growth—are consistent with the ones they are using to assess other potential SSA initiatives. This consistency will go a long way in ensuring that comparisons of the various options are accurate and useful.
We display the OMB expiration date on all survey materials sent to respondents, including the advance. We also make it accessible in the computer-assisted instruments when a respondent requests the information.
SSA is not requesting an exception to the certification requirements at 5 CFR 1320.9 and related provisions at 5 CFR 1320.8(b)(3).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | MEMORANDUM |
| Author | Dawn L. Patterson |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |