Supporting Statement Part B - PROMISE

Supporting Statement for The PROMISE Evaluation

B. Collection of Information Employing Statistical Methods

The staff interview and participant focus group data collection for which SSA is seeking approval in this submission will not employ any statistical methods. Additional information about the interview and focus group data collection methods is provided below.

B1. Respondent Universe and Sampling Methods

We will conduct a set of staff interviews and participant focus groups with all six PROMISE projects.

1. Staff Interviews

We will determine the PROMISE project and partnership organization staff to be interviewed in consultation with each PROMISE project director. In consultation with the PROMISE project director, Mathematica will identify key administrators and directors of partnering agencies and the appropriate project staff who can provide a range of perspectives on the implementation experiences of the PROMISE project. Examples of program directors and managers include: the PROMISE project director and principal investigators; administrators of state government agencies that participate in PROMISE; and executive directors of non-governmental or community-based organizations that provide services to PROMISE participants and other youth with disabilities. Examples of PROMISE project staff include: recruiters; case managers; employment specialists; benefits counselors; vocational rehabilitation counselors; and educational instructors and coordinators. Although a precise estimate of the number of administrative and service provider staff who will be involved in the PROMISE projects is unavailable, SSA anticipates that the evaluator will interview approximately 80 percent of the administrators and managers, and about 50 percent of the service provider staff over the course of the two rounds of site visits and interviews.

2. Focus Groups

The universe for the focus group data collection consists of youth, and parents or guardians of these youth, who are recipients of SSI benefits; who are enrolled in the PROMISE demonstration; and who have been randomly assigned to the treatment group. The convenience sample approach that we will use to identify focus group participants precludes SSA from using any findings from the focus group discussions to generalize to the entire population of youth enrolled in PROMISE, or to youth receiving SSI benefits nationwide. That is, the focus groups will not contain, and are not intended to contain, a representative sample of any larger population. We expect the qualitative data we will collect during the focus groups to provide additional insight into the impact findings that the quantitative data we will collect via surveys and administrative data will generate.

The evaluator will conduct all of the recruitment for the focus groups, and will consult with local program staff to identify enrollees who might be interested in participating in the focus group discussions. A goal of the selection and recruitment effort will be to ensure that a diverse set of PROMISE youth and families participate in each focus group. The evaluator will use criteria such as age, type of disability, and length of time receiving child SSI payments, to select a diverse pool of youth and families. The evaluators will request a list of up to 50 youth (and parents or guardians) to which they will send the advance notification letter and follow-up by telephone to invite them to participate. The goal will be to have 10 attendees and 2 alternates (in case of attrition) for each session. Table B.1 shows the total number of persons contributing to the focus groups by project and group type.

Table B.1 Treatment Group and Focus Group Sample Sizes by PROMISE Project

		Youth Focus Groups		Parent or Guardian Focus Groups			Percentage of Treatment Group
State	Number of Families in Treatment Group	Summer 2014	Winter 2016	Summer 2014	Winter 2016	Total N Selected	Families Selected
Arkansas	1,000	10	10	10	10	40	0.04
California	1,539	10	10	10	10	40	0.03
Maryland	1,000	10	10	10	10	40	0.04
New York	1,000	10	10	10	10	40	0.04
Wisconsin	1,000	10	10	10	10	40	0.04
Six-State Consortium	1,000	30	30	30	30	120	0.12
Totals	6,539	80	80	80	80	320	0.05

B2. Procedures for the Collection of Information

1. Staff Interviews

The first step in the data collection will be to send a letter to each PROMISE site project director explaining the evaluation and seeking their cooperation with it. The letter will come from Jeffrey Hemmeter, the SSA Project Officer, to lend credibility to the study and further encourage cooperation. The evaluator will follow up with the PROMISE project director in a telephone call to describe further the information we will gather from PROMISE stakeholders during in-person interviews and a brief self-administered social network survey that we will ask them to complete at the end of each interview. We will ask project directors to identify individuals who will be able to provide the required information, and for information about their general schedule constraints. We will develop a schedule for the interviews that meets participants’ needs collaboratively with the PROMISE project and partner contacts. Approximately two weeks before the interviews are scheduled to take place, the evaluator will mail an information packet to the PROMISE project director containing the final interview schedule. The packet will also contain contact information for the evaluation team member who will conduct the interviews so that respondents can reach them in the event of a schedule change or other issues that may arise before the interviews. Providing the sites with adequate information ahead of time in a professional manner will help build rapport and ensure that the interviews go smoothly and that interviewees are available and responsive.

The evaluator will use an interview guide, based on the interview topic list provided in Attachment B, to conduct the staff interviews. The interviewer will be responsible for taking notes during each interview. Upon completion of all interviews conducted for a particular PROMISE project, the evaluator will develop a summary of the information collected during the site visit and phone interviews.

During the staff interviews, the evaluator will ask interviewees to complete a brief social network survey. Separate versions of survey will be administered to program managers or directors and project staff, tailored to their specific perspectives (Attachment C). The survey will be self-administered with pen and paper. The evaluator will enter the survey data into an Excel spreadsheet for analysis after completion of the site visit.

2. Focus Groups

After state PROMISE project staff confirms youth and their families as eligible for services, the project staff will obtain their informed consent and enroll them into the study. The consent process will address the program benefits; random assignment process; expectation to complete follow-up surveys; and the voluntary nature of participation in all study activities. The project staff will also disclose any potential risks of participation and the use of personal information. Once the state PROMISE project staff receives consent, they will enter the youth into the PROMISE random assignment database, enroll them in the demonstration, and assign them to either the treatment or control group.

We will conduct the focus groups with a convenience sample of youth and their parents or guardians who are PROMISE treatment group members. We will conduct the focus groups during site visits in summer 2014 and winter 2016. Separate but concurrent discussions will occur with 10 youth and 10 parents or guardians in each group. The evaluator will work closely with the PROMISE project staff to arrange the focus groups. Where possible, we will convene the groups in a facility of a PROMISE service provider familiar to participants.

The evaluator will use a recruitment script to introduce the evaluation, describe the purpose of the focus group, and confirm the willingness to participate. One week before the focus groups, the evaluator will send a reminder letter to each individual who agreed to participate along with directions to, and a map of, the focus group location. The evaluator will also send a reminder or confirmation mailing prior to the session, with a telephone reminder placed the day before the session.

A professional researcher on the staff of the evaluation contractor will conduct the focus groups using a semi-structured protocol to facilitate an informal group discussion. To ensure that all information is captured, the facilitator will record the discussion for later transcription. Participants will be informed about the recording and instructed that they may request that the recording be suspended at any time. No identifying information will be asked during the focus group, and the facilitator will only call group participants by their first names; thus no identifying information will be included in the tape recording. Each focus group participant will receive a $30 incentive in the form of a gift card, following completion of the session.

SSA recognizes that the small number of participants and small number of groups and sampling approach means that the focus group data collected cannot be used to extrapolate to the larger population of youth and parents or guardians enrolled in PROMISE, or families in the broader population of those receiving SSI payments. However, the focus groups will capture critical qualitative information about the experiences of PROMISE participants, their families, and project staff. The information we will collect during these interviews will complement the information we will gather through administrative data and through follow up surveys, providing more in-depth and qualitative understanding of the PROMISE projects. These focus groups are a critical piece of the process for PROMISE and essential for the evaluation team to assess whether and how the projects did or did not meet expectations.

B3. Methods to Maximize Response Rates and Deal with Nonresponse

In arranging the interviews we will conduct with PROMISE staff and the staff of partner organizations, the evaluator will work with the PROMISE project leadership to determine the most convenient times and formats (group versus individual; phone versus in-person) to convene the interviews. The evaluator will also limit the interviews to approximately one hour to ensure that the data collection imposes only a modest burden on respondents. The evaluator will use separate discussion guides for each potential respondent type so that respondents are not asked about activities or issues that are not applicable to them. In addition, data collectors will meet with in-person interview respondents in their own offices or at a location of their choice.

Because the focus group sample is a convenience sample, target response rates to ensure a representative population are not at issue. To address non-response by ensuring the groups contain approximately 10 parents or guardians and 10 youth each, the evaluator will recruit more participants than needed, based on prior experience that a portion of those initially recruited will not attend the group when it meets. Further, the evaluator will provide telephone and mail reminders to all recruited participants as the focus group date approaches. Finally, we will provide incentive payments to focus group participants (as noted in section B.2) to alleviate some barriers to focus group participation.

B4. Tests of Procedures or Methods to Be Undertaken

We will not conduct pre-tests of the interview or focus group protocols. The evaluator will make minor modifications to the data collection procedures and protocols, if necessary, based on the experiences of the early interviews and focus groups conducted.

B5. Individuals consulted on Statistical Aspects of the Design and on Collection and/or Analyzing Data

As discussed in A.8, SSA convened a technical advisory panel for the PROMISE evaluation. The panel provided input on the evaluation criteria and research design. It consisted of researchers and advocates who reflected expertise in youth transition, disability, and evaluation design. The external experts were:

• Burt Barnow, PhD, George Washington University

• Hugh Berry, US Department of Education

• Mark Donovan, Marriott Foundation for People with Disabilities

• David Johnson, PhD, University of Minnesota

• Jamie Kendall, US Dept. of Health and Human Services

• Jeffrey Liebman PhD, Harvard University

• Pamela Loprest, PhD , The Urban Institute

An interdisciplinary team of economists, disability policy researchers, and survey researchers, and information systems professionals on the staff of the evaluation contractor (Mathematica Policy Research and its subcontractor, BCT Partners) contributed to the design of the overall evaluation. These individuals include:

• Karen CyBulski, Mathematica

• Thomas Fraker, PhD, Mathematica

• Jacqueline Kauff, Mathematica

• Gina Livermore, PhD, Mathematica

• Holly Matulewicz, Mathematica

• Tonya Woodland, BCT Partners

References

Blacher, Jan, Bonnie Kraemer, and Erica Howell. “Family Expectations and Transition Experiences for Young Adults with Severe Disabilities: Does Syndrome Matter?” Advances in Mental Health and Learning Disabilities, vol. 4, no.1, 2010, pp. 3–16.

Cameto, R., P. Levine, and M. Wagner. Transition Planning for Students with Disabilities. A Special Topic Report of Findings from the National Longitudinal Transition Study-2 (NLTS2). Menlo Park, CA: SRI International, November 2004. Available at [http://www.nlts2.org/reports/2004_11/nlts2_report_2004_11_execsum.pdf]. Accessed July 22, 2013.

Carter, E., D. Austin, and A. Trainor. “Predictors of Postschool Employment Outcomes for Young Adults with Severe Disabilities. Journal of Disability Policy Studies, vol. 23, no. 1, 2012, pp. 1–14.

Chiang, Hsu-Min, Ying Kuen Cheung, Huacheng Li, and Luke Y. Tsai. “Factors Associated with Participation in Employment for High School Leavers Autism.” Journal of Autism Developmental Disorders, vol. 42, no. 5, 2012, pp. 685–696.

Emerson, Eric. “Poverty and People with Intellectual Disabilities.” Mental Retardation and Developmental Disabilities Research Reviews, vol. 12, no. 2, 2007, pp. 107–113.

Hasazi, S.B., L.R. Gordon, and C. A. Roe. “Factors Associated with the Employment Status of Handicapped Youth Exiting High School from 1979 to 1983.” Exceptional Children, vol. 51, 1985, pp. 455–469.

Hemmeter, Jeffrey, Jacqueline Kauff, and David Wittenburg. “Changing Circumstances: Experiences of Child SSI Beneficiaries Before and After Their Age-18 Redetermination for Adult Benefits.” Journal of Vocational Rehabilitation, vol. 30, no. 3, 2009, pp. 201–221.

Lee, Gloria, and Erik Carter. “Preparing Transition-Age Students with High-Functioning Autism Spectrum Disorders for Meaningful Work.” Psychology in the Schools, vol. 49, no. 10, 2012, pp. 988–1000.

Lindstrom, Lauren, Bonnie Doren, and Jennifer Miesch. Waging a Living: Career Development and Long-Term Employment Outcomes for Young Adults with Disabilities, Council for Exceptional Children, vol. 77, no. 4, 2011, pp. 423–434.

Lindstrom, Lauren, Bonnie Doren, Jennifer Metheny, Pam Johnson, and Claire Zane. “Transition to Employment: Role of the Family in Career Development.” Council for Exceptional Children, vol. 73, no. 3, 2007, pp. 348–366.

Loprest, Pamela J., and David C. Wittenburg. “Post-Transition Experiences of Former Child SSI Beneficiaries.” Social Service Review, vol. 81, no. 4, 2007, pp. 583-608.

Luecking, R. and N. Certo. “Service Integration at the Point of Transition for Youth with Significant Disabilities: A Model that Works.” American Rehabilitation, vol. 27, 2003, pp. 2–9.

Powers, L., T. Garner, B. Valnes, P. Squire, A. Turner, T. Couture, and R. Dertinger. “Building a Successful Adult Life: Findings from Youth-Directed Research.” Exceptionality, vol. 15, no. 1, 2007, pp. 45–56.

Shattuck, Paul, Sarah Carter Narendorf, Benjamin Cooper, Paul Sterzing, Mary Wagner, and Julie Lounds Taylor. “Postsecondary Education and Employment Among Youth with an Autism Spectrum Disorder.” Pediatrics, vol. 129, no. 6, 2012, pp. 1042–1049.

Simonsen, M., and D. Neubert. “Transitioning Youth with Intellectual and Other Developmental Disabilities: Predicting Community Employment Outcomes.” Career Development and Transition for Exceptional Individuals, 2013. doi: 1177/216543412469399.

U.S. Government Accountability Office. “Summary of a GAO Conference: Helping California Youths with Disabilities Transition to Work or Postsecondary Education.” GAO-06-759SP. Washington, DC: United States Government Accountability Office, 2006.

Wittenburg, D., T. Golden, and M. Fishman. “Transition Options for Youth with Disabilities: An Overview of the Programs and Policies That Affect the Transition from School.” Journal of Vocational Rehabilitation, vol. 17, 2002, pp. 195–206.

Wittenburg, David. Testimony for Hearing on Supplemental Security Income Benefits for Children.” Presented at the Subcommittee on Human Resources Committee on Ways and Means U.S. House of Representatives. Washington, DC: Mathematica Policy Research, 2011.