Memo

MEMORANDUM

Contract Number: CNS14A0007-0007

Contract Title: AmeriCorps State and National Opportunity Youth Evaluation Bundling

Submitted to: Adrienne DiTommaso

Submitted by: Nicole Vicinanza

Subject: Results of pilot testing for Opportunity Youth AmeriCorps Member Survey

Date: April 3, 2015

This memorandum is submitted in partial fulfillment of Task 6 of the work plan, which calls for JBS to develop a memo detailing the results of pilot testing for the Opportunity Youth AmeriCorps Member Survey.

Introduction and Overview of Findings

JBS carried out pilot testing for the Opportunity Youth AmeriCorps Member Survey, including proctored, online, and telephone interviews, as well as cognitive interviews with a subset of survey respondents. Data collection took place between March 10^th and March 27^th, 2015. The pilot test results indicate that in most cases, the items made sense to respondents, the response choices provided a sufficient range of options to show variability of response, and the respondents interpreted the items as intended. However, both the results of the mode effects analysis of the pilot data and information from proctors, phone interviewees, and cognitive interviewees suggest that participants had difficulty comprehending some questions and response options. JBS noted these problems and modified the instrument accordingly.

In all, 67 participants were selected for the pilot, 37 took the survey, and 36 completed the survey. Of the 36, nine participated in cognitive interviews. The respondents who participated in cognitive interviews reported few difficulties in understanding and responding to survey items.

JBS has identified some specific suggestions to improve the survey or the respondents’ survey experience from the three survey modalities and nine cognitive interviews. Recommendations based on this include:

• Change data collection modality options. Use online surveys as default alternative data collection option when proctored paper surveys are not feasible. Follow up via phone only as needed.

• Make minor modifications to select items to improve comprehensibility. This includes rewording of questions and response options that were identified as potentially confusing by cognitive interviewees. This may also include the addition of clarifications regarding how AmeriCorps members should consider their service when responding to items regarding education and employment.

• Consider drafting slight instrument variants for different phases of data collection and/or treatment vs. comparison groups. For example, the question “Are you currently employed?” may not be appropriate for AmeriCorps members at pre-test or initial post-test. An alternative question may be needed at pre-test to ascertain recent employment history and allow treatment and comparison group members to be appropriately matched.

• Consider eliminating or making major modifications to items that may provide unreliable data. The following items are of limited utility in analysis and may be unnecessarily confusing and/or burdensome to respondents:

  • Please assess if the follow factors are barriers to employment for you personally.

  • How much money did you personally earn in 2014?

Methodology

The JBS team conducted an assessment of the survey instrument including the following:

• Analysis of mode effects on items with potential for differential responses due to sensitivity, social desirability, or comprehension;

• Univariate descriptive statistics to ensure sufficient variability of responses; and

• Reliability and variability of scales.

Appendices A, B, and C present the results of statistical analyses, including frequencies and percentages of categorical responses, means and standard deviations of continuous responses, and internal consistency and variability of scales.

JBS carried out pilot testing in March 2015. Sixty-seven opportunity youth (OY) AmeriCorps members were recruited to pilot test the survey. Program staff conducted recruitment at eight of the nine grantee organizations that comprise the evaluation bundle. One grantee, City of Davenport, was unable to participate in pilot testing because the program only has members enrolled from June to August.

The 67 potential participants were randomized to take the survey in one of three modalities: proctored paper, telephone, or online. Program staff administered the survey to those assigned to take the proctored paper survey. Those assigned to the phone interview condition were called at least three times between March 10^th and 20^th.

Those assigned to the online survey were sent email invitations to complete the survey on SurveyMonkey. These participants were sent an initial email on March 10^th, inviting them to take the survey. Follow-up reminders were sent on March 13^th, 18^th, 20^th, 23^rd, and 25^th to individuals who had not taken the survey.

Cognitive interviews

The end of the survey included the option for respondents to enter a phone number where they could be reached for a brief conversation to help improve the survey experience for AmeriCorps members who would take the survey when it is officially launched. Similarly, at the end of telephone interviews, the interviewer asked respondents if they would be willing to provide feedback on the survey experience and obtained phone numbers from those who agreed to participate in a follow-up call. For the proctored survey, program staff asked pilot testers if they would be willing to participate in a follow-up call with JBS staff, and emailed JBS the phone numbers of respondents who agreed to participate. JBS completed cognitive interviews with nine respondents, three for each survey modality. Results of these cognitive interviews are summarized below under Cognitive Interview Results. Three participants were selected from each mode condition, for a total of nine interviewees.

JBS project staff asked questions regarding the following topics: demographics; education and employment history; perceived barriers to employment; self-efficacy and behaviors in the areas of education, career, and civic engagement; actions taken to use of social services; and involvement in the criminal justice system. Interviewers read specific items and response options back to respondents to remind them about the pertinent items. Interviewers asked respondents to talk about their responses to items considered more likely to pose challenges for them. Respondents were asked whether a question or the associated response options were unclear or confusing, why they had selected a particular response, and what they thought the question meant. JBS project staff asked additional probing questions whenever a respondent indicated a lack of clarity in a survey item or the associated response options.

The results of the survey and cognitive interviews are described in the next sections.

Survey Responses

Thirty-seven respondents completed at least part the survey for an overall response rate of 55 percent. For each survey mode, the response rates are as follows:

• Proctored: 17 out of 22 (77 percent)

• Online (email): 10 out of 23 (43 percent)

• Telephone Interview: 10 out of 22 (45 percent)

JBS project staff monitored response rates regularly for the online and phone surveys, and provided targeted feedback to grantees to encourage their members to complete the survey. JBS project staff also checked in with programs about the progress of the proctored survey during conference calls on March 18^th and 23^rd.

One respondent (three percent of those who began the survey) did not complete the survey. One other participant did not answer several of the early questions. This suggests that survey dropout will likely not pose a significant problem.

Survey participants were only required to answer three questions at the beginning of the survey designed to create a unique, confidential identifier that would be used to match surveys with the same individual at each stage of the data collection process (baseline, first follow up, and second follow up). For all of the remaining items, excluding those that were only asked of some participants conditioned on previous responses, there was minimal item-nonresponse.

The average participant took 20 minutes to complete the survey. Phone interviews took longer than other modalities, with an average time of 26.5 minutes compared to 17 minutes for the online survey, and 20 minutes for the proctored paper survey.

Analysis of Mode Effects

Though the small sample size was a somewhat limiting factor in detecting potential mode effects on participant responses, analyses were conducted on a subset of questions in order to detect whether there were substantial differences in responses based on modality. In particular, researchers were concerned with mode effects on comprehension and responses to potentially sensitive questions.

No effects of modality were found in response to questions pre-identified as sensitive or having high likelihood of exhibiting social desirability bias. However, there was a significant mode effect on the response to one question: “Are you currently employed?” Those answering online were significantly more likely to respond that they were employed than those answering via proctored paper or phone survey (see Appendix A). This was a potentially ambiguous question for pilot test respondents, as they may differ in whether they classify AmeriCorps service as employment. Participants assigned to the proctored survey or phone interview conditions had the opportunity to clarify this question. Information from proctors indicates that some members sought and received clarification regarding this question.

Item Variability

For most measures, there was sufficient variability in item responses. There were two items for which all respondents replied “No”: identification as primary caregiver of an adult and active military service. Cognitive interviews revealed some possible confusion about the term “primary caregiver of an adult,” but no confusion regarding active military service. Since both are measures intended to match participants and serve as controls, rather than outcomes, the lack of variability is not a concern. These items will be retained because, although they may have a low incidence, they will be useful in the matching process.

Among questions for which participants were given a checklist of responses, there were three checklist options that were never selected:

• How do you identify yourself in terms of ethnicity/race? Response: Other

• Are you currently using or visiting any of the following? Response: Mutual support or other assistance programs (e.g., AA, NA, AlAnon, grief support groups)

• Are you currently accessing any of the following federal or state government supports? Response: Other financial assistance (e.g., TANF, child care assistance programs)

Scale Reliability

All scales demonstrated satisfactory internal consistency, with Cronbach’s alpha ranges between .74 and .90 (see Appendix C).¹

Cognitive Interview Results

Nine opportunity youth AmeriCorps members from four programs completed the cognitive interview with a member of the JBS project team. Three cognitive interviews were completed for each of the three survey modalities. Cognitive interviews lasted approximately 20 minutes. Qualitative data from the nine cognitive interviews were supplemented with information reported by survey proctors on the proctoring worksheet.

Respondents did not feel that the survey was too long. The shortest completion time cognitive interviewees reported was 15 minutes and the longest reported time was 35 minutes; the average completion time reported for this group was 23 minutes. One respondent who reported taking 30 minutes to complete the telephone survey reported that if felt “a little long.” One respondent remarked that the purpose of the survey was “not clear.” Another respondent felt that some of the demographic questions, such as age and race/ethnicity, are “very personal” and may not be the best questions with which to lead.

The remainder of this section describes results for specific sets of survey items.

Confidential Tracking Code (3 items)

These three items asked respondents to create a confidential tracking code by entering: (a) the first three letters of the city or town in which they were born, (b) the number of letters in their last name, and (c) the first two letters of their mother’s first name. A few members did not know the answer to the third question. In future, the instructions for this item will be to enter “AA” when the answer is unknown.

Demographic information (7 items)

This set of items asked respondents for basic demographic information including date of birth, gender, parent/caregiver status, military service, ethnicity/race, education, city/town name, and zip code for city of residence. Most respondents found the demographic items to be clear and to have satisfactory response options. One respondent sought clarification from the survey proctor about the meaning of the question asking if they are the primary caregiver of a parent or other adult.

Education history (4 items)

This set of items asked respondents to indicate the highest level of education completed, month and year when high school diploma or GED was attained (if appropriate), month and year when last enrolled in school, and school most recently attended (high school, post-secondary, vocational, etc.). Most respondents found these items to be clear and to have satisfactory response options. One respondent asked the survey proctor to clarify whether “school most recently attended” included an educational program in which he was currently enrolled as part of his AmeriCorps activities.² Two respondents were confused by the instruction to “skip to question 11”—to avoid additional employment-related questions—after answering “no” to a question asking if they are currently employed.

Employment history (5 items)

These items asked respondents if they are currently employed (outside of AmeriCorps) and, if so, the type of employment they have (full-time or part-time) and how long they have held their current position. Respondents who indicated they are not currently employed were also asked if they are currently looking for work; pursuing school or training; unable to work due to disability; or engaged in a full-time or part-time volunteer position, internship, or apprenticeship. Respondents reported no difficulties with these items.

Perceived barriers to employment (11 items)

This matrix of items asked respondents to characterize a series of potential reasons for not finding employment as "Not a barrier to employment," "A barrier but can be overcome," or “A large barrier to employment." Two respondents were confused by the term "barrier to employment," but were able to answer the individual items in this matrix. Respondents reported no other difficulties with these items.

Personal income in 2014 (1 item)

One item asked respondents to identify how much money they personally earned in 2014. Several respondents noted that they had to think about the answer for several minutes. One survey proctor noted that two respondents struggled to recall how much they earned in 2014, and expressed doubt about the accuracy of their estimates. Respondents typically had to think about their income in terms of a weekly or monthly salary and then extrapolate to cover their period of employment in 2014.

Self-efficacy: solving problems and achieving personal goals (10 items)

This set of items asked respondents to assess their ability to overcome obstacles and solve problems to achieve personal goals. Items asked respondents to answer using a 5-point Likert-type scale (Strongly disagree; Disagree; Neither agree nor disagree; Agree; Strongly agree) to respond to statements about persevering in the face of adversity and relying on one's coping skills to overcome difficulties. Most respondents reported no difficulties with these items. However, one respondent had difficulty understanding the phrase "means and ways" in the statement, "If someone opposes me, I can find the means and ways to get what I want." Another respondent found this set of items "a little awkward" and remarked that they required substantial thought.

Addressing problems in the community (8 items)

This set of items asked respondents to indicate their ability to perform specific actions in the context of addressing a problem in the community. Actions included, creating a plan to address the problem, expressing one's views in front of a group, and contacting an elected official about the problem. Respondents used a 5-point scale (I definitely could do this; I probably could do this; Not sure, I could not do this; I definitely could not do this). Most respondents understood these items and could answer them easily. One respondent found these items difficult to answer without knowing the type of community problem being addressed.

Confidence to pursue educational and career goals (18 items)

This set of items asked respondents to assess their confidence to undertake actions to advance their educational and career goals, such as completing a college or trade school application, preparing a good resume, and successfully managing the job interview process. Respondents used a 5-point scale (No confidence at all; Very little confidence; Moderate confidence; Much confidence; Complete confidence). Most respondents were able to answer these items without difficulty, and some noted that they had already undertaken some of the actions listed. One respondent was unclear on the meaning of "obtain technical or vocational certification."

Community involvement (6 items)

These items consist of five statements to which respondents provided answers on a 5-point Likert-type scale ranging from strongly disagree to strongly agree. The statements are designed to gauge community attachment, awareness of community issues, and sense of obligation to contribute to the community. The sixth item asked respondents to indicate whether they can trust none, some, most, or all people in their neighborhood. Respondents reported no difficulties with these items.

Intentions to pursue further education and career training (4 items)

These items consist of four statements to which respondents may answer on a 5-point Likert-type scale ranging from strongly disagree to strongly agree. The statements assess whether an individual has clear intentions about their educational goals and career plans. Respondents reported no difficulties with these items.

Recent employment-related activity (2 items)

The first of these two items presented respondents with a list of employment-related activities (e.g., sending a resume or completed a job application; talking with a person employed in a field in which they are interested; enrolling in a college, trade school, or a certification course) and asked them to indicate which, if any, they have done in the last six months. The second item asked respondents if they have looked for part-time work, full-time work, an internship or apprenticeship, or a volunteer position in the last six months. Some respondents reported unfamiliarity with the terms “alternative school” and “FAFSA application”. Given that respondents are AmeriCorps members, most indicated that they had looked for a volunteer position in the last six months.

Use of social services (2 items)

The first of these two items presented respondents with a list of community-based services and resources (e.g., local employment development division; domestic violence center; mutual support or other assistance program, such as AA, NA, AlAnon, or grief support groups), and asks them to indicate which, if any, they currently use. The second items asked respondents to indicate if they are currently accessing federal or state government supports (food assistance, such as WIC or SNAP; health care assistance, such as Medicaid; housing assistance, such as housing vouchers; other financial or practical assistance, such as TANF or child care assistance programs). Some respondents were not familiar with response options listed in the first item, including local employment development division, housing center, adult school, and community college extension program).

Criminal history (2 items)

The first of these two items asked respondents if they have ever been convicted as an adult, or adjudicated as a juvenile offender, of any offense by either a civilian or military court, other than minor traffic violations (no/yes). The second item asked respondents if they are currently facing charges for any offense or on probation or parole (no/yes). One respondent indicated that he was uncomfortable answering these questions, although he did answer them. No respondents reported any difficulty with understanding these questions and selecting a response.

Limitations due to physical, mental, or emotional limitations (1 item)

This item asked respondents whether they are limited in any way in any activities because of physical, mental, or emotional problems (no/yes). Respondents reported no difficulties with understanding or answering this question.

Voting in last presidential election (2 items)

These two items asked respondents to indicate whether they were registered to vote in the last presidential election and whether they had voted in the last presidential election (Yes; No; No, not eligible to vote, Do not know). One respondent noted some confusion about whether to choose “No” or “No, not eligible,” as she was old enough to vote but had not registered. Another respondent reported that he felt tempted to give a positive response regardless of whether he had participated in the election.

Civic participation in last 12 months (5 items)

These five items asked respondents how frequently they have engaged in various civic activities in the last 12 months, such as participating in community organizations (school, religious, issue-based, recreational), helping to keep the community safe and clean, and donating money or goods to a cause or issue that they care about (Not at all; Less than once a month; Once a month; A few times a month; A few times a week; Every day). Respondents reported no difficulties with understanding or selecting an answer.

Programs and services (1 item)

The last item in the survey asked respondents if they are participating in AmeriCorps or similar national or community service programs, employment supports (e.g., job training), or educational supports (e.g., tutoring, GED classes, college enrollment assistance). Given that pilot-test respondents are AmeriCorps members, nearly everyone checked the first item. Some members also checked the items for employment and educational supports in light of AmeriCorps-based opportunities in which they are participating. One respondent specifically asked the survey proctor whether the educational and employment supports that AmeriCorps provides were included in this question and was (correctly) informed that they were not included.

Conclusions and Considerations for Survey and Administration Modification

Differential response rates based on mode-confirmed expectations from program leaders that the use of proctored paper surveys would lead to the highest response rates, and that these surveys should be used when feasible.

The results also indicated that comparable response rates could be obtained for phone interviews and online surveys. Rather than giving participants who cannot complete a proctored paper survey the option of choosing between phone interviews and online surveys, the researchers suggest using the online survey as a default option, and only use phone interviews with participants that do not have email addresses or do not respond to emailed invitations and emailed or texted reminders to complete the online survey.

The low response rates for both phone interviews and online surveys portend potential challenges in data collection for AmeriCorps members at the six month follow up and for comparison group members at all time-points. This corroborates the need for a comprehensive recruitment and retention plan that includes multiple strategies, including incentives and ongoing communication with participants.

The respondents who participated in cognitive interviews noted few difficulties in understanding and responding to survey items. In some cases, respondents had difficulties recalling specific information or understanding specific terms, as noted in Table 1.

Table 1. Cognitive Interview

Topical Area	Issue
Demographics	Comprehension: Primary caregiver of a parent or other adult
Education history	Applicability: Knowing whether “school most recently attended” included an educational program in which the respondent was currently enrolled as part of AmeriCorps activities Instruction to skip questions after answering “No” to a question
Perceived barriers to employment	Comprehension: Barrier to employment
Personal income in 2014	Recall accuracy: Remembering how much money earned in 2014 (Respondent had to think about his or her income in terms of a weekly or monthly salary and then extrapolate to cover period of employment in 2014.)
Self-efficacy: solving problems and achieving personal goals	Comprehension: “Means and ways" in the statement "If someone opposes me, I can find the means and ways to get what I want" Sensitivity: Items found to be "a little awkward"
Addressing problems in the community	Applicability: Items found to be difficult to answer without knowing the type of community problem being addressed
Confidence to pursue educational and career goals	Comprehension: Obtain technical or vocational certification
Recent employment-related activity	Comprehension: Alternative school, FAFSA application
Use of social services	Comprehension: Local employment development division, housing center, adult school, and community college extension program
Criminal history	Sensitivity: Discomfort in answering these questions (but not refusal)
Voting in last presidential election	Comprehension, Social desirability
Programs and services	Applicability: Understanding whether the educational and employment supports AmeriCorps provides are included in the question

Based on the results of the pilot testing, JBS will make minor revisions to the following items:

• Are you a primary caregiver of a parent or other adult?

  • Clarify question by giving examples of such an adult in the response option text (“e.g. sick or disabled relative”).

• How much confidence do you have that you could:

  • Obtain technical or vocational certification?

    • Clarify the definition of technical or vocational certification by giving examples in the response option text (“e.g., online school or trade school”).

• In the last 6 months, have you done any of the following?

  • Completed a FAFSA application (Free Application for Federal Student Aid)

    • Clarify FAFSA by spelling out the acronym in the response option.

• Are you currently using or visiting any of the following?

  • Local employment development division

    • Clarify the purpose of such an employment development division or center by adding the following language to the response option in brackets: “for unemployment insurance or for help with finding a job”.

  • Housing center

    • Clarify the function of such a center by adding the following language to the response option in brackets: “for help with finding housing”.

• Did you vote in the last presidential election?

  • “No, I was not eligible to vote” repetitive between Q.30 and Q.31.

    • Clarify the question by making Q.31 (Vote in last presidential election?) dependent on Q.30 (registered to vote?).

Additional modifications will be made to the following items:

• Pilot Q.10: Are you currently employed?

  • Question 10 modified to assess prior employment. This was done to both verify opportunity youth status and to assess recent employment, and to mitigate mode effects. Pilot test question 10 revised to become final instrument Q. 11a-c.

  • Text modified for AmeriCorps members’ version of the survey to instruct that AmeriCorps service does not qualify as employment.

• Pilot Q.10d: Please assess if the follow factors are barriers to employment for you personally.

  • Eliminated in final instrument due to comprehensibility.

• Pilot Q.11: How much money did you personally earn in 2014?

  • Revise question text to ask for an estimated wage more easily recalled from participants. The question on the final instrument (Q.12) will now read “In your most recent job, what was your hourly pay rate (in dollars)? ________”

Appendix A: Analysis of Mode Effects

The Freeman-Halton extension of Fisher’s exact test for 2x3 contingency tables, which is appropriate for small sample sizes, was used to examine the effect of survey modality on items that were identified as potentially threatening due to comprehension problems, sensitivity, or social desirability.

Threat #1: Comprehension

Significant mode effects found: Yes (p<.001)

Notes: Respondents answering online were much more likely to respond “Yes” to the question “Are you currently employed?” As all participants were AmeriCorps members, this is assumed to be a misunderstanding of the question and a misclassification of AmeriCorps service as employment.

Are you currently employed?
			Modality: Phone, online, or proctored			Total
			Phone	Online	Proctored
Are you currently employed?	No	Count	7	2	15	24
		% within Modality: Phone, online, or proctored	77.8%	20.0%	88.2%	66.7%
	Yes	Count	2	8	2	12
		% within Modality: Phone, online, or proctored	22.2%	80.0%	11.8%	33.3%

Threat #2: Sensitivity

Significant mode effects found: No.

Notes: Participants in the online condition, which offers the greatest level of anonymity, were the least likely to state that they had been involved with the criminal justice system. This runs counter to what would be expected if sensitivity were a threat.

Have you ever been convicted as an adult, or adjudicated as a juvenile offender, of any offense by either a civilian or military court, other than minor traffic violations?
			Modality: Phone, online, or proctored			Total
			Phone	Online	Proctored
Have you ever been convicted as an adult, or adjudicated as a juvenile offender, of any offense by either a civilian or military court, other than minor traffic violations?	No	Count	5	6	7	18
		% within Modality: Phone, online, or proctored	55.6%	66.7%	43.8%	52.9%
	Yes	Count	4	3	9	16
		% within Modality: Phone, online, or proctored	44.4%	33.3%	56.3%	47.1%

Are you currently facing charges for any offense or on probation or parole?
			Modality: Phone, online, or proctored			Total
			Phone	Online	Proctored
Are you currently facing charges for any offense or on probation or parole?	No	Count	7	8	13	28
		% within Modality: Phone, online, or proctored	77.8%	88.9%	76.5%	80.0%
	Yes	Count	2	1	4	7
		% within Modality: Phone, online, or proctored	22.2%	11.1%	23.5%	20.0%

Threat #3: Social desirability

Significant mode effects found: No.

Notes: Online respondents most often responded with “Yes” when asked if they had voted in the last presidential election. This runs counter to expectations of what would occur if there were mode effects due to social desirability.

Did you vote in the last presidential election?
			Modality: Phone, online, or proctored			Total
			Phone	Online	Proctored
Did you vote in the last presidential election?	No	Count	2	5	8	15
		% within Modality: Phone, online, or proctored	22.2%	55.6%	47.1%	42.9%
	No, I was not eligible to vote	Count	5	1	6	12
		% within Modality: Phone, online, or proctored	55.6%	11.1%	35.3%	34.3%
	Yes	Count	2	3	3	8
		% within Modality: Phone, online, or proctored	22.2%	33.3%	17.6%	22.9%

Appendix B: Univariate Pilot Survey Results

Descriptive Statistics
	N	Minimum	Maximum	Mean	Std. Deviation
Age (years)	34.0	16	33	21.1	3.6
$	27.0	0.0	15000.0	4707.4	3885.3
Valid N (listwise)	10.0

Gender
		Frequency	Percent	Valid Percent	Cumulative Percent
	Male	26	70.3	70.3	70.3
	Female	11	29.7	29.7	100.0
	Total	37	100.0	100.0
Are you a parent or primary caregiver of a child?
		Frequency	Percent	Valid Percent	Cumulative Percent
	No	31	83.8	83.8	83.8
	Yes, and all of my children live with me	5	13.5	13.5	97.3
	Yes, and none of my children live with me	1	2.7	2.7	100.0
	Total	37	100.0	100.0
Are you a primary caregiver of a parent or other adult?
		Frequency	Percent	Valid Percent	Cumulative Percent
	No	37	100.0	100.0	100.0
Have you served on active duty in the military?
		Frequency	Percent	Valid Percent	Cumulative Percent
	No	37	100.0	100.0	100.0

How do you identify yourself in terms of ethnicity/race? Please select all that apply.
		Frequency	Percent	Valid Percent	Cumulative Percent
	Black, not of Hispanic origin	18	48.6	100.0	100.0
	American Indian or Alaskan Native	1	2.7	100.0	100.0
	White, not of Hispanic origin	19	51.4	100.0	100.0
	Other	0	0.0
	Hispanic	2	5.4	100.0	100.0
Please check the highest level of education that you have completed.
		Frequency	Percent	Valid Percent	Cumulative Percent
	Some high school	14	37.8	38.9	38.9
	High school diploma or GED	18	48.6	50.0	88.9
	Some college or post-secondary technical, vocational, or app	3	8.1	8.3	97.2
	4-year college degree (BA, BS)	1	2.7	2.8	100.0
	Total	36	97.3	100.0
Missing	System	1	2.7
Total		37	100.0
What school did you most recently attend?
		Frequency	Percent	Valid Percent	Cumulative Percent
	High school	23	62.2	65.7	65.7
	Post-secondary vocational/technical/alternative school	4	10.8	11.4	77.1
	Community college	7	18.9	20.0	97.1
	4-year college	1	2.7	2.9	100.0
	Total	35	94.6	100.0
Missing	System	2	5.4
Total		37	100.0

Are you currently employed?
		Frequency	Percent	Valid Percent	Cumulative Percent
	No	24	64.9	66.7	66.7
	Yes	12	32.4	33.3	100.0
	Total	36	97.3	100.0
Missing	System	1	2.7
Total		37	100.0
Please describe your current employment status:
		Frequency	Percent	Valid Percent	Cumulative Percent
	Part time 21-39 hours per week	8	21.6	66.7	66.7
	Full-time 40 or more hours per week	4	10.8	33.3	100.0
	Total	12	32.4	100.0
Missing	System	25	67.6
Total		37	100.0
How long have you held your current position?
		Frequency	Percent	Valid Percent	Cumulative Percent
	Less than 1 month	2	5.4	16.7	16.7
	1 to 3 months	1	2.7	8.3	25.0
	4 to 6 months	5	13.5	41.7	66.7
	More than 6 months	4	10.8	33.3	100.0
	Total	12	32.4	100.0
Missing	System	25	67.6
Total		37	100.0
Please describe your current employment status:
		Frequency	Percent	Valid Percent	Cumulative Percent
	Pursuing school or training instead of work	1	2.7	11.1	11.1
	Looking for work	5	13.5	55.6	66.7
	Not looking for work	1	2.7	11.1	77.8
	Engaged in full-time volunteer position, internship, or appr	2	5.4	22.2	100.0
	Total	9	24.3	100.0
Missing	System	28	75.7
Total		37	100.0
Have you ever done any of the following?
		Frequency	Percent	Valid Percent	Cumulative Percent
	A paid part-time job	25	67.6	100.0	100.0
	A paid full-time job	19	51.4	100.0	100.0
	A paid internship	5	13.5	100.0	100.0
	An unpaid internship	5	13.5	100.0	100.0
	Volunteer work that was not part of a requirement for high s	21	56.8	100.0	100.0
	Received a stipend for attending a program (note: does not i	7	18.9	100.0	100.0
	Baby-sitting, yard-work or chores that you were paid for by	21	56.8	100.0	100.0
In the last 6 months, have you done any of the following?
		Frequency	Percent	Valid Percent	Cumulative Percent
	Sent in a resume or completed a job application	23	62.2	100.0	100.0
	Written or revised your resume	19	51.4	100.0	100.0
	Interviewed for a job	21	56.8	100.0	100.0
	Contacted a potential employer	20	54.1	100.0	100.0
	Talked with a person employed in a field you are interested	18	48.6	100.0	100.0
	Taken a GED test	8	21.6	100.0	100.0
	Completed a course in high school, college, or an alternative school	8	21.6	100.0	100.0
	Completed a college or trade school application	5	13.5	100.0	100.0
	Completed a FAFSA application (Free Application for Federal	9	24.3	100.0	100.0
	Enrolled in a college, trade school, or a certification course	6	16.2	100.0	100.0
	None of the above	2	5.4	100.0	100.0

In the last 6 months, have you looked for any of the following?
		Frequency	Percent	Valid Percent	Cumulative Percent
	Full-time work	19	51.4	100.0	100.0
	Part-time work	21	56.8	100.0	100.0
	Internship or apprenticeship	7	18.9	100.0	100.0
	Volunteer position	8	21.6	100.0	100.0
	None of the above	5	13.5	100.0	100.0
Are you currently using or visiting any of the following?
		Frequency	Percent	Valid Percent	Cumulative Percent
	Local employment development division	3	8.1	100.0	100.0
	Housing center	2	5.4	100.0	100.0
	Job center	2	5.4	100.0	100.0
	Domestic violence center	1	2.7	100.0	100.0
	Food bank	1	2.7	100.0	100.0
	Community health clinic	5	13.5	100.0	100.0
	Adult school / community college extension programs	5	13.5	100.0	100.0
	Mutual support or other assistance programs	0	0.0
	None of the above	7	18.9	100.0	100.0
Are you currently accessing any of the following federal or state government supports?
		Frequency	Percent	Valid Percent	Cumulative Percent
	Food assistance (such as WIC or SNAP)	20	54.1	100.0	100.0
	Health care assistance (such as Medicaid)	12	32.4	100.0	100.0
	Housing assistance (such as housing vouchers)	1	2.7	100.0	100.0
	Other financial assistance	0	0.0

Have you ever been convicted as an adult, or adjudicated as a juvenile offender, of any offense by either a civilian or military court, other than minor traffic violations?
		Frequency	Percent	Valid Percent	Cumulative Percent
	No	18	48.6	52.9	52.9
	Yes	16	43.2	47.1	100.0
	Total	34	91.9	100.0
Missing	System	3	8.1
Total		37	100.0
Are you currently facing charges for any offense or on probation or parole?
		Frequency	Percent	Valid Percent	Cumulative Percent
	No	28	75.7	80.0	80.0
	Yes	7	18.9	20.0	100.0
	Total	35	94.6	100.0
Missing	System	2	5.4
Total		37	100.0
Are you limited in any way in any activities because of physical, mental, or emotional problems?
		Frequency	Percent	Valid Percent	Cumulative Percent
	No	31	83.8	88.6	88.6
	Yes	4	10.8	11.4	100.0
	Total	35	94.6	100.0
Missing	System	2	5.4
Total		37	100.0
Were you registered to vote in the last presidential election?
		Frequency	Percent	Valid Percent	Cumulative Percent
	No	13	35.1	37.1	37.1
	No, I was not eligible to vote	7	18.9	20.0	57.1
	Yes	15	40.5	42.9	100.0
	Total	35	94.6	100.0
Missing	System	2	5.4
Total		37	100.0

Did you vote in the last presidential election?
		Frequency	Percent	Valid Percent	Cumulative Percent
	No	15	40.5	42.9	42.9
	No, I was not eligible to vote	12	32.4	34.3	77.1
	Yes	8	21.6	22.9	100.0
	Total	35	94.6	100.0
Missing	System	2	5.4
Total		37	100.0
What programs are you participating in or services are you receiving?
		Frequency	Percent	Valid Percent	Cumulative Percent
	AmeriCorps or similar national or community service program	34	91.9	100.0	100.0
	Employment supports (such as job training)	7	18.9	100.0	100.0
	Educational supports (such as tutoring, GED classes, college	5	13.5	100.0	100.0
Modality: Phone, online, or proctored
		Frequency	Percent	Valid Percent	Cumulative Percent
	Phone	10	27.0	27.0	27.0
	Online	10	27.0	27.0	54.1
	Proctored	17	45.9	45.9	100.0
	Total	37	100.0	100.0

Appendix C: Scale Reliability

Scale
	# Items	Cronbach's alpha
General self-efficacy	10	0.90
Community self-efficacy	8	0.80
Work self-efficacy	9	0.81
Academic self-efficacy	6	0.81
Support self-efficacy	3	0.74
Sense of community	5	0.82
Goals - work	3	0.80
Community engagement	5	0.81

1 For the purposes of assessing scale reliability and validity, missing values were replaced with mean values.

2 Some programs assist members in obtaining a GED or high school diploma.

Draft: 4/3/2015 Page 1