Staff survey

INSTRUMENT 4

survey of program staff

This page intentionally left blank for double sided copying

OMB No.: xxxx-xxxx

Expiration date: xx/xx/xxxx

Staff Survey

STREAMS Evaluation

February 2016

The Paperwork Reduction Act Statement: Public reporting burden for this collection of information is estimated to average 30 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: XXX ATTN: XXX (xxxx--xxxx). Do not return the completed form to this address.

INTRODUCTION

The Office of Planning, Research, and Evaluation within the U.S. Department of Health and Human Services, Administration for Children and Families (ACF) has contracted with Mathematica Policy Research to complete an evaluation of programs offered by selected Healthy Marriage/Relationship Education (HMRE) grantees funded by the Office of Family Assistance. The evaluation will describe the interventions that grantees put into place, the resources available to support them, and their impacts.

You are asked to complete this survey because you were identified as a staff member who works directly with HMRE participants. Your participation is important to helping us understand the characteristics of the staff and organizations implementing HMRE programs.

Participation in the survey is completely voluntary and you may choose to skip any question. Your responses will be kept private and used only for research purposes. They will be combined with the responses of other staff and no individual names will be reported. While there are no direct benefits to participants, your participation will help the U.S. Department of Health and Human Services learn how to better teach youth and adults healthy relationship skills. There is minimal risk related to taking part in this study. In the unlikely event of a data breach, your participation in the evaluation could become known.

The length of this survey is different for different people, but on average it should take about 30 minutes. Not all response options may apply to you or your organization. Please choose the best answer to each question. You may also choose not to answer any question.

If you have any questions about the survey, please contact the team at Mathematica by calling

1-xxx-xxx-xxxxx (toll-free) or emailing xxxxxxx@mathematica-mpr.com.

Thank you for taking part in this survey. By completing the survey and submitting your responses, you are confirming that you understand the information you provide will be kept private, used only for research purposes, and that your answers will be combined with the responses of other staff so that no individuals are identified. Please print a copy of this consent screen for your future reference.

I1. Could you please confirm whether you work for [HMRE PROGRAM] at [ORGANIZATION]?

MARK ONE ONLY

1 □ Yes, I work for [HMRE PROGRAM] at [ORGANIZATION]

0 □ No Thanks for your time, we do not need you to complete the survey.

A. YOUR WORK ROLE AND EXPERIENCE

A1. What is your current job title at [ORGANIZATION]?

d □ Don’t know

A2. How long have you been employed at [ORGANIZATION]?

Please include the total time you have been employed at the organization, not just the time you have been in your current position.

| | | MONTHS OR | | | YEARS

d □ Don’t know

A3. How long have you been providing services to participants in HMRE programs?

Please include all work you have done for current and past organizations related to providing services to participants in HMRE programs.

0 □ I have not done any work related to providing services to participants of HMRE programs

| | | MONTHS OR | | | YEARS

d □ Don’t know

A4. The next questions are about your work activities at [ORGANIZATION]. How often do you do each of the following activities as part of your job?

Please answer thinking about your job as a whole, not just activities related to the HMRE program.

	SELECT ONE RESPONSE PER ROW
	MORE THAN MONTHLY	MONTHLY	LESS THAN MONTHLY	NOT AT ALL	DON’T KNOW
a. Recruit potential program participants	1 □	2 □	3 □	4 □	d □
b. Screen potential participants to determine which services they are eligible to receive	1 □	2 □	3 □	4 □	d □
c. Conduct participant intake, enroll participants and inform them of their program status after random assignment	1 □	2 □	3 □	4 □	d □
d. Conduct assessments of participant’s needs or skills	1 □	2 □	3 □	4 □	d □
e. Provide healthy relationship education classes	1 □	2 □	3 □	4 □	d □
f. Provide parenting education services	1 □	2 □	3 □	4 □	d □
g. Provide case management services	1 □	2 □	3 □	4 □	d □
h. Help participants find work	1 □	2 □	3 □	4 □	d □
i. Provide financial management or financial literacy education	1 □	2 □	3 □	4 □	d □
j. Monitor how screening and assessment tools are used, and assess the quality of those tools	1 □	2 □	3 □	4 □	d □
k. Monitor how healthy relationship education classes are conducted, and assess the quality of those classes	1 □	2 □	3 □	4 □	d □
l. Monitor how other program services are delivered, and assess the quality of those services	1 □	2 □	3 □	4 □	d □
m. Coordinate services for participants with other partner agencies	1 □	2 □	3 □	4 □	d □
n. Manage or supervise other staff at your organization	1 □	2 □	3 □	4 □	d □
o. Train other staff at your organization	1 □	2 □	3 □	4 □	d □
p. Conduct administrative activities (for example, paperwork)	1 □	2 □	3 □	4 □	d □
q. Other activities (specify)	1 □	2 □	3 □	4 □	d □

A5. Before coming to work for [ORGANIZATION], did you ever have any of the following experiences?

	SELECT ONE RESPONSE PER ROW
	YES	NO	DON’T KNOW
a. Working with youth	1 □	0 □	d □
b. Working with single adults	1 □	0 □	d □
c. Working with adult couples	1 □	0 □	d □
d. Delivering relationship education	1 □	0 □	d □
e. Providing case management services	1 □	0 □	d □
f. Providing parenting education	1 □	0 □	d □
g. Helping individuals prepare for or obtain employment	1 □	0 □	d □
h. Helping individuals with financial management or economic stability	1 □	0 □	d □
i. Working with individuals who have experienced domestic violence	1 □	0 □	d □
j. Recruiting or enrolling participants into a voluntary program	1 □	0 □	d □
k. Coordinating services with staff in partner agencies	1 □	0 □	d □
l. Managing or supervising staff	1 □	0 □	d □
m. Any other relevant experience (specify)	1 □	0 □	d □

The next question asks about your caseload. Please answer thinking about your own caseload, not the caseload of [HMRE Program] as a whole.

A6. Currently, about how many participants are on your caseload?

| | | PARTICIPANTS

0 □ I don’t carry a caseload

d □ Don’t know

B. USING an evidence-based program

Aarons - EBPAS

B1. The following statements give feelings someone might have about using a new intervention. By an intervention, we mean a program with specific components or guidelines that are meant to be followed in a set way. An intervention could be described in a manual and protocols or explained in a training.

How much do you agree with each statement about using new interventions?

	SELECT ONE RESPONSE PER ROW
	NOT AT ALL	TO A SLIGHT EXTENT	TO A MODERATE EXTENT	TO A GREAT EXTENT	TO A VERY GREAT EXTENT	DON’T KNOW
a. I like to use new interventions to help my clients	0 □	1 □	2 □	3 □	4 □	d □
b. I am willing to try new types of interventions even if I have to follow a program manual	0 □	1 □	2 □	3 □	4 □	d □
c. I know better than academic researchers how to serve my clients	0 □	1 □	2 □	3 □	4 □	d □
d. I am willing to use new and different types of interventions developed by researchers	0 □	1 □	2 □	3 □	4 □	d □
e. Research based interventions are not practical for programs like mine	0 □	1 □	2 □	3 □	4 □	d □
f. Program experience is more important than using interventions outlined in a manual	0 □	1 □	2 □	3 □	4 □	d □
g. I would not use a intervention that must follow a manual and prescribed protocols	0 □	1 □	2 □	3 □	4 □	d □
h. I would try a new intervention even if it were very different from what I am used to doing	0 □	1 □	2 □	3 □	4 □	d □

Aarons - EBPAS

B2. If you received training in an intervention that was new to you, how likely would you be to use it if…

	SELECT ONE RESPONSE PER ROW
	NOT AT ALL	TO A SLIGHT EXTENT	TO A MODERATE EXTENT	TO A GREAT EXTENT	TO A VERY GREAT EXTENT	DON’T KNOW
a. it seemed right for your participants?	0 □	1 □	2 □	3 □	4 □	d □
b. it “made sense” to you?	0 □	1 □	2 □	3 □	4 □	d □
c. it was required by your supervisor?	0 □	1 □	2 □	3 □	4 □	d □
d. it was required by [ORGANIZATION]?	0 □	1 □	2 □	3 □	4 □	d □
e. it was required by your state?	0 □	1 □	2 □	3 □	4 □	d □
f. it was being used by colleagues who were happy with it?	0 □	1 □	2 □	3 □	4 □	d □
g. you felt you had enough training to use it correctly?	0 □	1 □	2 □	3 □	4 □	d □

Panzano – Implementation Climate Assessment, excerpted from RPG

B3. Organizations have a “personality” that is reflected in the day to day work of the organization and the way staff members view their work. These items ask about how [HMRE CURRICULUM] might be used in organizations. Thinking about [ORGANIZATION], please indicate how much you disagree or agree with each statement. Within the past six months…

	SELECT ONE RESPONSE PER ROW
	STRONGLY DISAGREE	DISAGREE	AGREE	STRONGLY AGREE	DOES NOT EXIST IN OUR ORGANIZATION	DON’T KNOW
a. Staff members were adequately trained to use [HMRE CURRICULUM] at this organization	1 □	2 □	3 □	4 □	n □	d □
b. Top administration strongly supported the use of [HMRE CURRICULUM]	1 □	2 □	3 □	4 □	n □	d □
c. Staff members got positive feedback and/or recognition for their efforts to use [HMRE CURRICULUM]	1 □	2 □	3 □	4 □	n □	d □
d. Top administrators minimized obstacles and barriers to using[HMRE CURRICULUM] at this organization	1 □	2 □	3 □	4 □	n □	d □
e. This organization created clear and specific goals related to using [HMRE CURRICULUM]	1 □	2 □	3 □	4 □	n □	d □
f. There were systems to track how well [HMRE CURRICULUM] was used at [ORGANIZATION]	1 □	2 □	3 □	4 □	n □	d □
g. Training and technical assistance were readily available to staff members involved in using [HMRE CURRICULUM]	1 □	2 □	3 □	4 □	n □	d □
h. Staff members had the resources they needed to use [HMRE CURRICULUM] as designed	1 □	2 □	3 □	4 □	n □	d □
i. Staff members were encouraged to express concerns that came up in the course of using [HMRE CURRICULUM]	1 □	2 □	3 □	4 □	n □	d □

The next questions in this section are about your experiences using [HMRE CURRICULUM].

If you do not facilitate or supervise the use of [HMRE CURRICULUM], please go to question C1.

If you facilitate or supervise the use of [HMRE CURRICULUM], please continue to question B4.

B4. When using a program, organizations often make changes to meet the needs of participants, the timeline, available resources, or some other reason. Has [ORGANIZATION] changed [HMRE CURRICULUM] for any reason?

1 □ Yes

0 □ No

d □ Don’t know

B5. What kinds of changes were made to [HMRE CURRICULUM]?

MARK ALL THAT APPLY

1 □ Changed policies or procedures

2 □ Changed the sequence of sessions

3 □ Increased the number of sessions

4 □ Decreased the number of sessions

5 □ Changed the length of sessions

6 □ Changed the target population

7 □ Changed program content

8 □ Changed for cultural relevance

9 □ Other (specify)

d □ Don’t know

Adaptations scale. Moore, J., Bumbarger, B., Rhoades Cooper, B. (2013, April 19). Examining adaptations of evidence-based programs in natural contexts. Journal of Primary Prevention.

B6. There are several possible reasons why an organization might choose to make changes to a program. To what extent did the following factors contribute to any changes being made to [HMRE CURRICULUM]?

	SELECT ONE RESPONSE PER ROW
	NOT AT ALL				PRIMARY REASON FOR CHANGE	DON’T KNOW
a. Difficulty recruiting participants	1 □	2 □	3 □	4 □	5 □	d □
b. Difficulty retaining or engaging participants	1 □	2 □	3 □	4 □	5 □	d □
c. Difficulty finding the right staff	1 □	2 □	3 □	4 □	5 □	d □
d. Lack of or limited resources (such as space or time)	1 □	2 □	3 □	4 □	5 □	d □
e. Lack of time or competing demands on time	1 □	2 □	3 □	4 □	5 □	d □
f. Resistance from staff	1 □	2 □	3 □	4 □	5 □	d □
g. Need for a more culturally appropriate program	1 □	2 □	3 □	4 □	5 □	d □
h. Requests for changes by participants	1 □	2 □	3 □	4 □	5 □	d □

C. SUPERVISION, COACHING, AND SUPPORT

The next questions ask about coaching or supervision you may receive as a staff member for [HMRE PROGRAM]. If you have more than one supervisor, please answer these questions about the supervisor you work with the most in the [HMRE PROGRAM].

C1. Is there at least one person at [ORGANIZATION] whom you regard as your supervisor?

MARK ONE ONLY

1 □ Yes

0 □ No

d □ Don’t know

C2. In the past 12 months, how often did you have formal, one-on-one supervision meetings?

MARK ONE ONLY

1 □ Never

2 □ Daily

3 □ Weekly

4 □ Twice per month

5 □ Monthly

6 □ Once every few months

7 □ Yearly

d □ Don’t know

C3. In the past 12 months, how often did you have group supervision meetings with other staff members?

MARK ONE ONLY

1 □ Never

2 □ Daily

3 □ Weekly

4 □ Twice per month

5 □ Monthly

6 □ Once every few months

7 □ Yearly

d □ Don’t know

Dickinson and Painter

C4. Please read the following statements and decide how strongly you disagree or agree with each statement. My supervisor…

	SELECT ONE RESPONSE PER ROW
	STRONGLY DISAGREE	DISAGREE	SOMEWHAT DISAGREE	SOMEWHAT AGREE	AGREE	STRONGLY AGREE	DON’T KNOW
a. encourages staff to spend time mentoring new employees	1 □	2 □	3 □	4 □	5 □	6 □	d □
b. encourages staff to help each other with work problems	1 □	2 □	3 □	4 □	5 □	6 □	d □
c. cares about me as a person	1 □	2 □	3 □	4 □	5 □	6 □	d □
d. provides emotional support to me in difficult situations with HMRE program participants	1 □	2 □	3 □	4 □	5 □	6 □	d □
e. is appropriately flexible when it comes to applying rules	1 □	2 □	3 □	4 □	5 □	6 □	d □
f. has an attitude that helps me be enthusiastic about working in social services	1 □	2 □	3 □	4 □	5 □	6 □	d □
g. supports me in balancing the demands of my job with my personal life	1 □	2 □	3 □	4 □	5 □	6 □	d □
h. provides the help I need to do my job	1 □	2 □	3 □	4 □	5 □	6 □	d □
i. knows effective ways to work with HMRE program participants	1 □	2 □	3 □	4 □	5 □	6 □	d □
j. is willing to help me complete difficult tasks	1 □	2 □	3 □	4 □	5 □	6 □	d □
k. encourages creative solutions	1 □	2 □	3 □	4 □	5 □	6 □	d □
l. reinforces the training I receive	1 □	2 □	3 □	4 □	5 □	6 □	d □
m. helps me learn and improve	1 □	2 □	3 □	4 □	5 □	6 □	d □
n. is available when I ask for help	1 □	2 □	3 □	4 □	5 □	6 □	d □
o. has expectations for my work that are challenging but reasonable	1 □	2 □	3 □	4 □	5 □	6 □	d □
p. gives me clear feedback on my job performance	1 □	2 □	3 □	4 □	5 □	6 □	d □
q. has helped staff develop into an effective team	1 □	2 □	3 □	4 □	5 □	6 □	d □

C5-C7 developed for STREAMS

C5. Coaching helps staff be successful in their role through supervision, teaching, assessment, reflective feedback, and emotional support.

Is there someone either at your [ORGANIZATION] or outside of your [ORGANIZATION] who provides you coaching to support your work on the HMRE program?

MARK ONE ONLY

1 □ Yes

0 □ No

d □ Don’t know

C6. Who provides you with coaching support?

MARK ALL THAT APPLY

1 □ My supervisor coaches me.

2 □ I receive coaching from a staff member at my organization that is not my supervisor.

3 □ I receive coaching from an individual outside of my organization.

d □ Don’t know

C7. In the past 12 months, how often did you interact with the person providing you with coaching?

MARK ONE ONLY

1 □ Never

2 □ Daily

3 □ Weekly

4 □ Twice per month

5 □ Monthly

6 □ Once every few months

7 □ Yearly

d □ Don’t know

C8. Please read the following statements and rate how dissatisfied or satisfied you are with each with regard to [HMRE CURRICULUM]. Overall, how satisfied are you that...

Developed in consultation with Allison Metz from NIRN, adapted from RPG

	SELECT ONE RESPONSE PER ROW
	VERY DISSATISFIED	SLIGHTLY DISSATISFIED	NEITHER SATISFIED NOR DISSATISFIED	SLIGHTLY SATISFIED	VERY SATISFIED	DON’T KNOW
a. the information you received when being hired reflects the work you are being asked to do?	1 □	2 □	3 □	4 □	5 □	d □
b. the training you received prepared you to work effectively with participants?	1 □	2 □	3 □	4 □	5 □	d □
c. the coaching you receive is improving your skills and abilities to work effectively with participants?	1 □	2 □	3 □	4 □	5 □	d □
d. the challenges you encounter in providing effective services are understood in your organization?	1 □	2 □	3 □	4 □	5 □	d □
e. the challenges you encounter in providing effective services are understood by the HMRE program leadership?	1 □	2 □	3 □	4 □	5 □	d □
f. the challenges you encounter in providing effective services are being actively addressed?	1 □	2 □	3 □	4 □	5 □	d □
g. your immediate supervisor helps you develop your [HMRE curriculum] skillset?	1 □	2 □	3 □	4 □	5 □	d □
h. your organization’s administrators provide the support you need to work effectively with participants?	1 □	2 □	3 □	4 □	5 □	d □

D. TRAINING OPPORTUNITIES

The next questions ask about training activities you have attended as a staff member at [ORGANIZATION]. These training may have been held at [ORGANIZATION] or outside of [ORGANIZATION].

D1. During the past 12 months, have you attended any training activities related to your work, either at your workplace or somewhere else?

1 □ Yes

0 □ No

d □ Don’t know

D2. Why did you attend these training activities?

	SELECT ONE RESPONSE PER ROW
	YES	NO	DON’T KNOW
a. Required by your employer, specifically because of the [HMRE program] program	1 □	0 □	d □
b. Required by your employer, not specifically because of the [HMRE program] program	1 □	0 □	d □
c. Interest in topics	1 □	0 □	d □
d. To work towards additional credentials, certificates, or degrees	1 □	0 □	d □
e. To update my skills	1 □	0 □	d □
f. Other (specify)	1 □	0 □	d □

D3. Thinking about the content of the training activities you have attended during the past 12 months, how helpful was this training in guiding how you do your work?

MARK ONE ONLY

1 □ Not at all helpful

2 □ A little helpful

3 □ Somewhat helpful

4 □ Very helpful

5 □ Extremely helpful

d □ Don’t know

D4. During the past 12 months, did you attend training activities about the following topics?

	SELECT ONE RESPONSE PER ROW
	YES	NO	DON’T KNOW
a. Policy and procedures for the [HMRE PROGRAM ] program	1 □	0 □	d □
b. Healthy relationship education curriculum	1 □	0 □	d □
c. Workforce development	1 □	0 □	d □
d. Parenting or child development	1 □	0 □	d □
e. Youth development	1 □	0 □	d □
f. Child support	1 □	0 □	d □
g. Domestic violence	1 □	0 □	d □

D5. Please think about the training you have received on each of the following topics during the past 12 months at [ORGANIZATION]. How much has the training in each area helped you in your staff role?

		SELECT ONE RESPONSE PER ROW
	NOT AT ALL HELPFUL	A LITTLE HELPFUL	SOMEWHAT HELPFUL	VERY HELPFUL	EXTREMELY HELPFUL	DON’T KNOW	DID NOT RECEIVE TRAINING ON THIS TOPIC
a. Policy and procedures for the [HMRE PROGRAM ] program	0 □	1 □	2 □	3 □	4 □	d □	9 □
b. Healthy relationship education curriculum	0 □	1 □	2 □	3 □	4 □	d □	9 □
c. Workforce development	0 □	1 □	2 □	3 □	4 □	d □	9 □
d. Parenting or child development	0 □	1 □	2 □	3 □	4 □	d □	9 □
e. Youth development	0 □	1 □	2 □	3 □	4 □	d □	9 □
f. Child support	0 □	1 □	2 □	3 □	4 □	d □	9 □
g. Domestic violence	0 □	1 □	2 □	3 □	4 □	d □	9 □

E. ORGANIZATIONAL CLIMATE

Dickinson and Painter

E1. Please read the following statements and decide how strongly you disagree or agree with each statement with regard to [ORGANIZATION].

	SELECT ONE RESPONSE PER ROW
	STRONGLY DISAGREE	DISAGREE	SOMEWHAT DISAGREE	SOMEWHAT AGREE	AGREE	STRONGLY AGREE	DON’T KNOW
a. The mission of this organization is clear to me	1 □	2 □	3 □	4 □	5 □	6 □	d □
b. My work reflects the organization’s purpose	1 □	2 □	3 □	4 □	5 □	6 □	d □
c. I feel good about what this organization does for participants	1 □	2 □	3 □	4 □	5 □	6 □	d □
d. In this organization, there is more emphasis on the quality of services than on the number of participants served	1 □	2 □	3 □	4 □	5 □	6 □	d □
e. I am satisfied with the salary I receive from this organization	1 □	2 □	3 □	4 □	5 □	6 □	d □
f. I am paid fairly considering my education and training	1 □	2 □	3 □	4 □	5 □	6 □	d □
g. I am paid fairly considering the responsibilities I have	1 □	2 □	3 □	4 □	5 □	6 □	d □
h. I am satisfied with the physical work environment at this organization	1 □	2 □	3 □	4 □	5 □	6 □	d □
i. I am proud to tell others that I am part of this organization	1 □	2 □	3 □	4 □	5 □	6 □	d □
j. The administration shows concern for staff	1 □	2 □	3 □	4 □	5 □	6 □	d □
k. Employees of this organization are respected by other community professionals	1 □	2 □	3 □	4 □	5 □	6 □	d □
l. This organization is committed to my personal safety in the office	1 □	2 □	3 □	4 □	5 □	6 □	d □
m. This organization is committed to my personal safety when working off-site	1 □	2 □	3 □	4 □	5 □	6 □	d □
n. My professional opinions are respected in this organization	1 □	2 □	3 □	4 □	5 □	6 □	d □
o. I am able to give my input on the policies that effect how I do my job	1 □	2 □	3 □	4 □	5 □	6 □	d □
p. There are strong, positive relationships between this organization and other community resource providers	1 □	2 □	3 □	4 □	5 □	6 □	d □
q. I have the support to make work-related decisions when appropriate	1 □	2 □	3 □	4 □	5 □	6 □	d □
r. Organizational management shares leadership roles with staff	1 □	2 □	3 □	4 □	5 □	6 □	d □
s. This organization effectively responds to public criticism when it occurs	1 □	2 □	3 □	4 □	5 □	6 □	d □

F. DEMOGRAPHICS

These next questions ask about your background.

F1. Are you Hispanic or Latino?

MARK ONE ONLY

0 □ No

1 □ Yes

d □ Don’t know

F2. What is your race?

MARK ALL THAT APPLY

1 □ American Indian or Alaska Native

2 □ Asian

3 □ Black or African American

4 □ Native Hawaiian or other Pacific Islander

5 □ White

6 □ Other (specify)

d □ Don’t know

F3. What is the highest level of education you have completed?

MARK ONE ONLY

1 □ Did not complete high school or General Educational Development

2 □ High school diploma

3 □ General Educational Development

4 □ Some college/some postsecondary vocational courses

5 □ 2-year or 3-year college degree (Associate’s degree)

6 □ Vocational school diploma

7 □ 4-year college degree (Bachelor’s degree)

8 □ Some graduate work/no graduate degree

9 □ Graduate or professional degree (for example, MA, MBA, Ph.D., JD, or MD)

d □ Don’t know

F4. What is your profession or area of work?

MARK ALL THAT APPLY

1 □ Counseling

2 □ Education

3 □ Vocational rehabilitation

4 □ Juvenile justice

5 □ Psychology

6 □ Social work/human services

7 □ Medicine

8 □ Administration

9 □ Student

10 □ Other (specify)

11 □ None of these

d □ Don’t know

F5. Are you male or female?

1 □ Male

2 □ Female

F6. Is there anything else about your experiences implementing your HMRE program that you would like to add?

Thank you for completing the STREAMS Staff Survey!