On - line survey Leadership and Line / Other Frontline Staff

P articipant Survey Scales

Cross-site Evaluation of the National Training Initiative on Trauma-Informed Care (TIC) for Community-Based Providers from Diverse Service Systems

HHSP23320095624WC

September 4, 2014

Prepared for:
Adrienne Smith
OWH/OPS
200 Independence Ave S.W.
Room 728F
Washington, DC 20201

Submitted by:
Abt Associates
55 Wheeler St.
Cambridge, MA 02138

In Partnership with:
Rutgers’ School of Criminal Justice

1. Brief Literature Overview of the Use of Surveys to Assess Knowledge Uptake, Skills Acquisition, Attitudes and Beliefs 1

2. Introduction and Participant Demographic Information 5

3. Assessment of Values and Beliefs Scale 8

4. Assessment of Knowledge Scale 10

5. Assessment of Skills Scale 16

1. Brief Literature Overview of the Use of Surveys to Assess Knowledge Uptake, Skills Acquisition, Attitudes and Beliefs

The U.S. Department of Health and Human Services (HHS) Office of Women’s Health (OWH) National Training Initiative on Trauma Informed Care (TIC) for Community-Based Providers, referenced hereafter as the OWH Trauma Informed Care Training and Technical Assistance (TTA) initiative, employed an extensive curriculum of material on the: prevalence, conditions, and populations impacts of trauma; values and objectives of trauma informed care; and methods for implementing trauma informed care in health and human services programs. The training was provided in live sessions over a two-day period, with those in remote locations (Guam and Hawaii) participating by phone. In the two year implementation phase (2012 to 2014), participating organizations also received technical assistance on site or in telephonic sessions. At the end of the two-day training, participants were asked to complete a Participant Feedback Form. The TTA protocol did not include any pre- or post-assessments of participants’ knowledge, skills, attitudes and beliefs. Given the lack of a baseline measure, pre-test or post-test of knowledge, skills, attitudes, and beliefs, it is critical to assess knowledge uptake, skills development, and attitude and belief changes as part of the OWH TIC TTA Evaluation.

OWH TIC TTA participants will be asked to complete an online survey, which will be administered after the training and technical assistance program is complete. The survey is designed to gather demographic and professional information from participants, and collect responses to questions on survey scales devised to measure attitudes and beliefs, knowledge uptake and skills acquired as a result of the TIC TTA initiative. The survey employs binary, Likert, and other multiple choice scales, consistent with the literature on survey methods, summarized below. The evaluation will not be able, given the lack of pre-testing, to calculate a degree of change in the amount of knowledge each training participant has; therefore, a post-test-only design is appropriate. This type of investigational design measures experimental groups, and can compare control groups, after completion of a training program (Health Services Research Methods, 2014). It is suitable to use a post-test-only design when looking at whether or not participants have reached an identified outcome, such as adopting a certain value or belief or acquiring new information.

Five-point Likert scales (e.g., 1 = all the time, 2 = some of the time, 3 = rarely, 4 = I did not adopt this value or belief, 5 = I did this prior to the training) are proven to provide sufficient discrimination among levels so that accurate conclusions can be drawn from participant responses (Goodwin, 2009). Although Likert scales tend to yield more descriptive results than binary (i.e., yes or no) scales, there are times when using dichotomous items are preferred (i.e., when it is not necessary for survey or questionnaire respondents to discriminate between a range of options) (DeVellis, 2003). An article in The Canadian Journal of Program Evaluation describes the “process of developing measures to assess knowledge exchange outcomes using the dissemination of document addressing best practice in type 2 diabetes care as a specific example” (Skinner, 2007, p. 49). One of the ultimate goals of the research was to identify the measurement tool that proved to be the most effective at assessing whether or not knowledge uptake occurred. Since the tool was not intended to measure opinions, beliefs, or attitudes, but rather whether or not a subject acquired information, a binary scale was used.

Existing research indicates that knowledge uptake and skill acquisition, as well as changes in personal values and beliefs regarding a topic of interest, can reliably be assessed by examining participant responses on measurement scales administered after the training intervention. Through the use of appropriate and dependable experimental designs in addition to statistically sound measurements scales, reliable conclusions about the success (or failure) of an intervention can be drawn. Examples of studies employing these methods are described below.

The Public Health Agency of Canada recently conducted a study on the knowledge, attitudes, beliefs, and behaviors of older adults about pneumococcal immunization. Participants were administered a survey which assessed these topics, and multi-variable logistic regression was used to determine whether or not the factors were associated with pneumococcal vaccine receipt. The survey measured the subject’s knowledge of infection and immunization, individual beliefs regarding pneumococcal infection, and individual attitudes towards vaccines (Schneeberg et al., 2014). “Having been offered the pneumococcal vaccine by a health care provider, having been told by a healthcare provider about the pneumonia vaccine, and believing their healthcare provider thought receipt of the vaccine was a good idea were all positively associated with vaccine receipt” (Schneeberg et al., p.3). Both Likert and binary scales were used to measure knowledge, beliefs, and attitudes; Likert scales are very common in public health evaluation when the goal is to measure opinions, beliefs, and attitudes (DeVellis, 2003). They are typically used – and preferred over alternatives – with attitude, value, belief, and behavior items because they are more descriptive than other analyses.

A survey of general practitioners, psychiatrists, and internists examined different physicians’ attitudes towards suicidal behavior and their perceived competence to care for suicidal patients. “Five-point Likert scales were used to measure self-perceived competence, level of commitment, empathy, and irritation felt towards patients with somatic and psychiatric diagnoses” (Grimholt, Haavet, Jacobsen, Sandvik, & Ekeberg, 2014, p. 1). Responses for the five-point Likert scale items on the Understanding Suicidal Patients Scale (UPS) were scored from 1 (I agree completely) to 5 (I disagree completely). The Cronbach’s alpha calculated for the UPS-scale was 0.69, indicating acceptable reliability.

Over the past two decades, domestic violence training has been integrated into the curricula of most medical and postgraduate programs in the United States (Ramsay et al., 2012). In order to assess whether training and support programs increased (1) one’s ability to identify a women experiencing domestic violence, as well as, (2) one’s general knowledge of domestic violence, a cross-sectional survey was carried out by several researchers. Participants of the study included general practitioners (GPs) and practice nurses. The Physician Readiness to Manage Intimate Partner Violence Survey (PREMIS) is a valid, reliable, and well-recognized questionnaire in the United States (Ramsay et al., 2012). It includes sections on perceived preparedness and knowledge, actual knowledge, and opinions, among others. The PREMIS survey is a seven-point Likert scale, ranging from 1 = not prepared to 7 = well prepared for perceived preparation items; 1 = nothing to 7 = very much for perceived knowledge items; 1 = strongly disagree to 7 = strongly agree for opinion items. Training programs have been evaluated by comparing results of the PREMIS survey before and after program implementation. These programs have proven to be very effective, generally “increasing the knowledge, attitudes, and skills of students and clinicians in relation to domestic violence” (Ramsay et al., 2012, p. 648).

Most studies geared towards evaluating knowledge uptake, skill acquisition, and personal changes in values and beliefs focus on comparing the results of a questionnaire taken immediately after an intervention, and the results of the same questionnaire taken after a set amount of time (follow-up). In a follow-up assessment of Integrated Management of Neonatal and Childhood Illness (IMNCI) training, primary health care workers were trained on IMNCI using either conventional 8-day training or interrupted 5-day training methods, tested on composite knowledge and skills, and then re-tested 3 years after the initial training. The two scores were compared to see how much information had been retained after 3 years. Regardless of the training program (8- or 5-day), knowledge and skills acquired declined significantly (Venkatachalam, Kumar, Gupta, & Aggarwal, 2011). While no pre-test was conducted to detect how much a participant learned from the actual training program, it is evident that some knowledge was acquired and subsequently forgotten over time. It was concluded that the training program was effective, but refresher trainings should be held in order to prevent this decline.

In a study on the long-term effectiveness of parent education using the “baby oral health” model on the improvement of oral health of young children, caregivers were exposed to an interactive audio-visual aid about oral health and then asked to complete a questionnaire relating to the information presented. In this particular study, there were four groups: a study group enrolled at baseline, a study group follow-up at 18 months, a comparison group, and a comparison group enrolled at follow-up. Not surprisingly, parents in the study group acquired more knowledge about oral health than those in the control group. More importantly, however, is the amount of knowledge retained over the 18-month period by participants who were exposed to the health information; no significant loss of knowledge was identified in the study group follow up over the 18-month study period for the majority of items on the questionnaire (Kulkarni, 2013). The measurements of the follow-up study are indicative of the effectiveness of the intervention. Although this study contained a control group, it was not necessary. The most important comparison was not between the study group and the control group, but rather it was between the study group questionnaire responses immediately after the presentation and the study group questionnaire responses after 18 months.

References

DeVellis, R.F. (2003). Scale development: Theory and applications (2nd ed.). Thousand Oaks, CA: Sage.

Goodwin, C.J. (2009). Research in psychology: Methods and designs (6^th ed.). Hoboken, NJ: Wiley.

Grimhold, T., Haavet, O., Jacobsen, D., Sandvik, L., & Ekeberg, O. (2014). Perceived competence
and attitudes towards patients with suicidal behaviour: a survey of general practitioners, psychiatrists and internists. BMC Health Services Research, 14.

Health Services Research (HSR) Methods. (2014). Posttest only control group design. Academy Health. Retrieved from: http://www.hsrmethods.org/Glossary/Terms/P/Posttest%20Only%20Control%20Group%20Design.aspx

Kulkarni, G. (2013). Long-term effectiveness of parent educating using the “baby oral health” model on the improvement of oral health of young children. International Journal of Dentistry. Retrieved from: http://dx.doi.org/10.1155/2013/137048

Ramsay, J., Rutterford, C., Gregory, A., Dunne, D., Eldridge, S., Sharp, D., & Feder, G. (2012). Domestic violence: Knowledge, attitudes, and clinical practice of selected UK primary healthcare clinicians. British Journal of General Practice. 647-655.

Shneeberg, A., Bettinger, J., McNeil, S., Ward, B., Dionne, M., & Cooper, C. (2014). Knowledge, attitudes, beliefs, and behaviours of older adults about pneumococcal immunization, a Public Health Agency of Canada/Canadian Institutes of Health Research Influenza Research Network (PCIRN) investigation. BMC Public Health, 14.

Skinner, K. (2007). Developing a tool to measure knowledge exchange outcomes. The Canadian Journal of Program Evaluation, 22(1). 49-73.

Venkatachalam, J., Kumar, D., Gupta, M., & Aggarwal, A. (2011). Knowledge and skills of primary health care workers trained on integrated management of neonatal and childhood illness: Follow-up assessment 3 years after the training. Indian Journal of Public Health, 55(4). 298-302.

2. Introduction and Participant Demographic Information

As a participant in the U.S. Department of Health and Human Services (HHS) Office of Women’s Health (OWH) Trauma Informed Care (TIC) Training and Technical Assistance initiative provided in your HHS Region, you attended a two-day training event, followed, in some cases, by two technical assistance visits or calls. During the two-day training event, you were presented with a curriculum designed to share information about trauma, its prevalence and effects on individuals and communities. The curriculum also covered topics on the use of trauma informed care to support individuals with traumatic experiences in using health and human services in a manner that avoids creating additional traumatic effects on them. In this survey, which is part of the evaluation of the OWH TIC Training and Technical Assistance initiative, we ask you for some basic background information and then ask you questions about values and beliefs, as well as knowledge and skills, related to trauma and trauma informed care. Participation in this survey is voluntary. It is designed to be brief, and should take you approximately 25 to 30 minutes to complete. We appreciate your assistance in completing this survey and contributing to the evaluation. For any questions, we can be reached at: ENTER NAME AND NUMBER AND EMAIL ADDRESS.

Participant Demographic and Background Review

1. When and where did you participate in the OWH TIC Training and Technical Assistance?

• Region 1: Worcester, MA on July 30-31, 2012

• Region 2: Albany, NY on April 22-23, 2013

• Region 2: Newark, NJ on April 15-16, 2014

• Region 3: Philadelphia, PA on June 4-5, 2013

• Region 3: Henrico, VA on April 30-May 1, 2014

• Region 4: Columbia, SC on May 7-8, 2013

• Region 5: Brooklyn Center, MN on May 21-22, 2013

• Region 6: West Monroe, LA on January 31-February 1, 2013

• Region 6: Fort Worth, TX on June 10-11, 2014

• Region 7: Kansas City, MO on February 20, 2013

• Region 8: Denver, CO on June 19-20, 2013

• Region 9: San Francisco, CA on May 1-2, 2013

• Region 10: Seattle, WA on July 30-31, 2012

• Region 10: Nampa, ID on May 21-22, 2014

2. How old are you?

• Under 30 years

• 31 – 40 years

• 41 – 50 years

• Over 50 years

3. What is your gender?

• Female

• Male

• Other (Please specify)

4. What is the highest level of education you have completed?

• High School Diploma or GED Certification

• University or College Diploma

• Associates Degree

• Bachelor’s Degree

• Specialized or Professional Graduate or Post-graduate Degree (e.g. MBBS, MSc, MPH, PhD)

• Other (Please specify)

5. What type of organization do you currently work in?

• Peer or Consumer Advocacy or Support Organization

• Violence Against Women Program

• Reproductive Health Program

• Maternal and Child Health Program

• Minority Health Program

• Immigrant or Refugee Health Program

• Other Community Health Center or Clinic

• Community Behavioral Health Center or Clinic

• Community or General Hospital

• Specialty or Psychiatric Hospital

• Court or Victim/Witness Aid Program

• Criminal Justice Program

• Other (Please specify)

6. Which of these job categories best describes your current role or job?

• Consumer Affairs or Advocacy

• Support Services

• Clinical Services

• Case Management

• Clinical Management

• Program Management

• Administration

• Other (Please specify)

7. How long have you been working in this current position?

• < 1 year

• 1 – 4 years

• 5 – 9 years

• 10 – 14 years

• ≥ 15 years

8. How long have you been working in your organization?

• < 1 year

• 1 – 4 years

• 5 – 9 years

• 10 – 14 years

• ≥ 15 years

9 How long have you been working in your field?

• < 1 year

• 1 – 4 years

• 5 – 9 years

• 10 – 14 years

• ≥ 15 years

10. Prior to participating in the OWH TIC Training and Technical Assistance, had you attended a lecture, seminar, or workshop on trauma informed care?

• Yes (Please specify)

• No

11. After participating in the OWH TIC Training and Technical Assistance, have you attended a lecture, seminar, or workshop on trauma informed care?

• Yes (Please specify)

• No

3. Assessment of Values and Beliefs Scale

Values and Beliefs Assessment
	As a result of the TIC Training, to what extent do you agree with and use in your work any of the following values or beliefs?					Let us know if you held any of these values or beliefs before the TIC Training
	Strongly agree and it always guides my work	Agree and it sometimes guides my work		Disagree and it rarely guides my work	Strongly disagree and it never guides my work	I knew and understood this concept prior to the training
	1		2	3	4
01. Exposure to trauma is common.						Yes No
02. Trauma exposure has no boundaries and spans generations.						Yes No
03. Trauma is a defining and organizing experience.						Yes No
04. Trauma lives in the body.						Yes No
05. Recovery from trauma is possible for all.						Yes No
06. Trauma-informed practice creates conditions for safety, healing, and recovery.						Yes No
07. Peers and program participants are the experts in their own recovery.						Yes No
08. Healing happens in safe, stable, nurturing relationships.						Yes No
09. Healing from trauma is trans- formative (i.e., changes all aspects of a person’s life).						Yes No
10. Paths to healing are personal and diverse.						Yes No
11. Healing builds strength in the “broken places.”						Yes No
12. Informed choice is central to healing from trauma.						Yes No
13. Physical and emotional safety are essential for healing.						Yes No

4. Assessment of Knowledge Scale

Part One: Providing Your Feedback on the Content of the Training and TA

Please answer the questions in the text boxes below in your own words.

01. What were the most important things you learned from the OWH Trauma Informed Care Training?

02. What were the most important things you learned from the OWH Trauma Informed Care Technical Assistance?

03. What impact did the OWH Trauma Informed Care Training and TA have on your agencies' practice, service recipients and community?

04. Was there other information you would like to have learned about in the TIC TTA?

05. Do you have any changes you would recommend be made for future TIC Training and TA sessions?

Part Two: Rating Familiarity with Knowledge Domains
To what extent was the information about trauma and trauma informed care that was provided during the TIC Training new to you?
	Most or All of this information was new to me	Much of this information was new to me	I learned some new information	I knew much of this information already	I knew all of this information it added nothing	I do not know this information
	1	2	3	4	5	6
01. Violence against women, including prevalence, age, economic and cultural status of those affected
02. Traumatic events and their effects on emotional and mental wellbeing
03. Traumatic events and their effects on physical health
04. Traumatic events and their effects on the brain and perception
05. Psychological techniques to reduce the effects of trauma
06. Physiological techniques to reduce the effects of trauma
07. The components of trauma informed care and how to implement it
08. Engagement, perspective and roles of peers in personal recovery and trauma informed care

Part Three: True/False Questions

Can you tell us if the following statements about trauma, its effects on people,
and trauma informed care are true or false, or if you are unsure?

01. Trauma often involves being emotionally or physically hurt by a trusted person like a friend or family member or someone who works at an institution, such as a school, church or hospital.

• True

• False

• Not Sure

02. Young and poor women are the most likely of all groups to experience or be victims of violence.

• True

• False

• Not Sure

03. It takes time for people to recover a sense of safety, security and happiness after trauma.

• True

• False

• Not Sure

04. Trauma-specific services are the same as trauma informed care.

• True

• False

• Not Sure

05. Trauma informed care is an organizational structure and treatment framework that involves understanding, recognizing, and responding to the effects of all types of trauma.

• True

• False

• Not Sure

06. Trauma informed care emphasizes physical, psychological and emotional safety for both recipients of services and providers

• True

• False

• Not Sure

07. Trauma informed care helps survivors rebuild a sense of control and empowerment.

• True

• False

• Not Sure

08. The impact of trauma is experienced across the lifespan and across generations.

• True

• False

• Not Sure

09. Cultural sensitivity honors all healing traditions.

• True

• False

• Not Sure

10. Re-traumatization, or being traumatized again, is always intentional.

• True

• False

• Not Sure

Part Four: Multiple Choice Questions

Can you select the correct answer to each of the following questions about trauma,
its effect on people, and trauma informed care?

01. What did you learn about adverse childhood experiences (ACE)?

• They happen to people in the first five years of life

• They happen only to a small number of people

• They affect people in poorly run schools

• They are commonplace and affect many people

02. How do we know whether situations or people are safe, dangerous, or life threatening?

• Because danger signs are posted

• Because our brains are wired to signal that information

• Because other people warn us

03. What happens when the nervous system senses danger?

• We think about the best way to cope

• We call for help

• We feel fear that overrides logical thinking

04. Traumatic events can cause serious and lasting changes in people, including changes to which of the following things?

• Emotions

• Thinking

• Memory

• All of the above

05. Which of these is a result of adverse childhood experiences?

• Poor access to healthcare

• Lack of jobs for teenagers

• Communities with school problems

• Health problems in adulthood

06. Re-traumatization causes a person to experience overwhelming emotions and reactions associated with a previous traumatic event. Which is true?

• For someone to be re-traumatized, the person must experience the same situation and interactions in the same environment as the original traumatic episode.

• Anything that reminds the person of their original trauma, including situations, attitudes, interactions and environments could re-traumatize someone

• Situations, attitudes, interactions and/or environments cannot cause re-traumatization

07. In trauma informed systems, what do people do with their knowledge, skills, and values?

• Review them

• Reflect on them

• Revise them when needed

• All of these

• None of these

08. Which of these core competencies are required to provide trauma informed care?

• Knowledge

• Skills

• Beliefs

• All of these

09. Which of these tasks is necessary to provide trauma informed care?

• Provide for physical safety

• Address emotional safety

• Help survivors of trauma/peers rebuild a sense of control and empowerment

• All of these

10. Which of these are essential to trauma-informed practice?

• Programs that teach life skills

• The wisdom, experience, and expertise of people with lived experience

• Information about community resources

• Benefits advocacy

11. Where can re-traumatization occur?

• In institutional settings

• In community-based settings

• In either type of setting

12. When we recall a traumatic life event, we might not be aware of what we are experiencing. How do researchers describe that experience?

• Implicit memory that is experienced in the body as something physical or somatic

• Affective memory that is experienced as something that is emotional or a feeling

• Both implicit memory and affective memory

• Neither implicit memory or affective memory

13. What is “person-first language” and why is it important to use in trauma informed programs?

• Recognizing that people always come first

• Talking about the person before you talk about their condition

• Saying, for example, that someone is “a person with PTSD”, rather than saying that someone is a “PTSD case”

14. Trauma informed systems require which of these things?

• Intentional and authentic peer and staff collaborations

• Sharing of wisdom and expertise of peers and staff alike

• Honoring and understanding both peer and staff perspectives

• All of the above

5. Assessment of Skills Scale

Assessment of Skills Scale
	As a result of the TIC Training, to what extent did you implement any of the following skills in your practice?
	All the time	Some of the time	Rarely	I did not implement this skill	I did this prior to the training	Does not apply to my job or role
	1	2	3	4	5	6
01. Explain trauma and trauma reactions.
02. Recognize “symptoms” as adaptive coping mechanisms of trauma.
03. Explain trauma informed systems of care.
04. Ask “What happened?” instead of “What’s wrong?”
05. Identify when women are frozen or revved up for fight or flight.
06. Help women to shift from threat responses to more calm responses.
07. Understand the components of effective communication.
08. Explain what safety is.
09. Identify spaces, conditions, and practices that may cause women to react as if they are re‑experiencing the original trauma (i.e., be re-traumatized).
10. Use “universal precautions” to avoid re-traumatization.
11. Use person-first, non-clinical language.
12. Support peer or participant skill development by sharing power.
13. Support peer or participant involvement by providing opportunities for program participants to facilitate, organize, and coordinate activities.
14. Establish and maintain transparency in actions and interactions.
15. Share information in an ongoing, consistent manner.
16. Establish trusting relationships with colleagues.
17. Establish trusting relationships that are respectful, collaborative, and inclusive with peers and participants.
18. Make appropriate referrals with timely follow-up.
19. Engage peers and participants with empathy, warmth, and sincerity.
20. Practice self-care in an intentional, consistent manner.
21. Maintain confidentiality.
22. Ask for help when needed from supervisor, peers and participants, and colleagues.
23. Offer peers and participants choices, and honor their decisions.
24. Coach peers and participants to know their own strengths and talents.
25. Demonstrate culturally appropriate respect.
26. Tailor approach to the unique and personal goals and needs of peers and participants.