Design/Screen Shots

Attachment A: NCTSI Evaluation Design and Data Collection
Instruments
The various components of the redesigned NCTSI evaluation and associated instrumentation are
described below.
Component 1: Descriptive and Clinical Outcomes
To address the GPRA goals of increasing access to services and improving outcomes, we
propose continuing the use descriptive and clinical outcome data to describe the characteristics
of children in formal treatment at NCTSN-funded centers, monitor the type and amount of
services that they receive, and assess whether children’s outcomes improve over time. The focus
is on children and families receiving intensive treatment for trauma exposure. This approach is
built on the Core Data Set (CDS) (see Attachment B), which includes instruments specifically
designed for this initiative as well as standardized checklists from the field. The CDS provides
both baseline and longitudinal data on the children being served by the program at each of the
funded NCTSN centers on various domains including, but not limited to:








Demographics
Residential placement/domestic environment
Family characteristics
Service use
Trauma exposure
‘Real world’ functioning
Standardized assessments
o Emotional/behavioral problems: Child Behavior Checklist (Achenbach, 2000, 2001)
and Children’s Depression Inventory-2 Short (Kovacs, 1992)
o Trauma Sequelae: Trauma Symptom Checklist for Children-Alternate Version (Briere,
1989, 1995) and Trauma Symptom Checklist for Young Children (Briere, 2005)
o Trauma symptoms: UCLA PTSD Index for DSM IV (Pynoos, Rodriguez, Steinberg,
Stuber, & Fredrick, 1998)
o Substance Use: Global Appraisal of Individual Needs Modified Short Screener
(Dennis, Chan, & Funk, 2006)
o Parental Stress: Parenting Stress Index Short Form (Abidin, 1995)

All NCTSN centers that provide direct clinical services using NCTSN funds are required to
collect data using the CDS and submit the data to the NCTSI evaluator. Data are collected by
grantees at baseline and at three-month intervals through the end of treatment. The purpose of the
CDS measures is to provide client-level baseline and longitudinal outcome information to
evaluate the impact of services provided at NCTSN funded centers. CDS data help the NCTSN
to clarify the kinds of traumas children experience, which children are being served, what types
of treatment are being provided, and the relationship between child traumatic stress and
contextual factors. The CDS has also helped to enhance the capacity of centers to administer
standardized measures, and collect and analyze data using such measures on a routine basis.

Component 2: Access to High Quality, Trauma-informed Services
The NCTSI mission is to expand access to high quality, trauma-informed services for traumaexposed children and adolescents and their families nationwide. This component of the
evaluation is designed to assess NCTSI program progress in achieving this mission by collecting
and analyzing data from a variety of sources addressing the question of whether access to high
quality, trauma-informed services has improved and for which demographic groups. This
component includes three subcomponents focused on: 1) child-serving systems targeted by
NCTSN center training efforts; 2) children served by NCTSN centers; and 3) NCTSI program
national reach. Descriptions of each subcomponent follow.
Related to child-serving systems (i.e., mental health, child welfare, juvenile justice, education,
and primary care), the newly proposed Evidence-based Practice (EBP) and Trauma-informed
Systems Change Survey (ETSC) (see Attachment C) is a Web-based survey designed to assess
the extent to which NCTSN training and other dissemination activities have enhanced the
knowledge base and use of trauma-informed services (TIS) within child-serving agencies,
centers and organizations that are not a part of the NCTSN but rather have received training from
the NCTSN as well as to assess the extent to which such services are evidence-based. The survey
branches into two versions adapted for project directors/administrators and human service
providers (e.g., mental health providers, child welfare case workers, teachers, primary care health
care providers and others), allowing for questions tailored to the professional orientation and
activities of each group. The survey collects information on the following domains:









Adoption/implementation of evidence-based practices
Adoption/implementation of assessment measures
Degree to which agency or organization screens for trauma
Degree to which agency or organization assesses trauma impact
Degree to which agency or organization provides or refers for trauma treatment
Degree to which clinical practice is trauma-informed
Degree to which organizations are trauma-informed
Facilitators and barriers to providing trauma-informed care

The ETSC survey will be used to assess the extent to which NCTSN training and dissemination
activities have improved access to high quality, trauma-informed services for trauma-exposed
children and their families that are served through such child-serving systems. The ETSC survey
is administered twice over the grant period of the NCTSN centers (i.e., during the first year and
third year of the 4-year grant period).
Related to children directly served by NCTSN centers, this component will examine disparities
in receipt of quality care for children served by NCTSN centers by assessing the characteristics
of the children served (age, gender, race, income, insurance status, assessment and diagnosis),
service utilization, and consumer satisfaction. Secondary analysis of data in the CDS (described
above, see Attachment B), will be conducted to examine service use by different subgroups of
children and youth served by the NCTSN. Service use will be assessed based on characteristics
such as age, gender, race, income, insurance status, involvement in other child-serving sectors,
and presenting problems. Such data will be used to compare the use of services and emergency
rooms prior to and after entry into NCTSN services. In addition, a review of literature and non-

NCTSN datasets will be conducted to identify data that may allow for a comparison of NCTSN
and non-NCTSN service use. It may be feasible to compare NCTSN service use to data on
overall access to care or data on access to care among specific subgroups (e.g., children in child
welfare). A thorough literature review will be conducted to determine available data sources and
to assess their appropriateness for comparison. If appropriate data sources are identified,
summary statistics on NCTSN service use will be compared to summary statistics on nonNCTSN service use. In addition to the CDS secondary analysis, to assess the disparities in
receipt of quality care within the NCTSN program, information on characteristics of the children
and youth who are waitlisted to receive services and on average how long clients are on the
waitlist will be extracted from the newly proposed Online Performance Monitoring Report
(OPMR) (see Attachment D and description of this report below).
In addition, related to children directly served by NCTSN centers, secondary analysis of data
elements from the Transformation Accountability (TRAC) system and the CDS will be
conducted to assess consumer perspectives on services. TRAC items regarding the perception of
care by caregivers and youth, particularly around access to services, participation in treatment,
cultural sensitivity, appropriateness, and client satisfaction will be drawn from the Perception of
Care instrument. These data will be analyzed in relation to child and youth characteristics from
both the TRAC system and the CDS. Characteristics examined may include client demographics,
outcomes, service use, and dropout rates.
Related to national reach, this component will also assess the extent to which the NCTSN has
impacted the knowledge and awareness, policies, planning, programs, and practices related to
trauma-informed care among state and national child-serving organizations external to the
NCTSN centers. A revised version of the OMB-cleared, Web-based NCTSI National Reach
Survey (see Attachment E) will be used to assess the extent to which the NCTSN has impacted
trauma-informed services information/knowledge, policy, and practices among mental health and
non-mental health child-serving organizations external to the Network. As a clarification,
respondents for the ETSC Survey are 1) administrators who interact with NCTSN centers who
are from agencies, centers, and organizations that provide a variety of types of services directly
to children and adolescents and 2) providers from similar agencies, centers and organizations
who are trained by NCTSN centers. Respondents for the National Reach Survey are the
directors of state and national professional associations and organizations (specific respondents
will be identified by the NCCTS and NCTSN centers). The survey is administered in alternate
years of the evaluation contract to all state and national associations representing the child
serving systems. The National Reach Survey collects information on the following domains:









Extent of familiarity with NCTSN centers
Types of collaboration with these centers
Familiarity with NCTSN products
NCTSN products implemented
Extent to which staff members in these organizations have knowledge about childhood
trauma (i.e., consequences, treatment needs, and interventions)
Organization’s use of specialized interventions for child trauma
Policies, screening and assessment procedures
Plans for developing services, training materials, management information, funding, and
dissemination of information to support trauma-exposed children and adolescents



Interaction with the NCTSN on the above plans

As described, the Web-based National Reach Survey was developed as part of the currently
OMB-approved evaluation. Revisions to the survey are described, along with other revisions to
the evaluation, in Section 2.b.
Component 3: Training, Evidence-based Practices (EBPs), and Family/Consumer
Partnerships
As described above, a major goal of the NCTSN is to enhance the capacity of administrators and
service providers from agencies, centers and organizations associated with child-serving systems
(including mental health, child welfare, juvenile justice, education and primary care) to use
trauma-informed services (TIS) with trauma-exposed children and their families. NCTSN centers
promote the use of TIS within child-serving systems to increase public awareness and knowledge
about trauma exposure, trauma impact, and the range of trauma-informed assessments and
services that are available. Specifically, NCTSN centers are involved in activities including
training, dissemination of resources, education, and consultation regarding topics such as traumarelated evidence-based practices, interventions, and methods. As described above, as part of
Component 2, the ETSC Survey will be used to assess the extent to which NCTSN training and
dissemination activities have improved access to high quality, trauma-informed services for
trauma-exposed children and their families. For Component 3, the ETSC Survey will be used to
assess the ways in which agencies, schools, and organizations that are a part of child-serving
systems have changed their service approach as a result of participating in educational activities
or trainings supported by NCTSN centers; in particular, this survey will assess the extent to
which the respondents’ service activities have become evidence-based and trauma-informed as a
result of the dissemination efforts of the NCTSN.
While the ETSC Survey assesses change in practice among agencies, centers and organizations
that are trained by (or otherwise educated by) NCTSN centers, the OPMR (see in-depth
description of this report below) collects similar information about implementation of evidencebased practices and trauma-informed practice from administrators of NCTSN centers. As part of
Component 3, this data from the OPMR will be used to assess changes in practice within
NCTSN centers related to the implementation of evidence-based practices as well as
organizational shifts within NCTSN centers resulting in greater attention to and use of traumainformed services.
Two additional forms will support data collection for Component 3, including a Training
Summary Form (TSF) (see Attachment F) that will be completed by professionals associated
with the NCTSN who are delivering the trainings provided by NCTSN centers. In general, these
trainers are NCTSN or NCCTS center staff members. Information related to training will be
collected on the TSF at the conclusion of trainings. This form will collect information on the
number of participants trained, the type of training (including the trauma types addressed in the
training), and the topics emphasized in the training. In addition, a Training Sign-In Sheet
(TSIS) (see Attachment G) will be completed by the participants of NCTSN-sponsored trainings.
Participants will provide their names; agency, organization or center for which they work; their
roles; and contact information including an email addresses. In addition, they will be asked to
indicate whether the evaluation may contact them for participation.

Finally, data will be collected through the OPMR to assess the extent to which, and the specific
ways in which, consumer/family-driven approaches to care and consumer/family involvement
are incorporated into the design, development and implementation of grant activities among
NCTSN centers. This study will assess activities pertaining to both clinical practice and
organizational/program practice. SAMHSA is committed to fostering consumer and family
involvement in substance abuse and mental health policy and program development. A key
component of that commitment is involvement of consumers and family members in the design,
development and implementation of projects funded through SAMHSA’s grant programs, which,
in the case of the NCTSI program, could result in improved, higher quality care for traumaexposed children and their families.
Informal, preliminary assessment suggests that only a portion of NCTSN centers actively involve
consumers/family members in grant planning and implementation; however some are engaged in
innovative approaches that are not being captured (e.g., focus groups with consumers/family
members to solicit opinions on approaches to training service providers). Such approaches, if
more thoroughly documented, could serve as an example for other centers. In addition, it is
important to assess formally the range of activities centers are engaged in to address this goal.
In addition to assessing centers’ activity related to promoting consumer/family involvement in
the planning and implementation of grant activities, the data gathered through the OPMR will
provide information on centers’ efforts to engage families and promote retention in service
delivery. Responses will be compared to the CDS to assess whether centers’ efforts to promote
retention are resulting in higher quantities of follow-up data and to understand center needs
around training and technical assistance or other supports needed to improve retention and
follow-up.
Lastly, the data collected through the OPMR on consumer/family involvement, along with data
collected through the TRAC system regarding consumer satisfaction, will be analyzed and the
findings used to design a qualitative study that includes consumer/family in the target respondent
group. The qualitative study will include either interviews or focus groups and will be submitted
in Year 1 of the NCTSI Evaluation for OMB clearance. The qualitative investigation is needed
both to conduct an in-depth assessment of consumer satisfaction and to assess the
consumer/family perspective on the strengths and weaknesses of NCTSN family partnership
activities and the impact on service delivery.
Component 4: Sustainability
Assessing the sustainability of the progress made by the NCTSN and its partners is a key
evaluation priority identified by stakeholders advising on the redesign of the NCTSI Evaluation.
Therefore, while this issue was not addressed as part of the previous evaluation design, it has
been included as a new area of importance for future NCTSI evaluation. This component of the
evaluation focuses on understanding the degree to which NCTSI grant activities continue after
funding has ended and the factors associated with the continuation of—or lapse in—grant
activities such as the implementation of evidence-based practices or approaches to strengthen
trauma-informed service provision. The newly proposed Web-based Sustainability Survey—
For Funded Centers (see Attachment H.1) collects data from administrators and evaluators of

funded NCTSN centers annually to assess plans for sustainability and factors associated with
such planning activity among currently funded centers. A second version of the survey, nearly
identical, the Web-based Sustainability Survey—For Affiliate Centers (see Attachment H.2),
will be administered annually to administrators of affiliate centers, which are centers that no
longer receive SAMHSA funding but have continued involvement with the NCTSN and are
defined by SAMHSA as affiliates. These surveys collect information on the following domains:









NCTSI grant history
Funding sources and fiscal strategies
Program mission
Infrastructure
Service delivery
Continuation of practices and programs
NCTSN involvement and collaborations
Involvement with non-NCTSN centers

Additional Evaluation and Monitoring Activity
SAMHSA is also requesting clearance for the OPMR (Attachment D), which is primarily a
mechanism for SAMHSA to monitor centers’ progress towards achieving stated goals and a
fulfillment of SAMHSA requirements for accountability and performance monitoring. In
addition, as a result of collaborative efforts to streamline, consolidate, and reduce data collection
required of grantees to report on grant activities for national monitoring and evaluation purposes,
this form will also serve as an important data source informing several components of the NCTSI
evaluation. The form incorporates evaluation domains that were previously a part of either the
currently OMB-approved evaluation or other monitoring and evaluation efforts conducted by
SAMHSA or the NCCTS, each of which impacted NCTSI grantees (detailed information about
the consolidation of monitoring and evaluation activities in the OPMR is included in Section 2.b)
The OPMR will be completed by NCTSN center administrators on a quarterly (Q), annual (A),
or ongoing (O) basis using an online reporting system (ORS). The form collects data related to
the following domains:















Major project goals, activities, and accomplishments (Q)
Local evaluation activities (Q)
Project accomplishments (Q)
Public Awareness activities (Q)
Service Capacity (A)
Number of clients served with Direct clinical, Client-related, and Family services provided
by the Center (Q)
Trauma-Informed Practices and Interventions (Q)
Products Developed under the Auspices of the NCTSI Grant (Q)
Collaborative Activities at the local level (Q)
Interagency planning and coordination (A)
Collaborative activities within the NCTSN (Q)
Workgroup participation (A)
Sustainability of services (A)
Data collection progress (Q)



Financing (A)

Attachment B: NICON Screen Shots
Race/Ethnicity Variables and Child Behavior Checklist (CBCL)
Issue Summary:
The NCTSI Evaluation will not use the race/ethnicity variable from the Child Behavior Checklist (CBCL). Rather, data regarding
race/ethnicity is collected using the baseline Core Clinical Characteristics form. Following are screen shots from the NICON system
displaying the Core Clinical Characteristics Form and CBCL.

NICON screen shot of the Core Clinical Characteristics form, which includes questions about race and ethnicity:

NICON screen shot of the Child Behavior Checklist, which excludes questions about race and ethnicity: