RTRC Measures

Year 2 Project 2 - Provide guidance on measures for the School-Based
Telehealth Network Grant Program evaluation
Research
& Policy2017
Brief
Report – August

School-Based Telehealth Network Grant
Program Measures: Results and
Recommendations
Prepared by the RTRC University of Southern Maine Team:
Kimberley Fox, MPA
Amanda Burgess, MPPM
Karen Pearson, MLIS, MA
George Shaler, MPH
With input from the RTRC SB TNGP Advisory Team:
Marcia M. Ward, PhD, University of Iowa
Kimberly Merchant, MA, University of Iowa
Steve North, MD, Center for Rural Health Innovation
Christopher Shea, PhD, University of North Carolina – Chapel Hill

Rural Telehealth Research Center
Partners:
University of Iowa
University of North Carolina – Chapel Hill
University of Southern Maine

1

Acknowledgement
The authors would like to thank our fellow project team members Marcia Ward and Kim Merchant of
the University of Iowa, Christopher Shea of the University of North Carolina at Chapel Hill, and Steve
North of the Center for Rural Health Innovations for their expertise and guidance in conceptualizing and
developing this project and scoring measures. We also would like to thank Hayley Love and her
colleagues at the School-Based Health Alliance, Paula Wiegel and A. Clinton MacKinney of the University
of Iowa, and Andrew Soloman and Michael Edwards of the Northeast Telehealth Resource Center for
offering their expertise and participating in Round 2 scoring. Finally, we want to thank all of the SB TNGP
grantees that volunteered their time to score and provide input on potential SB TNGP measures.

2

EXECUTIVE SUMMARY
In September 2016, the Federal Office of Rural Health Policy (FORHP) Office for the Advancement of
Telehealth (OAT) awarded grants through its Telehealth Network Grant Program (TNGP) to 21 grantees
across the country to demonstrate how telehealth can expand access to, and coordinate and improve
the quality of, health care services offered through school-based health centers (SBHCs). Grants were
targeted to rural, frontier, and underserved communities providing telehealth services for children, with
a particular focus on five clinical areas: asthma, behavioral health, diabetes, obesity reduction and
prevention, and oral health.
As part of this initiative, FORHP commissioned the Rural Telehealth Research Center (RTRC) to provide
guidance on a set of measures that could be used for a cross-grantee evaluation of the School-Based
Telehealth Network Grant Program (SB TNGP). These measures supplement and build off existing
measures and research to assess school-based telehealth’s effectiveness in expanding and enhancing
SBHC access, quality, and cost effectiveness. To that end, the principal goal of this project was to define
measures to evaluate school-based telehealth that will inform future policy changes and sustainability
efforts by engaging in the following activities:


Development of an inventory of potential SB TNGP measures based on related school-based
health, child health, and/or telehealth measures recommended by key stakeholders and
grantees or identified in the literature;



Defining a methodology for evaluating this inventory of measures to determine which are most
relevant and applicable for evaluating the SB TNGP initiative and, using this methodology;



Identifying a list of core recommended measures that could be collected and reported by SB
TNGP grantees for a cross-grantee evaluation.

Steps in measure inventory development
After clarifying our process and goals with FORHP and OAT staff and becoming familiar with the grantee
proposals, we gathered potential measures. We conducted an environmental scan to identify relevant
measures from stakeholder agencies and organizations. We also reviewed the grant applications to
determine what grantees currently track or plan to track under the SB TNGP grant, and conducted a
review of the evidence-based and grey literature to supplement measures identified through the
environmental scan. The measures extracted from our environmental scan and literature review were
compiled in a spreadsheet and reviewed for scoring.
Prior to scoring, the inventory of measures was categorized into 23 measure “domains” based in part on
schemes used by the National Quality Measures Clearinghouse and the School-Based Health Alliance,
including the five clinical focus areas of the grant (asthma, behavioral health, diabetes, obesity reduction
and prevention, and oral health), other clinical areas (e.g. acute care, substance use), and relevant nonclinical topics (e.g. access, school performance, telehealth process and structure, cost effectiveness).
Measures were sorted into domains, and then sub-domains, through an iterative review process.
Throughout the scoring process the research team collapsed domains and sub-domains as needed,
based on scoring outcomes.
Measure evaluation and results
Our method for identifying candidate SB TNGP evaluation measures was guided by the following key
principles that ultimately informed the evaluation criteria by which measures were scored. To the extent
possible within our inventory of existing measures, we strove to identify a core set of measures that
achieves the following six objectives:
3

1. Addresses goals and objectives of the SB TNGP initiative including increasing, expanding or
improving access to, coordination, and quality of, health care delivered through SBHCs in rural
communities; training of health care providers; quality of health information available to health
care providers, patients, and families; clinical care for specific childhood conditions including
asthma, behavioral health, diabetes, obesity reduction and prevention, and oral health; and cost
effectiveness and return on investment.
2. Builds off or enhances existing Performance Improvement Measurement System (PIMS)
measures, with a focus on clinical outcomes.
3. Relates to conditions or interventions commonly treated or provided in school-based or
telehealth settings, conditions that if left untreated place children at high risk, and conditions or
interventions for which there is opportunity for improvement in access and/or quality through
telehealth.
4. Aligns with existing measures and is usable for quality improvement efforts by grantees and
their partners/stakeholders to support sustainability.
5. Meets evaluation criteria related to reliability and validity, measure specification, and feasibility
of data collection for students receiving telehealth, and has the potential to be collected for
students not receiving telehealth services.
6. Minimizes burden of data collection by grantees.
Our inventory of measures for scoring began with a total of 1,220 measures—533 clinical measures in 11
domains and 687 non-clinical measures in 12 domains. After three rounds of scoring by the research
team, external experts, and the grantees, the final recommended set of 27 SB TNGP performance
measures included 17 clinical measures and 10 non-clinical measures (see Table below).
Table. Final recommended measures by domain
Domain

Number of
Measures

Targeted clinical measures (N = 17)
Asthma
Behavioral Health
Diabetes
Healthy Weight
Oral Health
Non-clinical measures (N = 10)
Access
Prevention
Telehealth Process and Structure
School Performance
Cost effectiveness/Cost saving
TOTAL

4
3
3
3
4
4
1
1
1
3
27

This report describes the methodology and process for identifying the inventory of potential SB TNGP
measures and domains of focus; the multi-round review process including the specific criteria, scoring
processes, and minimum thresholds used in each round to evaluate the measures for inclusion in an
evaluation of SB TNGP grantees; and the final list of recommended SB TNGP measures identified
through this process. We also identified gaps in current measures to assess the benefit of school-based
telehealth services. A separate report discusses lessons learned from this effort that may be helpful for
future FORHP and OAT initiatives to identify measures for inclusion in future FOAs or TNGP cooperative
agreements.

4

I.

INTRODUCTION

In September 2016, the Federal Office of Rural Health Policy (FORHP) Office for the Advancement of
Telehealth (OAT) awarded grants through its Telehealth Network Grant Program (TNGP) to 21 grantees
across the country to demonstrate how telehealth can expand access to, and coordinate and improve
the quality of, health care services offered through school-based health centers (SBHCs). As specified in
the Funding Opportunity Announcement (FOA HRSA-16-102), grants were targeted to rural, frontier,
and underserved communities providing telehealth services for children, with a particular focus on five
clinical areas: asthma, behavioral health, diabetes, obesity reduction and prevention, and oral health.
As part of this initiative, in September 2016 FORHP commissioned the Rural Telehealth Research Center
(RTRC) to provide guidance on a set of measures that could be used for a cross-grantee evaluation of the
School-Based Telehealth Network Grant Program (SB TNGP).
The goals of this effort were to:


Develop an inventory of potential SB TNGP measures based on related school-based health,
child health, and/or telehealth measures recommended by key stakeholders and grantees or
identified in the literature.



Define a methodology for evaluating this inventory of measures to determine which are most
relevant and applicable for evaluating the SB TNGP initiative and, using this methodology.



Identify a list of core recommended measures that could be collected and reported by SB TNGP
grantees for a cross-grantee evaluation.

This report describes the methodology and process for identifying the inventory of potential SB TNGP
measures and domains of focus; the multi-round review process including the specific criteria, scoring
processes, and minimum thresholds used in each round to evaluate the measures for inclusion in an
evaluation of SB TNGP grantees; and the final list of recommended SB TNGP measures identified
through this process. We also identified gaps in current measures to assess the benefit of school-based
telehealth services. A separate report discusses lessons learned from this effort that may be helpful for
future FORHP and OAT initiatives to identify measures for inclusion in future FOAs or TNGP cooperative
agreements.

II.

BACKGROUND AND GUIDING PRINCIPLES

Despite their proliferation in both urban and rural areas over the last two decades, evaluations of SBHCs
have found that many schools still face barriers in getting needed services or follow-up referrals for
children and adolescents due to limited capacity to provide necessary specialty services (i.e. mental
health, oral health), challenges with reimbursement, difficulties engaging parents in on-site preventive
health education, and/or lack of funding to support a comprehensive SBHC model in all schools.1 These
problems are magnified in rural, underserved areas where parents have to drive long distances and take
time off work to bring children to follow-up services. School-based telehealth offers a potential
opportunity to expand and enhance access to services,2,3 but its use is still relatively limited (0.2% of
SBHCs).4 As a result, further evidence regarding the effectiveness of providing rural telehealth generally,
and specifically in school-based settings, is needed.5-7
FORHP’s SB TNGP grants that seek to expand telehealth in school-based settings could help to increase
the availability and use of these services. Critical to these efforts, however, is the need to design
rigorous evaluations and monitoring measures that build off and supplement existing measurement and
research to assess school-based telehealth’s effectiveness in expanding and enhancing SBHC access,
5

quality, and cost effectiveness. While studies of on-site school-based health care have demonstrated
increased student access to health and preventive service use, high rates of student and parent
satisfaction, and some improvements in chronic care management,8 many have called for more rigorous
studies and evaluations to establish a standardized set of SBHC health indicators and to determine
which SBHC components—including telehealth—are most effective in meeting the needs of the
communities they are designed to serve.9-12 To that end, the principal goal of this project was to define
measures to evaluate school-based telehealth that will inform future policy changes and sustainability
efforts.
Our method for identifying candidate SB TNGP evaluation measures was guided by the following key
principles that ultimately informed the evaluation criteria by which measures were scored. To the extent
possible within our inventory of existing measures, we strove to identify a core set of measures that
achieves the following objectives:
1. Addresses goals and objectives of the SB TNGP initiative as identified by FORHP in the original
solicitation and during project planning. These include to increase, expand, or improve:


Access to, coordination, and quality of health care delivered through SBHCs in rural
communities



Training of health care providers



Quality of health information available to health care providers, patients, and families



Clinical care for specific childhood conditions including asthma, behavioral health,
diabetes, obesity reduction and prevention, and oral health



Cost effectiveness and return on investment

Within these goals, we prioritized outcome measures in the five targeted clinical categories
(asthma, behavioral health, diabetes, obesity reduction and prevention, and oral health), with
the goal of ultimately recommending at least two measures in each clinical category.
2. Builds off or enhance existing Performance Improvement Measurement System (PIMS)
measures, with a focus on clinical outcomes.
3. Relates to conditions or interventions commonly treated or provided in school-based or
telehealth settings, conditions that if left untreated place children at high risk, and conditions or
interventions for which there is opportunity for improvement in access and/or quality through
telehealth.
4. Aligns with existing measures and is usable for quality improvement efforts by grantees and
their partners/stakeholders to support sustainability.
5. Meets evaluation criteria related to reliability and validity, measure specification, and feasibility
of data collection for students receiving telehealth, and has the potential to be collected for
students not receiving telehealth services.
6. Minimizes burden of data collection by grantees.
Although guided by these principles, our efforts were constrained by several factors. Most importantly,
we were limited to evaluating existing performance measures. We did not have the resources to
develop and test new measures specific to capturing the impact of school-based telehealth in rural
areas. In our review of the literature, we were also limited by the level of detail describing measure
specifications included in published articles, which in some cases meant that measures culled from the
6

literature did not meet the minimum measure specification threshold scores, eliminating them from
later rounds of review. We also lacked the resources to conduct an independent review of the scientific
evidence for each measure considered for this project. If the recommended measures prove
inadequate, we may need to do a more comprehensive review of the evidence and/or consider
development of new measures. Finally, as evidence is evolving in the area of school-based research,
given the relatively limited number of scientifically tested measures, particularly for non-clinical process
measures, we had to relax several of the guidelines noted above to ensure sufficient numbers of
measures in priority non-clinical domains. While we also tried to develop clear scoring criteria
definitions, many of the evaluation criteria required subjective judgment by reviewers that may not
have been consistently applied.

III.
MEASURE INVENTORY DEVELOPMENT AND EVALUATION
METHODOLOGY
We began our process by meeting with FORHP and OAT staff to review the goals of the SB TNGP
initiative and reviewing grantee proposals and summary descriptions to assess cross-grantee target
populations, clinical focus areas, and proposed use of telehealth services. We reviewed the current OAT
PIMS measures that SB TGNP grantees are required to report, and we used them as a starting point for
our measures inventory. We also investigated other related school-based mental health and SBHC
measure initiatives being supported under cooperative agreements from the Health Resources and
Services Administration’s (HRSA’s) Maternal Child Health Bureau, including efforts at the School-Based
Health Alliance (SBHA) and the University of Maryland’s Center for School Mental Health. Building off
this existing measures work, we then conducted an environmental scan of measures required or
recommended by other key stakeholder groups (e.g, state-level agencies, National Association of School
Nurses, California Telehealth Resource Center, etc.) and a literature review to identify studies of
pediatric telehealth generally or specifically in school-based settings to develop an inventory of potential
candidate measures for the SB TNGP initiative. This environmental scan was also used to assess the
degree to which measures aligned across key stakeholders to minimize reporting burden on grantees, as
discussed in more detail in Section D.
A. Environmental scan of existing stakeholder measures
To establish a list of relevant measures developed and/or recommended by stakeholder groups beyond
those required or recommended by OAT/PIMS, SBHA, and the University of Maryland’s Center for
School Mental Health the research team first developed a list of organizations involved in the
development or endorsement of performance and outcome measures and/or focused on the
advancement and improvement of SBHCs and/or telehealth. The research team started the process of
developing this list by reviewing the list of organizations that were reviewed for measures for the
University of Iowa’s Evidence-Based Tele-emergency Network Grant Program project. Members of the
SB TNGP research team recommended additional organizations focused on the development of SBHC or
telehealth measures for review, yielding a list of 31 agencies and organizations for review (see Appendix
A for the full list of stakeholder groups/agencies reviewed).
A researcher reviewed the websites for each of the 31 agencies and organizations. Review included
comprehensive browsing of the websites for relevant measures and searches for specific search terms:
telehealth, telemedicine, pediatric, and school-based health center. For the measure sources listed
below, we pre-screened measures using the following exclusion criteria:
1. Agency for Health Research and Quality (AHRQ)—Given that the initial search of AHRQ’s
National Quality Measure Clearinghouse using the identified search terms yielded thousands of
7

results, we further refined our search to relevant measures from this targeted list of
organizations included in the Clearinghouse:


Child and Adolescent Health Measurement Initiative (including the National Survey of
Children’s Health and the Young Adult Health Care Survey)



Dental Quality Alliance



Health Resources and Services Administration (HRSA) Health Disparities Collaborative
(including the asthma, depression, and diabetes collaboratives)



Maternal and Child Health Bureau of HRSA



Physician Consortium for Performance Improvement



Quality Measurement, Evaluation, Testing, Review, and Implementation Consortium

2. Children's Health Insurance Program Reauthorization Act (CHIPRA) Core Measures—We further
refined search criteria of CHIPRA Core Measures by excluding all maternal and perinatal health
measures.
3. Healthcare Effectiveness Data and Information Set (HEDIS)—The project team excluded all
HEDIS measures that the National Committee for Quality Assurance identified as only being
applicable to Medicare.
Ultimately, 427 measures were collected from 24 stakeholder groups for whom we were able to identify
related measures that are required or recommended.
Based on our guiding principle to align SB TNGP measures, to the extent possible, with existing reporting
requirements, we also sought to identify measures that grantees may be required to report as SBHCs
within their states. To assist in this process, SBHA provided a matrix of state-mandated SBHC measures
in 10 states that they had identified as part of a measure search process in 2015. The measures are
required to be collected and reported by SBHCs to school-based health center program offices in these
states (i.e. Colorado, Connecticut, Illinois, Louisiana, Maine, Massachusetts, Michigan, New Mexico, New
York, and Oregon.) From the measures required in these 10 states, the research team identified 88
relevant potential SB TNGP measures.
We also reviewed the 21 SB TNGP grant applications for measures that grantees indicated they currently
track or plan to track under the SB TNGP grant. Each proposal was searched for the following terms:
measure, indicator, and evaluation. In addition, two sections of each proposal—“evaluation and
technical support capacity” and “work plan”—were closely reviewed for measures. This review of grant
applications yielded 315 potential measures.
All relevant measures that were identified from stakeholder organizations, states, and grantees through
this process were compiled into a spreadsheet database that listed, when available, the following
information for each measure: citation, recommending/sponsoring organization, measure description,
calculation (numerator and denominator when available), rationale, and data source/collection method.
B. Literature review
The goal of our literature review was to supplement measures identified through the environmental
scan by searching evidence-based literature related to school-based or pediatric health care to identify
additional measures that could be used for a cross-grantee evaluation of the SB TNGP. Our search
strategy encompassed the published peer-reviewed literature available through existing article
databases, including PubMed, CINAHL, Cochrane Database, and ERIC. Additionally, we searched Google
8

Scholar and the grey literature for additional studies, reports, and presentations to help inform our
selection of SBHC-focused measures. We used a wide variety of search terms related to telehealth and
schools, including telehealth, telemedicine, school(s), school-based health center, health, health
services, adolescent(s), and pediatric health. We also included search terms for rural, data collection,
measures, and the specific SB TNGP focus conditions of asthma, obesity, diabetes, oral health,
behavioral or mental health in combination with the telehealth and school search terminology. We
limited our search to studies published in English. Screening criteria included whether the articles
focused on telehealth and school-age children as we were primarily looking for studies that provided
measures that intersected these topics, targeting empirically-based studies.
We examined the records retrieved through these broad and comprehensive searches without limiting
our search by date to give us an overall sense of the trend in the literature on reporting measures
relevant to SBHCs and telehealth. Reference lists from relevant articles and systematic reviews were
searched by hand for additional articles to inform our measure selection. As a result of our
conversations with FORHP during the project, we also conducted a search on cost-effectiveness
literature relative to telehealth to glean potential measures relevant to this project.
Because the evidence-based literature for telehealth in schools is not robust, we broadened our search
to include general studies of the impact of telehealth more broadly to assess whether these measures
could be applied to the school-based setting. In addition to the searches on the databases referenced
above we also searched literature identified in the Telebehavioral Health Institute’s extensive
bibliography on telemental health and behavioral telehealth.
In total, our research team reviewed 250 titles, abstracts, reports, web sites, presentations, and full-text
articles. From these, the team identified 63 unique articles for further review, which yielded 556
measures (see Appendix B for the list of articles selected for inclusion). Measures identified in the
literature were added to the measures inventory.
C. Categorization by domain and sub-domain
Prior to scoring we categorized the inventory of measures into 23 measure “domains,” based in part on
categorization schemes used by the National Quality Measures Clearinghouse and the SBHA. The 23
domains encompassed the five clinical focus areas of the grant (asthma, behavioral health, diabetes,
obesity reduction and prevention, and oral health), other clinical areas (e.g. acute care, substance use),
and relevant non-clinical topics (e.g. satisfaction, school performance).
Each domain was made up of sub-domains, or categories of closely related measures. For example, the
behavioral health domain was made up of sub-domains including “antipsychotics,” “anxiety screening,”
“PHQ-9 utilization,” “suicide risk assessment,” and “trauma screening” among others. Leading up to the
first round of scoring there were 148 sub-domains across the 23 domains, including an uncategorized
sub-domain within each domain for measures that did not easily fit into a sub-domain.
Measures were sorted into domains, and then sub-domains, through an iterative review process
between four researchers on the University of Southern Maine research team. Throughout the scoring
process the research team collapsed domains and sub-domains as needed, based on scoring outcomes.
D. Scoring criteria and review process for clinical and non-clinical measures
Our methodology for evaluating the inventory of measures builds off selection criteria guidelines used
by the National Quality Forum (NQF) with some modifications tailored to address SB TNGP goals and
recognize the limitations of evidence-based research in this area to meet NQF scientific criteria.
Modifications were informed by criteria utilized by the University of Iowa’s Evidence-Based Teleemergency Network Grant Program project and input received from the SBHA. In total, we identified
9

four broad categories that included 10 specific selection criteria for assessing the benefit of potential
measures for SB TNGP evaluation (see Table 1 for full list of criteria and descriptions). Measures were
scored on these criteria using a three-round scoring process for both clinical and non-clinical measures.
Table 1. SB TNGP Measure Selection Criteria and Scoring Method
Criteria

Description

Scoring method

Criteria 1: Importance to measure and report for school-based telehealth practice
Amenable to
Measure is related to an
Yes/No
telehealth
intervention for which the use
of telehealth technology has
been or could be applied.
High patient risk Measure is related to an
Yes/No
intervention where the child is
at high risk if not treated.
High volume
Measure is related to an
Yes/No
intervention that is commonly
provided in school-based
settings.
Opportunity for Measure assesses an
 High (2) = Significant opportunity to
improvement
intervention for which access
improve access, quality, or reduce costs
to, cost, or quality of care can
 Medium (1) = Some opportunity to
be improved.
improve access, quality, or reduce costs
 Low (0) = Minimal opportunity to improve
access, quality, or reduce costs
Criteria 2: Sensitivity to school-based health or telehealth services
Rigor
Measure can accurately
 High (2) = Likely accurately captures what
capture what it is intended to
it is intended to measure
measure
 Medium (1) = Likely mixed results on
measure accuracy
 Low (0) = Likely does not accurately
capture what it is intended to measure
Criteria 3: Feasibility of collecting
Data collection
Information is routinely
 High (2) = Feasibility of collecting and
generated and/or can be
reporting is high
collected and reported by
 Medium (1) = Feasibility of collecting and
school-based or telehealth
reporting is moderate
settings without undue burden
 Low (0) = Feasibility of collecting and
reporting is minimal
Measure
Measure has a clearly defined
 2 = Fully specified
specification
set of specifications for the
 1 = Partially specified
data elements required to
 0 = Not at all specified
calculate the measure
Criteria 4: Usability for quality improvement and FORHP evaluative needs
Alignment
Measure is used by other
Score is based on the number of
federal/state agencies or
organizations represented within each subschool-based or telehealth
domain.
associations to which grantees
 3 = 6+ alignments (7+ organizations)
may need to report/are
 2 = 3-5 alignments (4-6 organizations)
reporting
 1 = 1-2 alignments (2-3 organizations)
 0 = no alignment (1 organization)

Scoring
Round
2

2

2

2

2

2&3

1

1

10

Utility for
intended
stakeholders

Measure is useful to
grantees/providers/payers to
measure value of telehealth in
SB setting.

Utility for
study/grant
objectives

Measure is useful to assess
expanded access, quality
and/or reduced cost of care
and can be used to assess the
business case/ROI of schoolbased telehealth services for
sustainability

 High (2) = Very useful in measuring value
of telehealth in SB setting
 Medium (1) = Somewhat useful in
measuring value in SB setting
 Low (0) = Not useful or of limited use in
measuring value
 Access: Useful to assess expanded health
access in rural areas.
 Training: Useful to assess whether more
health care providers have been trained.
 Quality: Useful to assess improved quality
of health and health information to
support patient care/decision making.
 Clinical Outcomes: Useful to assess
improvements in targeted clinical
areas/reduce disparities.
Scoring: Yes/No

3

1

SUMMARY OF MEASURE EVALUATION RESULTS

IV.

During the first round of scoring four members of the University of Southern Maine SB TNGP research
team scored a total of 1,220 measures—533 clinical measures in 11 domains and 687 non-clinical
measures in 12 domains. For Round 1, reviewers scored measures on three criteria: measure
specification, alignment, and utility for study/grant objectives (see Table 1 for criteria definitions and
scoring method). Non-clinical measures were scored in Round 1 using the same criteria as those used in
Round 1 for clinical measures with the exception that we slightly modified the scoring scale for utility
from a Yes/No scale to a three-point scale to allow for greater flexibility and variability in defining the
level of utility. Each domain was randomly assigned to two team members for scoring. Scores for
measure specification and alignment were averaged between the two scorers. Any disagreements
between the two scorers on the utility criteria (a Yes/No scale) were discussed by the scorers and
reconciled.
The research team established the following minimum threshold for moving a measure on to Round 2:


Measure specificity: Average score greater than one (i.e. measure was partially or fully specified)



Utility: Yes (clinical measures) or average score greater than one (non-clinical) (i.e. useful
measure for targeted FORHP SB TNGP goals)



Alignment: Average score greater than zero (i.e. more than one organization uses this measure
or recommends using this measure)

A measure was moved on to Round 2 if the measure met all three of these minimum criteria. In the
process of each round of review, some domains or subdomains were eliminated or consolidated due to
the limited number of measures and/or overlap of measures in related domains/subdomains (e.g. the
chronic care management domain was eliminated during Round 1 because measures in that category
did not meet the minimum threshold scoring criteria cutoff.) As shown in Table 2, 455 measures met the
Round 1 minimum threshold criteria and were moved to Round 2.

11

Table 2. Number of Measures Scored in Each Round, by Domain
Number of measures scored in each round, by domain
Domain
Access/Enrollment
Access*
Acute Care
Asthma
Behavioral Health
Care Coordination
Care Substitution
Chronic Care Management
Cost Effectiveness/ Cost Saving
Diabetes
Health Education
Obesity/Healthy Weight
Oral Health
Prevention**
Process-Structure
Reproductive Health/STDs
Risk Screening/Needs Assessment
Satisfaction
School Performance
Substance Use
Sustainability
Telehealth Process & Structure
Utilization
Well-Child Visit/Prevention
Workforce
Total

Round 1

Round 2

66

11

Round 3

13
2
8
12

Recommended
Measures
4

9
91
125

4
45
100

56
15
4
155
37
14

13
6

3

38
17

8
5

3
3

97
46

47
15

5
6
19

3
4
1

70
31
21

6
16
8

76
14
31
13
101
84

12
3
27
10
26
21

8
1

1

1
8

1

41
23

28
2

1

1,220

455

100

4
3

27

*Includes the former "Access-Enrollment" and "Utilization" domains.
**Includes the former "Reproductive Health," "Risk Screening-Needs Assessment," "Substance Use," and "Well Child VisitPrevention" domains.

The second round of scoring was conducted by the research team members at the University of Iowa
and University of North Carolina, clinical subject expert Steve North, MD, and representatives from the
SBHA and the Northeast Telehealth Resource Center. Seven reviewers scored the clinical measures on
six criteria including the degree to which the measure was related to an intervention 1) commonly
provided in school-based settings (high volume), 2) where the child is at high risk if not treated (high
patient risk, 3) for which the use of telehealth technology has been or could be applied in a SB setting
(amenable to telehealth), and 4) for which access to, cost, or quality of care can be improved
(opportunity for improvement). The reviewers also considered if a measure 5) can accurately capture
what it is intended to measure (rigor), and if 6) information is routinely generated and/or can be
collected and reported by school-based or telehealth settings without undue burden (data collection).
12

Measure scores and reviewer comments from seven scorers were combined and then divided by the
number of scorers who scored each measure (scorers were not required to score measures for which
they felt they did not have expertise). Based on average reviewer scores, review of comments, and
discussion with the review team, we established the following minimum thresholds for moving
measures on to Round 3:








High volume: Average score greater than .50 (Yes-No/0-1 scale)
High patient risk: Average score greater than .50 (Yes-No/0-1 scale)
High volume: Average score greater than .75 (Yes-No/0-1 scale) OR High patient risk: Average
score greater than .75 (Yes-No/0-1 scale)1
Amenable to telehealth: Average score greater than zero (Yes-No/0-1 scale)
Opportunity for improvement: Average score greater than 1.5 (0-2 scale)
Rigor: Average score greater than 1.25 (0-2 scale)
Data collection: Average score greater than 1 (0-2 scale)

Given duplication/similarities in measures across domains and sub-domains, prior to moving measures
forward for Round 3 grantee review, the research team conducted a cross-domain consolidation process
whereby domains with similar or duplicate measures identified or where there were relatively few
measures, were consolidated. We also reviewed the full list of measures from prior rounds of review for
any clinical outcome measures that may have been deleted in earlier rounds. After this process, 100
measures in 15 domains moved on for Round 3 grantee review.
For Round 3, all grantees were invited to voluntarily participate in scoring. Ultimately, 18 of the 21 SB
TNGP grantees participated in the scoring process. Grantees were asked to score measures on two
criteria using a three-point high, medium, low scale: 1) the measure is routinely generated and/or can
be collected and reported by school-based or telehealth settings without undue burden (data collection)
and 2) the measure is useful to grantees and their partners to measure value of telehealth in SBHC
setting for sustainability (utility for intended stakeholders). Grantees were not requested to score all
measures, but only those for which they felt they had expertise or that were pertinent to their grant
area of focus (e.g. grantees focused solely on oral health could only score on oral health measures).
Grantees were also asked to provide comments on the specific measures to indicate any concerns or
make suggestions for measure improvement. Grantee scores were combined and averaged based on the
number of scorers for each measure. The full research team used these scores in selecting the final list
of recommended measures to inform an evaluation of the SB TNGP initiative.
The final review process by the SB TNGP full research team involved reviewing averaged grantee scores
across and within criteria, considering the number of grantees that scored each measure, and the
comments provided on the measures. In our review of comments, we identified measures or measure
concepts that had merit but may need further clarification in terms of specification, which may have
contributed to lower grantee scores than would have been the case with the clarification.
For PIMS measures that moved forward through the three-round review process, we also looked at data
currently reported by grantees to assess whether and how definitions for these measures could be
modified in order to improve reporting accuracy. Reflecting on our initial guiding principles for measure
1

For Round 3 we established two sets of minimum thresholds related to high-volume and patient risk scores to
assess how many measures would be moved forward under each scenario (using a .5 or .75 cut off point on both
criteria). Since the initial .5 cut off point for both criteria eliminated too many measures, the team ultimately
decided to merge the minimum threshold to include any measure that scored over .75 on either of the criteria. For
example, a measure could get a .8 on high volume and a .4 on high patient risk and it would still make it through to
the next round.

13

selection and our intention to enhance PIMS measures with a particular focus on clinical measures, we
sought to include measures that covered all of the SB TNGP goals including the five clinical conditions of
focus and other SB TNGP goals of improving access, quality of health information, training, and cost
effectiveness.

V.

RECOMMENDED MEASURES

Table 3 presents the recommended set of 27 SB TNGP performance measures for evaluation. The final
recommended set includes 17 clinical measures—three to four in each of the five targeted clinical focus
area of the SB TNGP grants—and 10 non-clinical measures, including four related to access, one related
to prevention, one measuring telehealth process, one related to measuring school attendance and
ability to stay in school, and three related to cost savings.
Table 3. Description of recommended measures

Brief description (N = 27)
#
1
2
3
4
5

6
7
8
9
10
11
12

13

Targeted clinical measures (N = 17)
Percentage of students enrolled at the SBHC with asthma that have asthma
severity classification assessed in the measurement period
Percentage of students enrolled at the SBHC with asthma that have an asthma
action plan on file in the measurement period
Percentage of students enrolled at the SBHC with persistent asthma who are on
appropriate medication in the measurement period
Percentage of students enrolled at the SBHC with asthma with rescue medication
on file at the SBHC in the measurement period
Percentage of students enrolled at the SBHC who have been screened in the
measurement period with an age appropriate risk assessment that includes a
depression screening and follow-up is documented if necessary
Percentage of students enrolled at the SBHC identified as being depressed who
self-report improved mental health in follow-up counseling or medical visits in the
measurement period
Percentage of students enrolled at the SBHC with a diagnosis of major depressive
disorder with an assessment for suicide risk in the measurement period
Percentage of students enrolled at the SBHC with diabetes with documented selfmanagement goals in the measurement period
Percentage of students enrolled at the SBHC with diabetes with a documented
HbA1c test done in the measurement period
Average HbA1c value during the measurement period for students enrolled at the
SBHC with diabetes
Percentage of students enrolled at the SBHC who have been diagnosed as obese
(i.e., a BMI-for-age >85th percentile) in the measurement period
Percentage of students enrolled at the SBHC with a BMI greater than or equal to
85th percentile who had a blood pressure percentile documented and classified as
normal or abnormal in the measurement period
Percentage of students enrolled at the SBHC who had an outpatient visit with a
PCP or OB/GYN in the measurement period and who had evidence of the following
during the measurement:
1. Percentage of patients with height, weight, and BMI percentile documentation

Domain

Asthma
Asthma
Asthma
Asthma

Behavioral Health

Behavioral Health
Behavioral Health
Diabetes
Diabetes
Diabetes
Healthy Weight

Healthy Weight

Healthy Weight

14

14
15
16
17

18
19
20
21
22
23
24

25

26
27

2. Percentage of patients with counseling for nutrition
3. Percentage of patients with counseling for physical activity
Percentage of students enrolled at the SBHC who received an oral health
evaluation/ screening in the measurement period
Percentage of students enrolled in the SBHC who received a school-based dental
screening in the measurement period and were diagnosed with tooth decay
Percentage of students enrolled in the SBHC who were referred for follow-up oral
health services in the measurement period
Percentage of students enrolled in the SBHC who received a sealant on a
permanent second molar tooth as a school-based dental service in the
measurement period
Non-Clinical Measures (N = 10)
Percentage of students enrolled in the SBHC in the measurement period
Percentage of students enrolled in the SBHC receiving telehealth services by
service type and setting in the measurement period
Number of SBHC telehealth encounters by service type and site in the
measurement period
Percentage of students enrolled in the SBHC with an identified PCP in the
measurement period
Percentage of student enrolled in the SBHC who completed a comprehensive risk
assessment in which the provider discussed common health risk behaviors in the
measurement period
Percentage of SBHC telehealth visits that were not completed due to technical
issues in the measurement period
Number of school days missed in the measurement period for students enrolled in
the SBHC
Percentage of SBHC patient encounters in the measurement period where student
is:
1) returned to class;
2) is sent home;
3) transferred to emergency/urgent care;
4) referred to PCP;
5) referred to specialty care; and
6) no subsequent face-to-face care is necessary within a defined measurement
period
Patient travel miles saved through the use of telehealth in the measurement
period; estimated associated costs
Estimated reduction or avoidance in patient travel costs as a result of avoided
face-to-face post telehealth care in the measurement period

Oral Health
Oral Health
Oral Health

Oral Health

Access
Access
Access
Access
Prevention
Telehealth Process
and Structure
School
Performance

Cost effectiveness/
Cost saving
Cost effectiveness/
Cost saving
Cost effectiveness/
Cost saving

A. Condition-specific clinical measures
Recommended condition-specific clinical measures include both process and outcome measures where
possible. The identification of clinical outcome measures was constrained both by limitations of the
current research and development of pediatric and SBHC outcome measures and based on input
provided by grantees about their utility and/or feasibility of gathering and reporting these data in
school-based settings.

15

Asthma
Measures 1 through 4 are asthma measures that assess the degree to which students enrolled in the
SBHCs have access to recommended evidence-based diagnosis, care, and treatment for this condition.
They include the percentage of students enrolled that 1) are diagnosed with asthma with a severity
classification assessed, 2) have an asthma action plan on file/in place, 3) for those with persistent
asthma, the percentage on appropriate asthma medication, and 4) the percentage who have rescue
medication on file with the SBHC. All of these asthma measures received the highest average grantee
score in terms of utility and ability to collect the data. Other proposed asthma measures scored in
Round 3 that measure patient outcomes—such as pulmonary function test scores or student reported
asthma symptom free days—ultimately were not included in our recommended list due to significantly
lower average grantee scores and grantee comments that raised concern about the sensitivity to
telehealth of the proposed intervention and/or the undue burden it would place on grantees to collect
(e.g. acquiring spirometry scores when not collected through telehealth program). In response to
grantee comments, we also clarified measure specifications to indicate when the measure denominator
was all children enrolled in the SBHC.
Behavioral health
Measures 5-7 are behavioral health process and outcome measures that measure access to
standardized recommended risk assessments, mental health screenings, and needed follow-up care;
regular suicide risk assessment for students at high-risk; and symptom relief for those suffering with
depression. Measure 5 is the percentage of SBHC users screened with an age-appropriate risk
assessment that includes a depression screening and for which a follow-up plan is documented for those
with a positive screen. Measure 7 is the percentage of patient visits for SBHC patients with a diagnosis of
major depressive disorder with an assessment for suicide risk. Both of these measures received average
scores of 3.0 or higher by grantees in Round 3. In response to grantee comments, we modified Measure
5 to be less proscriptive regarding the specific screen required, such as the PHQ-9 or PHQ-2, as not all
grantees are using this tool and substituted broader language to allow for any age-appropriate risk
assessment. Measure 6 is the percentage of SBHC enrolled students identified as being depressed who
self-report improved mental health in follow-up counseling or medical visits. This measure received
higher Round 3 grantee scores than other behavioral health outcome measures that referenced specific
tools or specific point increases to measure improvement between visits because certain tools are not
universally used by grantees.
Diabetes
Measures 8 through 10 focus on diabetes care for children in SBHCs and are intended to measure the
degree to which access to appropriate testing and treatment is improved as a result of the SB TNGP
grant program. In general, fewer grantees scored diabetes measures in Round 3 due to the fact that
fewer grantees targeted this clinical condition as part of their telehealth intervention. As with the
previous recommended clinical measures, diabetes measures would only be required to be reported by
grantees targeting care for this clinical condition as part of their telehealth intervention.
Measure 8 is the percentage of diabetic patients enrolled in the SBHC with documented selfmanagement goals in the measurement period. This measure received an average score of 3.3 by
grantees in Round 3 review and, along with Measure 10—the average HbA1c value for those identified
diabetic SBHC patients in the clinical information system—was the highest scored diabetes measures by
grantees that scored on these measures. Measure 9—the percentage of SBHC diabetic patients with a
documented HbA1c test in the measurement period—is broadly agreed upon as a measure of quality for
diabetes care and can be used to measure whether access to appropriate testing has improved.
16

In our final selection process, we also looked at the existing PIMS diabetes outcome measure to assess
whether it should be included. This measure had been eliminated in previous rounds of scoring due to
low scores on the high-volume and feasibility of data collection criteria. In reviewing PIMS data
submitted by SB TNGP grantees, only three grantees reported on the PIMS diabetes measure and for
these, the numbers reported appeared much larger than is likely and may reflect misinterpretation of
what should be reported. Given anticipated small numbers of patients with diabetes and even lower
numbers likely to have HbA1c levels above 7%, we believe collecting the average HbA1c value (Measure
9) rather than number of patients above 7% will yield larger numbers for evaluation.
Healthy weight
Measures 11 through 13 are process and outcome measures of childhood obesity that will meet our
study objective of measuring whether obesity prevention, identification, and follow-up care have
improved as a result of the SB TNGP program. Measure 11 measures the prevalence of SBHC users
diagnosed as obese (i.e. Body Mass Index (BMI) greater than the 85th percentile) within the
measurement period. Measure 12 is the percentage of SBHC users with a BMI greater than or equal to
the 85th percentile who had blood pressure percentile documented. Measure 13 includes three submeasures including the percentage of SBHC users with an outpatient visit with a primary care provider
(PCP) or obstetrician/gynecologist (OB/GYN) who had 1) evidence of height, weight, and BMI percentile
documented, 2) received nutrition counseling, and 3) received counseling for physical activity.
These healthy weight measures were generally among the highest scored by grantees for this domain.
However, several grantees raised concerns about the sensitivity of these measures to telehealth and
their ability to get data from the primary care provider (PCP) if the well-child visit was not done at the
SBHC, which may require further clarification during the measure specification process.
Oral health
Measures 14 through 17 are process measures to assess whether access to oral health screening,
diagnosis, early identification, and treatment has increased as a result of the SB TNGP grant program. As
with other clinical measures, these would only be required to be reported by grantees that are providing
telehealth services related to oral health.
All four of these measures focus on the degree to which recommended screenings are being provided,
including the percentage of children enrolled in the SBHC who received an oral health evaluation or
screening within the measurement period (Measure 14), received a school-based dental health
screening and were diagnosed with tooth decay (Measure 15), who were referred for follow-up oral
health services (Measure 16), and who received a sealant on a permanent molar tooth through the
school-based dental service (Measure 17).
B. Non-clinical measures
Per our guiding principles, in addition to clinical measures, we also sought to include measures to
address whether SB TNGP grants were successfully able to address other non-clinical goals including:


Access to, coordination, and improved quality of health care delivered through SBHCs in rural
communities



Training of health care providers



Quality of health information available to health care providers, patients, and families



Cost effectiveness

17

Access
Measures 18 through 21 are access measures that assess the degree to which SB TNGP services have
increased access to school-based health and primary care services in general and specifically to services
delivered through telehealth. Measure 18 is the percentage of students enrolled in the SBHC by school
site during the measurement period, which, with expanded availability of telehealth services, is
expected to increase over time. Measure 19 is the percentage of students enrolled in the SBHC that
received telehealth services during the measurement period by type of service type (i.e. asthma,
behavioral health, oral health, obesity, or diabetes) and site. Measure 20 is the number of telehealth
encounters provided during the measurement period by service, setting, and site, which can help assess
variation in utilization of specific telehealth services. Measure 21 is the percentage of enrollees with an
identified PCP, a widely recognized measure of child health access. All of these access measures
received the highest average grantee scores in terms of utility and ability to collect the data. The two
access-related PIMS measures that were scored by grantees in Round 3 (i.e. the number of consultants
providing care in the reporting period compared to the previous period and the number of encounters
by specialty/service, by patient care setting, and by type of telemedicine encounter) received lower
average scores due to grantees’ concerns that the measures did not capture what they were intended to
measure (i.e. # of consultants providing care across periods) or that the measures overlapped with
another measure that was less specific and thus would be easier to collect (i.e. # of encounters by
patient care setting and by type of telemedicine encounter vs. # of encounters by service, setting and
site).
Prevention
Measure 22 measures the degree to which SB TNGP services have helped improve the quality of
preventive care provided to students enrolled in the SBHC as measured by the percentage of student
who completed a comprehensive risk assessment in which the provider discussed common health risk
behaviors. This measure received slightly lower average scores by grantees than immunization and wellchild visit preventive measures. However, in contrast to comprehensive risk assessment, grantees’
comments suggested that while immunization and well-child visit rates were useful to measuring schoolbased health, it was not clear how they measured the effectiveness of telehealth.
Telehealth process and structure
Given that most SB TNGP grantees are in the early stages of implementing telehealth services in school
sites, and many are using telehealth for the first time, telehealth process and structure measures are
useful for assessing technical telehealth implementation issues. Measure 23—the percentage of SBHC
telehealth visits that were not completed due to technical issues during the measurement period—
received the highest average score of 3.3 from grantees. This measure will be useful in assessing the
degree to which technical issues are reduced over time.
School performance
One potential benefit of school-based telehealth services that could support their sustainability is the
degree to which they are able to provide needed clinical services in the school setting, allowing students
to remain in school rather than missing school to seek face-to-face medical care. Measures 24 focuses
on school attendance and measures the number of school days missed within the measurement period
for students enrolled in the SBHC. As the only measure included in this domain for Round 3 review, it
was seen as an important measure concept to include. However, it did receive lower average scores
than many other recommended measures in part due to concerns about the data collection source (e.g.
getting school records on absences or administering surveys) which may need to be modified in the final
measure specification process.
18

Cost effectiveness/Cost savings
Measures 25 through 27 measure the cost effectiveness of school-based telehealth in terms of the
disposition of the patient and the travel costs required for students to get face-to-face services at the
remote site if telehealth services were not available. Measure 25 is the percentage of SBHC patient
encounters within the measurement period where the student is returned to class, sent home, referred
to emergency/urgent care, primary care, or specialty care, or where no face-to-face follow-up is
necessary. Measures 26 and 27 are measures that are adapted from the existing PIMS avoided travel
measure, which received the highest average score by grantees of cost-effectiveness measures in Round
3. Based on our review of PIMS data submitted by grantees to date, we found that some grantees were
not accurately reporting or over-reporting the travel miles saved by using an average distance between
sites rather than site-specific distances for specific encounters. Thus we adapted the PIMS avoided/
saved measure into two separate measures. Measure 26 is patient travel miles saved through the use of
telehealth based on distance to the remote site that was avoided as a result of telehealth. Measure 27 is
the estimated reduction or avoidance of patient travel costs associated with follow-up care post
telehealth based on patient disposition captured in Measure 25.
Proposed non-clinical measures are expected to be reported by all grantees, while clinical measures
would only be required for grantees targeting the associated clinical condition. For example, grantees
that are solely focused on teledentistry would only be required to report on the oral health measures.
During the tool development phase, we will add a response category of “not applicable” for grantees
that do not provide the services related to the measure.
Non-clinical measures not recommended
While we were able to recommend at least one measure related to most of FORHP’s SB TNGP goals (e.g.
all five targeted clinical conditions, access, quality, and cost effectiveness), no telehealth training process
or outcome measures met the minimum threshold criteria to be recommended for inclusion. As a result
FORHP may want to continue to require grantees to report on the existing PIMS training measures
and/or review the training-related data submitted by grantees thus far to assess the value of
maintaining this as a required measure for reporting. We would note that some of the clinical quality
measures (e.g. whether suicide risk is conducted for children diagnosed with major depressive disorder)
could be proxies for measuring telehealth training impact in terms of whether the clinicians are
following appropriate recommended guidelines.
Measures of patient and provider satisfaction scored highly during Round 3 grantee review, however,
the research team ultimately did not recommend that patient or provider satisfaction measures be
included in required SB TNGP measures. Based on our review of the literature, we found considerable
existing evidence in the telehealth literature generally and in the school-based or other pediatric
telehealth literature, that patients and providers are satisfied with telehealth services relative to face-toface, so we did not believe collecting satisfaction-related measures would contribute to the evidencebase. We also found that satisfaction tools used for existing studies were not well-specified in the
published articles and would require additional contact with authors that our timeframe did not permit.
If FORHP were to allow grantees to voluntarily report satisfaction measures, the research team could
conduct a more thorough review of satisfaction tools to recommend a standard tool that could be used
across grantees.
C. Input on existing PIMS measures and suggested enhancements
As indicated above, we included measures currently collected in the PIMS for OAT in our inventory of
measures. Existing PIMS measure specifications were drawn from an OAT-provided document titled,
“Performance Improvement Measurement System for the Office for the Advancement of Telehealth.”
19

Grantees are currently required to report PIMS measures on an aggregate basis bi-annually. Existing
PIMS measures are primarily process measures, but also include a few clinical measures. As part of our
review of PIMS measures, we analyzed PIMS data reported by the grantees for the first six months of
their projects, to assess the degree to which they were being reported and the need for greater
clarification on measures that may have been misinterpreted in reporting.
During the multi-round review process, most PIMS measures did not meet the minimum threshold for
specified criteria. For example, two of the PIMS measures were eliminated in the first round of scoring
because they did not meet the minimum threshold for the “utility” criterion. An additional eight PIMS
measures did not meet minimum threshold to advance to the third round of scoring—primarily due to
low scores on the “data collection” and “opportunity for improvement” criteria or because they were
similar or duplicative of other higher scored measures in that domain and were eliminated in the final
consolidation process. The four remaining PIMS measures were scored by grantees during the third
round of scoring, and only two of the measures (patient travel miles saved and number of telehealth
encounters by service) were included in the final recommended list of measures (with some
modification to definitions). The average scores for the four PIMS measures that made it to Round 3
review ranged from 2.5 to 3.8 on a four-point scale.
D. Key challenges and measure gaps in SB TNGP measure selection
In evaluating potential SB TNGP measures, we were limited by the state-of-the-art in child health
measurement generally and specifically in school-based health or telehealth. We found that validated
measures that had been tested and used in other environments were not necessarily applicable or
appropriate for school-based settings or for assessing child health and needed to be adapted. While
some of the measures have been validated in some type of telehealth setting, other measures have not
been rigorously tested. Some of the measures were chosen because the author, source document,
measure specificity, description and/or rationale were detailed and they appeared to align with the
initiative. Further clarification should be sought for these measures when it comes to instrument
development and data collection methodologies.
We also found it challenging to identify detailed selection criteria given the diversity of measures
identified. The broad selection and scoring criteria used placed more responsibility on reviewers who
had a wide range of expertise to interpret them, which may have increased subjectivity and decreased
inter-rater reliability. For some measures, there was wide variation across reviewers’ scores and/or
differences in scoring for relatively similar measures by the same reviewer. The prioritization of
including the alignment criteria in the first round of measure review may also have unintentionally
removed some potentially good measures from further review because only one organization
recommended the measure.
Finally, the ability to collect comparison group measurements was not factored into the scoring process.
To evaluate the benefit of SB telehealth, comparison groups within the grantee programs will be
necessary. For example to demonstrate the value of SB telehealth for diabetic patients a comparison
group of diabetic patients that does not access telehealth services will be needed. Ultimately some of
the measures may need to be revised if a comparison group cannot be located.

VI.

NEXT STEPS FOR SB TNGP EVALUATION

Once FORHP approves the final SB TNGP list of measures, in the next stage of our work, the research
team will develop a research protocol and define the measure specifications of each of these
performance measures and the data elements needed to calculate them. While some of these
20

specifications are available for NQF-endorsed and other SB TNGP selected measures, we will have to
develop or define our own specifications for measures that are not as clearly specified. We also plan to
create or modify the data collection tool by which grantees will report the measures and establish a
process and timeline for beginning data collection from grantees.
As part of the specification process, we will also identify whether measures will be collected at the
individual patient level or at the aggregate program or site level and will identify the mechanism or tool
for grantees to report them – either through modifications to the existing PIMS reporting system or
through a separate Microsoft Excel-based tool to be developed, building off the existing tool being used
for the Evidence-Based Tele-emergency Network Grant Program that members of the research team are
currently overseeing. The Excel tool will provide a structure for grantees to calculate their performance
results, including reporting the de-identified patient data needed to calculate performance measures
based on every student that received SB TNGP services, including those receiving telehealth services and
those treated through face-to-face SB services.
References
1.
Silberberg M, Fox K, Quinn W, Cantor J. Evaluation of the Newark School-Based Youth Services
Program: Part 1 Report of Stakeholder Perceptions. New Brunswick, NJ: Rutgers University, Center for
State Health Policy;2000.
2.
Alverson D, Hall-Barrow J, Dion D, et al. 15 Million Kids in Health Care Deserts: Can Telehealth
Make a Difference? New York, NY: Children's Health Fund; April 21, 2016.
3.
Martin A, North S. School-Based Health Care Via Telemedicine. Online webinar presentation:
Health-e-Schools and Center for Rural Health Innovation; July, 2015.
4.
School-Based Health Alliance. 2013-14 Digital Census Report. 2015. Available at:
http://censusreport.sbh4all.org/#growth. Accessed August 29, 2016.
5.
Gilman M, Stensland J. Telehealth and Medicare: Payment Policy, Current Use, and Prospects for
Growth. Medicare Medicaid Res Rev. 2013;3(4):E1-E14.
6.
Lambert D, Gale J, Hansen A, Croll Z, Hartley D. Telemental Health in Today's Rural Health
System. Portland, ME: Maine Rural Health Research Center; December, 2013.
7.
Rural Health Advisory Committee, Work Group on a New Rural Health Care Delivery Model.
Rural Health Care: New Delivery Model Recommendations. St. Paul, MN: Office of Rural Health and
Primary Care; January, 2009.
8.
Crespo RD, Shaler GA. Assessment of School-Based Health Centers in a Rural State: The West
Virginia Experience. J Adolesc Health. Mar 2000;26(3):187-193.
9.
Bersamin M, Garbers S, Gold M, et al. Measuring Success: Evaluation Designs and Approaches to
Assessing the Impact of School-Based Health Centers. J Adolesc Health. 2016;58(1):3-10.
10.
Keeton V, Soleimanpour S, Brindis CD. School-Based Health Centers in an Era of Health Care
Reform: Building on History. Curr Probl Pediatr Adolesc Health Care. 2012;42(6):132-156.
11.
Mason-Jones AJ, Crisp C, Momberg M, Koech J, De Koker P, Mathews C. A Systematic Review of
the Role of School-Based Healthcare in Adolescent Sexual, Reproductive, and Mental Health. Syst Rev.
2012;1:49.
12.
Silberberg M, Cantor JC. Making the Case for School-Based Health: Where Do We Stand? J
Health Polit Policy Law. Feb 2008;33(1):3-37.
21

APPENDIX A. STAKEHOLDER GROUPS
Stakeholder Groups
Federal entities
Agency for Healthcare Research & Quality

National Quality Measures Clearinghouse
Centers for Medicare and Medicaid Services

2016 Core Set of Children’s Health Care Quality Measures for Medicaid and CHIP
Health Resources and Services Administration



Health Center Uniform Data System
Office for the Advancement of Telehealth, PIMS measures

States with SBHC measures identified by SBHA
Colorado
Connecticut
Illinois
Louisiana
Maine
Massachusetts
Michigan
New Mexico
New York
Oregon
Stakeholders
California School-Based Health Alliance
California Telehealth Resource Center (of the Consortium of Telehealth Resource Centers)
Center for Health and Health Care in Schools
Center for School Mental Health
Colorado Association of School-Based Health Care
Connecticut Association of School-Based Health Centers
National Association of School Nurses
National Committee for Quality Assurance

Healthcare Effectiveness Data and Information Set
National Quality Forum
School-Based Health Alliance
University of Iowa
Stakeholders reviewed that yielded no applicable measures
American Telemedicine Association
Indiana School Health Network
Kentucky Youth Advocates
School-Based Health Alliance State Affiliates (Arkansas, Georgia, Illinois, Louisiana,
Maryland, Michigan, New Mexico, New York, North Carolina, Oregon, Washington, West
Virginia)
Institute for Healthcare Improvement
Texas Association of School-Based Health Centers
The Children’s Partnership

22

APPENDIX B. ARTICLES REVIEWED FOR SCORING
Allison MA, Crane LA, Beaty BL, Davidson AJ, Melinkovich P, Kempe A. School-Based Health Centers:
Improving Access and Quality of Care for Low-Income Adolescents. Pediatrics. Oct 2007;120(4):e887894.
Assimacopoulos A, Alam R, Arbo M, et al. A Brief Retrospective Review of Medical Records Comparing
Outcomes for Inpatients Treated Via Telehealth Versus in-Person Protocols: Is Telehealth Equally
Effective as in-Person Visits for Treating Neutropenic Fever, Bacterial Pneumonia, and Infected Bacterial
Wounds? Telemed J E Health. Oct 2008;14(8):762-768.
Banos RM, Cebolla A, Oliver E, Alcaniz M, Botella C. Efficacy and Acceptability of an Internet Platform to
Improve the Learning of Nutritional Knowledge in Children: The Etiobe Mates. Health Educ Res. Apr
2013;28(2):234-248.
Belmont JM, Mattioli LF. Accuracy of Analog Telephonic Stethoscopy for Pediatric Telecardiology.
Pediatrics. 2003;112(4):780-786.
Bergman DA, Sharek PJ, Ekegren K, Thyne S, Mayer M, Saunders M. The Use of Telemedicine Access to
Schools to Facilitate Expert Assessment of Children with Asthma. Int J Telemed Appl. 2008:159276.
Botasso M, Sanches SG, Bento RF, Samelli AG. Teleaudiometry as a Screening Method in School Children.
Clinics (Sao Paulo). Apr 2015;70(4):283-288.
Broder E, Manson E, Boydell K, Teshima J. Use of Telepsychiatry for Child Psychiatric Issues: First 500
Cases. CPA Bulletin. 2004;36(3):11-15.
Bynum AB, Irwin CA, Cranford CO, Denny GS. The Impact of Telemedicine on Patients' Cost Savings:
Some Preliminary Findings. Telemed J E Health. Winter 2003;9(4):361-367.
Calear AL, Christensen H, Mackinnon A, Griffiths KM, O'Kearney R. The Youthmood Project: A Cluster
Randomized Controlled Trial of an Online Cognitive Behavioral Program with Adolescents. J Consult Clin
Psychol. Dec 2009;77(6):1021-1032.
Cunningham DL, Connors EH, Lever N, Stephan SH. Providers' Perspectives: Utilizing Telepsychiatry in
Schools. Telemed J E Health. Oct 2013;19(10):794-799.
Davis AM, James RL, Boles RE, Goetz JR, Belmont J, Malone B. The Use of Telemedicine in the Treatment
of Paediatric Obesity: Feasibility and Acceptability. Matern Child Nutr. Jan 2011;7(1):71-79.
Doolittle GC, Williams AR, Cook DJ. An Estimation of Costs of a Pediatric Telemedicine Practice in Public
Schools. Med Care. Jan 2003;41(1):100-109.
Elford DR, White H, St John K, Maddigan B, Ghandi M, Bowering R. A Prospective Satisfaction Study and
Cost Analysis of a Pilot Child Telepsychiatry Service in Newfoundland. J Telemed Telecare. 2001;7(2):7381.
Elford R, White H, Bowering R, et al. A Randomized, Controlled Trial of Child Psychiatric Assessments
Conducted Using Videoconferencing. J Telemed Telecare. 2000;6(2):73-82.
Eminovic N, Dijkgraaf MG, Berghout RM, Prins AH, Bindels PJ, de Keizer NF. A Cost Minimisation Analysis
in Teledermatology: Model-Based Approach. BMC Health Serv Res. Aug 25 2010;10:251.
Fox KC, Connor P, McCullers E, Waters T. Effect of a Behavioural Health and Specialty Care Telemedicine
Programme on Goal Attainment for Youths in Juvenile Detention. J Telemed Telecare. 2008;14(5):227230.
23

Gallagher TE. Augmentation of Special-Needs Services and Information to Students and Teachers
"Assist"--a Telehealth Innovation Providing School-Based Medical Interventions. Hawaii Med J. Oct
2004;63(10):300-309.
Garrett CC, Hocking J, Chen MY, Fairley CK, Kirkman M. Young People's Views on the Potential Use of
Telemedicine Consultations for Sexual Health: Results of a National Survey. BMC Infect Dis. 2011;11:285.
Glueckauf R, Fritz S, Ecklund-Johnson E, Liss H, Dages P, Carney P. Videoconferencing-Based Family
Counseling for Rural Teenagers with Epilepsy: Phase 1 Findings. Rehabil Psychol. 2002;47(1):49-72.
Grogan-Johnson S, Alvares R, Rowan L, Creaghead N. A Pilot Study Comparing the Effectiveness of
Speech Language Therapy Provided by Telemedicine with Conventional on-Site Therapy. J Telemed
Telecare. 2010;16(3):134-139.
Guo JJ, Jang R, Keller KN, McCracken AL, Pan W, Cluxton RJ. Impact of School-Based Health Centers on
Children with Asthma. J Adolesc Health. 10// 2005;37(4):266-274.
Harper DC. Telemedicine for Children with Disabilities. Child Health Care. Winter2006 2006;35(1):11-27.
Hilty DM, Ferrer DC, Parish MB, Johnston B, Callahan EJ, Yellowlees PM. The Effectiveness of Telemental
Health: A 2013 Review. Telemed J E Health. Jun 2013;19(6):444-454.
Himle MB, Freitag M, Walther M, Franklin SA, Ely L, Woods DW. A Randomized Pilot Trial Comparing
Videoconference Versus Face-to-Face Delivery of Behavior Therapy for Childhood Tic Disorders. Behav
Res Ther. 2012;50(9):565-570.
Horn BP, Barragan GN, Fore C, Bonham CA. A Cost Comparison of Travel Models and Behavioural
Telemedicine for Rural, Native American Populations in New Mexico. J Telemed Telecare. Jan
2016;22(1):47-55.
Izquierdo R, Morin PC, Bratt K, et al. School-Centered Telemedicine for Children with Type 1 Diabetes
Mellitus. J Pediatr. Sep 2009;155(3):374-379.
Jones M, Luce KH, Osborne MI, et al. Randomized, Controlled Trial of an Internet-Facilitated Intervention
for Reducing Binge Eating and Overweight in Adolescents. Pediatrics. Mar 2008;121(3):453-462.
Kennedy C, Yellowlees P. A Community-Based Approach to Evaluation of Health Outcomes and Costs for
Telepsychiatry in a Rural Population: Preliminary Results. J Telemed Telecare. 2000;6 Suppl 1:S155-157.
Kennedy C, Yellowlees P. The Effectiveness of Telepsychiatry Measured Using the Health of the Nation
Outcome Scale and the Mental Health Inventory. J Telemed Telecare. 2003;9(1):12-16.
Lancaster P, Krumm M, Ribera J, Klich R. Remote Hearing Screenings Via Telehealth in a Rural
Elementary School. Am J Audiol. Dec 2008;17(2):114-122.
Lessard JA, Knox R. Telehealth in a Rural School-Based Health Center. J Sch Nurs. Apr 2000;16(2):38-41.
Loane MA, Bloomer SE, Corbett R, et al. A Randomized Controlled Trial Assessing the Health Economics
of Realtime Teledermatology Compared with Conventional Care: An Urban Versus Rural Perspective. J
Telemed Telecare. 2001;7(2):108-118.
Malasanos TH, Burlingame JB, Youngblade L, Patel BD, Muir AB. Improved Access to Subspecialist
Diabetes Care by Telemedicine: Cost Savings and Care Measures in the First Two Years of the FITE
Diabetes Project. J Telemed Telecare. 2005;11 Suppl 1:74-76.
Mansour ME, Rose B, Toole K, Luzader CP, Atherton HD. Pursuing Perfection: An Asthma Quality
Improvement Initiative in School-Based Health Centers with Community Partners. Public Health Rep.
24

November 1, 2008 2008;123(6):717-730.
March S, Spence SH, Donovan CL. The Efficacy of an Internet-Based Cognitive-Behavioral Therapy
Intervention for Child Anxiety Disorders. J Pediatr Psychol. Jun 2009;34(5):474-487.
Marcin JP, Ellis J, Mawis R, Nagrampa E, Nesbitt TS, Dimand RJ. Using Telemedicine to Provide Pediatric
Subspecialty Care to Children with Special Health Care Needs in an Underserved Rural Community.
Pediatrics. Jan 2004;113(1 Pt 1):1-6.
McConnochie K, Wood N, Herendeen N, ten Hoopen C, Denk L, Neuderfer J. Integrating Telemedicine in
Urban Pediatric Primary Care: Provider Perspectives and Performance. Telemed J E Health. Apr
2010;16(3):280-288.
McConnochie KM, Ronis SD, Wood NE, Ng PK. Effectiveness and Safety of Acute Care Telemedicine for
Children with Regular and Special Healthcare Needs. Telemed J E Health. Aug 2015;21(8):611-621.
McConnochie KM, Wood NE, Herendeen NE, ten Hoopen CB, Roghmann KJ. Telemedicine in Urban and
Suburban Childcare and Elementary Schools Lightens Family Burdens. Telemed J E Health. Jun
2010;16(5):533-542.
McConnochie KM, Wood NE, Kitzman HJ, Herendeen NE, Roy J, Roghmann KJ. Telemedicine Reduces
Absence Resulting from Illness in Urban Child Care: Evaluation of an Innovation. Pediatrics.
2005;115(5):1273-1282.
McIntosh S, Cirillo D, Wood N, Dozier AM, Alarie C, McConnochie KM. Patient Evaluation of an Acute
Care Pediatric Telemedicine Service in Urban Neighborhoods. Telemed J E Health. Dec
2014;20(12):1121-1126.
Namakian M, Subar P, Glassman P, Quade R, Harrington M. In-Person Versus "Virtual" Dental
Examination: Congruence between Decision-Making Modalities. J Calif Dent Assoc. Jul 2012;40(7):587595.
Nelson E-L, Barnard M, Cain S. Treating Childhood Depression over Videoconferencing. Telemedicine
Journal And E-Health: The Official Journal Of The American Telemedicine Association. 2003;9(1):49-55.
Nelson E-L, Barnard M, Cain S. Feasibility of Telemedicine Intervention for Childhood Depression.
Counselling & Psychotherapy Research. 2006;6(3):125-129.
Nelson E-L, Duncan AB, Peacock G, Bui T. Telemedicine and Adherence to National Guidelines for Adhd
Evaluation: A Case Study. Psychol Serv. 2012;9(3):293-297.
O'Kearney R, Gibson M, Christensen H, Griffiths KM. Effects of a Cognitive-Behavioural Internet Program
on Depression, Vulnerability to Depression and Stigma in Adolescent Males: A School-Based Controlled
Trial. Cogn Behav Ther. 2006;35(1):43-54.
O'Kearney R, Kang K, Christensen H, Griffiths K. A Controlled Trial of a School-Based Internet Program for
Reducing Depressive Symptoms in Adolescent Girls. Depress Anxiety. 2009;26(1):65-72.
O'Reilly R, Bishop J, Maddox K, Hutchinson L, Fisman M, Takhar J. Is Telepsychiatry Equivalent to Face-toFace Psychiatry? Results from a Randomized Controlled Equivalence Trial. Psychiatr Serv. Jun
2007;58(6):836-843.
Pretorius N, Arcelus J, Beecham J, et al. Cognitive-Behavioural Therapy for Adolescents with Bulimic
Symptomatology: The Acceptability and Effectiveness of Internet-Based Delivery. Behav Res Ther. Sep
2009;47(9):729-736.
Romano MJ, Hernandez J, Gaylor A, Howard S, Knox R. Improvement in Asthma Symptoms and Quality
25

of Life in Pediatric Patients through Specialty Care Delivered Via Telemedicine. Telemed J E Health.
Winter 2001;7(4):281-286.
Ruskin PE, Silver-Aylaian M, Kling MA, et al. Treatment Outcomes in Depression: Comparison of Remote
Treatment through Telepsychiatry to in-Person Treatment. Am J Psychiatry. 2004;161(8):1471-1476.
Shaikh U, Cole SL, Marcin JP, Nesbitt TS. Clinical Management and Patient Outcomes among Children
and Adolescents Receiving Telemedicine Consultations for Obesity. Telemed J E Health. Jun
2008;14(5):434-440.
Shealy KM, Davidson TM, Jones AM, Lopez CM, de Arellano MA. Delivering an Evidence-Based Mental
Health Treatment to Underserved Populations Using Telemedicine: The Case of a Trauma-Affected
Adolescent in a Rural Setting. Cogn Behav Pract. 8// 2015;22(3):331-344.
Shore JH, Brooks E, Savin DM, Manson SM, Libby AM. An Economic Evaluation of Telehealth Data
Collection with Rural Populations. Psychiatr Serv. Jun 2007;58(6):830-835.
Sousa P, Fonseca H, Gaspar P, Gaspar F. Controlled Trial of an Internet-Based Intervention for
Overweight Teens (Next.Step): Effectiveness Analysis. Eur J Pediatr. 2015;174(9):1143-1157.
Spaulding R, Belz N, DeLurgio S, Williams AR. Cost Savings of Telemedicine Utilization for Child Psychiatry
in a Rural Kansas Community. Telemed J E Health. Oct 2010;16(8):867-871.
Stain HJ, Payne K, Thienel R, Michie P, Carr V, Kelly B. The Feasibility of Videoconferencing for
Neuropsychological Assessments of Rural Youth Experiencing Early Psychosis. J Telemed Telecare.
2011;17(6):328-331.
Stensland J, Speedie SM, Ideker M, House J, Thompson T. The Relative Cost of Outpatient Telemedicine
Services. Telemed J. Fall 1999;5(3):245-256.
Storch EA, Ehrenreich May J, Wood JJ, et al. Multiple Informant Agreement on the Anxiety Disorders
Interview Schedule in Youth with Autism Spectrum Disorders. J Child Adolesc Psychopharmacol. Aug
2012;22(4):292-299.
Wootton R, Bloomer SE, Corbett R, et al. Multicentre Randomised Control Trial Comparing Real Time
Teledermatology with Conventional Outpatient Dermatological Care: Societal Cost-Benefit Analysis.
BMJ. May 06 2000;320(7244):1252-1256.
Yellowlees PM, Hilty DM, Marks SL, Neufeld J, Bourgeois JA. A Retrospective Analysis of a Child and
Adolescent Emental Health Program. J Am Acad Child Adolesc Psychiatry. 2008;47(1):103-107.
Young TL, Ireson C. Effectiveness of School-Based Telehealth Care in Urban and Rural Elementary
Schools. Pediatrics. Nov 2003;112(5):1088-1094.
Zaylor C. Clinical Outcomes in Telepsychiatry. J Telemed Telecare. 1999;5 Suppl 1:S59-60.

This study was supported by the Federal Office of Rural Health Policy (FORHP), Health Resources and
Services Administration (HRSA), U.S. Department of Health and Human Services (HHS) under cooperative
agreement number 6 UICRH29074-01-01. The information and conclusions in this report are those of the
authors and no inferred endorsement by FORHP, HRSA, or HHS.
Research partners: University of Iowa, University of North Carolina, and University of Southern Maine.

26