Ssb_avert_072523

SUPPORTING STATEMENT: PART B

February 6, 2023

Advancing Violence Epidemiology in Real-Time (AVERT)

New

Point of Contact:

Yushiuan Chen

Contact Information:

Centers for Disease Control and Prevention

National Center for Injury Prevention and Control

4770 Buford Highway NE

Atlanta, GA 30341

Phone: (303) 249-9057

Email: ukt9@cdc.gov

CONTENTS

Section Page

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1. Respondent Universe and Sampling Methods 3

B.2. Procedures for the Collection of Information 3

B.3. Methods to Maximize Response Rates and Deal with Nonresponse ...7

B.4. Tests of Procedures or Methods to be Undertaken……………………8

B.5. Individuals Consulted on Statistical Aspects and Individuals

Collecting and/or Analyzing Data…………………………………….9

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B1. Respondent Universe and Sampling Methods

The purpose of the Advancing Violence Epidemiology in Real-Time (AVERT) system is to identify ED visits related to firearm injuries (regardless of intent), other violence-related injuries, and mental health conditions to improve local, state, and national situational awareness of trends to improve violence prevention and response efforts at the local and state levels. This goal will be accomplished by standardizing and enhancing sharing of existing ED data locally collected by a minimum of 10 funded health departments with CDC. CDC requires that participating health departments share ED data with CDC through the NSSP BioSense platform. The NSSP BioSense platform is the required mechanism because it minimizes burden on participating health departments by allowing CDC to complete the ED violence data form while maximizing data quality by enabling collaborative analyses between CDC and health departments. In CDC’s Advancing Violence Epidemiology in Real-Time Notice of Funding Opportunity (AVERT, CDC-RFA-CE-23-0007), DVP included the following required eligibility criteria: (1) Use of the national ESSENCE platform for their syndromic surveillance data management; and (2) Collects and accesses ED visit data from a minimum of 80% of ED facilities in their jurisdiction. This includes ED visit data from a minimum of 90% of Level 1-3 trauma centers.

B2. Procedures for the Collection of Information

The health departments participating in AVERT will share the following type of data with CDC on an ongoing basis: Counts of ED visits for overall firearm injury, intentional self-directed firearm injury, unintentional firearm injury, assault-related firearm injury, intimate partner violence, sexual violence, suspected child abuse and neglect, youth violence, and mental health conditions that occurred each month by county and by age, sex, and race/ethnicity. Data will be shared with CDC on a bimonthly basis using a standardized Excel form, the ED violence data form (Attachment D), and standard CDC case definitions of overall firearm injury, intentional self-directed firearm injury, unintentional firearm injury, assault-related firearm injury, intimate partner violence, sexual violence, suspected child abuse and neglect, youth violence, and mental health conditions. Also, measures of data quality and metadata will be collected and are described in the SSA and Attachment D.

The specific procedures CDC will use to collect the ED violence data form from the participating health department is described below.

Procedures for collecting the ED violence data form (Attachment D) on a bimonthly basis

Step 1: Participating health departments will use the following data source to complete the ED violence data form.

• National Syndromic Surveillance Program (NSSP) BioSense Platform (OMB #0920-0824): State and local health departments share preliminary case-level ED data in near-real time with CDC. These data include the chief complaint of the patient seeking care at the ED (e.g., “firearm injury”) and/or diagnosis codes, primarily the International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) diagnosis codes. Chief complaints tend to be submitted within 24 hours of the ED visit while ICD-10-CM codes may take a few weeks.

Step 2: Using access to case-level ED data provided to CDC through the NSSP BioSense platform, CDC will complete the vast majority of the ED violence data form each month including calculating the total number of overall firearm injury, intentional self-directed firearm injury, unintentional firearm injury, assault-related firearm injury, intimate partner violence, sexual violence, suspected child abuse and neglect, youth violence, and mental health conditions that occurred each month by county and by age, sex, and race/ethnicity. CDC, however, will need to consult with the participating health department when completing metadata and will also share the ED violence data form with the participating health department to validate the CDC analysis.

Step 3: CDC will upload the completed ED violence data form for participating health departments to access through the National Center for Injury Prevention and Control (NCIPC) interface hosted on the CDC Secure Access Management Service (SAMS) Partner’s Portal, referred to as the NCIPC Partner’s Portal. The NCIPC Partner’s Portal will conduct automatic data quality checks on submitted files to verify that required data is submitted without major data quality issues. This process will improve data quality and reduce the need for multiple data submissions by state and local health departments. The CDC SAMS partner portal which hosts the NCIPC Partner’s Portal is a web site designed to provide secure centralized access to external users such as public health departments to data and computer applications operated by CDC. It can also be used to securely exchange data between CDC and the participating health departments.

Step 4: Up to one month will be taken for CDC to collaborate with participating health departments to validate their submitted data for dissemination. Data with no significant issues are expected to be validated much quicker than a month. In contrast, data with significant issues may have to be resubmitted and/or excluded from analyses on a case-by-case basis. On average, CDC is currently able to validate data and close out each data submission within two and a half weeks. Key parts of the validation process include:

• Ensuring the data submission is internally consistent (i.e., total ED visits broken down by sex, age, and race/ethnicity match those broken down by county).

• Identify and discuss with submitting health departments large changes in total ED visits or data quality (e.g., large drop in the number of ED visits or a large drop in the number of ED visits with a valid ICD-10-CM code) identified in the metadata or review of aggregate data reported to CDC.

Step 5: CDC will convene a workgroup of participating health departments at least once every quarter to identify ways to improve the data sharing process, data quality, case definitions and analytic approach of AVERT.

Proposed analyses used on data

The following types of statistical methods will be used to analyze the data:

• Descriptive analyses such as analyzing the percent change in the rate of ED visits (number of firearm injury ED visits divided by total ED visits per time period multiplied by 100,000) related to firearm injuries (regardless of intent), other violence-related injuries, and mental health conditions by US region, state/territory, county, sex, race/ethnicity, and age group.

• Long-term trends analyses using Joinpoint regression or hierarchical linear modeling.

• Identifying and tracking possible upticks within states by working with health departments to analyze trend changes in related to firearm injuries (regardless of intent), other violence-related injuries, and mental health conditions at the county level or districts level (e.g., districts are combinations of counties within state designed by states).

Estimation Procedures

The data collection will not use statistical weighting because:

• The census will include a minimum of 80% of all ED visits (i.e., a significant portion of all ED visits).

• The primary goal of syndromic surveillance is to provide situational awareness and inform response. To achieve these goals, regional and state surveillance needs to be able to track and validate large changes in violence-related ED visits by examining the specific local area or hospital data driving the large change.

• The primary goal of AVERT is to detect rapid changes over time in participating hospitals instead of estimating the incidence of violence through estimation procedures. Consequently, the key concern is monitoring for substantial changes in ED participation or data quality over time that would bias detection of upticks or changes in firearm injuries (regardless of intent), other violence-related injuries, and mental health conditions (See Unusual Problems for a description for how AVERT works to track and respond to this potential bias).

• Participating hospitals are not randomly selected and limited information is available on non-participating hospitals. This coupled with the small percent of non-participating hospitals would make accurate adjustments for non-response difficult and expensive.

Degree of Accuracy

This issue does not apply to this methodology.

Unusual Problems

ED syndromic systems are designed to collect rapid preliminary data on changes in illness and injuries, such as firearm injuries, other violence-related injuries, and mental health conditions. These systems, however, may not provide an accurate estimate of the full burden of illnesses and injuries because they are based on preliminary data. AVERT addresses this limitation in the following ways:

1. CDC analyses will primarily focus on detecting upticks and rapid shifts in ED visits related to firearm injuries (regardless of intent), other violence-related injuries, and mental health conditions to inform prevention and response. Consequently, public health departments can share data from different ED data sources because the key data requirement is the ability to detect change over time within each jurisdiction (e.g., data consistently collected within the same jurisdiction overtime) and not comparing absolute counts and rates of violence-related ED visits across participating health departments (e.g., one jurisdiction may report a slightly higher firearm injury rate than another jurisdiction because it captures 95% of ICD-10-CM codes on ED visits within a month of the date of the ED visit compared to the other jurisdiction which captures only 60%).

   1. CDC will not compare numbers or rates of firearm injuries (regardless of intent), other violence-related injuries, and mental health conditions calculated using data ED violence data form across jurisdictions (e.g., say one state has a higher firearm injury rate than another state) because the data sources and data quality vary across jurisdictions (e.g., one jurisdiction may report a higher firearm injury rate than another jurisdiction because it EDs commonly describe patient’s chief complaint with two to three sentences compared to another jurisdiction where chief complaints are commonly described with two or three words).

   2. CDC will not analyze rates (e.g., ED visits related to a firearm injury divided by total number of ED visits in a state) with fewer than 20 cases in the numerator (e.g., number of ED visits related to a firearm injury) because of possible statistical instability of rate estimates. For instance, CDC will not report the percent change in firearm injury rates from January to February 2022 if only 19 ED visits involved a firearm injury in January 2022.

2. CDC will monitor the preliminary ED data submitted on the ED violence data form by each jurisdiction for large monthly changes in hospital participation or data quality (e.g., percent of ED visits missing chief complaint and/or ICD-10-CM diagnosis information). This is critical to identify and respond to major changes in ED syndromic systems such as transmission delays associated with the implementation of new EHR system by a major health care provider or a large health care provider joining or exiting the local ED surveillance. Responses to these types of problems will be customized and could include delaying submission of data until a problem is resolved or suppressing data from counties or state(s) if the data has major and/or multiple data quality problems. Other steps may be investigated if major variation in the percent of ED visits missing data on chief complaint and diagnosis codes (e.g., ICD-10-CM diagnosis codes) is found to consistently occur over time in a significant number of counties or states.

B3. Methods to Maximize Response Rates and Deal with Non-response

One primary issue to conducting rapid violence surveillance of all ED visits in the US is that NSSP BioSense does not provide national coverage or maximum state coverage, where not all EDs participate in the system.

CDC will engage in four strategies to improve the coverage of the AVERT system and address this issue. AVERT coverage is defined as the percent of non-federal ED visits reported to CDC in the ED violence data form.

1. AVERT can be rapidly implemented and scaled to more states and the District of Columbia because it relies on sharing data already being collected by state and local health departments through the NSSP BioSense Platform instead of attempting to establish a new ED data collection.

2. CDC’s AVERT notice of funding opportunity (CDC-RFA-CE-23-0007) included the following required eligibility criteria for jurisdictions applying: (1) Use of the national ESSENCE platform for their syndromic surveillance data management; and (2) Collects and accesses ED visit data from a minimum of 80% of ED facilities in their jurisdiction. This includes ED visit data from a minimum of 90% of Level 1-3 trauma centers.

3. CDC’s AVERT notice of funding opportunity (CDC-RFA-CE-23-0007) requires all participating state health departments and the District of Columbia to spend a portion of their funding to support efforts to maintain and enhance collection of rapid ED data for this program. This funding allocation is designed to ensure that sufficient support is provided to the staffing unit collecting the data and can include use of funds for support staff or infrastructure, which over time should improve the coverage and quality of data collected by AVERT.  

4. CDC’s AVERT notice of funding opportunity (CDC-RFA-CE-23-0007) also allowed funding to be used to implement general improvements to the recipient’s ED syndromic surveillance system (e.g., increase hospital participation or submission of ICD-10-CM codes, improve completeness and quality of race/ethnicity fields) if these enhancements support improving the timeliness, completeness, or quality of violence syndromic surveillance.

B4. Tests of Procedures or Methods to be Undertaken

Case definitions of ED visits involving overall firearm injury, intentional self-directed firearm injury, unintentional firearm injury, assault-related firearm injury, intimate partner violence, sexual violence, suspected child abuse and neglect, youth violence, and mental health conditions have been extensively tested in the following ways:

• Under previous CDC funding (i.e., Firearm Injury Surveillance Through Emergency Rooms or FASTER), health departments developed state-based case definitions for firearm injuries. The commonalities across these state-based definitions have been critical in informing the development of national definitions. In addition, differences across state-based definitions coupled with intense consultations with CDC have resulted in refinements of the national definitions that enhance the ability these national case definitions to detect ED visits involving a firearm injury.

• CDC has collaborated with staff of the NSSP ESSENCE team to analyze and review case-level ED visits identified by each of the case definitions. Analytic techniques have included identifying words and ICD-10-CM codes that are commonly connected within ED visit (e.g., ED visits with a chief complaint that includes the abbreviation “GSW”). Manual review of the ED chief complaint and ICD-10-CM codes of ED visits identified by CDC staff have been critical in excluding inappropriate visits and expanding the violence-related definitions (e.g., capture common misspellings) to detect previously undetected cases.

• The case definitions were informed by a review of other ICD-9-CM and ICD-10-CM violence coding systems.

The design of the ED violence form has been informed by feedback from the 10 state health departments and the District of Columbia participating in FASTER.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following consulting efforts were made:

1. AVERT staff has intensely collaborated with the Division of Health Informatics (DHIS), the CDC division that operates NSSP BioSense. A key benefit from this collaboration is the expertise of CDC staff with extensive experience analyzing syndromic ED data, including a candid assessment of its strengths and weakness.

2. As part of FASTER, CDC convened a workgroup where CDC discussed data quality and analytic approaches with 10 state health departments and the District of Columbia on approximately a monthly basis.

3. FASTER-funded health departments engaged with CDC staff to assist in the development and refinement of firearm injury and violence-related injury case definitions for syndromic data. This work will continue with AVERT-funded states.

4. As part of AVERT, CDC will convene a quarterly workgroup meeting where CDC and funded health departments discuss data quality issues and the development of case definitions as well as data dissemination efforts.

5. CDC has also worked closely with the Council for State and Territorial Epidemiologists (CSTE) to ensure health department epidemiologists are well-equipped with the skills to analyze ED data collected within their state by conducting regional trainings.

One CDC senior epidemiologist, two epidemiologists, and a data manager will support the analysis of AVERT data. This includes intensive work evaluating case definitions and the quality of the preliminary ED data both internally and with state health department staff. This group can consult with another senior scientist and/or request statistical support on an as needed basis.