Draft CCWIS Data Quality Self-Assessment Tool 2021-08-04

C CWIS Self-Assessment Tool Data Quality

OMB # 0970-0568

Expiration Date: 04/30/2024

Data Quality Self-Assessment Tool v1.0

Introduction

The quality of data collected by a title IV-E agency supports the safety, permanence, and well-being of children and helps meet the expectations of an efficient, economical, and effective system. High quality data enables the agency to: meet federal, and state or tribal data reporting requirements; provide reliable information to guide casework decisions; assess and evaluate service needs and delivery, build user confidence in data integrity; and assess performance and program accountability.

Agencies improve data quality through a continuous and structured process that includes strong governance, a committed strategy, leadership buy-in, data ownership by all stakeholders, regular assessment, and rigorous corrective action. Each CCWIS data quality requirement in 45 CFR §1355.52(d) is an essential and unique dimension that collectively promotes quality CCWIS data. Per 45 CFR §1355.52(d)(3) and (5), the agency must submit an annual Data Quality Plan and conduct a Biennial Data Quality Review.

CCWIS Technical Bulletin (TB) #6 provides guidance to title IV-E agencies to develop a Data Quality Plan. This self-assessment tool does not duplicate the Data Quality Plan, the Biennial Review, or any Continuous Quality Improvement (CQI) processes, rather it provides a framework for the agency to self-assess the effectiveness of its data quality efforts.

Tool Format

This self-assessment tool is divided into sections as outlined on the chart below. Every question and additional consideration has a unique Element # for easy reference. Please refer to the instructions in Technical Bulletin #7 or contact your federal analyst if you have questions about the tool or a specific element.

Section	Element #
Overview and Background Information	K.A.xx
Self-Assessment – Data Quality Goals	K.B.xx
Resources and Additional Considerations	K.Cx.xx

In the Overview and Background Information section, agencies collect information on the overall CCWIS (or collection of modules and external systems if applicable) and its environment. If a question is not applicable to the system or module, indicate “N/A” and provide a reason.

K.A.01 Date this assessment was completed.

K.A.02 Is the CCWIS or external system(s) accessed by users who are not employees of the agency (or the county in a county-administered system), such as private providers, staff of other state or tribal agencies, court staff, or contractors? Please identify external user groups, including CWCAs. Please add more rows as needed.

User Group	Purpose of Use

K.A.03 What automations or other technology changes does the agency have to improve data quality in planning or development? Agencies may provide an existing plan or document as a response to this question.

In this section, the agency may document evidence for each of the listed goals and requirements. We encourage agencies to simplify their responses by referencing submitted documentation, such as the Data Quality Plan, Biennial Review findings, CCWIS Self-Assessment Tools, periodic data quality reviews, APDs or attach screenshots, system documentation (for example, a data dictionary), training materials, project or governance charters, and agency policy or procedures.

If a question or goal is not applicable to the CCWIS, indicate “N/A” and provide a reason.

Data Quality Goals

This section identifies data quality goals that support an agency’s efforts to meet CCWIS Data Quality requirements as defined in 45 CFR §1355.52(d). Assess whether the agency meets each data quality goal. Linking this self-assessment tool back to the Data Quality Plan and Biennial Reviews creates a consistent cycle of plan, measure, assess, and correct.

#	Data Quality Goal	Evidence that Supports the Goal
K.B.01	The agency addresses data quality issues identified through CCWIS Biennial Reviews.	Typically demonstrated with reference documentation such as the CCWIS Data Quality Plan, agency communications, and action plans with defined timeframes and name of the responsible person(s). The agency can provide metrics showing progress and effectiveness of technical enhancements to improve accuracy, timeliness, and completeness of CCWIS data and program outcomes.
K.B.02	The agency integrates CCWIS data quality strategies with other agency efforts to improve outcomes. Efforts to address CCWIS data collection and reporting do not occur in isolation and are considered when addressing CFSR, NYTD, AFCARS, NCANDS, and other federal, and state or tribal data quality efforts.	Typically demonstrated with documents that describe agency data quality and continuous quality improvement efforts, CFSR Systemic Factor conformity determinations, federal assessment reports, and other CCWIS self-assessment tools. The agency can provide examples how CCWIS data is relied upon and used to support program evaluation needs.
K.B.03	The system generates reports of continuing or unresolved CCWIS data quality issues and prompts the user to identify and correct data errors.	Typically demonstrated with sample reports, screenshots of dashboards, and descriptions of functionality.
K.B.04	The agency has a defined procedure and prioritization schedule to introduce patches, software fixes, minor system changes, and updates for known data quality issues.	Typically demonstrated with a schedule, prioritization framework, and procedure documents that describe the agency’s processes.
K.B.05	The agency has standards and protocols for ensuring system changes that may affect downstream reports in data warehouses remain accurate, timely, and complete.	Typically demonstrated with recent examples, and procedure documents or system documentation that describe the agency’s standards and protocols.
K.B.06	The agency’s training curriculum addresses the connection between the CCWIS and agency data quality priorities.	Typically demonstrated with training plans, curricula content, training evaluations, and interviews with stakeholders.
K.B.07	The agency updates the training curriculum and in-system help functions to keep current with CCWIS functionality.	Typically demonstrated with screenshots and examples of in-system help functionality. The agency synchronizes the deployment process for production and training environments to ensure both are current and support program needs.
K.B.08	The agency maintains a comprehensive and current data dictionary that uses plain English.	Typically demonstrated with policy language that addresses timely updates and a description of the data dictionary structure maintained (e.g., file/system structure, types of data maintained, etc.) and screenshots or sample content from the data dictionary.
K.B.09	The teams and entities responsible for CCWIS development and maintenance share a common data dictionary.	Typically demonstrated with agency policy or with examples of contract language if using contracted vendors.
K.B.10	The agency uses a data model to define business rules and requirements to ensure that all required data elements are accurately represented.	Typically demonstrated with the agency’s data model and description of the efforts taken to create the data model.
K.B.11	The agency is maximizing automation to support data quality efforts.	Typically demonstrated with screenshots and descriptions of functionality. Data quality plans and biennial review activities demonstrate regular assessment of data quality controls in the CCWIS and progress with supporting high quality data.
K.B.12	The agency includes data quality requirements in data sharing agreements to manage accuracy and consistent usage. This includes data imported through required exchanges or external systems and data exported to other entities.	Typically demonstrated with examples of data sharing agreements. The agency has a centralized repository of data sharing agreements and methods to ensure they are up to date and meet confidentiality and security requirements.
K.B.13	The agency uses an exchange standard with CWCAs and external IV-E systems to prevent duplication or data quality issues when transmitting or receiving data. When possible, automation is used to identify and correct errors as data is exchanged.	Typically demonstrated with system documentation, data sharing agreements, or by reference to the CCWIS Design Requirements and CWCA Self-Assessment Tools.
K.B.14	The system has error messaging that clearly describes the problem and gives the user constructive advice to fix. Messaging is highly noticeable, preserves the user’s work, provides links to help the user understand the nature of the problem, and is consistent with current best practices in user design.	Typically demonstrated with system documentation, screenshots, descriptions of functionality, or by reference to the User Experience Self-Assessment Tool.
K.B.15	The agency involves CWCAs and stakeholders in ongoing data quality efforts. When a quality issue is identified, the agency has an established process to fix it.	Typically demonstrated with agency policy, data sharing agreements, or by reference to the CWCA Self-Assessment Tool.
K.B.16	The system supports program implementation and collects data to evaluate performance over time.	Typically demonstrated with sample reports, examples of data analysis using CCWIS data, or by reference to the Reporting Self-Assessment Tool.

The resources below are technical assistance examples that title IV-E agencies may consider in developing data quality functions. The lists provided are not exhaustive and are not requirements. Agencies may add examples of best practices from their system they wish to highlight to add to the CCWIS body of knowledge.

Resource 1 – Data Quality Characteristics

Resource 2 – Governance Model

Resource 3 – Additional Considerations

Resource 1 – Data Quality Characteristics

The quality of data the agency collects supports the safety, permanence, and well-being of children and helps meet the expectations of an efficient, economical, and effective system. Completeness, timeliness, and accuracy are CCWIS data quality requirements required in 45 CFR § 1355.52. However, there are other data quality characteristics agencies may consider to strengthen data quality when developing a CCWIS. Agencies can use this list of characteristics to self-assess their systems and processes as part of an ongoing cycle of continuous quality improvement.

#	Characteristic	Description
K.C1.01	Accessible	Approved users can retrieve up-to-date data in a useful and easy-to-understand format to make decisions or extract for reporting and statistical analysis. “Accessible” in this context does not mean “accessibility” as defined in the User Experience Self-Assessment Tool.
K.C1.02	Accurate	The data is free of identifiable errors and represents the original source. Original sources can include first person accounts, imported data from external systems, and data from source documents, among other things. Users are entering data as intended.
K.C1.03	Complete	The agency collects required federal, and state or tribal data elements. Narrative fields include agency-required details and the user design supports the collection of all necessary data.
K.C1.04	Defined	The agency defined each data element with precision, including acceptable values or range of values in a data dictionary. For example, “year of birth” is four digits.
K.C1.05	Relevant	The agency established a purpose for recording each data element in a data dictionary, screen-level help, training materials, and policy. System documentation and training reinforce and support this purpose. Users understand why data is collected and how it is used.
K.C1.06	Reliable	Data is consistent and does not change depending on when or how it is displayed or stored.
K.C1.07	Responsive	The agency solicits and acts upon user feedback to collect and showcase relevant data, which increases user confidence in the quality of data. See the User Experience Self-Assessment Tool for more information about user engagement.
K.C1.08	Secure	Data at rest, in transit, and in use is maintained and exchanged confidentially. This includes data exchanged internally between automated functions and/or CCWIS modules and data exchanged between CCWIS and external systems.
K.C1.09	Timely	Data entry is timely to support assessment, service, and payment decisions.
K.C1.10	Unduplicated	Data elements comprising information such as person, resource, and case records are distinct and unduplicated. The agency has business practices and automated tools to prevent duplication. The agency has a process to allow merging or linking of records.
K.C1.11	Unique	Data elements in the CCWIS are stored in only one place. On every screen and report where the data element is used, it is retrieved from that storage location.

Resource 2 – Governance Model

CCWIS Data Quality Plans outline the title IV-E agency’s data governance and data management strategies to ensure CCWIS data quality. If the agency addresses or includes these practices in the governance of the CCWIS, please write “Yes” in the “Included in CCWIS Governing Model?” column.

#	Included in CCWIS Governing Model?	Governance Practice
K.C2.01		The governing body coordinates information technology efforts with overall agency governance to address data quality goals and priorities.
K.C2.02		The CCWIS data governance framework involves the whole organization and meets current agency data quality needs and future business objectives.
K.C2.03		Cross-functional agency representatives and partner stakeholders actively and consistently participate to strengthen data quality practices.
K.C2.04		Governance members’ roles and responsibilities in supporting data quality and promoting agency data quality priorities are clearly defined. This includes identification of all data owners.
K.C2.05		The governing body creates and maintains data standards that contribute to the safety, permanence, and well-being of children and families.
K.C2.06		The governing body resolves key issues with urgency and timeliness to promote high data quality standards.
K.C2.07		The agency and governing body define data quality priorities to improve the quality of data and support agency goals.
K.C2.08		The governance body regularly and consistently communicates to ensure that all data owners provide high quality data by adhering to agency business practices, policies, and processes.
K.C2.09		Baseline data quality measures are part of a regular cycle of using data to demonstrate improvement and support decision-making by the governing body.
K.C2.10		The governing body has authority and flexibility to prioritize efforts to ensure the CCWIS maintains high quality data.

Resource 3 – Additional Considerations

An agency operating a CCWIS has the flexibility to determine which system features it will use to monitor and support data quality. This section lists useful system feature(s) agencies may wish to incorporate into the CCWIS to promote, monitor, and measure data quality. If the agency is including these additional considerations in the CCWIS, please write “Yes” in the “Included in Agency’s CCWIS?” column.

#	Included in Agency CCWIS?	Additional Considerations
K.C3.01		Data Validation Edits. CCWIS uses business rules to validate the minimum data required to complete a business process.
K.C3.02		Data Exchange Error Reporting. CCWIS generates reports to identify when data provided via an exchange requires validation or correction because it did not meet the data quality standards.
K.C3.03		Data Validation. CCWIS uses public or third-party application program interfaces or web services to validate standardized data and common data sets like addresses, industry standard codes (e.g., DSM-5, ICD-10, etc.), and geographic mapping or geocoding for integrated maps.
K.C3.04		Generation of Forms/Documents. Data entry is efficient and streamlined to assist staff with producing accurate forms/notices.
K.C3.05		Intelligent Automation. Automated processes are executed periodically or as users enter data to identify and proactively address data quality risks.
K.C3.06		Data Matching. Automated tools match records using approaches such as external databases, Soundex, fuzzy logic, or machine learning to prevent duplication.
K.C3.07		Duplicate Record Prevention or Resolution. CCWIS includes features to assist users with identifying potential duplicate persons and cases to prevent the introduction of duplicate records. CCWIS includes features to merge or otherwise resolve existing duplicate persons and cases.
K.C3.08		Digital Asset Management. The agency uses a standard for naming, storing, and searching metadata for images, scanned documents, and other electronic assets accessed by the CCWIS.
K.C3.09		Examination of Underlying Data. Users can easily access the case data contributing to an outcome measure to verify its accuracy and increase confidence in the report findings. See the Reporting Self-Assessment Tool for more information about using data for reports.
K.C3.10		Stable Reporting Data. CCWIS freezes/locks data elements to prevent changes. The agency has established business rules that determine when data can be edited and how changes are recorded to support audits. See the Reporting Self-Assessment Tool for more information about using data for reports.
K.C3.11		Error Messaging. CCWIS generates meaningful automated error messages across the system according to agency policy and/or business rules. For example, these messages are generated when: Data entered is incorrect. Data entry is required for the business process. Data is inconsistent.
K.C3.12		Mandatory Data Fields. The CCWIS user interface identifies mandatory data fields.
K.C3.13		Auto Posting to Dashboard or Calendar. CCWIS automatically generates messages, activities, or tasks and posts them to a user’s dashboard or calendar as a reminder for task completion or a scheduled activity.
K.C3.14		Escalation. CCWIS automatically notifies supervisor (or designated individuals) if a deadline is near or missed.
K.C3.15		Workflow Initiation. CCWIS uses the agency automated business rules to initiate the next activity in the workflow of the business process. An “approve,” “save,” or “submit” button is automatically enabled as a validation that all required data has been completed, timely entered, and is accurate.
K.C3.16		Address Verification. CCWIS validates addresses entered by users against the USPS Address Management database. If a discrepancy exists, the user is given the opportunity to select the USPS address before moving to the next screen or task.

We encourage agencies to add examples of additional considerations from their data quality function(s) and feature(s) they wish to highlight.

#	Agency-Submitted Additional Considerations

PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN: Through this information collection, the Administration for Children and Families (ACF) is collecting information to document that title IV-E agencies have planned and developed their system’s conformity to federal CCWIS and Advance Planning Document requirements. Public reporting burden for this collection of information is estimated to average 10 hours per title IV-E agency choosing to develop and implement a CCWIS system, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. This is a voluntary collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB # is 0970-0568 and the expiration date is 04/30/2024.