Comprehensive Child Welfare Information System (CCWIS) Review and Technical Assistance Process Self-Assessment Tool: Data Quality and Reporting

C CWIS Self-Assessment Tool Reporting

OMB # 0970-0568

Expiration Date: 04/30/2024

Reporting Self-Assessment Tool v1.0

Introduction

A title IV-E agency’s CCWIS must support the efficient, economical, and effective administration of the title IV-B and title IV-E programs to ensure the safety, permanency, and well-being of children and to assist families.¹ A CCWIS must be able to generate or contribute to title IV-B and IV-E federal reports that are required by federal statutes and regulations, and reports required by state or tribal child welfare laws, regulations and policies, and audits and reviews that support title IV-B and IV-E programs.² Title IV-E agency (agency) staff and other stakeholders use reports to manage casework, assess client needs, monitor placements and other services, evaluate program effectiveness, determine and manage budgets, collaborate with other human service programs, assess compliance with laws and policies, and improve decision-making.

While federal requirements specify the information and the format for some reports, agencies have great flexibility in designing reports for their programs’ needs. To ensure reports meet the needs of stakeholders, agencies should consider reporting needs and requirements early in the system development life cycle and establish ongoing processes to ensure targeted users and other stakeholders are included in the design, development, and use of reports.

The Reporting Self-Assessment Tool will help agencies assess:

• Compliance with CCWIS reporting requirements; and

• the effectiveness of report design, development, and training practices.

Tool Format

This self-assessment tool is divided into sections as outlined on the chart below. Every question and additional consideration has a unique Element # for easy reference. Please refer to the instructions included in Technical Bulletin #7: The CCWIS Technical Assistance and Compliance Review Process or contact your federal analyst if you have questions about the tool or a specific element.

Section	Element #
Overview and Background Information – System Information	H.A.xx
Self-Assessment – Part 1 – Reporting Goals	H.B1.xx
Self-Assessment – Part 2 – Foundational Requirements	H.B2.xx
Resources and Additional Considerations	H.C.xx

In the Overview and Background Information section, agencies document information on the overall CCWIS (or collection of modules if appropriate) and its environment. Please answer each question fully. If a question is not applicable to your system, indicate “N/A” and provide a reason it is not applicable.

System Overview

H.A1.01 Date this assessment was completed.

H.A1.02 Name of the CCWIS or module(s) included in this self-assessment.

H.A1.03 Brief description of the CCWIS or module(s), its status (such as planning, development, production, operational), and target completion date. Agencies may attach or reference documents such as APDs, project plans, and responses provided in other CCWIS self-assessment tools. Where applicable, indicate when referenced documents were last updated.

H.A1.04 Brief description of the reporting environment, including report tools, report infrastructure, and report archiving. Agencies may attach or reference documents such as report tool overviews, system architectural diagrams, and report inventories. Where applicable, indicate when referenced documents were last updated.

H.A1.05 Brief description of the process for generating each of the ongoing federal reports listed below. List the general, high-level report generation steps describing how data is gathered, formatted, and submitted. Also indicate the system providing the data for each report. If multiple systems contribute data, list all systems and the challenges, if any, of collecting report data from different systems. For example, the agency may note that CCWIS collects National Youth in Transition Database (NYTD) case management data and that a separate youth portal collects the survey data, which is then provided to CCWIS to generate the NYTD report for submission to ACF. Or the agency may note that child welfare contributing agency systems provide AFCARS data. If the title IV-E agency does not submit a specific report, indicate that it is “Not submitted.”

Agencies may attach or reference documents such as APDs, system architectural diagrams, and process flows. If applicable, indicate when referenced documents were last updated.

• AFCARS:

• National Child Abuse and Neglect Data System (NCANDS):

• NYTD:

• Form CB-496:

• Title IV-E Prevention Program Data Elements:

• Child and Family Services Plans & Annual Progress and Services Reports:

• Monthly Caseworker Visits:

Part 1 – Reporting Goals

This section of the self-assessment describes general goals applicable to all reports: user access, report design, and training. Reporting goals for specific programs are addressed in other self-assessment tools. For example, the Case Management self-assessment tool addresses case management reporting goals.

In this section, the title IV-E agency may document components, factors, processes, and design elements of the system that support CCWIS reporting goals. We encourage agencies to simplify their responses by referencing submitted documentation, such as APDs or attaching screen shots, system documentation, training materials, and agency policy or procedures.

#	Reporting Goal	Evidence that CCWIS Supports the Reporting Goal
H.B1.01	User access. Authorized stakeholders can easily access reports.	Typically demonstrated with documents describing user access, report organization, report search tools (such as searching, filtering, and sorting reports), and user options to set selection criteria.
H.B1.02	Report design. Reports include features, such as clear titles and data labels, to support clear understanding and interpretation.	Typically demonstrated with report examples, descriptions of the project’s design standards, and processes for incorporating user feedback into subsequent report designs.
H.B1.03	User training. The agency promotes report understanding and use with training.	Typically demonstrated with training plans, training schedules, and curricula content.
H.B1.04	Support casework and supervision. CCWIS reports and alerts provide caseworkers with actionable information for decision-making and case monitoring. CCWIS reports support efficient supervisor oversight.	Typically demonstrated with sample reports, screen shots of dashboards, and descriptions of functionality that enable users to generate reports tailored to specific tasks and clients.
H.B1.05	Data analysis. CCWIS reports and, if applicable, data extracts, support program administration with tools to manage work, monitor outcomes, assess program strengths and weaknesses, support research, and provide evidence and justification for budget requests.	Typically demonstrated with sample reports and examples of data analysis using CCWIS data to measure program priorities (such as effective resource allocation) and emerging issues (such as disproportionality in foster care).
H.B1.06	Data dictionary. The agency maintains a comprehensive data dictionary to support report development, training, and CCWIS maintenance.	Typically demonstrated with the data dictionary, the policy or procedures for timely updating of the data dictionary, or by reference to the Data Quality Self-Assessment Tool.

Part 2 – Foundational Requirements

Foundational requirements identify conditions to comply with CCWIS Project requirements at 45 CFR § 1355.52(b), (c), and (d). These foundational requirements apply whether staff enter data directly into the CCWIS, or data is imported through a data exchange.

In this section, the title IV-E agency may document components, factors, and design elements of the function(s) or exchanges that support CCWIS foundational requirements. We encourage agencies to simplify their responses by referencing submitted documentation such as APDs or attach screen shots, system documentation, training materials, and agency policy or procedures.

#	Foundational Requirement	Evidence that CCWIS Supports the Foundational Requirement
H.B2.01	Federal reports. CCWIS provides data for required ongoing federal child welfare reports.	Typically demonstrated with documents describing successful report submissions or steps the agency is taking to address known report issues, such as report submission logs, data compliance reports, or federal assessment reports (for example, in progress or finalized AFCARS Improvement Plans).
H.B2.02	Federal audits. Reports from CCWIS support federal audits, reviews, and other monitoring activities.	Typically demonstrated with reference documentation such as report inventories/catalogs, and listings of data elements to support each type of federal audit/review, such as title IV-E reviews and the Child and Family Service Reviews.
H.B2.03	State/Tribal reports. Reports from CCWIS support state or tribal child welfare laws, regulations, policies, practices, reporting requirements, audits, and reviews for programs and services described in title IV-B and title IV-E.	Typically demonstrated with evidence that CCWIS reports are meaningful and responsive to stakeholder needs. Such evidence may be provided by tools to gather user feedback (i.e., user meetings, survey data, focus groups) and CCWIS statistics, such as report usage data.
H.B2.04	Report reliability. Reports from CCWIS are complete, timely, and accurate. Agency staff have confidence in reports and use the reports to guide decision-making and casework.	Typically demonstrated with reference documentation such as the agency’s data quality plan, findings from CCWIS biennial data quality reviews, supervisory reviews, CQI team reviews, report usage data, and other self-assessment tools.

The resources below are technical assistance examples that title IV-E agencies may consider in developing reports. The lists provided are not exhaustive and are not requirements. Agencies may add examples of best practices from their system they wish to highlight to add to the CCWIS body of knowledge.

Resource 1 – Ongoing Federal Child Welfare Report References

Resource 2 – Additional Considerations – CCWIS Report Development Practices and Processes

Resource 1 – Ongoing Federal Child Welfare Report References

CB maintains information on ongoing federal child welfare reports that CCWIS must support on the Monitoring and Reporting Systems pages of the CB website.³ We encourage agencies to use this information to confirm they have met the data, submission, and other requirements for each report.

Resource 2 – Additional Considerations – CCWIS Report Development Practices and Processes

The Additional Considerations section describes useful features agencies may wish to incorporate into the CCWIS design/features. If the agency is including these additional considerations in the CCWIS, please write “Yes” in the “Included in Agency’s CCWIS?” column.

The table includes practices and processes on topics such as report development, training, usage, and culture. These topics cover several issues addressed in rows grouped together:

• Report testing

• Report documentation

• Report training

• Reports and change management

• Report design

• User access to data and reports

#	Included in Agency’s CCWIS?	Additional Considerations
H.C2.01		Including complete report requirements. The agency documents comprehensive report requirements to ensure reports meet user needs. Agencies report that commonly collected requirements for each report may include: Intended audience or users of the report, such as: caseworkers; supervisors and managerial staff; children, youth, and families served; agency administrators; courts; legislative and executive staff providing oversight; and community partners. Report purpose or goal. Report format/layout. Report data elements, including values retrieved from the CCWIS database, provided by other systems, or calculated fields. Selection criteria, such as filters, conditions, sorting, and grouping, that are: hard-coded and unchanging for all report runs; user-determined at run time; and both (some reports with selection criteria hard-coded, other reports with selection criteria user-determined). Action triggering report execution, such as: User initiation. A defined schedule of event(s), such as deadlines for investigations, home visits, eligibility determinations, or financial reporting. System-detected problems, such as missing or inconsistent data, or data not entered in specified timeframes. “Drill down” features to permit users to compare regions/counties/offices/units and to view the case data comprising statistics. Functions to export report/data to different formats (e.g., PDF, Excel, charts, and graphs). Report distribution platform/method, such as: reporting server/cloud; desktop/laptop; mobile device; data extract; and hard copy. Report archival requirements. Requirements for a data analysis/statistical package for use by skilled analysts and researchers working with CCWIS data extracts.
H.C2.02		Managing report requests. The agency has an established, documented process to efficiently vet, prioritize, and approve report requests. Some common procedures used by agencies include: Assessing requests against certain factors (such as estimated level-of-effort, addressing a critical need, stakeholder priority, or the impact upon other modules). Determining, as part of initial vetting, if a report request duplicates or is similar to an existing report (by means such as searching the report inventory/catalog to find all reports using the same data or serving a similar purpose).
H.C2.03		Report testing. The agency follows testing procedures that verify report requirements are met, reports are accurate, and stakeholders have confidence in reports. This may include: Testing standards that address: developing test data and scripts; unit, system, and regression testing; prioritizing results; and tracking issue resolution and retesting. Procedures to ensure testing verifies: the accuracy of report calculations; that all records that should be included in a summary report are included; and that records that should not be in the report are excluded. Different testing environments for different purposes. Processes for updating testing environments. Automated testing tools.
H.C2.04		Report testing. If appropriate, testing is flexible and accommodates variations with different testing protocols for different report categories, such as: federal reports; standard CCWIS reports; ad-hoc reports; and data extracts.
H.C2.05		Stakeholder management. The agency encourages stakeholder support for CCWIS with bi-directional communication strategies to ensure reports meet stakeholder needs. To support stakeholder-to-project communication, the agency may train stakeholders on procedures: to request help finding or understanding reports; for providing feedback on reports; and for requesting new reports or enhancements to existing reports. To support project-to-stakeholder communication, the agency may implement processes to: update stakeholders on the status of their report requests; alert stakeholders to possible report errors; and provide updates on the status/resolution of report errors.
H.C2.06		Report documentation. The agency follows a report documentation standard. Agencies report that project report documentation generally: catalogs all requirements (such as those listed in H.C2.01); provides requirements traceability; defines report purpose; documents report designs; documents data sources and calculations; and includes test cases, expected results, and actual results.
H.C2.07		Report documentation. The report inventory/catalog is searchable so the project may easily determine which reports include specified data elements and if report requests duplicate existing reports.
H.C2.08		Report documentation. If appropriate, report documentation standards vary to accommodate different development approaches for different report categories, such as: federal reports; standard CCWIS reports; departmental ad-hoc requests; executive (governor) requests; legislative requests; media requests; and data extracts.
H.C2.09		Report training. The agency provides training to support proper and effective report usage. The project accommodates users with training venues and formats, such as: classroom training; training at work sites; online training; interactive training; refresher training; online screen aids; and online help manuals.
H.C2.10		Report training. The CCWIS report training curriculum: Is integrated with training on child welfare policies and agency practice (for example, intake workers are simultaneously trained on intake policies, processes, and related reports while learning how CCWIS helps them do their work). Helps workers understand the purpose and scope of each report. Introduces workers to the statistical concepts needed to understand reports. Demonstrates how to use reports for problem solving. Explains how to use several reports together to analyze a problem or evaluate issues from different perspectives.
H.C2.11		Report training. The project continuously improves training. The project regularly assesses training effectiveness using tools and metrics such as: training evaluations; user surveys; help desk calls; and report usage statistics. The project revises training to address evaluation findings.
H.C2.12		Reports and change management. CCWIS project staff and program staff collaborate in planning for new program implementations to ensure reports support changing child welfare laws, policies, and practices.
H.C2.13		Reports and change management. The agency promptly informs the project of new report requirements so the project can factor any new screens/procedures/data elements into planning, design, procurements, and APDs. For example, new reporting requirements may be necessary as the result of Child and Family Services Review PIP, findings from title IV-E reviews, or new federal statutes and regulations.
H.C2.14		Reports and change management. The CCWIS project notifies program staff when new or updated data exchanges provide new data so the program can identify opportunities to incorporate the data into practice (such as when CCWIS incorporates a data exchange with the Social Security Administration).
H.C2.15		Reports and change management. The agency has tools to assess scope and identify report changes due to programmatic or data changes (such as new data, revised data definitions, or new picklist values).
H.C2.16		Report design. Report designs support clear, unambiguous understanding and interpretation of reports with features, such as: Report titles that state the report’s purpose and/or question(s) the report addresses. Clear and precise column headings and field labels. Clarifying footnotes that: specify the population included in the report explain report calculations Including run date and time. Permitting users to set selection criteria at runtime. Listing selection criteria on reports (such as the report covering a specified period or including a subset of children/families/providers). Reporting records with missing values for data elements.
H.C2.17		Report design. CCWIS promotes worker efficiency with alerts/ticklers. Automated features may include: CCWIS automatically generates alerts/ticklers based on business rules. CCWIS assists users in efficiently processing alerts/ticklers with features such as: Visual cues to prioritize critical or past due alerts/ticklers (such as color coding or priority ranking). Permitting users to set customized alerts. “One-click” links to the associated task. Automatically clearing of alerts/ticklers when the task is completed.
H.C2.18		Maintaining and improving report quality. The agency continuously works to improve reports using methods such as: Designing CCWIS to automatically gather report-use statistics. Gathering user feedback. Determining if users develop their own reports because CCWIS does not meet the need. Removing or archiving obsolete reports.
H.C2.19		User access to data and reports. The agency has a standard, documented process for approving/terminating users and setting user access levels to data and reports.
H.C2.20		User access to data and reports. CCWIS helps users locate needed reports with strategies, such as topical groupings of reports or links in policy guidance to reports useful for implementing a policy.
H.C2.21		User access to data and reports. Users have options to view summarized data in different formats (e.g., spreadsheets, graphs, and charts).
H.C2.22		User access to data and reports. CCWIS provides support for the agency’s restricted/confidential cases. For example: CCWIS limits access to sensitive/high-profile cases, per agency policy. CCWIS reports include/exclude sensitive/high-profile case data in summary statistics, per agency policy. CCWIS reports include/exclude sensitive/high-profile cases in reports with the capacity to “drill-down” to case data.
H.C2.23		Agency culture. The agency fosters a culture that values high-quality data and reliable reports to support work, decision-making, and policy. Cultural indicators include: Agency leadership reinforces the importance of data quality by using CCWIS reports to manage the program, guide policy, and provide evidence for budget requests. Staff understand the impact reliable data have upon outcomes. Staff are motivated to collect quality data. Staff (e.g., field workers, supervisors, and administrators) have confidence that reports accurately portray the program and activities. User trusts CCWIS-generated performance measures.

We encourage agencies to add examples of additional considerations from their reporting function they wish to highlight.

#	Agency-Submitted Additional Considerations

1 See 45 CFR 1355.52(a).

“Reports” for the purposes of this tool includes, in addition to federal data submissions (such as the Adoption and Foster Care Analysis and Reporting System (AFCARS)), any presentation of data to a user, including formats such as hard-copy and online reports, graphs, spreadsheets, online dashboards, and data extracts provided to researchers and data analysts.

2 See 45 CFR 1355.52(c)

3 45 CFR 1355.52(c)(1).

PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN:  Through this information collection, the Administration for Children and Families (ACF) is collecting information to document that title IV-E agencies have planned and developed their system’s conformity to federal CCWIS and Advance Planning Document requirements.  Public reporting burden for this collection of information is estimated to average 10 hours per title IV-E agency choosing to develop and implement a CCWIS system, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information.  This is a voluntary collection of information.  An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number.  The OMB # is 0970-0568 and the expiration date is 04/30/2024.