SSA - PAE Usability Testing

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Usability Testing of Resources to Support the Identification and Care of Children with Prenatal Substance or Alcohol Exposure in the Child Welfare System

Formative Data Collections for Program Support

0970 – 0531

Supporting Statement

Part A

January 2022

Submitted By:

Children’s Bureau

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Sharon Newburg-Rinn, Ph.D.

Social Science Research Analyst

Data Analytics and Reporting Team

Children’s Bureau, ACF, HHS

330 C Street SW Room 3042

Washington, DC 20201

Sharon.Newburg-Rinn@acf.hhs.gov

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Formative Data Collections for Program Support (0970-0531).

• Description of Request: This proposed information collection effort will gather feedback from end users of a toolkit of resources sponsored by the Children’s Bureau in collaboration with the Centers for Disease Control and Prevention under an interagency agreement. The toolkit is intended to support child welfare agency staff in the identification and support of children living with prenatal exposure to alcohol and other substances. The information collected (via interviews) will tell us the extent to which agency staff believe the toolkit resources would enhance child welfare practice and be easy to use, their attitudes toward use of the resources, their perspectives on resources and supports needed to implement toolkit-supported tasks, and their perspectives on resource content that should be modified. This feedback from the field will be shared with the resource development team, to allow them to make needed refinements and modifications to the toolkit before it is rolled out and evaluated. We do not intend for this information to be used as the principal basis for public policy decisions.

A1. Necessity for Collection

The Children’s Bureau (CB) in the Administration on Children, Youth and Families (ACYF), Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS) is partnering with the Centers for Disease Control and Prevention (CDC) to support child welfare agency staff in the identification and support of children in the child welfare system living with prenatal substance exposure (PSE), particularly including prenatal alcohol exposure (PAE). To this end, we are developing a toolkit of resources that child welfare agencies can use to:

• Increase awareness, understanding, and knowledge of PAE/PSE; and

• Plan and implement internal and cross-system processes, in partnership with key stakeholders, that help identify, assess/evaluate/screen, share information about, and provide care and support to children with prenatal exposure and their families. 

The toolkit comprises multiple components (i.e., sections). As components of the toolkit are being developed, feedback is needed from child welfare staff in the field to determine the usability of each component before the toolkit is finalized, rolled out, and evaluated.

There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

The purpose of the proposed data collection effort is to solicit feedback from the intended audience of the toolkit for the purposes of usability testing. A core function of usability testing is to quickly assess the adequacy of the toolkit components and to detect deficiencies that require correction by the toolkit development team before the toolkit is finalized and rolled out for implementation. The key activities will provide information on users’ perspectives about the usefulness and ease of use of the toolkit; identify implementation processes that are anticipated to be challenging or critical to implementing the toolkit with integrity; and gather feedback about how the toolkit can be improved. The goal of the usability testing process is to get to a stable, usable version of the toolkit and an understanding of the technical supports that are needed for implementation of the toolkit. The toolkit can then be piloted through formative evaluation and, eventually, summative evaluation. (Note: data collection activities for the formative and summative evaluations will be submitted through a separate information collection request at a later date).

Information will be synthesized by the study team and shared with the toolkit development team, who will use the information to improve the toolkit. Improved sections of the toolkit may then be shared back with the users for additional feedback, if needed.

This proposed information collection meets the following goals of ACF’s generic clearance for formative data collections for program support (0970-0531):

• Delivery of targeted assistance related to program implementation or the development or refinement of program and grantee processes.

• Planning for provision of programmatic or evaluation-related training or technical assistance (T/TA).

• Use of rapid-cycle testing activities to strengthen programs in preparation for summative evaluation.

Guiding Questions

The proposed information collection is guided by five usability testing questions, which will be applied to user review of each of the eight toolkit components:

1. Do toolkit users believe the toolkit component would enhance child welfare practice?

2. What are users’ perspectives on the component’s ease of use?

3. What are users’ perspectives on whether they are likely to use the component on the job?

4. What are users’ perspectives on the implementation resources and supports that may be needed to implement the policy, process, and practice-improving approaches proposed in the toolkit component?

5. Is there content in the component that users believe should be modified?

Study Design

The proposed descriptive study will use data collected through interviews to respond to the five guiding questions above (see B1 in Supporting Statement Part B for additional detail). To conduct this study, two to three local child welfare agencies (i.e., sites) within two states will be recruited to participate in the usability testing process (see section B2. in Supporting Statement Part B for additional detail). The study team will work closely with agency directors to identify individual staff or teams who represent the types of agency roles that might typically use the toolkit (e.g., supervisors, investigation/intake staff, ongoing case management staff). The study team will then work with agency leadership to develop a written approach for each agency, detailing how agency staff will sequence and review the toolkit content and document their feedback.

Feedback will be collected from toolkit users via interviews. Child welfare agency directors and staff in specialist roles will be interviewed individually. Child welfare agency supervisors and staff – who typically work together in supervisory teams – will be interviewed in groups. The purpose of the interviews and the protocol of interview questions is the same for individual and group interviews. The key difference between them is the group interviews are expected to take longer than individual interviews due to the number of participants (see table A-1).

The feedback collected from toolkit users via interviews will be synthesized by the study team and the synthesized data will be shared with the toolkit resource development team and the federal sponsors for quality improvement purposes. In limited situations, improvements to the toolkit that are made by the resource development team in response to this feedback may be shared back with the users for additional review and reflection. This cyclic process will be complete once perspectives on toolkit usability appear to converge, and a stable version of the toolkit is ready for a standard formative evaluation.

This study design is appropriate for usability testing because it (1) engages end users to provide perspectives on the likely functionality of the toolkit in the field; (2) minimizes the burden on study respondents through the use of interviews that are intended to target feedback on the key aspects of toolkit quality; and (3) aims to improve the validity of the toolkit as an intervention, prior to the start of a subsequent evaluation of its outcomes¹. The findings from this study are not intended to be representative of or generalizable to a larger population, yet the project’s intent is to gain useful information regarding utility and feasibility of the toolkit and its components and to learn about how it may work in different child welfare agency contexts.

The specific instrument used to gather data for usability testing is outlined in table A-1 (see appendix A for a copy of this instrument). More information about the collection of these data is available in section B4 of Supporting Statement B.

Table A-1. Data collection activities

Data Collection Activity	Instrument(s)	Respondent, Content, Purpose of Collection	Mode and Duration
Interviews with child welfare agency staff to assess usability features and provide feedback on select toolkit components	Usability testing interview – Individual	Respondents: Child welfare agency directors (3 total) and specialist roles (e.g., data staff, substance use response program manager) (6 total) Content: Perceptions of usefulness, ease of use, attitudes toward use, implementation resource needs, and areas for improvement Purpose: To provide practice-informed perspectives on toolkit content and how best to implement the toolkit in child welfare contexts. This information will be used to improve the toolkit before evaluation.	Mode: Individual interview Duration: 1 hour
	Usability testing interview – group	Respondents: Child welfare agency supervisors and staff (e.g., intake/investigation, ongoing case management, prevention, permanency) (12 total) Content: Perceptions of usefulness, ease of use, attitudes toward use, implementation resource needs, and areas for improvement Purpose: To provide practice-informed perspectives on toolkit content and how best to implement the toolkit in child welfare contexts. This information will be used to improve the toolkit before formal evaluation.	Mode: Group interview Duration: Administered over 5 sessions of 1.5 hours each

Other Data Sources and Uses of Information

The information collection described here represents the whole of the study; no other sources of information (e.g., administrative data sources) would be collected or used in the study.

A3. Use of Information Technology to Reduce Burden

Data primarily will be collected through interviews conducted via telephone or video conference technology. With the permission of informants, interviews will be audio recorded and transcribed to maximize detailed and accurate notes and to minimize the need to go back to informants to clarify what was said.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

The proposed information collection represents a unique source of data that is not already available. The toolkit is an innovation still under development; as such, there are no existing sources of data that could be used to understand the usability of this specific resource.

A5. Impact on Small Businesses

No small businesses will be involved with this information collection.

A6. Consequences of Less Frequent Collection

This study will use rapid-cycle improvement processes over the course of several months (see section A16); feedback collected from toolkit users will be used to improve the toolkit and, in some limited cases, users may be asked to review and comment on the improvements. Not engaging respondents in a review of any subsequent modifications to the toolkit could reduce the utility of the usability testing process, which is intended to engage users in a feedback loop to develop a stable version of the toolkit that is ready for formative evaluation.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published two notices in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for formative information collection. The first notice was published on October 13, 2020, Volume 85, Number 198, page 64480, and provided a sixty-day period for public comment. The second notice was published on December 28, 2020, Volume 85, Number 248, page 84343, and provided a thirty-day period for public comment. ACF did not receive any substantive comments.

Consultation with Experts Outside of the Study

The study team has worked with selected experts and stakeholders regarding states they perceived were demonstrating strong practice in identifying and caring for children with PAE/PSE. The team took these perspectives into account when developing a preliminary list of state agencies to engage in discussions about participating in this study.

A9. Tokens of Appreciation

Non-monetary program support will be provided to participating agencies, as described in section A13.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

No personally identifiable information will be collected through this data collection instrument.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data. They will be informed that their participation is voluntary, and that their information will be kept private to the extent permitted by law. Participants will be informed that interviews will be audio recorded only with their permission. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information. The full study protocol including recruitment, consenting process, administration, analysis, and reporting will be approved by the study Institutional Review Board.

Data Security and Monitoring

The contract team in this study has developed a Data Safety and Monitoring Plan that assesses all protections of respondents’ Personally Identifiable information (PII). The Contractors ensure that all their employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.

As specified in the evaluators' contracts, the Contractors shall use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. Contractors shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractors shall: ensure that this standard is incorporated into the Contractors’ property management/control system; establish procedures to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractors must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or PII that ensures secure storage and limits on access.

A11. Sensitive Information ²

No questions of a sensitive nature are included in these evaluations.

A12. Burden

Explanation of Burden Estimates

Table A-2 includes the estimates of the response burden for the usability testing study. The total annual response burden is estimated to be 375 hours. The interview protocol has multiple sections, each corresponding to a component of the toolkit. The response burden includes the time study participants will spend reviewing components of the toolkit and forming their impressions of the usability of those components (this is estimated to take 2 hours per component) plus the time it would then take to participate in an individual or group interview about the reviewed components. The estimates also include time to re-review up to two components that will be updated based upon participants’ original feedback and group interview about the revised components. The number of components reviewed will vary by participant role in their agency:

• Local child welfare agency directors or site managers (3 total) will review two toolkit components and participate in 1-hour individual interviews.

• Specialist child welfare staff (e.g., data staff) (6 total) will review two toolkit components and participate in 1-hour individual interviews.

• Other child welfare staff (e.g., frontline, supervisors) (12 total) will review up to 8 toolkit components each (with additional review of up to 2 revised components) and participate in 5 group interviews (group interviews will cover 2 components at a time).

Estimated Annualized Cost to Respondents

After applying hourly wage estimates to burden hours in each respondent category, the current annual cost to the respondents is $11,241.75. This cost information is based on the most current data available (May 2020). For labor categories, the mean hourly wage for Local Child Welfare Agency Director ($60.45) (comparable BLS category is 11-1021 “General and Operations Manager”) and Child Welfare Specialist ($48.60) (comparable BLS category is 15-1245 “Database Administrators”) were used for respondents participating in the individual interviews and Local Child Welfare Staff ($26.90) (comparable BLS category is 21-1020 “Social Workers”) was used for respondents participating in the group interviews. Labor categories and wage information were obtained from the following website: https://www.bls.gov/oes/current/oes_nat.htm#00-0000,%2021-0000,%20Community%20and%20Social%20Service%20Occupations%20mean%20hourly%20=%2021.79

Table A-2. Estimated Annualized Burden and Cost to Respondents

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total/ Annual Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Usability Testing Interview Protocol – Individual administration	9	1	5	45	$60.45 (3 directors) $48.60 (6 specialists)	$2,364.75
Usability Testing Interview Protocol – Group administration	12	1	27.5	330	$26.90	$8,877.00
Total				375		$11,241.75

A13. Costs

To enhance motivation to participate, the study team will provide non-monetary supports in the form of access to expert consultants and recognition in published reports (e.g., including names of reviewers on the credit page) to agency sites participating in the usability testing. While the qualitative data from semi-structured interviews are not intended to be statistically generalizable, the study’s findings will be more relevant for practice if the study team is able to secure participation from a range of agency staff teams, including agency staff teams that may face time barriers to study participation. The non-monetary supports are intended to compensate agencies for the time and opportunity costs of participation in the study (e.g., staff time away from other necessary work).

A14. Estimated Annualized Costs to the Federal Government

The estimated costs to the federal government for the data collection are indicated in table A-3. The estimates include the loaded costs and fees of study team staff and federal project leadership time on field work (including site participant review of toolkit components and participation in interviews) and study team administration of the interviews and analysis of data. The annual cost to the federal government for this collection is $24,845.09.

Table A-3. Estimated annualized costs to the federal government

Cost Category	Estimated Costs
Field work (site participant review, feedback, interviews)	$11,241.75
Study team administration and analysis	$13,603.34
Total costs over the request period	$24,845.09

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella formative generic clearance for program support (0970-0531).

A16. Timeline

The usability testing study will be conducted across 5 months. Participants will review two toolkit components at a time and will typically participate in an individual or group interview within 7-10 days of completing their review. (The total number of components a given participant will review is detailed in Explanation of Burden Estimates above.) The final interview will gather usability information about the full toolkit. The anticipated pace of the study is for participants to complete the review/interview for a pair of toolkit components every 4-5 weeks.

A17. Exceptions

No exceptions are necessary for this information collection.

1 A separate information collection request for these data collection activities will be submitted.

2

9