Usability Testing of Resources to Support the Identification and Care of Children with Prenatal Substance or Alcohol Exposure in the Child Welfare System

Appendix A. PAE usability testing interview protocol

OMB Control No.: 0970-0531

Expiration Date: 07/31/2022

THE PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN:

Public reporting burden for this collection of information is estimated to average 5 hours per response for local child welfare agency directors, site managers, and specialist child welfare staff, who will review two toolkit components and participate in a single 60-minute interview. Burden is estimated to average 27.5 hours (administered in 5 parts of 5.5 hours each) for other child welfare staff (e.g., frontline staff and supervisors), who will review up to eight toolkit components each and participate in five 90-minute group interviews. These estimates include the time for reviewing instructions, gathering, and maintaining the data needed, and reviewing the collection of information. This is a voluntary collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #0970-0531, Exp. 07/31/2022. If you have any comments on this collection of information, please contact Sharon Newburg-Rinn, Ph.D., ACF, Administration on Children, Youth, and Families by email at Sharon.Newburg-Rinn@acf.hhs.gov.

Identification and Care of Children with Prenatal Alcohol and Other Drug Exposures: Focus Group/Interview Questions for Toolkit Usability Testing

• Interviewer initials:

• Notetaker initials:

• Site #/Participant ID(s):

• Date/Time:

• Toolkit components reviewed during interview/focus group:

[Brackets]=instructions for interviewer

Italics= to be read by interview facilitator

Data Collection Overview

[Briefly discuss the purpose of the interview/focus group]: The purpose of our conversation is to gather feedback on select components of the Toolkit: Identification and Care of Children with Prenatal Alcohol and Other Drug Exposures. This toolkit is currently under development, and we will be asking you questions about your perceptions regarding the toolkit’s: usefulness, ease of use, your attitudes towards using the toolkit, any anticipated implementation resource needs, and areas for improvement. Your feedback is valued and will be used to identify needed refinements to the toolkit.

[Convey to each interview participant our privacy policy]: (1) the [insert interview/focus group] is voluntary; (2) you can decline to answer any questions, or you can stop participating in the at any time; (3) feedback from this [insert interview/focus group] will be held in private by the project team and your name will not be associated with any feedback you provide. The [insert interview/focus group] should take [insert estimate time for interview/focus group] minutes.

[Ask permission to tape record the interview/focus group]: In order to ensure we capture the discussion accurately and completely, I would like to ask for you permission to audio record the session. Only our project team will have access to this recording. Will you allow us to record this interview?

[If respondent(s) decline, the notetaker will still be taking notes]

[If respondent agrees, the interviewer asks the notetaker to start the recording.]

Thank you, the recording has started.

[Ask if they have any questions for you before you begin. Ask them to provide their verbal consent to participate in the interview.]

Respondent background/role questions

1. Please describe your education and position at your agency.

2. Briefly tell me about the activities you currently conduct as part of your job.

3. How many years have you worked in at this agency?

4. How many years have you worked in child welfare?

Toolkit component: The role of child welfare in helping families affected by prenatal substance exposures [Data collection administration: Group 1]

The learning objectives of the “Role of Child Welfare in Helping Families Affected by Prenatal Alcohol Exposures” toolkit component include:

1. Gain background knowledge on the prevalence of prenatal alcohol exposures in the child welfare population and learn how they affect child welfare outcomes

2. Hear about the current approaches being used in child welfare to address prenatal alcohol exposures

3. Learn about the benefits of identifying and providing services to children with prenatal alcohol exposures who are involved in the child welfare system

4. Understand the role of child welfare professionals and agencies in supporting children and families affected by prenatal alcohol exposures

1. To what extent were these learning objectives reflected in the content you reviewed? Probe for each learning objective.

2. Did you think the structure of this toolkit component was clearly organized? Why or why not?

3. Was this toolkit component content easy to understand? Why or why not?

4. Was the amount of detail in the toolkit component appropriate? Probe: was the amount of detail provided in the toolkit actionable? Did the toolkit motivate you to take action? Why or why not?

5. Do you understand how the toolkit component is intended to be used? Probe: do you understand what is intended to be achieved by using this toolkit section?

6. This toolkit component is primarily focused on providing information and resources related to prenatal alcohol exposure (or PAE). Do you think the toolkit should place a greater emphasis on providing information and resources related to prenatal exposure to other types of substances? If so, how can we achieve a better balance?

7. In what ways, if any, do you think the toolkit component could be used to enhance your work?

8. What is the likelihood that you would implement the toolkit component in your work?

9. Do you have any other suggestions for ways toolkit component could be improved? Probe: Any content that is missing? Confusing?

Toolkit component: Agency self-assessment

[Data collection administration: Group 1]

The learning objectives of the “Agency Self-assessment” toolkit component include:

1. Help agency leaders assess their processes, policies, and partnerships for the identification and care of children, youth, and families affected by PAE/PSE.

2. Help determine the sections of the toolkit relevant to the agency’s needs.

3. Self-identify what pieces of the toolkit to use within their agency.

2. To what extent were these learning objectives reflected in the content you reviewed? Probe for each learning objective.

3. Did you think the structure of the “Agency Self-assessment” toolkit component was clearly organized? Why or why not?

4. Are the instructions/guidance for the agency self-assessment easy to understand? Why or why not?

5. Does the agency self-assessment help to assess an agencies current processes, policies and partnerships? Probe: can you think of any additional questions that could be added to help an agency assess their current processes, policies, and partnerships?

6. Does the agency self-assessment help determine the sections of the toolkit relevant to the agency’s needs? Probe: When the agency self-assessment identified a section of the toolkit that was relevant to the agency’s needs, was it easy to find that section of the toolkit to utilize?

7. Is the agency self-assessment easy to use without the need for technical assistance? Probe: If technical assistance was needed to use the agency self-assessment, what type of technical assistance would be useful? Are you likely to complete the entire assessment?

8. What would it take for your agency to implement this component? Probe: How long would it take? how feasible is it? Who would you need buy in or approval from, etc.?

9. What obstacles, if any, would need to be overcome in order for your agency to implement this component? Probe: lack of resources? Buy-in from staff?

10. Do you have any other suggestions for ways the “Agency Self-assessment” toolkit component could be improved? Probe: any content missing? Confusion?

Toolkit Component: Screen and identify

[Data collection administration: Group 2]

The learning objectives of the “Screen and identify” toolkit component include:

1. Understand the role of child welfare agency leaders, supervisors, and caseworkers in screening and identifying children and youth prenatally exposed to alcohol and/or other types of substances.

2. Learn about the role of partners in other systems and the value of collaborative approaches to identify and evaluate children and youth prenatally exposed to alcohol.

3. Learn about working in partnership with families and caregivers to screen for prenatal alcohol exposure. [If clarification is needed, for the purposes of the toolkit, “caregivers” include parents, foster parents, kinship caregivers, foster-to-adopt parents, and adoptive parents of children involved with the child welfare system and in out-of-home care.]

1. To what extent were these learning objectives reflected in the content you reviewed? Probe for each learning objective.

2. Did you think the structure of the “Screen and identify” toolkit component was clearly organized? Why or why not?

3. Was the “Screen and identify” toolkit component content easy to understand? Why or why not? Probe: Would the screening script be easy to use with families? Why or why not?

4. Was the amount of detail in the “Screen and identify” toolkit component appropriate?

Probe: was the amount of detail provided in the toolkit actionable? Did the toolkit motivate you to take action? Why or why not?

5. Do you understand how the “Screen and identify” toolkit component is intended to be used? Probe: do you understand what is intended to be achieved by using this toolkit section?

6. Will the tools and resources provided in the toolkit support policy and procedure changes? Are the instructions and examples helpful? Why or why not? Probe: specific types of policy and procedures changes focused on in this section:

   1. Are the tools and resources provided in the toolkit sufficient to make policy and procedure changes for collecting information on maternal alcohol and drug use during pregnancy when working with children of all ages?

7. What is the likelihood that you would implement the “Screen and identify” toolkit component in your work? Probe if not mentioned: Likelihood to use the resources and scripts to screen for prenatal substance exposure? Likelihood to document the results of the prenatal exposure screening script?

8. Are the instructions for using the screening sample scripts clear? Probe: are they prescriptive enough? Too prescriptive?

9. [Ask of supervisors] Would you have the information you would need to coach staff on using the screening sample scripts? Please explain.

10. What would it take for your agency to implement this component? Probe: How long would it take? how feasible is it? who would you need buy in or approval from, etc.?

11. What obstacles, if any, would need to be overcome for your agency to implement this component? Probe: lack of resources? Buy-in from staff?

12. Do you have any other suggestions for ways “Screen and identify” toolkit component could be improved? Probe: Any content that is missing? Confusing?

Toolkit Component: Refer and integrate services

[Data collection administration: Group 2]

The learning objectives of the “Refer and Integrate Services” toolkit component include:

1. Learn how to identify FASD and other services in your community—including culturally responsive and specific services—and how to determine when to refer children and families to those services

2. Determine best practices for developing or enhancing universal referral forms for neurodevelopmental and FASD evaluations and services

3. Hear how to effectively engage caregivers in a way that encourages participation in neurodevelopmental and FASD evaluations and services.

[If clarification is needed, for the purposes of the toolkit, “caregivers” include parents, foster parents, kinship caregivers, foster-to-adopt parents, and adoptive parents of children involved with the child welfare system and in out-of-home care.]

4. Understand when and how to exchange information with caregivers and service providers relevant to the screening, evaluation, and diagnosis of FASDs

1. To what extent were these learning objectives reflected in the content you reviewed? Probe for each learning objective.

2. Did you think the structure of the “Refer and Integrate Services” toolkit component was clearly organized? Why or why not?

3. Was the “Refer and Integrate Services” toolkit component content easy to understand? Why or why not?

4. Was the amount of detail in the “Refer and Integrate Services” toolkit component appropriate? Probe: was the amount of detail provided in the toolkit actionable? Did the toolkit motivate you to take action? Why or why not?

5. Do you understand how the “Refer and Integrate Services” toolkit component is intended to be used? Probe: do you understand what is intended to be achieved by using this toolkit section?

   1. Is this toolkit component useful in helping to understand the types of services that are effective for children with FASDs? Why or why not?

   2. Is this toolkit component useful in helping to learn about the FASD services available in the community? Why or why not?

   3. Is the toolkit component useful in building out the service array? Why or why not?

6. Will the tools and resources provided in the toolkit support policy and procedure changes? Are the instructions helpful? Why or why not? [Probe: specific types of policy and procedures]. Are the tools and resources provided in the toolkit sufficient to make policy and procedure changes for:

   1. the development or the modification of a universal referral form that incorporate consent and release of information? Why or why not?

   2. changes on determining which families need services and what services these families need? Why or why not?

   3. providing caseworkers with the guidance on how to engage families with supportive services? Why or why not?

7. What is the likelihood that you would implement the “Refer and Integrate Services” toolkit component in your work? Probe (if not mentioned): Likelihood to develop or modify a universal referral form that can be used with all providers?

8. Are the instructions for the resources and tools in this toolkit component helpful to decide which families need to be referred to services and, if referrals are made, what issues need attention? Why or why not?

9. Does the toolkit provide clear instructions on what the critical pieces of information are needed when documenting service referrals?

   1. Are the instructions in this toolkit component on the documentation of services compatible with agency requirements for casework documentation?

10. What would it take for your agency to implement this component? Probe: How long would it take? how feasible is it? who would you need buy in or approval from, etc.?

11. What obstacles, if any, would need to be overcome for your agency to implement this component? Probe: lack of resources? Buy-in from staff?

12. Do you have any other suggestions for ways “Refer and Integrate Services” toolkit component could be improved? Probe: Any content that is missing? Confusing?

Toolkit Component: Partner with and support caregivers

[Data collection administration: Group 3]

The learning objectives of the “Partner with and Support Caregivers” toolkit component include:

[If clarification is needed, for the purposes of the toolkit, “caregivers” include parents, foster parents, kinship caregivers, foster-to-adopt parents, and adoptive parents of children involved with the child welfare system and in out-of-home care.]

1. Understand the importance of developing and providing training and written materials to caregivers to help them recognize and understand the effects of prenatal alcohol exposure, and to care for children who have experienced it.

2. Learn about resources available to evaluate children with possible prenatal alcohol exposure and how to assist families in accessing those services.

3. Access resources that can be shared with caregivers about the signs and symptoms of possible prenatal alcohol exposure, the screening and evaluation process for possible FASDs, and the benefits of a diagnosis.

4. Access resources that can be shared with caregivers to increase their knowledge of the effects of prenatal alcohol exposure on child development, parenting strategies, and support for children/youth living with an FASD

5. Access resources that can be shared with youth living with an FASD to help them understand their diagnosis.

1. To what extent were these learning objectives reflected in the content you reviewed? Probe for each learning objective.

2. Do you think the structure of the “Partner with and Support Caregivers” toolkit component was clearly organized? Why or why not?

3. Was the “Partner with and Support Caregivers” toolkit component easy to understand? Why or why not?

4. Was the amount of detail in the “Partner with and Support Caregivers” toolkit component appropriate? Probe: was the amount of detail provided in the toolkit actionable? Did the toolkit motivate you to act? Why or why not?

5. Do you understand how the “Partner with and Support Caregivers” toolkit component is intended to be used? Probe: do you understand what is intended to be achieved by using this toolkit section?

6. Are the tools and resources provided in the toolkit sufficient to develop and provide training and written materials to caregivers? Why or why not? Probe: Will the training be sufficient to build knowledge and skills for caregivers? What additional supports or resources are needed for the further development of knowledge and skills for caregivers?

7. What is the likelihood that you would implement the “Partner with and Support Caregivers” toolkit component in your work? Probe (if not mentioned): likelihood of giving the written materials out to caregivers. Was there sufficient guidance in the toolkit on the steps needed to create resources be available in languages that align with the caregiver’s native language? Was there sufficient information in the toolkit on how to align the written materials with caregivers culturally?

8. Did the toolkit provide sufficient information about the level of engagement expected of families? Please explain.

9. What would it take for your agency to implement this component? Probe: How long would it take? how feasible is it? who would you need buy in or approval from, etc.?

10. What obstacles, if any, would need to be overcome in order for your agency to implement this component? Probe: lack of resources? Buy-in from staff?

11. Do you have any other suggestions for ways “Partner with and Support Caregivers” toolkit component could be improved? Probe: Any content that is missing? Confusing?

Toolkit Component: Collect, document, and share information

[Data collection administration: Group 3]

The learning objectives of the “collect, document, and share information” toolkit component include:

1. Understand the importance of collecting and documenting accurate information on prenatal alcohol exposure in child welfare case files.

2. Learn how to consistently document and use information in child welfare case files to improve the identification of—and service provision for—families affected by prenatal alcohol exposure.

3. Hear how sharing case file information with service providers can enhance service provision for families affected by prenatal alcohol exposure and learn effective information-sharing strategies.

4. Gain insight into how continuous quality improvement (CQI) and evaluation efforts can improve child welfare practice around prenatal alcohol exposure.

1. To what extent were these learning objectives reflected in the content you reviewed? Probe for each learning objective.

2. Did you think the structure of the “collect, document, and share information” toolkit component was clearly organized? Why or why not?

3. Was the “collect, document, and share information” toolkit component content easy to understand? Why or why not?

4. Was the amount of detail in the “collect, document, and share information” toolkit component appropriate?

Probe: was the amount of detail provided in the toolkit actionable? Did the toolkit motivate you to take action? Why or why not?

5. Do you understand how the “collect, document, and share information” toolkit component is intended to be used? Probe: do you understand what is intended to be achieved by using this toolkit section?

6. Will the tools and resources provided in the toolkit support policy and procedure changes? Are the instructions and examples helpful? Why or why not? [Probe: specific types of policy and procedures]. Are the tools and resources provided in the toolkit sufficient to make policy and procedure changes for:

1. including information on and documenting in the case record maternal drug and alcohol use during pregnancy when working with children of all ages? Please explain.

2. including information on documenting children who may have been exposed to substances prenatally? Please explain.

7. In what ways, if any, do you think the “collect, document, and share information” toolkit component could be used to enhance your work?

8. What is the likelihood that you would implement the “collect, document, and share information” toolkit component in your work?

9. [Ask of agency leadership such as supervisor/managers/director/administrators]: What is the likelihood that you would incorporate the toolkit content into your policy, practices, and procedures to include:

   1. Providing training or addressing HIPAA rules for sharing information. Please explain. ______

   2. Building new data elements into your data system. Please explain. ______

   3. Developing new processes or revising existing processes to use data for strategic planning. Please explain. ______

   4. Developing a plan for how they would use data gathered through screening for evaluation/CQI purposes. Please explain. ______

10. [Ask of agency leadership such as supervisor/managers/director/administrators]: Are instructions for documentation compatible your agency’s CWIS technologies?

Please explain._________

11. Is the documentation content instructive for others not directly involved with the investigation/assessment (e.g., supervisor, caregivers, providers)? Is it adequate for planning and follow-up? Please explain.___________

12. What would it take for your agency to implement this component? Probe: How long would it take? how feasible is it? who would you need buy in or approval from, etc.?

13. What obstacles, if any, would need to be overcome in order for your agency to implement this component? Probe: lack of resources? Buy-in from staff?

14. Do you have any other suggestions for ways the “collect, document, and share information” toolkit component could be improved? Probe: Any content that is missing? Confusing?

Toolkit component: Plan

[Data collection administration: Group 4]

The learning objectives for the “Plan” toolkit component include:

1. Learn about strategies to effectively partner with key stakeholders to plan and implement cross-system processes to screen, identify, evaluate, and support children and families who experienced prenatal alcohol and other drug exposures.

2. Learn how to effectively partner with key stakeholders and develop cross-system processes to identify, evaluate, and support children and families affected by prenatal alcohol exposure.

2. To what extent were these learning objectives reflected in the content you reviewed? Probe for each learning objective.

3. Do you think the structure of the “Plan” component was clearly organized? Why or why not?

4. Was the “Plan” toolkit component easy to understand? Why or why not?

5. Was the amount of detail in the “Plan” toolkit component appropriate? Probe: was the amount of detail provided in the toolkit actionable? Did the toolkit motivate you to take action? Why or why not?

6. Do you understand how the “Plan” toolkit component is intended to be used? Probe: do you understand what is intended to be achieved by using this toolkit section? Is the toolkit clear about the implementation team’s purpose?

7. Are tools and resources provided in the toolkit sufficient to provide guidance on which key stakeholders are needed? Are the tools and resources sufficient in providing guidance on how to get key stakeholders involved? Are the instructions and examples helpful? Why or why not? Probe: for whether the information about the engagement of families was sufficient.

   1. Is the toolkit clear about who to include in the implementation team and why they are relevant?

   2. For team members not yet engaged, does the toolkit provide adequate instructions on how to get them involved?

8. Are there any key stakeholders missing from the toolkit that should be added? If so, who are those key stakeholders and why should they be added?

9. What would it take for your agency to implement this component? Probe: How long would it take? how feasible is it? who would you need buy in or approval from, etc.?

10. What obstacles, if any, would need to be overcome in order for your agency to implement this component? Probe: lack of resources? Buy-in from staff?

11. What is the likelihood that you would implement the “Plan” toolkit component in your work? Probe if not mentioned: Likelihood to engage an implementation team in supporting their implementation process.

12. Do you have any other suggestions for ways “Plan” toolkit component could be improved? Probe: Any content that is missing? Confusing?

Toolkit component: Train

[Data collection administration: Group 4]

The learning objectives for the Train toolkit component include:

1. Understand the importance of child welfare staff training that will improve and support the screening, identification, and evaluation of and service provision for families affected by prenatal alcohol and other drug exposures.

2. Access training modules developed for child welfare staff that includes content on substance use, addiction, the effects of prenatal alcohol exposure on child development, and how prenatal exposure may affect child safety, permanency, and well-being.

3. Access training modules on the importance of screening for and identifying prenatal alcohol exposures in addition to other drugs.

4. Learn how to implement policies, processes, and tools to improve the identification of and service provision for families affected by prenatal alcohol exposure.

1. To what extent were these learning objectives reflected in the content you reviewed? Probe for each learning objective.

2. Did you think the structure of “Train” toolkit component was clearly organized? Why or why not?

3. Was the “Train” toolkit component easy to understand? Why or why not?

4. Was the amount of detail in the “Train” toolkit component appropriate? Probe: was the amount of detail provided in the toolkit actionable? Did the toolkit motivate you to take action? Why or why not?

5. Do you understand how the “Train” toolkit component is intended to be used? Probe: do you understand what is intended to be achieved by using this toolkit section?

6. What is the likelihood you would incorporate the training modules and resources into training? Probe for the following:

   1. If a new training is needed to be developed at your agency, what is the likelihood that the training modules and resources provided in the “Train” toolkit would be used to develop this new training?

   2. If training already exists at your agency, what is the likelihood that the training modules and resources provided in the “Train” toolkit would be used and incorporated into or adapted for that existing training?

   3. Do the training modules and resources provided in the “Train” toolkit need any additional customization for use? Why or why not?

   4. Were the training modules and resources within the toolkit in a format that could be adapted and used in a cost-efficient way? Why or why not?

7. Do the training modules and resources within the “Train” toolkit component have the right amount of detail or was it too much or too little. Please explain.

8. What would it take for your agency to implement this component? Probe: How long would it take? how feasible is it? who would you need buy in or approval from, etc.?

9. What obstacles, if any, would need to be overcome in order for your agency to implement this component? Probe: lack of resources? Buy-in from staff?

10. Do you have any suggestions for ways “Train” toolkit component could be improved? Probe: Any content that is missing? Confusing?

Toolkit component: Overall toolkit design/content

[Data collection administration: Group 5]

We will be asking you questions about the overall toolkit and ask you to share specific examples from the toolkit in explaination.

[Required questions for toolkit component wireframes]

1. As you reviewed the toolkit, was the overall toolkit clear? Please explain.

Probe: What do you think is the purpose of the toolkit?

The overall objectives of the toolkit include:

1. To increase awareness, understanding, and knowledge of prenatal alcohol and other substance exposure; and,

2. To plan and implement processes that help identify, evaluate, share information, and provide care and support to families and children living with and FASD.

2. To what extent were these learning objectives reflected in the toolkit? Probe for each learning objective.

3. Do you understand who the intended audience is for the toolkit? Please explain.

4. Would an online toolkit with downloadable content/resources be easy for you to access? Why or why not?

[Optional questions depending on toolkit roll-out at the time of usability testing]

5. Does the toolkit have an appealing look and feel? Please explain. Probe: Did the toolkit format include the right balance of text and graphics? Why or why not?

6. Does the toolkit have a professional appearance? Why or why not?

7. Was the toolkit easy to navigate? Were you able to easily find topics related to your work? Why or why not?

8. Was the language used in the toolkit person-first and inclusive? Please explain. [If clarification is needed, “person first language” puts the person before the disability, and describes what a person has, not who a person is (e.g., children with an FASD vs. “disabled children”. “Inclusive language” aims to avoid expressions that imply ideas that are sexist, racist, or otherwise biased, prejudiced, or denigrating to any particular group of people.]

Thank you for your time and participation!

16