SSB - Prog Supp GenIC: PREP PYP Curriculum Adaptation

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Data Collection to Inform a Curriculum Adaptation for the Personal Responsibility Education Program (PREP): Promising Youth Programs (PYP) Project

Formative Data Collections for Program Support

0970 - 0531

Supporting Statement

Part B

April 2022

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Selma Caal (COR)

Project Monitor:

Kathleen McCoy

Part B

B1. Objectives

Study Objectives

This study aims to collect data on the implementation of a two-lesson curriculum for youth with intellectual disabilities (IDD), which was adapted from a curriculum originally designed for a general school population. The findings from the data collection will be used to refine and finalize the adapted curriculum to ensure it is appropriate and accessible for youth with IDD, who are served by PREP grantees, and ultimately inform future technical assistance efforts provided to PREP grantees serving these youth.

Generalizability of Results

This study is intended to present internally valid description of the implementation of the curriculum adaptation in chosen sites, not to promote statistical generalization to other sites or service populations. However, the data will be used to refine and finalize the content of the curriculum adaptation to ensure it can be implemented with the youth with IDD. The final curriculum adaptation will be available for ACF grantees and the public to use to educate youth with IDD on Internet safety.

Appropriateness of Study Design and Methods for Planned Uses

The activities proposed within this request are appropriate to meet the intended purpose and use of the data, as outlined in supporting statement A and above. The following provides details about the data collection and the appropriateness of each information collection activity.

The data collection will include youth focus groups, facilitator interviews, and fidelity logs.

• Focus groups provide a unique opportunity to collect information about participant opinions and reflections, capturing their voices in a way that other data collection methods, like survey instruments, cannot. For example, unlike surveys, the focus groups will allow us to delve deeper into youths’ feedback and perceptions of the lesson content and delivery, their understanding of the lesson content, the extent of youths’ participation and engagement in the lessons, and their recommendations for lesson refinement. This design lends itself well to this population because youth with IDD can have lower literacy levels and may not be able to complete a survey. In addition, respondents may need questions reframed or clarified to understand the intent of the question. A focus group provides an opportunity for the moderator to provide additional supports and accommodations to youth during data collection, such as asking questions verbally and reframing questions as needed, while still collecting high quality data that captures the experiences and perspectives of the participants.

• One-on-one virtual or in-person facilitator interviews will allow the study team to collect data capturing facilitators’ reflections on their implementation experience, their satisfaction with the lessons, their perceptions of youth engagement with the lessons, and any adaptations they recommend making to the lesson plans or facilitator instructions. This design will allow facilitators to provide nuanced feedback specific to the population of students that they teach. This is important for youth with IDD because the range of IDD among the youth may vary from site to site. Collecting broad qualitative feedback from facilitators in multiple sites will allow the study team to make the most tailored refinements to the lessons for youth with a variety of disabilities. The study team will also ask facilitators whether they prefer an in-person or virtual interview to accommodate the busy and dynamic schedules many facilitators have.

• Facilitator fidelity log data will be helpful in identifying specific refinements needed for each lesson, including any changes required to improve facilitator guidance or modifications to lesson activities. Online feedback directly from facilitators after each lesson’s implementation will provide real-time insight into the facilitators’ experience with the facilitator guidance and curriculum content for each lesson. This design is well-suited to facilitators, especially special education teachers, who may appreciate providing specific feedback in a short online form to accommodate their busy schedules.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information. Key limitations will be included in written products associated with the study.

B2. Methods and Design

Target Population

We propose to conduct up to sixteen focus groups with high-school aged youth with IDD who participate in the implementation of the Internet safety lessons in their schools or community-based organizations (CBO). Our intent is to recruit youth served by geographically and racially/ethnically diverse schools or CBOs and with different disabilities and severity of IDD to get a broad range of perspectives, as PREP grantees serve geographically and racially/ethnically diverse youth, as well as serve youth with varying disabilities. This approach will provide more feedback from youth on the lesson content, delivery, and proposed improvements they would like to see in the curriculum. This will provide the study team with information on the utility of the lessons with a broader population of youth that reflects the population of PREP grantees. Each focus group will have up to six participants for a maximum of 96 participants across the sixteen sites.

The target population for the facilitator interviews and fidelity logs are facilitators at schools or community-based organizations (CBOs) who implement the Internet safety lessons with youth with IDD (up to 24 facilitators). We estimate there will be at least one facilitator in each of the sixteen recruitment sites but recognize that many special education classrooms and programs for youth with IDD have support staff, such as classroom aides who may help to co-facilitate a lesson. Aides will also be invited to participate in interviews (with a max of 8 aides selected for interviews).

Sampling and Site Selection

The study team will recruit a diverse set of up to sixteen high schools or CBOs serving high-school aged youth with IDD to participate in the study. The sites either need to have dedicated special education classrooms or programs or be a school or CBO that only serves special education students. We plan to reach out to several high schools with which the study team has previously worked but will also identify several additional schools or CBOs to increase the geographic and demographic diversity of the participants. The study team also plans to identify current or former PREP grantees working with youth with IDD as potential implementation sites. Once potential recruitment sites are identified the study team will hold a virtual meeting to describe feasibility of implementation, benefits of participation, data collection activities, and timeline. Additional meetings to discuss site feasibility will be held as needed.

Once a school or CBO has agreed to participate in the data collection, the study team will ask the site to identify facilitator(s) to implement the two lessons. These staff will likely be special education teachers or staff serving youth with IDD in a CBO. The study team will work with the facilitator to distribute parental consent forms to parents or legal guardians of potential participants in the classroom or setting where the two lessons will be implemented regardless of their age.¹ During implementation, facilitators will be asked to complete fidelity logs and following implementation they will be asked to participate in a 60-minute facilitator interview. The study team will use non-probability sampling to identify potential respondents who can provide information on their experience facilitating the lessons. Because facilitators will be purposively selected based on the school or CBO where they are employed, they will not be representative of the population of facilitators that may eventually implement the lessons. Instead, we aim to obtain variation in facilitators’ experiences implementing the lessons.

The sampling frame for the focus group will be the roster of youth who have participated in at least one of the lessons and who have parental consent to participate.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

The study will collect data on participant engagement and satisfaction with the adapted curriculum to contribute to the refinement and finalization of the adaptation. All instruments were developed in collaboration with ACF and Mathematica staff and were designed to be concise and streamline data collection to only collect the information necessary to refine the curriculum adaptation.

The fidelity log (Instrument 1) was developed to capture information on lesson dosage, student engagement, facilitator adaptations, facilitator satisfaction, and overall reactions to the lessons. The facilitator topic guide (Instrument 2) was developed to address four main topic areas for lesson refinement: (1) satisfaction with the lessons, (2) recommendations for adaptations or improvements, (3) student engagement and comprehension, and (4) perceptions on whether the lessons met their intended objectives. The youth with IDD focus group protocol (Instrument 3) was developed to address four main topic areas for lesson refinement: (1) comprehension of lesson content, (2) change in knowledge of Internet safety, (3) satisfaction and with the lesson content and facilitation of the lessons, (4) engagement in the lessons, (5) recommendations for modifications or improvements to the lessons.

For development of the data collection instruments, the study team drew from Mathematica focus group protocols and fidelity logs for the evaluations of the Real Essentials Advance relationship education program, Love Notes curriculum, and the evaluation of the Personal Responsibility Education Program.

B4. Collection of Data and Quality Control

ACF is contracting with Mathematica for this data collection. The researchers at Mathematica have extensive experience collecting fidelity log data from facilitators and conducting focus groups with youth populations, including youth with IDD for prior studies sponsored by the U.S. Department of Health and Human Services and other federal agencies. The study protocols also received research ethics review approval on March 8, 2022 in accordance with the requirements of the US Code of Federal Regulations for the Protections of Human Subjects through HML IRB.

Participants will be recruited from up to sixteen schools or CBOs serving high-schools aged youth with IDD. A team of two trained researchers working on the study team will serve as focus group moderators and conduct data collection. The researchers have extensive experience collecting qualitative data from youth, including youth with IDD. Moderators will be trained on the youth focus group guide and ensure that data are collected in a consistent way across focus groups.

Fidelity Logs (Instrument 1). The study team will ask facilitators to complete an online consent form agreeing to complete a short online fidelity log following each implementation session. Completing fidelity logs following each session will give them an opportunity to document which activities they were able to complete, the time it took to complete activities or lessons, and any necessary modifications. For example, facilitators could note whether they needed to pre-teach any topics to help students be prepared for the lessons or adapt any activities for youth with specific disabilities within their classroom. We will use the information from the fidelity logs and observations to refine lesson content and facilitator instructions and guidance—for example, by more accurately estimating the amount of time each lesson will take and making modifications to activities to make them accessible to more youth. Facilitators will enter logs into a web-based platform. The fidelity log is a valuable tracking tool that will minimize staff burden by including checkboxes and limiting open-ended responses as feasible. 

Facilitator Interviews (Instrument 2). All curriculum facilitators (and co-facilitators) will be invited to complete one semi-structured in-person or virtual interview following lesson implementation. A team of two researchers working on the study team will conduct the interview (one lead interviewer and one note-taker). Before the interview, the researchers will ask for oral consent from each respondent to participate in the interview and to record the interview. Interviews will focus on facilitators’ reflections about their satisfaction with the lessons, how their students responded to the lessons, and any adaptations they recommend making to the lessons. The researchers will ask the facilitator if they consent to having the interview recorded and will not record if they do not consent.

Focus groups or interviews (Instrument 3). Each site will host one focus group with up to six participants that will last about 60 minutes. The study team will send parents a letter informing them of the study and giving them the opportunity to opt their child out of one or both of the lessons, should they choose. Before collecting study data from the focus group from youth participants, the study team will seek active consent from a parent or legal guardian. The online consent form will explain the purpose of the study, the data to be collected, and use of the data. The form will also state that answers will be kept private, that participation is voluntary, and that youth may refuse to participate at any time without penalty. The study team will provide the link to the online consent form for facilitators to email to parents prior to implementation. The study team will reduce burden on sites, parents, and youth by using online consent forms through a secure website.

Due to the needs of youth with IDD, we expect the facilitators to prefer in-person data collection, but we will accommodate sites if they want virtual data collection because of the COVID-19 pandemic or for other reasons. We will ask facilitators the best mode to collect youth feedback—semi-structured focus groups, individual interviews, or a combination of both. If a focus group will be in-person, the study team will work with site staff to arrange the sessions at convenient times and locations and to recruit up to seven youth for each group. A team of two researchers working on the study team will conduct the focus groups (one lead facilitator and one note-taker). On the day of the focus group or interview, the study team member will explain to youth participants that they will have the opportunity to fill out an assent form (or consent form if age 18 or older) before participating in the focus group or interview. The study team member will also read the assent form to participants, providing them with a chance to opt out of the focus group if they so desire, without penalty. We are aware that youth with IDD have special ethical considerations when conducting consent/assent such as using plain language to explain what the research will involve and the benefits to the youth and society; using multiple communication techniques (for example, verbal and written); and respecting the autonomy of the youth throughout the process.^2,3 The study team will adhere to these practices to ensure the youth are providing fully informed assent/consent. The researchers will ask the youth if they consent to having the focus group recorded. If all youth consent the session will be recorded to facilitate notetaking, but if one youth does not consent, we will not record the session.

During the focus groups, moderators will ask questions in a semi-structured way, providing opportunities for youth to ask clarifying questions and respond as appropriate. Throughout the sessions, moderators will probe participants to elaborate on their comments and encourage more in-depth comments as needed. Moderators will also encourage participants to read and engage with others in the group. Recordings of all interviews and focus groups will be saved at the end of the sessions to support quality and accuracy of focus group data collection.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The data collection activities—interviews, focus groups and fidelity logs—are not designed to produce statistically generalizable findings and participation is wholly at the respondents’ discretion. Response rates will not be calculated or reported.

NonResponse

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. The study team will document take up rates and attrition among the samples.

B6. Production of Estimates and Projections

The data will not be used to generate population estimates, either for internal use or dissemination.

B7. Data Handling and Analysis

Data Handling

Focus group, interview, and facilitator log data and associated recordings will be saved on a secure drive only accessible to Mathematica study team members. Study team members will monitor and review fidelity log submissions and follow-up with facilitators if they see any errors in data entry or need facilitators to clarify their qualitative feedback. Focus group moderators will review their notes to fix spelling and grammar issues, fill in any missing words or concepts, and explain unclear terms or phrases in preparation for qualitative analysis and interpretation.

Data Analysis

We will analyze the quantitative data collected through fidelity logs to assess dosage and pacing of lessons. We will organize and code qualitative data collected through fidelity logs and youth focus groups to identify themes around the various challenges or successes of each lesson. We will use this information to create a recommended list of modifications for each lesson plan and revise the lessons and facilitator guidance accordingly.

Data Use

The data will provide ACF with respondent feedback on how the curriculum adaptation was received and recommendations that will influence refinements to the Internet safety lessons. With this critical information, ACF will be able to finalize the curriculum and have formative implementation data to share with federal staff and grantees that may be interested in implementing the curriculum in the future.

Ultimately, this data will allow ACF to improve future training and technical assistance to PREP grantees serving youth with IDD, by providing needed resources that would serve this population. Findings from the data collection will also be shared in a report and journal article that outlines the process for the curriculum, the formative findings from the study, and best practices for entities implementing the adapted curriculum, such as state agencies, federal grantees, or schools. The report will discuss limitations of the data and include information on how to interpret the findings.

Once the curriculum adaptation is final ACF staff could notify grantees of the availability of the curriculum and findings associated with the formative evaluation to incorporate into their programming. The curriculum will be available online for grantees to access. The adapted Internet safety lessons will be hosted on Common Sense Media’s website as well as FYSB’s The Exchange website which provides resources to both grantees and the general public.

B8. Contact Persons

Table B8 lists the federal and contract staff responsible for the study, including each individual’s affiliation and email address.

Table B8. Individuals Responsible for Youth with IDD Data Collection Procedures

Name	Affiliation	Email address
Selma Caal	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	Selma.Caal@acf.hhs.gov
Kathleen McCoy	VPD Government Solutions Staff, Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	Kathleen.McCoy@acf.hhs.gov
Jean Knab	Mathematica	JKnab@mathematica-mpr.com
Katie Adamek	Mathematica	kadamek@mathematica-mpr.com

Attachments

Appendix A: Consent form for parents of youth with IDD

Appendix B: Consent form for youth with IDD over 18

Appendix C: Assent form for youth with IDD participants

Appendix D: Consent for facilitators

Instrument 1: Youth with IDD fidelity log

Instrument 2: Youth with IDD facilitator interview topic guide

Instrument 3: Youth with IDD focus group guide

1 Even though some youth with intellectual and developmental disabilities may be 18 or older, the recruitment organization may require us to obtain parental consent.

2 Yan, Eric and Kerim M Munir (2004). Regulatory and Ethical Principles in Research Involving Children and Individuals with Developmental Disabilities. Ethics & Behavior, vol. 14 (1): 31–49.

3 McDonald, Katherine E., Nicole M. Schwartz, Colleen M. Gibbons, and Robert S. Olick (2015). “You can’t be cold and scientific”: Community views on ethical issues in intellectual disability research. Journal of Empirical Research on Human Research Ethics, Vol 10, 196-208

7