Permanency Innovations Initiative
Pretesting of Evaluation Surveys
(OMB 0970‑0355)
Supporting Statement Part B
Submitted By:
Office of Planning, Research and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
7th Floor, West Aerospace Building
370 L’Enfant Promenade, SW
Washington, D.C. 20447
Project Officer: Maria Woolverton
The CAPP sample universe is children in Family Reunification status in Fresno County that have received the CAPP practice model from CAPP-trained caseworkers. There are 157 biological parents and legal guardians in this universe, and 297 cases (or children). Of these, approximately 20 are biological parents and legal guardians of African American or American Indian children and the remainder are biological parents or legal guardians of non-African American or non-American Indian children. From this latter group, 80 will be sampled, for a total of 100 adults. Biological parents and legal guardians interviewed will be 18 years of age or older.
The sample for ADES includes 50 children and their caseworkers randomly assigned from (1) a population in care for 2 or more years, which comprises children ages 13-17.5 (N=308), and (2) a population in care for at least 1 year but less than 2 years (N=674), which comprises children ages 5 to 17.5 who have been in out-of-home care for 12 months or more (consecutively), and who have one or more of the following risk factors:
Multiple placements (3+) in the first 100 days of care,
Parents not available due to abandonment, relinquishment, or incarceration,
Substance abuse (children or parents),
Mother refuses services for mental health or substance abuse, or
Child is a chronic runaway (no contact for up to 3 months).
Approximately half of the children will be from each population. Of the 50 children in the pretest sample, about 30 will receive the treatment and about 20 will receive services as usual.
The random sample for the DCFS pretest includes 100 youth from the universe of those ages 11-16 who are placed in traditional, relative, and specialized foster homes throughout the State of Illinois who, upon reaching the 2-year anniversary of entering foster care, are experiencing mental health symptoms and/or have had at least one placement change. DCFS youth will meet the age criteria for participation in the program on a rolling basis. Thus, DCFS will project cases 1-2 months ahead that will soon meet the established eligibility criteria based on age and time in foster care. Then, DCFS will confirm the youth’s eligibility by examining administrative data on the youth’s current mental health and trauma-related symptoms and previous placement changes. DCFS will then consult the caseworker to verify the appropriateness of the youth to participate in the study and will obtain consent from the DCFS Guardian, who has legal authority over all children in foster care in the State of Illinois, for each individual youth to participate in the study (intervention and data collection). Biological parents eligible to participate in the study include those: (1) who are the parent of a program-eligible youth and (2) for whom DCFS is still pursuing reunification of the youth and parent. DCFS will make the determination as to the viability of reunification between the youth and the parent. In the rare event that it is not the biological parent but a legal guardian or an adoptive parent who DCFS is seeking to reunify with a study youth, this parent will be eligible to participate in the intervention and data collection. Foster parents are eligible to participate in the study if they are the current caregiver of a program-eligible youth.
Sampling Procedures
For CAPP, the 100 biological parents and legal guardians who are interviewed will include all parents of African American and American Indian children in family reunification status that are being served by CAPP-trained caseworkers. Approximately 20 potential parents/legal guardians are in this population. The sample will also include a simple random sample of 80 biological parents or legal guardians of other non-African American or non-American Indian children, also being served by CAPP-trained caseworkers, using a random number generator. This will be a total of 100 adults. All biological parents and legal guardians sampled will have provided passive consent for the evaluation contractor to have access to their contact information. CAPP providers will obtain passive consent from the biological parents and legal guardians for their contact information to be provided to the data collection contractor. When the data collector subsequently telephones the parent/guardian, a verbal audiotaped consent for participating in the interview will be administered prior to the interview being conducted.
ADES has randomly assigned all 215 of their caseworkers to treatment (60%) or services-as-usual (40%) groups. All caseworker positions were listed in an Excel spreadsheet in numeric order by position number and a random number was assigned to each position using the random number generator in Excel. The randomized numbers were then saved and the positions were aligned in numeric order by the randomized number. The top 60% were identified as treatment workers and the remaining were identified as control workers. The children who meet the program eligibility requirements on the caseworker’s caseload will either receive (1) treatment or (2) services-as-usual based on their caseworker’s assignment. There will be a total of 30 children who will receive the intervention (15 from each population) and 20 who will be in the control group (10 from each population). All participating caseworkers will provide informed consent and all children/youth will be asked for assent for the data collection procedures.
DCFS will use a computer program to randomly assign youth to treatment or services-as-usual groups. DCFS will assign the foster and biological parents to a condition based on the youth’s assignment. There will be a total of 100 youth randomized to the treatment and services-as-usual group (50 in each group). We anticipate enrolling 100 foster parents and 100 biological parents in the pretest. All youth and foster and biological parents will be asked to consent/assent to data collection.
Data Collection Procedures: CAPP
For CAPP, the data collection period for the telephone interviews with parents/guardians will begin when OMB clearance is received and conclude approximately 8 months later. Data extraction from case records will occur toward the end of that period. Data collection involves the following steps:
Informed Consent: Fresno County, the service provider agency, will send a Frequently Asked Questions and Consent Letter to Parents and Guardians with Bill of Rights (see Attachment A1) by U.S. mail to all parents and legal guardians of children in family reunification status that are being served by CAPP-trained caseworkers (N=approximately 157) to obtain passive consent for the evaluation contractor to have access to their contact information. For each parent and legal guardian who allows contact information to be shared by way of passive consent, Fresno County DSS will complete and upload a Parent Study Contact Information Form (see Attachment A2 for the Parent Study Contact Information Form) including name, address, and current and alternative phone numbers onto the evaluation contractor’s secure server. Parents and legal guardians who contact Fresno County DSS to indicate that they do not want to share their contact information with the evaluation contractor will not have their contact information uploaded and shared.
Data Collection for Interviews with Biological Parents and Legal Guardians: The evaluation contractor will administer the telephone interview in its entirety to all parents or legal guardians in the sample who provided passive consent to the study. (See Attachment A3 for the Parent/Guardian Interview with Consent Script.). The telephone interview will be administered in its entirety, and will be administered once to each respondent. Respondents will provide verbal consent for the interview, which has been incorporated into the estimated .33 hours/20 minutes burden for the telephone interview. Interviews will be conducted by trained interviewers.
Data Collection for the Case Record Reviews: A trained data collector from the evaluation contractor will extract information from the case files for all families in the study (see Attachment A4 for the Case Record Review Instrument).
Data Collection Procedures: ADES
For ADES the data collection period for the child interviews will begin after OMB clearance and last for approximately 6 months. The caseworker interviews will begin towards the end of the pretest period.
Consent Process for Children: The state of Arizona is providing consent for children to be invited to participate in the study. ADES will send an FAQ letter to the out-of-home care provider of each child in the study, describing the evaluation contractor’s interest in interviewing the child in their care twice over a 6-month period (see Attachment B3 for the Care Provider FAQ). The letter also informs them that a data collector will be contacting them soon to schedule an interview with the child. ADES, which has legal authority over all children in foster care in the state, has provided the evaluation contractor with approval to obtain the contact information for the out-of-home care provider for the children in the study. ADES will upload the contact information to the evaluation contractor’s secure server using the Youth and Care Provider Study Contact form (see Attachment B1 for the Youth, Care Provider, and Caseworker Study Contact Form).
A data collector will follow up with the out-of-home care provider by phone to schedule the interview. Once at the home, the data collector will meet with the child in a private setting and review the assent form (if age 7-17.5) or read the assent script (if age 5-6). Children between the ages of 7 and 17.5 will provide written assent. We will use verbal assent for children ages 5 and 6 since they are typically unable to comprehend and sign forms such as this one. (See Attachment B2 for the Child Assent Forms.)
Data Collection for Children: Children enrolled into the study will be invited to participate in an in-person interview at the start of their participation in the study and 6 months afterwards. The interviews are estimated to take about 30 minutes to complete. These interviews will examine the number and types of informal supports the child has in their lives, utilizing the Supportive Connections Inventory, and the child’s readiness for permanency, utilizing the Readiness for Permanency Inventory. Two versions of this inventory will be used, based on the child’s age (ages 5-10, ages 11-17.5). The instruments are standardized. See Attachment B4 for the Child-Youth Interview.
Consent Process for Caseworkers: ADES has already provided approval for the evaluation contractor to obtain contact information for the caseworkers. The contact information will be uploaded to the evaluation contractor’s secure server using the study contact form (see Attachment B1). ADES will send introductory e-mails (Attachments B5 and B6) notifying caseworkers about the study being conducted in the state of Arizona. The e-mails will describe the Prospective-Homes-Found Interview and the Decision-Making Interview. ADES will attach caseworker consents to the e-mails, which will be completed with the data collector during a single interview session in order to minimize burden for the pretest (see Attachment B7 for Caseworker Consents and Interviews). The data collectors will audio-record all verbal agreements to participate.
Data Collection for Caseworkers: Toward the end of the pretest, a data collector will contact the caseworker and review the informed consent information. For caseworkers who verbally consent to be interviewed, the Prospective-Homes-Found Interview and the Decision-Making Interview will be administered during a single interview session (see Attachment B7 for the Caseworker Consents and Interviews).
Data Collection Procedures: DCFS
The DCFS pretest will run for about 7 months, beginning after OMB clearance.
Informed Consent for Biological Parents: DCFS will send a study FAQ letter to the biological parents (see Attachment C2, Biological Parent FAQ Letter and Consent Script). The letter will include informed consent information and will inform potential respondents that if they do not contact DCFS by a certain date, DCFS will release their contact information to Westat so that Westat can tell them more about the study. After the target date, DCFS will provide Westat the Parent Study Contact Form information (see Attachment C1, Biological Parent Study Contact Form) for all the parents who provided passive consent for their information to be released to Westat. The contact information will be uploaded to Westat’s secure server using study contact forms. A data collector will then call the parent to discuss the study, review the study consent form, and answer any questions about the study (see Attachment C2 for the Biological Parent FAQ Letter and Consent Script). During the consent process, the data collectors will ask prospective respondents to separately consent to: (1) participation in the phone interviews and (2) Westat merging their interview responses with DCFS administrative data on their family. Parents can decline consenting to the latter and still participate in the interviews. The data collectors will audio record all verbal agreements to participate.
Data Collection for Biological Parents: We will invite program-eligible biological parents to participate in two 15-minute telephone interviews: at the start of their enrollment in the study and 6-months later. The interviews will assess their abilities to regulate their emotions and behaviors and the quality of their support system. Please note that in the rare event that it is not the biological parent but a legal guardian or an adoptive parent who DCFS is seeking to reunify with a study youth, Westat will invite this parent to participate in the data collection using identical instruments. The instruments administered during the biological parent interview are the Abbreviated Dysregulation Inventory and the Social Provisions Scale (see Attachment C3, Biological Parent Interview). Both are standardized instruments.
Informed Consent for Foster Parents: Once a youth is randomly assigned to a treatment or control condition, DCFS will send a FAQ letter to the youth’s foster parent (see Attachment C4 for the Foster Parent FAQ Letter and Consent Form) and the youth (see Attachment C5 for the Youth FAQ Letter and Assent Form). The FAQ letters will include informed consent information. The foster parent FAQ letter will also inform potential respondents that if they do not contact DCFS by a certain date, DCFS will release their contact information to Westat so that Westat can tell them more about the study. After the target date, DCFS will provide Westat the contact information of all the parents who provided passive consent for their information to be released to Westat. The Youth and Foster Parent Study Contact Form information (see Attachment C6) will be uploaded to Westat’s secure server using a study contact form. If the foster parent does not agree to their contact information being released to Westat, DCFS will not release their contact information and the foster parent will not participate in data collection. If the foster parent passively agrees to their contact information being released to Westat, DCFS headquarters will upload the youth and foster parent’s name and contact information to Westat’s secure server using a study contact form. A Westat data collector will then call the foster parent to review the FAQ and determine their interest in participating in the study (see Attachment C4 for the Foster Parent FAQ Letter and Consent Form). Then the data collector will schedule a date to meet with both the youth and the foster parent during a visit to the home. Once at the foster home, the data collector will meet with the potential respondents individually to review the study assent/consent form, answer any questions, and initiate the baseline interview for those who agree to participate. In addition to consenting to the interviews, the data collector will seek consent from foster parents for Westat to merge their interview responses with DCFS administrative data on the youth’s family. Foster parents can decline the latter and still participate in the interviews.
Data Collection for Foster Parents: Program-eligible foster parents will be invited to take part in a 40-minute Computer-Assisted Self-Interview (CASI) session at the start of their enrollment in the study and 6 months later. A Westat data collector will set up the CASI and be available to assist the parent as needed throughout the CASI session. The CASI includes questions assessing the youth’s ability to regulate their emotions and behaviors from the parent’s perspective, their parenting skills in responding to youth’s emotions and behaviors, and the quality of their support system. The instruments administered during the foster parent interview are shown in Attachment C7 (Foster Parent Interview, including the Abbreviated Dysregulation Inventory-Parent Version, Social Provisions Scale, and the Parenting Practices Chicago Survey- Parent Version). These are all standardized instruments. (See Attachment C8 for the CASI measures and instruction screen shots.)
Informed Assent for Youth: The DCFS Guardian, who has legal authority over all children in foster care in the State of Illinois, will need to provide consent for a foster child under the age of 18 to participate in the study (both the intervention and data collection). As DCFS identifies eligible youth, they will seek individual consent from the Guardian to enroll each youth in the study. This approval will occur prior to randomly assigning a youth to treatment or control conditions. As previously described, DCFS will send a FAQ letter to the youth (Attachment C5) and the youth’s foster parent (Attachment C4). The foster parent and youth FAQ letter will include informed consent information. The foster parent FAQ letter will also inform potential respondents that if they do not contact DCFS by a certain date, DCFS will release their contact information to Westat so that Westat can tell them more about the study. After the target date, DCFS Headquarters will provide Westat the contact information of all the parents who provided passive consent for their information to be released to Westat. A Westat data collector will then call the foster parent to schedule a time to meet with the youth and tell them about the study. Once at the foster home, the data collector will meet with the youth to go over the Youth FAQ Letter and Assent Form (see Attachment C5), answer any questions, and initiate the baseline interview for those who agree to participate.
Data Collection for Youth: We plan to invite program-eligible youth ages 11-16 to take part in a 40-minute Audio Computer-Assisted Self-Interview (ACASI) session at the start of their enrollment in the study and 6 months later (see Attachment C8 for screen shots of the ACASI administration). A Westat data collector will set up the ACASI and be available to assist the youth as needed throughout the ACASI session. The ACASI includes questions that examine the youth’s trauma-related and mental health symptoms, their ability to regulate their emotions and behaviors, quality of their relationship with adults, and their capacity to form and maintain relationships. The instruments administered during the youth interview include three standardized instruments (Abbreviated Dysregulation Inventory, Social Provisions Scale, and the Trauma Symptoms Checklist for Children) and two brief measures modified from non-standardized instruments (the Biological Parent Contact with Youth Instrument and the Youth Emotional Permanency Instrument) (see Attachment C9 for the Youth Interview).
In order to maximize response rates, all three sites will provide incentives to respondents (see Section A9). Other ways to encourage participation in the data collection include involving stakeholders, agency staff, and service providers in selecting instruments and designing procedures, to increase their buy-in.
Prior to this information collection, in all sites the evaluation contractor will pretest instruments with fewer than 10 interviewees to confirm that test participants understand each item and comprehend the response options. The contractor also will determine how well data collection procedures work, timing, administration issues, and training issues. Any difficulties encountered with respect to the procedure, training materials, or instruments will be discussed with the sites, and suggested revisions to the evaluation plan will be outlined.
The team is led by Maria Woolverton, project officer; Andrea Sedlak, project director; and Mark Testa, principal investigator. Additional staff consulted on statistical issues at Westat include Barnali Das, Senior Statistician.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Molly Buck |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |