PII - CAPP Supporting Statement A

Permanency Innovations Initiative

Pretesting of Evaluation Surveys

(OMB 0970‑0355)

Supporting Statement Part A

January 2013

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

7^th Floor, West Aerospace Building

370 L’Enfant Promenade, SW

Washington, D.C. 20447

Project Officer: Maria Woolverton

A. JUSTIFICATION

The Permanency Innovations Initiative (PII), funded by the Children’s Bureau, Administration for Children and Families, is building the evidence base for innovative interventions that improve permanency outcomes for children and youth who are in or at risk for long-term foster care. Six grantees were funded during an initial planning year to select, design, or develop interventions and work with an evaluation contractor to develop site-specific evaluation plans for the subsequent four years. A major emphasis of the PII is the design of rigorous evaluations that will provide credible evidence and replicable interventions for achieving faster permanency for children and youth in foster care.

The proposed PII Evaluation includes multiple components and two phases. The components include:

• a cross-site evaluation, including implementation study and administrative data study;

• six site-specific impact evaluations; and

• a cost study.

Phase I of data collection received OMB clearance on 8/17/2012 (OMB# 0970-0408) and includes the cross-site implementation study (covering all six sites) and the site-specific impact evaluations for two grantees (the State of Kansas and Washoe County, Nevada) that began full implementation of their interventions in August 2012. A second phase, for which an information collection request will be submitted in the future, includes the site-specific impact evaluations for the four grantees that will begin full implementation of their interventions at a later point in time, as well as collection of data for the cost study and administrative data study.

Three grantees, the California Department of Social Services’ California Partnership for Permanency (CAPP), the Arizona Department of Economic Security (ADES), and Illinois Department of Children and Family Services (DCFS) are ready to begin pretesting their data collection instruments and procedures. ACF requests permission to conduct pretesting with the purpose of evaluating CAPP’s parent/guardian, ADES’s child and caseworker, and DCFS’s child, biological parent, and foster parent interview measures and procedures. The interview batteries and procedures need to be tested for acceptability to respondents and sensitivity to change. The information collected will be used for internal purposes only and will not be released to the public; it will be used for adjusting the evaluation plans so that the data collection is acceptable to families and achieves a high response rate.

A.1. Circumstances Making the Collection of Information Necessary

Although the Adoption and Safe Families Act of 1997 included provisions focused on moving children and youth quickly into permanent families while maintaining their safety, many jurisdictions continue to experience growing populations of children who age out of foster care without achieving permanency. The PII grantees are implementing innovative interventions to address site-specific issues and help achieve timely permanency for more children and youth. A key component of the PII is the collection of data that will demonstrate linkages between interventions and outcomes.

A.2. Purpose and Use of the Information Collection

CAPP’s initiative, known as the “CAPP Child and Family Practice Model” or “CAPP Model,” is being pretested in Fresno County, California. The CAPP Model addresses all children in or entering foster care at elevated risk of a non-permanent exit (e.g., aging out) or long-term foster care, with a targeted effort to help children who are in care the longest and experience the worst outcomes. According to statewide data, these sub-populations include children of African American and American Indian heritage. The purpose of the information collection is to determine whether implementation of the CAPP Model eliminates or reduces disparities between children of African American and American Indian descent and non-African American and non-American Indian children with respect to the risk of long-term foster care. The pretest will involve a telephone survey administered to one biological parent or legal guardian for each African American and American Indian child (N=approximately 20) and a random sample of 80 biological parents or legal guardians of other non-African American or American Indian children. Trained data collectors will conduct a 20-minute telephone interview designed to assess the following short-term outcomes: (1) involvement of extended family, community, and Tribes in problem posing, ongoing assessment, and developing solutions to achieve child safety, permanency, and well-being; (2) extent to which children and youth feel connected to important people in their lives; (3) degree of hopefulness that the family can be successful; and (4) improved caseworker-client relationships. Prior to conducting the interview, CAPP will distribute a CAPP Study Frequently Asked Questions (FAQ) by U.S. mail to eligible parents/legal guardians to obtain passive consent for the evaluation contractor to have access to their contact information. In addition, the pretest will include a case record data extraction to be conducted by the evaluation contractor to record information not stored in California’s child welfare electronic system. The data will include information on child welfare system contacts, collateral contacts, visitations, and meetings.

ADES is implementing an intervention that will provide services to two groups of children to prepare them for permanent placements: one group has been in care for 2 or more years and the other group has been in care for at least 1 year but less than 2 years. The pretest will follow a randomized design and consists of interviews with approximately 50 children/youth (randomly selected to be in the pretest) and their caseworkers. The pretest will assess the procedures and instruments for collecting data from children and caseworkers. All 215 caseworkers have been randomly assigned to treatment or control group; about 60 percent were randomly assigned to the intervention group and the others were randomly assigned to services as usual. Thus there are two levels of randomization: (1) caseworkers were randomized to treatment or control condition, and (2) children will be randomized into the pretest and then will be in the same study group (treatment or control) as their caseworkers. Trained data collectors will conduct interviews using a battery of instruments. The children’s interviews include a readiness for permanency inventory and a supportive connections interview; the caseworkers’ interviews include a prospective homes interview and decision-making interview. Children will be interviewed twice, once at the beginning of the study and once 6 months later; caseworkers will be interviewed once, near the end of the pretest period. Before the interviews are conducted, both children and care providers will be given a frequently asked questions (FAQ) information sheet and children’s assent will be obtained. The data from the pretest will be used to understand the time required to administer the total battery, child and caseworker receptivity to the batteries, and the instruments’ sensitivity to change, as well as any special challenges that arise during the administration.

DCFS will implement a 4-month intervention that will provide services to foster youth identified as being at risk of remaining in long-term foster care. The evaluation will follow a randomized design and consist of interviews with approximately 100 foster youth, 100 foster parents, and 100 biological parents. The pretest will assess the procedures and instruments for collecting data on foster youth, foster parents, and biological parents. Youth who meet the eligibility criteria will be randomly assigned, half will receive the intervention and half will receive services as usual. Youth eligible to participate have to meet the following criteria: be between the ages of 11-16; be placed in traditional, relative and specialized foster homes throughout the state of Illinois; be reaching the 2-year anniversary of entering foster care and experiencing mental health symptoms and/or have had at least one placement change. Based on historical data, there will be approximately 100 youth who will meet these criteria at the start of the pretest phase.¹ Thus, all of these youth will be randomly assigned for the purpose of participating in the pretest. DCFS will perform the random assignment for the youth, and the foster and biological parents will be assigned based on the youth’s assignment. Each respondent (youth, foster parent, and biological parent) will be asked to participate in two interviews: one at the start of their enrollment in the study and a second one 6 months later. The interviews include questions that examine the youth’s ability to regulate their emotions and behaviors from their perspective as well as from the foster parent’s perspective, parenting skills of the foster parent in responding to youth’s emotions and behaviors, the biological parent’s ability to regulate their emotions and behaviors, youth’s trauma-related and mental health symptoms, youth’s capacity to form and maintain relationships, and the quality of the youth’s, biological parent’s, and foster parent’s support system. Trained data collectors will conduct interviews using a battery of instruments. The data from the pretest will be used to understand the time required to administer the total battery, foster youth and foster and biological parent receptivity to the battery, and the instruments’ sensitivity to change, as well as any special challenges that arise during the administration.

A.3. Use of Improved Information Technology and Burden Reduction

To reduce burden on parents and legal guardians who participate in interviews for the CAPP pretest, Westat will administer the interview by telephone at a time that is convenient to the respondent. Through prior administration to fewer than 10 respondents, the telephone interview (including consent) has been timed to be no longer than 20 minutes.

To reduce burden on the child and caseworker for the ADES pretest, Westat plans to administer the instruments for youth in their current home placement and to the caseworkers by telephone at times convenient to them. The child interview will take approximately 30 minutes² and the caseworker interview will take approximately 45 minutes.³

To reduce burden on the biological parents for the DCFS pretest, Westat plans to administer the instruments by telephone at times convenient to them. The telephone interview (including consent) will take no longer than 20 minutes. Westat also plans to administer the foster parent instruments and youth instruments at the foster home during the same visit to the home. The visit will be at a time most convenient to the foster parent and youth. Each interview will take approximately 40 minutes (not including consent/assent). In addition, the foster parent interviews will be conducted via Computer-Assisted Self-Interview (CASI) methodology, and the youth interviews will be conducted via Audio-CASI (ACASI) to make the interviews easier and more efficient. A Westat data collector will set up the CASI and ACASI and be available to assist the parent and youth as needed throughout the interview sessions.

A.4. Efforts to Identify Duplication and Use of Similar Information

For all three sites, the proposed interviews are not currently used by the foster care agencies that will participate in the project.

A.5. Impact on Small Businesses or Other Small Entities

No small businesses are impacted by the data collection in this project.

A.6. Consequences of Collecting the Information Less Frequently

This is a onetime collection.

A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances requiring deviation from these guidelines.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

The first Federal Register notice to renew ACF’s generic clearance for pretesting was published in the Federal Register on June 10, 2011 (vol. 76, no. 112, p. 34077), inviting public comment on our plans to submit this request. ACF received no comments or questions in response to this notice.

The second Federal Register notice was published in the Federal Register on August 29, 2011 (vol.76, no. 167, p. 53682.

A.9. Explanation of Any Payment or Gift to Respondents

CAPP will provide all interview respondents with a $25 Target gift card upon completion of the parent/guardian interviews in order to encourage parents/guardians to participate and to improve the response rate. ADES will provide children with a $15 gift card to Target, Walmart, or similar store upon completion of the child interviews. ADES has a history of providing gift cards to children in care who participate in similar interviews and interview efforts. DCFS will provide $15 to biological parents and $20 to foster parents and youth upon completion of interviews in order to encourage parents and youth to participate and improve the response rate. These amounts are based on local experience with providing respondents with small monetary gifts and gift cards.

A.10. Assurance of Privacy Provided to Respondents

All sites have a full range of assurances for privacy, which were included in their submission to the Committee for the Protection of Human Subjects (CAPP’s IRB) and to the Westat IRB (for ADES and DCFS). These include:

• All respondents will provide informed consent, administered either through a written form or verbally over the telephone that will explain the evaluation process and assure them that their information will be private and securely stored.

• All project staff will follow strict policies and procedures for respondents’ privacy.

• All hard copies of documents will be secured behind two locks (e.g., locked file cabinet in locked room).

• All electronic content will be stored on secure servers. The server will be set with privileges that allow access only by specific individuals who have a username and password.

• All project data will be reported and presented at the aggregate level in order to prevent the identification of any individual respondent.

The PII evaluation has received a Certificate of Confidentiality from the National Institutes for Health. In addition, DCFS will be submitting a package to the DCFS IRB in December 2012.

A.11. Justification for Sensitive Questions

The CAPP data collection battery includes potentially sensitive questions, such as: “I’m getting the support I need for the stress I have felt in this situation” or “I feel I can count on myself to manage things well when my child(dren) come(s) home.” The ADES data collection battery also includes potentially sensitive questions, such as asking children to rate on a 1-5 scale their agreement/disagreement to the following statements: “I often feel lonely because I have few close friends with whom to share my concerns or worries” and “In many ways, I feel disappointed about my achievements in life. The DCFS data collection battery also includes potentially sensitive questions, such as asking children to rate on a 0-3 scale the frequency by which they agree with the following statements: “Feeling afraid somebody will kill me”, “Wanting to hurt myself”, and “Wanting to hurt other people.” This information is required to assess youth trauma symptoms and will help the evaluators assess whether the intervention decreases these symptoms among the treatment youth at a significant rate compared to youth who do not receive the intervention (a short-term outcome of the intervention). For all of these data collections, respondents will be assured of their privacy and told that they can skip any questions they do not want to answer and can stop the interview at any time.

A.12. Estimates of Annualized Burden Hours and Costs

Table A.1 contains the estimated burden hours for each type of respondent.  To compute the total estimated annual cost, the total burden hours were multiplied by the average hourly wage for each adult participant. The specific average hourly wages were as follows: $22.88⁴ for the parent/guardian interviews; $18.16⁵ for ADES caseworkers; $20.00⁶ for DCFS caseworkers; $26.78⁷ Fresno County (CA) caseworkers. Children and youth do not have an hourly wage. The total annual burden for this pretest activity is expected to be 521 hours.

TABLE A.1

ESTIMATED ANNUAL RESPONSE BURDEN AND ANNUAL COST

Attachment	Instrument	Number of Respondents	Number of Responses Per Respondent	Average Burden Hours per Response	Total Burden Hours				Average Hourly Wage				Total Annual Cost
	CAPP:
A1	CAPP Frequently Asked Questions and Consent Letter to Parents and Guardians with Bill of Rights	100	1.0	0.08	8		22.88				183.04
A2	CAPP Parent Study Contact Information Form	1	157	0.08	13		26.78				348.14
A3	CAPP Parent-Guardian Interview with Consent Script	100	1.0	0.33	33		22.88				755.04
	ADES:
B1	ADES Youth, Care Provider, and Caseworker Study Contact Form	1	50.0	0.08	4			18.16				72.64
B2	ADES Child Assent Forms	50	1.0	0.17	9			--				--
B4	ADES Child-Youth Interview	50	2.0	0.50	50			--				--
B7	ADES Caseworker Consents and Interview	50	1.0	0.92	46			18.16				835.36
	DCFS:
C1	DCFS Biological Parent Study Contact Form	1	100	0.08	8			20.00				160.00
C2	DCFS Biological Parent FAQ Letter and Consent Script	100	1.0	0.08	8			22.88				183.04
C3	DCFS Biological Parent Interview	100	2.0	0.25	50			22.88				1144.00
C4	DCFS Foster Parent FAQ Letter and Consent Form	100	1.0	0.08	8			22.88				183.04
C5	DCFS Youth FAQ Letter and Assent Form	100	1.0	0.08	8			--				--
C6	DCFS Youth and Foster Parent Study Contact Form	1	100	0.08	8			20.00				160.00
C7	DCFS Foster Parent Interview	100	2.0	0.67	134			22.88				3065.92
C9	DCFS Youth Interview	100	2.0	0.67	134			--				--
	Estimated Total	--	--	--	521			--				7090.22

A.13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

There are no direct monetary costs to respondents; they spend only their time to participate in the study.

A.14. Annualized Cost to Federal Government

The annualized cost to the federal government for the pretesting activities is $192,000.

A.15. Explanations for Program Changes or Adjustments

These will be new data collections in each of these sites. Pretesting is needed to assess the timing and flow of the instruments. This is an additional information collection request under generic clearance 0970-0355.

A.16. Plans for Tabulation and Publication and Project Time Schedule

There are no plans for tabulating and publishing the information gathered from this pretest process. The information that is collected will be for internal use only. CAPP’s pretest will be conducted for 8 months beginning at OMB clearance. ADES’s pretest will be conducted for 6 months beginning at OMB clearance. DCFS’s pretest will be conducted from for 7 months beginning at OMB clearance.

A.17. Display of Expiration Date for OMB Approval

The OMB number and expiration date will be displayed on the consent forms.

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this data collection.

1 Out of about 16,000 children in foster care across Illinois.

2 Children of different ages will respond to different items in the child instruments, but ADES has estimated that on average, the child interviews will take about 30 minutes.

3 There are two components of the caseworker interview, with individual email notifications and consent forms. However, order to reduce burden for the pretest, the two components will be administered as a single interview.

4 Average hourly wage is from Bureau of Labor Statistics, Current Employment Statistics Survey, 2011.

5 Average caseworker hourly wage was supplied by ADES via e-mail on 8/30/2012.

6 Average caseworker hourly wage was supplied by DCFS via e-mail on 10/22/2012.

7 Average caseworker hourly wage supplied by CAPP via e-mail on 10/12/2012.

31