SSA RISE CCT-admin-cost_6 24 14_clean

Permanency Innovations Initiative: Phase 3

OMB Information Collection Request

0970 - 0408

Supporting Statement

Part A

June 2014

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

7^th Floor, West Aerospace Building

370 L’Enfant Promenade, SW

Washington, D.C. 20447

Project Officer:

Maria Woolverton

A1. Necessity for the Data Collection

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval for data collection as part of the Permanency Innovations Initiative (PII) Evaluation. PII is a 5-year initiative funded by the Children’s Bureau (CB) within ACF. The overall goal of PII is to build the evidence base for innovative interventions that enhance the well-being and improve permanency outcomes for children and youth who are at the highest risk for long-term foster care and who experience the most serious barriers to permanency. ACF’s Office of Planning, Research and Evaluation (OPRE) is overseeing the evaluation component of the PII.

The proposed PII Evaluation includes multiple components:

• Cross-site implementation study;

• Site-specific impact evaluations;

• Cost study; and

• Administrative data study.

OMB has approved information collection activities for the cross-site implementation study and four site-specific impact evaluations. This current request is for the cost study, the administrative data study, and for additional activities associated with one of the previously approved site-specific impact evaluations. Additionally, we are requesting approval of a revised site-specific instrument that has already received OMB approval.

Study Background

The Adoption and Safe Families Act of 1997 included provisions focused on moving children and youth quickly into permanent families while maintaining their safety. However, many jurisdictions continue to experience growing populations of children who age out of foster care without achieving permanency. Consequently, the CB released a grant announcement for a Permanency Innovations Initiative (HHS-2010-ACF-ACYF-CT-0022, CFDA No. 93.648) to “...fund demonstration projects that support the implementation and test the effectiveness of innovative intervention strategies to improve permanency outcomes of subgroups of children that have the most serious barriers to permanency....” Six grantees were funded under this initiative.

The PII grantees are developing and implementing innovative interventions to address site-specific barriers in order to achieve timely permanency for more children and youth. Overall, PII was designed and structured to address the scarcity of evidence-based programs and practices in the field of child welfare. Despite knowledge about the numerous barriers to permanency, the effectiveness of programs and strategies for achieving permanency has not been established. Using a mix of research methods, the various evaluation components will inform the federal government about the effectiveness of the PII interventions and provide information to help other child welfare agencies develop, implement, and strengthen interventions in the future.

Legal or Administrative Requirements that Necessitate the Collection

The legislative authority is Section 426 of the Social Security Act (42 U.S.C. Section 626). The PII grantees are required to engage in rigorous site-specific evaluations that will help improve services and demonstrate linkages between their interventions and outcomes. They are also required to participate in a cross-site evaluation of the initiative.

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

The PII grantees have been funded to identify local barriers to permanent placement and to develop and implement innovative strategies that mitigate or eliminate those barriers and reduce the likelihood that children will linger in foster care. The proposed PII Evaluation includes multiple components:

• Cross-site implementation study;

• Site-specific impact evaluations;

• Cost study; and

• Administrative data study.

Cross-site implementation study. The implementation study is documenting the status of grantees’ implementation of their planned interventions and addresses questions related to whether implementation status mediates or moderates the achievement of proximal outcomes and/or a reduction in long-term foster care, and whether grantees’ implementation capacity improves over the course of the grant period. A package for this study was approved in August 2012 by OMB (OMB #0970-0408).

Site-specific impact evaluations. Grantees have been funded to implement different interventions with different target populations. Due to the diversity in each site’s sample, as well as across interventions being implemented, each study has unique research questions and data collection tools. Interventions are also being implemented and evaluated on different timelines. Therefore, multiple OMB packages are being submitted to obtain approval for individual sites’ data collection tools.

OMB approved data collection instruments for two site-specific impact evaluations (Washoe County, Nevada and State of Kansas) in August 2012 (OMB #0970-0408). OMB approved data collection instruments for a third site-specific impact evaluation (Illinois DCFS) and the ORB component of the Los Angeles Gay and Lesbian Center’s RISE project in August 2013 (OMB #0970-0408).

The current package is seeking approval for data collection related to evaluation of the Los Angeles Gay and Lesbian Center’s RISE project, which comprises the care coordination team (CCT) and outreach and relationship building (ORB) components. (See below for additional information on this site.)

Cost study. The current package is seeking approval for data collection related to the PII cost study. Some information for the cost study will be obtained through a review of existing documentation (such as grantee expenditure reports). The available information allows for categorization of the first-level of program costs, namely line-item expenditures, including personnel, space, utilities, travel, and supplies. Additional information is required for the second-level categorization, which will comprise the components of personnel (labor) costs, typically the largest proportion of program costs. For personnel costs, we will distinguish various types of program staff activities, such as direct client services and project management and administration activities.

Administrative data study. The current package is seeking approval for data collection related to the administrative data study. State administrative data will serve as the key source of information on the long-term outcomes of importance to PII (e.g., permanency-related outcomes). These data will be used to determine whether interventions help improve permanency-related outcomes for youth in PII intervention groups as compared to youth in the comparison or control conditions. The administrative data study does not require new instruments for measurement and will make use of data currently reported by States under separate OMB clearances for the Adoption and Foster Care Analysis and Reporting System (AFCARS) (OMB Control # 0980-0267) and the National Child Abuse and Neglect Data System (NCANDS) (OMB Control # 0980-0229), as well as data maintained in State Automated Child Welfare Information Systems (SACWIS). This information collection request includes burden associated with preparing or delivering existing administrative data to the PII evaluators.

This information collection request is the third information collection request package submitted to cover different phases and components of the PII project. ACF expects to submit one additional request in the future.

Research Questions

The overarching research question for PII is whether various interventions can improve permanency outcomes (e.g., increase rates of permanency, or decrease time to permanency) for children in the foster-care system. However, there is considerable variability across the PII grantees, both in terms of their specific target populations and the interventions they implemented. Thus, each site has unique research questions and sets of proximal outcomes hypothesized to lead to the common permanency outcome(s). Data collection instruments for each site-specific evaluation are tailored to the research questions and proximal outcomes being measured (see below for more details on the research questions pertinent to the sites we are requesting approval for in this current OMB package). See Attachment A1 for instruments and research questions for previously-approved PII sites, and Attachments B1, C1, and D1 covering the site and studies in the present package (RISE, cost study, and administrative data study).

Los Angeles Gay and Lesbian Center RISE

The Los Angeles Gay and Lesbian Center is implementing an intervention that will address barriers to permanency and well-being for lesbian, gay, bisexual, transgender, questioning (LGBTQ), and gender-nonconforming children and youth, ages 5-19, who are in foster care in Los Angeles County. RISE comprises two components: (1) outreach and relationship building (ORB, which received OMB clearance in August 2013) aimed at creating LGBTQ competency and supportive strategies in Los Angeles Department of Children and Family Services (DCFS) offices and private foster care agencies; and (2) care coordination teams (CCTs) to provide wraparound services addressing the particular permanency needs of LGBTQ and gender-nonconforming children and youth and their families.

The main research question for the ORB component is: For staff in DCFS offices and private agencies, is receiving the ORB training associated with greater competence and changes in behavior regarding addressing issues of sexual orientation and gender nonconformity compared to their competence before receiving the ORB intervention? The ORB component instrumentation (RISE staff pre-test and post-test) was pre-tested as approved under the ACF/OPRE generic clearance, Pre-testing of Evaluation Surveys (0970-0355), and it was found that the instrument was vulnerable to self-reporting bias. Respondents rated themselves extremely high on the pretest, leaving little room for improved scores on the post-test (see the table below). Therefore, RISE revised the instrument to better capture changes in the staff’s knowledge, competence and behavior, as well as eliminate the ceiling effect that was occurring. The revised instrument is included for review in this request package.

Descriptive Statistics for LGBTQ Competence Staff Survey Subscales

		Pre-Test		Post-Test		Change Score
Subscales (1-5, higher scores are better)	N	Mean	Mode	Mean	Mode	Mean	Mode
Importance	236	4.55	5.00	4.55	5.00	-.002	0.00
Skills	239	4.04	5.00	4.20	5.00	.158	0.00
Organizational Support	243	3.70	4.00	4.07	4.00	.376	0.00

The main research question to be addressed in the evaluation of the CCT component is whether LGBTQ and gender-nonconforming children and youth in foster care achieve more timely and stable permanence if they and their families receive intensive wraparound services delivered by CCTs compared with LGBTQ children who receive usual services. A secondary research question examines the impact of RISE’s CCT component on proximal outcomes, including durable connections, enhanced well-being, and improved emotional permanency.

Cost Study

The findings from the cost study will inform the federal government about the organizational cost of operating PII interventions and provide information to help other child welfare agencies. For instance, future adopters of PII interventions will benefit from findings on cost-per-case that will enable estimates of operating cost based on locally projected personnel and non-personnel costs and caseload sizes.

The following questions will be addressed:

• What is the cost-per-case of providing PII intervention services? What accounts for intra-intervention variation among cases?

• What is the cost-per-case of providing intervention services for different child and family subgroups? What accounts for the variations among child and family subgroups?

• What is the number of incidents and amount of person-time expended on each type of PII project activity, including casework, supervision, service delivery management, and project administration? What accounts for the variation in the mix of activities (types, incidence, and person-time expended)?

• Does cost-per-case differ across PII projects? If so, what accounts for the variation?

• What is the relationship between costs and outcome effect sizes for each PII project and across PII projects?

The scope of the cost study encompasses a full range of costs incurred by an organization while operating a PII project during a period of continuing, stable operations. Grantee organization services and activities conducted outside the scope of the PII project are excluded from the cost study. The primary analysis unit is an individual case (child/family) that receives PII intervention services. Estimates of costs at the case level enable aggregate estimates, such as cost-per-child and family subgroup or cost-per-week or month of operation.

Administrative Data Study

The cross-site administrative data study seeks to answer the PII overarching research question: Do the grantee interventions improve permanency outcomes (e.g., increase rates of permanency, or decrease time to permanency) for children in the foster care system? The research questions of each PII grantee are unique to their target populations, their intervention, and their set of proximal outcomes. The administrative data study will examine a consistent set of permanency outcomes for all grantees utilizing AFCARS, NCANDS, and SACWIS data. The study subjects of each grantee will be matched to comparison cases drawn from those datasets.

Study Design

RISE

The evaluation of the RISE ORB component has already been previously reviewed and approved by OMB (for additional details, please see the information collection request approved in August 2013 under OMB #0970-0408).

The RISE CCT evaluation will utilize a comparison group design; proximal outcomes will include measures of youth’s permanency connections, self-acceptance, and support and rejection in their relationships. RISE plans for 80 children and youth to participate in the evaluation; 40 referrals will be assigned to receive CCT services and 40 referrals will be assigned to a comparison group and receive services as usual. Ultimately, data collected through the evaluation will provide information on the extent to which RISE achieved its goal of faster permanency for LGBTQ and gender-nonconforming children and youth.

The CCT instruments were reviewed by the Williams Institute for the RISE project, using the most current thinking on how to ask questions about sexual orientation and gender identity.  The Williams Institute is a national think tank at UCLA School of Law dedicated to conducting rigorous, independent research on sexual orientation and gender identity law and public policy.  They are most known for their work on population based research documenting demographics and economic wellbeing of LGBT people in the U.S. and for leading workgroups and publications designed to make recommendations for measuring sexual orientation and gender identity.  To this end, they previously published the Best Practices for Asking Questions About Sexual Orientation On Surveys (Sexual Minority Assessment Research Team, 2009) and are in the process of completing the “Best Practices for Asking Questions to Identify Transgender and Other Gender Minority Respondents on Population-Based Surveys” (Gender Identity in U.S. Surveillance workgroup, in preparation).  The Institute’s work on youth and human services include: Serving Our Youth: Findings from a National Survey of Service Providers Working with Lesbian, Gay, Bisexual and Transgender Youth Who Are Homeless or At Risk of Becoming Homeless (July 2012) and Provider Perspectives on the Needs of Gay and Bisexual Male and Transgender Youth of Color (August 2013).

Cost Study

The PII cost study is designed to systematically describe and analyze the cost-per-case incurred by four PII projects in providing PII services to children and families during a period of stable operation. The study features a single case research design without longitudinal or external control/comparison groups. Internal subgroup comparisons and relational analyses of costs and outcomes are achievable within the limits of the research design. The research design is methodologically appropriate considering the stage of development of the PII projects.

While a uniform approach to cost data collection will be applied across grantees, the detailed plan, particularly the contents of items contained in web-based activity logs (surveys of PII personnel), will be customized for each PII project.

With regard to financial information, the OMB-approved PII Implementation Case Study Protocol (Attachment C) is designed to capture the dollar amounts allocated to both labor and non-labor resources related to the implementation of the PII project at each site. Annual PII grant expenditure data obtained from the Children’s Bureau will provide the basis for follow-up discussions with grantee staff. For example, grantees will be asked under the PII Implementation Case Study Protocol to “describe how dollars were spent to support labor (what positions in the organizational structure were supported).” The evaluation team will not collect personally identifying information such as name or SSN.

Grantee expenditure reports usually contain aggregate data on resource category costs. The administrative and evaluation data systems might contain case-specific data on the incidence of some but not all client service activities. To the extent feasible, the data on PII activities from existing systems will be used and, when necessary, supplemented with data collected from weekly and monthly logs (web-based surveys) completed by PII personnel. The logs will record the occurrence of activities that are not captured in existing data systems.

With regard to the use of locally generated PII project activity data, grantees’ existing evaluation and administrative data systems capture data on some case-level activities such as client assessments, in-office meeting, and in-home visits by CPS social workers. For example, one grantee adapted an existing case management data system to include activities carried out by case workers that deliver PII intervention services. The system includes most, if not all, specific activities conducted with or on behalf of a specific family. To support local PII evaluations, two other grantees installed REDCAP systems that capture selected case-level activities. If obtained under existing or newly negotiated data-sharing agreements, grantees’ readily available data on PII project activities will reduce burden associated with PII project activity logs (web-based surveys). Only activities not available through grantees’ existing administrative and evaluation data systems will be collected using activity logs.

An early step in the cost study will be to identify PII activities captured in grantees’ administrative and evaluation data systems. Activity incidence data obtained from grantee data systems will be site- and case-specific. We will not obtain personal information such as names and SSN’s.

Administrative Data Study

The cross-site administrative data study will use a quasi-experimental design with retrospective matched comparison groups. Comparison groups for the subjects of each grantee intervention will be drawn from AFCARS, NCANDS, and SACWIS datasets. These datasets contain large numbers of cases with recorded outcomes and well-defined and well-understood field definitions, and they can be obtained with minimal burden on grantees. Statistical and machine-learning methods will be used to identify a set of characteristics that have objectively demonstrated strong associations to permanency outcomes in these datasets. Each intervention's comparison group will be matched to study subjects on this set of characteristics.

Record quality checks for missing, inconsistent, and out-of-range values are applied to all new AFCARS and NCANDS records. In addition, record linkage (de-duplication and cross-record) processing employs consistency checks.

Permanency outcome measures can be very sensitive to missing removal and discharge dates. Removal dates are rarely (<0.1%) missing, but discharge dates may be missing in as many as 30% of records in some states. We anticipate that missing discharge dates will be an issue in some jurisdictions, and that the probability of missing information may be associated with the intervention and other important covariates. Since records with truly missing discharge dates still contain partial permanency information, we can calculate approximate dates of loss-to-followup. We intend to treat records with missing discharge dates as right-censored observations at an approximate date of loss-to-followup. We expect that this censoring method will minimize bias, maximize accuracy, and have an insignificant effect on power.

Universe of Data Collection Efforts:

Previously Approved Data Collection Instruments

OMB approval for data collection for the Kansas, Washoe, Illinois, and RISE ORB grantee impact studies and the cross-site implementation study has already been received. See Attachment A1 for the Research Matrix for the Kansas, Washoe, Illinois, and cross-site implementation studies, showing a summary of their research designs, instruments, burden hours, and research questions. (Due to revisions in its instruments, RISE ORB is included in the current package and has a new research matrix even though part of the information collection were already approved.)

Current Request for Data Collection Instruments

RISE CCT

The data collection activities for the RISE CCT evaluation include interviews with children and youth ages 5-19, individuals identified as permanency resources, and current caregivers. See Attachment B1 for the RISE Research Matrix, a crosswalk of RISE research design, instruments, burden hours, and research questions covering both the CCT and the ORB components

For youth, ages 11-19, a trained data collector will administer the RISE Youth Assent Form and Script and the RISE Youth Interview (Attachment B2) to the youth at baseline and 1 year later.

All of the instruments used in the youth interview, though informed by existing instruments, were developed for RISE by RISE and evaluation contractor staff. This interview contains questions related to youth’s personal comfort with sexual orientation and gender identity, how much their LGBTQ identity is supported or rejected by others, the youth’s ability to identify adults in their lives that provide support, as well as debriefing questions due to the sensitive nature of the issues covered. Approximately six months after enrollment, the data collector will administer the Youth Qualitative Interview Assent and Script and the Youth Qualitative Interview Protocol, which assesses youth’s perceptions of their experiences with receiving services from RISE, as well as overall comfort discussing sensitive topic with his or her caseworker (Attachment B3; note there are separate protocols for the intervention and comparison groups).

For children ages 5-10, data collection will take place on the same schedule (baseline and 1 year later). CCT facilitators will administer the CCT Facilitator Interview to children ages 5-10 in the intervention group, which identifies adults in the child’s life that are committed to providing ongoing support (Attachment B4).

For all children and youth in the intervention group (ages 5-19), CCT facilitators will complete the CCT Facilitator Survey (see Attachment B5). This survey will be completed at baseline and 1 year later, and it captures the facilitator’s assessment of the child’s achieved level of permanency.

CCT facilitators regularly administer and complete the Child and Adolescent Functional Assessment Scale (CAFAS) on all clients for case planning purposes. We are using those data for analysis purposes. Data will be provided to the PII team at baseline and 1 year later by PII facilitators. The burden displayed in Table A2 is for the amount of time the facilitators will spend compiling and submitting the CAFAS data to the evaluation team. No additional burden is being placed on the facilitators, since the caseworkers routinely complete this measure independent of the PII Evaluation. (Note that the CAFAS is copyrighted so is not included in the instrument package; Attachment B6 covers the CAFAS submission.)

Data collectors will collect information from two types of adults who were involved in the CCT process for youth ages 11-19: a permanency resource and a current caregiver. A permanency resource is an adult involved in the CCT process who has expressed a willingness to provide a permanent home for the child or youth. A current caregiver, with whom the child or youth lives, may become the permanency resource. Only permanency resources and current caregivers for the intervention group with youth ages 11-19, will be interviewed. Data collectors will administer the Permanency Resource Consent Form and Interview (Attachment B7) or the Current Caregiver Consent Form and Interview (Attachment B8) to the respective respondents at approximately baseline and 1 year later. The Permanency Resource Interview assesses whether the permanency resource is committed to providing ongoing support to the youth, their comfort level with LGBTQ persons, the connections they have made with the youth, their attitude toward LGBTQ youth, and their attitudes towards gender and gender roles of person that do and do not fit typical masculine and feminine archetypes (Attachment B7). The Current Caregiver Interview for caregivers of youth in the intervention group measures the caregiver’s attitudes toward LGBTQ persons, as well as their attitudes towards gender and gender roles of person that do and do not fit typical masculine and feminine archetypes. (Attachment B8).

RISE ORB

Previously approved data collection activities for the ORB evaluation include administration of a pretest and posttest with foster care staff who participate in ORB training. We are requesting revisions to the pretest and posttest instruments in addition to clearance for a follow-up survey to be administered two months after training in order to assess whether there were changes in the staff’s behavior, as a result of the training. The proposed version was tested on nine respondents.

As described in the previous clearance, for staff randomly selected to participate in the study, RISE will administer a consent form and Staff Pretest immediately before the training begins (see Attachment B9a. Immediately after the training, staff will complete the Staff Posttest (Attachment B9b, same instrument as pretest, but different consent form). The pre- and posttest assess the staff’s knowledge of how to address issues of sexual orientation and gender nonconformity. Two months later, staff will receive an email invitation to complete the Staff Follow-Up Survey, which is a web-based instrument (see Attachment B10). The Follow-up Survey will assess whether or not staff are addressing issues of sexual orientation and gender nonconformity in a sensitive manner. Attachment B10 also includes a print invitation to complete the Staff Follow-Up Survey, which will be used when the email cannot be delivered due to erroneous email addresses.

Cost Study

The cost study research matrix is shown in Attachment C1. Prior to participating in Cost Focus Group discussions, case workers and supervisors identified to participate in a focus group will spend about 1.5 hours preparing for the focus group discussion by completing the Cost Study consent form and preparation table. These materials involve the group participant compiling information about time spent on various activities, such as advocacy, assessments, direct services, documentation, travel, mentoring, etc. (see Attachment C2 for consent form and preparation table). The participants will then participate in a Cost Focus Group, which will yield information used to inform the creation of customized instruments for use in log data collection at each grantee (see Attachment C3 for the focus group guide). Following the focus groups, each participant will complete the Trial Administration of Logs, which will be a customized instrument developed for that site (see Attachment C4) based on information learned during the focus group.

Web-based weekly casework activity logs (Attachment C5) will gather information from caseworkers that serve PII client children and families. For each PII case, caseworkers will be asked to indicate the number of activities that occurred during a specific week and the estimated total person-time devoted to PII activities. The activities include direct client service, indirect client service, and management and administration. Caseworkers’ reports of activities in the two client service categories will be case-specific. In other words, caseworkers will indicate the type and number of each client service activity carried out for a case during a reporting week. Direct client service refers to activities conducted with the client child and family, such as assessment, counseling and support, initiating referrals, and scheduling appointments. Indirect client service refers to collateral activities conducted on behalf of a client child and family, such as case management documentation/MIS data entry, locating resources, advocating without the client, and caseworker travel. The occurrence of management and administration activities, which are defined below, also will be captured in weekly casework activity logs.

Web-based weekly supervisor activity logs (Attachment C6) will gather information on individual supervision, group supervision, and management and administration activities. Individual supervision refers to supervisory activities conducted with an individual caseworker, such as case status review meetings, coaching sessions, and individualized training. Group supervision refers to supervisory activities conducted with multiple caseworkers, such as conducting team meetings.

Web-based monthly management and administration activity logs (Attachment C7) will gather information on service delivery management and program administration: activities of PII personnel that do not involve PII cases, but relate to the continuing operation of the PII project. Service delivery management refers to activities that support delivery of client services, such as leading and attending training sessions and team meetings. Program administration refers to activities that foster program development and maintenance, such as attending organization meetings, serving on committees or work groups, screening candidate referral agencies, outreach and marketing, and grant management.

Costs will be compared across sites on multiple dimensions including: total cost per case, total labor and non-labor cost per case, average cost per case, and range of cost per case. Of course, since the interventions are quite different across sites, the comparisons mainly are intended to inform potential future adopters of the interventions about cost differences. We will be able to compare grantee expenditure reports and cost study reports. For non-labor hours, we will collect data from grantee expenditure reports, as these serve as grantee’s official documentation of spending each month. We believe this will be the most accurate portrait of grantee’s spending on non-labor related items.

Grantee-specific casework, supervision, and management and administration activity reports will be generated within 90 days after activity log data collection ends for each grantee. The reports will be completed in the period June through August 2015. A final report on cost-per-case cost that includes activity log data will be completed in September 2015.

Administrative Data Study

The administrative data study does not require new instruments for measurement. Data for the administrative data study will be submitted on the same schedule that the grantees are already submitting their data to ACF. The research matrix is shown in Attachment D1. Instructions for submitting the administrative data are shown in Attachment D2, if the respondent is including no added fields, and Attachment D3, if the respondent is including added fields (see last sentence of Attachment D3).

Future Information Collection Requests

There will be one more PII OMB information collection request, which will cover data collection for the California Partnership for Permanency (CAPP) site.

A3. Improved Information Technology to Reduce Burden

Electronic data collection is not planned for RISE CCT because of small sample size and resource issues. All proposed RISE CCT instruments will be administered in-person and individually. The follow-up survey for the ORB component will be web-based. The cost study includes web-based data collection to help reduce the burden on respondents; whenever possible, data for the cost study will be collected from existing documentation or administrative data sources in order to reduce burden on the grantees. For the administrative data study, all states currently submit AFCARS data files every 6 months and NCANDS data files every 12 months to ACF. The PII evaluation team has created a secure online portal to accept those same data files from PII grantee states on the same schedule.

A4. Efforts to Identify Duplication

Except for the CAFAS, none of the proposed instruments for RISE CCT and ORB are currently being used on the target populations in the child welfare systems. In order to streamline the data collection process and reduce duplicative efforts, we reviewed all measures to confirm that no questions are repeated and the instruments are not redundant. Additionally, we reviewed all measures to ensure that they measured discrete concepts of interest for this research study. No two measures target the exact same concept. In addition, we will be using extant administrative data to measure distal outcomes such as achievement of permanency. For the cost study, all of the proposed instruments are new and not currently being used in the grantee sites; none of the information sought through the focus groups and activity logs will be available except through this information collection. For the administrative data study, the data collection schedule corresponds to exactly the same 6- and 12-month schedule that states follow for their routine AFCARS and NCANDS submissions to ACF, which minimizes the burden on PII grantee states.

A5. Involvement of Small Organizations

No small organizations are impacted by the data collection in this project.

A6. Consequences of Less Frequent Data Collection

For RISE CCT, the data collection schedule was developed to produce the information necessary for tracking families’ and children’s progress on the key outcomes of interest. In order to track the progress of the families in both intervention and comparison groups, the data need to be collected on the specified schedules. RISE added a follow-up survey for ORB in order to capture changes in behavior after receiving the training, as the pretest and posttest measure changes in knowledge. A 2-month follow-up period will provide sufficient time for practice changes to occur.

The cost study is designed to collect accurate information on case-level activities. Timely correspondence between the occurrence of an activity and collecting data on the activity ensures more accurate recall. The use of weekly logs for casework and supervision and monthly logs for management and administration will ensure accurate correspondence between the occurrence of an activity and the collection of data on the activity. Less frequent completion of logs will compromise quality of activity data because of respondents’ recall capabilities.

Data for the administrative data study will be submitted on the same schedule that the grantees are already submitting administrative data to ACF under other clearances.

A7. Special Circumstances

There are no special circumstances for the proposed data collection efforts.

A8. Federal Register Notice and Consultation

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on September 13, 2013, Volume 78, Number 178, page 56714-56715, and provided a sixty-day period for public comment. A copy of Federal Register notices is attached (see Attachment F). During the notice and comment period, we received and responded to a request for the draft instruments.

The following experts were consulted on methodological issues concerning evaluation issues:

• Patti Chamberlain, Ph.D. – expert on implementation and efficacy research, particularly implementation of parent-mediated interventions and scaling up best practices models.

• John Landsverk, Ph.D. – expert in cost calculation and implementation research in child welfare and mental health interventions.

• Andrew Barclay, M.S. – expert in database design, data analysis, and statistics, particularly as applied to child welfare.

• Linda Collins, Ph.D. – expert on optimization of behavioral interventions, particularly on adaptive designs for prevention.

A9. Incentives for Respondents

We will provide a token of appreciation to respondents for participating in the site-specific grantee evaluations. These amounts are based past experience in conducting interviews with child welfare populations and on local experience with providing respondents with small monetary gifts and gift cards. No individual incentives will be provided for the cost study or the administrative data study. For the RISE CCT evaluation, youth will receive a $15 gift card and caregivers and resources will receive a $20 gift card at each interview as a token of appreciation. Staff participating in ORB training will receive a gift (such as a water bottle) worth a maximum of $10 after completing the follow-up survey.

A10. Privacy of Respondents

All consent forms include assurances of privacy. These assurances, which are included in the IRB applications for all PII grantees, include:

• Respondents receive a written informed consent form that will explain the evaluation process and assure them that their participation is voluntary, and their information will be private to the extent permitted by law and securely stored.

• Respondents are assured that, without risk of penalty, they may discontinue participating or choose not to participate in an interview or group discussion or answer certain questions at any point.

• Strict policies and procedures for respondents’ privacy are followed by all project staff.

• All hard copies of documents are secured behind two locks (e.g., locked file cabinet in locked room).

• All electronic content is stored on secure servers. The server is set with privileges that allow access only by specific individuals who have a username and password.

• All project data are reported and presented at the aggregate level in order to prevent the identification of any individual respondent, and names of respondents will not appear in any report.

All data transmissions are over secure channels. All electronic content is stored on secure servers. The data and analysis servers can be accessed only by specific staff over channels secured through two-factor authentication. Study data will be reported and presented only at an aggregate level that protects individual privacy.

The Office of Planning, Research and Evaluation is working with our Office of Information Services to create and publish a Privacy Impact Assessment (PIA) to ensure that information handling conforms with applicable legal, regulatory, and policy requirements regarding privacy; determine the risks of collecting and maintaining PII; assists in identifying protections and alternative processes for handling PII to mitigate potential privacy risks; and communicates an information system’s privacy practices to the public. This PIA, titled ACF Research and Evaluation Studies, will be available online through the Department of Health and Human Services.

A11. Sensitive Questions

The data collection batteries for the grantees’ evaluations include questions that might be perceived as sensitive, including questions about gender identity and sexual orientation (see Attachment B2: Youth Survey). Questions about gender identity and sexual orientation are asked because the CCT intervention is designed to increase participating LGBTQ’s youth’s self-acceptance, and their perception of the support and rejection of others based on gender identity and sexual orientation. Respondents will be assured of their privacy, and they will be told that they can skip any questions that they do not want to answer and can stop the interview at any time. RISE recently conducted a survey of youth¹ that specifically asked the youth what their gender identity and sexual orientation were; RISE did not find that the youth became upset or stressed from the questions.

The focus groups and activity log data collection for the cost evaluation do not include questions that might be perceived as sensitive. Finally, the administrative data used in this study are gathered during the course of the state agency's routine business and do not pose sensitive questions to children and families.

A12. Estimation of Information Collection Burden

Previously Approved Information Collections

We have used 1,160 hours (30%) of the previously approved 3,810 hours, as shown in Table A1.

Table A1. Previously Approved Information Collections Approved Hours and Hours Remaining
Instrument	Previously Approved Annual Burden Hours	Hours Used	Annual Burden Hours Remaining
CROSS-SITE IMPLEMENTATION STUDY:
Survey of Organization/System Readiness	18	12		6
Implementation Drivers Web Survey	240	29		211
Grantee Case Study Protocol	240	40		200
Fidelity Data (Implementation Quotient Tracker)	24	4		20
Cross-Site Estimated Total	522	85		437
KANSAS:
Caregiver Initial Information Form	30	15		15
Family Assessment Battery	900	444		456
CAFAS/PECFAS	630	296		334
Caseworker discussions for NCFAS-G&R completion	315	148		167
Kansas Estimated Total	1875	903		972
WASHOE COUNTY:
Family Assessment Battery	525	118		407
Washoe Estimated Total	525	118		407
ILLINOIS DCFS:
DCFS Biological Parent Study Contact Form	17	2		15
DCFS Biological Parent Interview	86	2		84
DCFS Youth and Foster Parent Study Contact Form	23	5		18
DCFS Foster Parent Interview	342	21		321
DCFS Youth Interview	342	24		318
Illinois DCFS Estimated Total	810	54		756
RISE ORB:
RISE Staff Pre-Test (B9a: revised)	39	0		39
RISE Staff Post-Test (B9b: revised)	39	0		39
RISE ORB Estimated Total	78	0		78
TOTAL	3,810	1,160		2,650

Newly Requested Information Collections

Table A.2 contains the estimated annual burden hours and annual cost for each type of respondent. The total annual burden for this information collection is expected to be 3,792 hours.

Table A2. Total Burden Requested Under this Information Collection
Instrument	TOTAL Number of Respondents	Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	ANNUAL Burden Hours	Median Hourly Wage	Total Annual Cost
RISE CCT:
B2: Youth Interview (ages 11-19)	65	22	2	1	44	17.217	275.367
B3: Youth Qualitative Interview (ages 11-19)	65	22	1	1	22	17.217	137.687
B4: CCT Facilitator Interview (facilitator burden)8	4	2	4	0.2	2	20.57	41.14
B4: CCT Facilitator Interview (child burden)8	8	3	2	.5	3	--
B5: CCT Facilitator Survey	4	2	21	0.2	8	20.57	164.56
B6: Facilitator submission of CAFAS data9	4	2	21	0.1	4	20.57	82.28
B7: Permanency Resource Interview10	34	11	2	1.0	22	17.21	378.62
B8: Current Caregiver Interview11	34	11	2	0.6	13	17.21	223.73
RISE CCT burden total					118	--	1,303.37
RISE ORB:
B10: Staff Follow-Up Survey	470	157	1	0.3	47	20.57	966.79
RISE ORB burden total					47	--	966.79
Cost Study:
C2: Cost Study Focus Group Preparation	27	9	1	1.5	14	27.02	378.28
C3: Cost Focus Group Guide	27	9	1	4.0	36	27.02	972.72
C4: Trial Administration of Logs	27	9	1	1.5	14	27.02	378.28
C5: Weekly Casework Activity Log	369	123	52	0.4	2,558	20.57	52,618.06
C6: Weekly Supervision Activity Log	117	39	52	0.4	811	27.02	21,913.22
C7: Monthly Management/Administration Log	90	30	12	0.5	180	29.69	5,344.20
Cost Study burden total					3,613	--	81,604.76
Administrative Data Study:
D2: Data file submission, no added fields	2	1	12	0.3	4	20.57	82.28
D3: Data file submission with added fields	3	1	12	0.8	10	20.57	205.70
Administrative Data Study burden total					14	--	246.84
Estimated Annual Burden Total					3,792	--	84,121.76

⁷ Hourly wage and annual cost are only calculated for youth ages 18-19. There are expected to be 25 youth ages 18-19, or about 8 youth per year.

⁸The CCT facilitators, rather than the data collector, will conduct these interviews due to the young ages of the children in the intervention group. These interviews will be conducted only with children, ages 5-10, in the intervention group. Total n = 8.The same instrument will be administered twice. There is no annual cost for the child’s burden, because the child is younger than 18.

⁹The CAFAS is administered as part of case planning, so the only burden is in submitting the CAFAS data to the evaluation team.

¹⁰The interviews with permanency resources will be conducted with permanency resources only for youth in the treatment group (ages 11-19), not in the comparison group. It will not be possible to verify that interviewing permanency resources would be safe for youth in the comparison group.

¹¹The interviews with current caregivers will be conducted with current caregivers only for youth in the treatment group (ages 11-19), not in the comparison group. It will not be possible to determine that interviewing current caregivers would be safe for youth in the comparison group.

Total Burden

The total annual burden under OMB #0970-0408 will be 2,650 annual hours remaining for previously approved instruments (table A.1) in addition to 3,792 annual hours for instruments under review in this request. This is a total of 6,442 hours under OMB #0970-0408.

Total Annual Cost

To compute the total estimated annual cost, the total burden hours were multiplied by the median hourly wage for each adult participant. The specific median hourly wages were as follows: $17.21² for parents/guardians/caregivers/permanency resources/youth ages 18-19; $20.57³ for caseworkers and facilitators; $27.02⁴ for supervisors; and $29.69⁵ for managers/administrators. Children and youth under age 18 do not have an hourly wage

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The total cost for the data collection activities under this current request will be $903,781 for RISE CCT, the cost study, and the administrative data study; and $1,857,751 remaining for Kansas, Washoe, Illinois DCFS, RISE ORB, and the implementation study. Annual costs to the Federal government will be $981,671 for all proposed data collection activities under OMB #0970-0408.

A15. Change in Burden

This is an additional information collection request under OMB #0970-0408.

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

Analysis Plan

Cost Study

The PII cost study focuses on a grantee organization’s full cost of operating a PII intervention, which is also known as “cost allocation analysis.” “Societal costs” are excluded, such as the dollar value of client or volunteer time devoted to PII activities and the value of non-PII services received by PII clients. Cost is defined as the monetary value of resources expended on PII project activities. This primarily includes the cost of person-time used to deliver direct services (defined as activities conducted with, or on behalf of, a specific case), which is the basic building block for estimates of the full cost of a case. The definition of cost implies collection of data on personnel resources, including (a) the amount of person-time expended on PII activities (resource units expended) and (b) the dollar value on those person-hours (monetary value of resource units). The other components of cost-per-case include the cost of person-time used for supervision, management, and administration activities that support client services and general PII project operations as well as overhead costs such as facilities and equipment. The costs of person-time used for supervision, management, and administration, which is incurred at organization echelons above the front-line caseworkers, will be apportioned across cases based on pre-determined rules, such as allocation based on the amount of caseworker person-time expended on a case. Overhead is defined as PII project costs in non-personnel resource categories that generally are not assigned to a specific case. These costs include expenses for facilities, utilities, equipment, purchased services, supplies, and miscellaneous other expenditures. These non-personnel costs will also be apportioned across cases based on pre-determined rules.

A series of statistical analyses will be conducted to describe multiple aspects of person-time utilization and dollar cost. The first analysis will examine the distribution of person-time across the following major activity clusters: casework (activities conducted with or on behalf of a child/family), supervision (activities that involve individual and group oversight and support of case workers), and management and administration (activities that support delivery of client services and foster PII project development, implementation, and maintenance). Based on compensation data, the dollar cost of labor will be examined to establish an estimate of personnel cost per case (child/family). The estimated amounts of person-time used on a given activity (expressed in hours or fractions thereof) will be multiplied by the cost-per-hour for the personnel that carried out the activity, producing a monetary value of person-time expended on the activity. These basic calculations to establish monetary values for person-time expended on PII activities will enable aggregations of personnel costs (and person-time utilization) by case, activity type, client characteristic, and personnel echelon. Additional analyses will examine average weekly labor use and cost as well as the distribution of labor across weeks. Cross-case and cross-site comparisons will be used to investigate the extent to which labor use and costs are variably distributed across major clusters and types of activities. The final estimate of full cost per case will include personnel, non-personnel, and general agency administration costs.  For cases with accompanying data on outcome change scores, an exploratory analysis will be conducted of cost per increment of outcome change.

 

To assess factors that might be associated with variations in person-time use and cost among cases, cost study cases at each grantee and all grantees will be included in multivariate analyses. Child/family characteristics data from the PII administrative data study will be merged with cost study data. Child characteristics will include variables such as ethnicity, gender, age at PII project entry, and duration of out-of-home placement. Parent/family characteristics will include variables such as number of children under age 18 in household.

Site-specific Impact Evaluations and Administrative Data Study

Each PII site will be collecting data on proximal outcomes that will differ from site to site. These will be examined within each site for additional insights into the permanency process but will not be generalizable across sites. The primary distal outcome for all sites will be achievement of permanency and length of time needed to achieve permanency from entry into the child welfare system for the intervention and comparison groups of children. This type of data is censored because at end of the follow-up time there will be children in the study for whom permanency has not yet been achieved, and thus complete information as to the length of time needed to achieve permanency is not available. For this type of data, survival analysis techniques will be used to take into account the censored nature of the data. Also it is likely that other outcomes such as reentry to foster care will be censored to the duration of each study. The permanency outcomes of intervention and comparison groups will be analyzed and compared using techniques appropriate to censored data.

All sites will have AFCARS administrative data and this will provide important covariates that can be used to adjust for variation in permanency outcomes or in the length of stay measure, so that the true effect of the treatment - our primary focus in the study - may be gauged. In particular, we may want to adjust for the sex, race, and age of the child, and for any disability that the child may have. These are likely to impact both exit to permanency and the length of stay in the system. Other secondary information available includes reason for removal, abuse, number of previous removals, etc., and if appropriate, these data items may also be used to adjust for possible differences in the outcome being analyzed.

Because each PII grantee intervention has a unique target population and study selection criteria, study subjects must be matched to a non-study comparison group on a per-intervention basis. Thus, each PII grantee intervention will have an associated comparison group. These comparison groups will be selected from non-study AFCARS, NCANDS, and SACWIS cases using a balanced match on characteristics that, at the moment of entry into the study, would strongly predict the same outcome as a study subject in the absence of the PII intervention. Additionally, every site will be collecting data on proximal outcomes that will differ from site to site. These will be examined within each site for additional insights into the permanency process but will not be generalizable across sites.

Time Schedule and Publication

Table A3 outlines the estimated time schedule for data collection, analysis, and publications.

Table A3. PII Evaluation Timeline
Task	First Package (PII1)			Second Package (PII2)		Third Package (PII3)*			Fourth Package (PII4)**
	Cross-Site	Kansas	Washoe	DCFS	RISE ORB	RISE CCT	Admin	Cost	CAPP
Data Collection, Data Cleaning, and Quality Assurance	9/1/12 – 5/31/15			8/1/13 – 5/31/15		Upon OMB approval– 5/31/15		Upon OMB approval – 8/31/15	Upon OMB approval – 5/31/15
Create Analysis Files	1/1/15 – 5/31/15	4/1/15 – 5/31/15				Upon OMB approval – 5/31/15		Upon OMB approval – 8/31/15	Upon OMB approval – 5/31/15
Data Analysis	2/15 – 7/15								6/15 – 8/15
Draft Report	7/30/15								9/30/15
Final Report	9/30/15

*Current information collection request

**Future information collection request

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

References

Durso, L.E., & Gates, G.J. (2012). Serving our youth: Findings from a national survey of service providers working with lesbian, gay, bisexual, and transgender youth who are homeless or at risk of becoming homeless. Los Angeles: The Williams Institute with True Colors Fund and The Palette Fund.

Durso, L.E., Kastanis, A., Wilson, B. M., & Meyer, I. H.  (2013).  Provider perspectives on the needs of gay and bisexual male and transgender youth of color.  Retrieved from http://williamsinstitute.law.ucla.edu/wp-content/uploads/ACCESS-final-rev-aug-2013.pdf

Gender Identity in U.S. Surveillance Workgroup (2013).  Gender-related measures overview.  Retrieved from http://williamsinstitute.law.ucla.edu/wp-content/uploads/GenIUSS-Gender-related-Question-Overview.pdf

Sexual Minority Assessment Research Team. (2009).  Best practices for asking questions about sexual orientation on surveys.  Retrieved from http://williamsinstitute.law.ucla.edu/wp-content/uploads/SMART-FINAL-Nov-2009.pdf

1 This youth survey was not part of the evaluation, but was an activity that RISE did on their own and thus was not submitted for OMB review.

2 Median all-occupation hourly wage is from Bureau of Labor Statistics, Occupational Employment Statistics ($16.71 in 2012, increased 3% for 2014). http://www.bls.gov/oes/current/oes_nat.htm

3 Median caseworker and facilitator hourly wage is from O*NET OnLine ($19.97 in 2012, increased 3% for 2014). http://www.onetonline.org/link/summary/21-1021.00

4 Median supervisor hourly is from O*NET OnLine ($26.23 in 2012, increased 3% for 2014). http://www.onetonline.org/link/summary/21-1029.00

5 Median manager/administrator hourly wage is from O*NET OnLine ($28.83 in 2012, increased 3% for 2014). http://www.onetonline.org/link/summary/11-9151.00

31