Regional Partnerships Grants (RPG) National Cross- Site Evaluation and
Evaluation Technical Assistance
Supporting Statement, Part B
For OMB Approval
November 25, 2013
In response to federal legislation (Child and Family Services Improvement and Innovation Act of 2011; Pub. L. 112-34) in October 2012, under the Regional Partnership Grant (RPG) program, the Children’s Bureau (CB) within the Administration for Children and Families issued 17 grants to develop interagency collaborations and provide services designed to increase well-being, improve permanency, and enhance the safety of children who are in an out-of-home placement or are at risk of being placed in out-of-home care as a result of a parent's or caretaker's substance abuse. The overall objective of the RPG Cross-Site Evaluation is to describe and document the performance of the RPG grantees, the outcomes for families enrolled in RPG, and the effectiveness of approaches taken by the grantees, as stated in the legislation. To meet these evaluation goals, the RPG Cross-Site Evaluation includes three study components: (1) an implementation and partnership study; (2) an outcomes study; and (3) an impact study. The implementation and partnership study will collect data on RPG programs—with a special focus on 10 selected evidence-based programs—and on the collaborative partnerships. The outcomes study will describe the characteristics of and changes over time in children, adults, and families who participate in the RPG programs. The impact study will include a subset of seven RPG grantees that are implementing rigorous comparison local evaluation designs and can provide outcome data on both treatment and comparison group members.
All 17 RPG grantees will participate in the implementation and partnership study. In this ICR, clearance is sought for five instruments associated with this study. Before describing the instruments and their respondents, we describe the sample of RPG program services on which some of these instruments will focus.
Focal EBPs. To meet the needs of adult and child family members and ensure the comprehensive assistance needed to support adult recovery from substance abuse, ensure child safety, and stabilize families, each grantee will provide multiple services to participants. In its grant announcement, CB emphasized the importance of basing these services on evidence-based or evidence-informed program and practice models (EBPs). Across all grantees combined, 51 different EBPs will be used, with some grantees implementing a dozen or more. To reduce the burden and complexity of data collection that would be necessary if detailed service delivery data were collected on all 51 EBPs, the RPG Cross-Site Evaluation will gather service delivery data on a subset of 10 EBPs (referred to as “focal” EBPs).
CB selected these focal EBPs based on four criteria: (1) the EBP is being implemented as a primary service of at least two RPG grantees; (2) the EBP is a session-based program for which service use data can be collected; (3) each of the 17 grantees is implementing at least one of the focal EBPs; and (4) the set of focal EBPs includes at least one model from each of the five types of interventions RPG grantees are implementing (child-caregiver therapy, counseling, family strengthening, response to trauma, and substance abuse treatment). On average, each RPG grantee is implementing and will report on 3 of the 10 focal EBPs.
Clearance is requested for the following five instruments associated with this component of the evaluation:
Grantee and partner staff topic guide. During site visits to each of the 17 grantees, semi-structured interviews will be conducted with the RPG grantee director. Small group interviews will be conducted with managers and supervisors for each of the focal EBPs, some of which may be implemented by the grantee and others by partner organizations. Individual interviews will be conducted with two frontline staff and two supervisors or managers of focal EBPs. If an EPB is delivered by more than two staff of each type, we will randomly select two to participate in the individual interviews.
Semi-annual progress reports. All RPG grantee project directors or their designees will submit written semi-annual progress reports, each describing implementation and enrollment progress, program changes, and adherence to focal EBP models during the prior six months.
Enrollment and service log. All RPG grantees will record program enrollment and exit dates, characteristics, and service enrollment and exit dates for all RPG participants in a web-based log. Frontline staff or their designees will document actual service delivery (such as length and content of sessions) for all RPG participants who are enrolled in any of the 10 focal EBPs.
Staff survey. All frontline staff who provide direct services to children, adults, and families for the 10 focal EBPs will be asked to complete a web-based survey. “Frontline staff” are those who work directly with any member(s) of the RPG case, and may include therapists, social workers, home visitors, or similar staff. We estimate that an average of 20 staff members will fit this criterion for each of the 17 RPG grantees.
Partner survey. Lead staff of the RPG grantee and RPG partner organizations will be asked to complete a web-based survey. Partner organizations are defined as organizations other than the grantee that provide RPG services to families enrolled in the program and/or coordinate their services with the grantee. We estimate that up to 20 individuals will fit this criterion for each of the 17 RPG grantees.
The outcomes study will include all 17 grantees. In this ICR, clearance is sought for one instrument associated with the outcomes study component of the cross-site evaluation:
Outcomes study master instrument. This includes standardized instruments CB has selected for use by the grantees to measure selected outcomes, a household roster, and specifications for administrative data elements grantees will obtain for additional outcome measures. Outcome data on child well-being, safety, and permanency will be collected on one focal child in each participating family, by the child’s primary caregiver. Each grantee will select a focal child at enrollment based on their target populations and planned services (for example, some grantees plan to serve families with infants or toddlers, while others plan to serve adolescents or teens.) Grantees will administer instruments that collect data on family functioning/stability to the focal child’s primary caregiver, if the child has not been removed from the home at RPG enrollment. If the child has been removed from the home, the instruments will be administered to the adult in the child’s original home who served as the child’s primary caregiver prior to removal. Instruments that collect data on adult recovery from substance abuse will also be administered to that adult, unless she or he is not receiving RPG services but a separate adult is receiving RPG services; in that situation, recovery data will be collected for that separate adult who is receiving RPG services.
The impact study will include seven grantees that are implementing rigorous comparison local evaluation designs and can provide outcome data on both treatment and comparison group members. In this ICR, clearance is sought to collect a subset of elements contained in the outcome master study instrument from families in the comparison groups.
Impact study master instrument. This includes 4 of the 10 standardized instruments from the outcomes study master outcome instrument, along with the household roster and specifications for all of the administrative data elements contained in the outcomes study master outcome instrument. These data will be collected on individuals in the comparison groups of the seven grantees that will participate in the impact study.
In this ICR, clearance is sought for the five instruments associated with the implementation and partnership study. The data collection procedures are described below:
Grantee and partner staff interview topic guide (Instrument #1). Two members of the RPG cross-site evaluation team will conduct two site visits to each grantee (one in year 2 of the grant program, the other in year 4). While on-site, they will conduct in-person interviews with grantee and partner staff. The interviews will be with individuals or small groups, depending on staff role. Evaluators will obtain written consent from each interviewee, including permission to audio record the interviews for later transcription. One team member will moderate the interview. If interviewees do not consent to auodiotaping, the second team member will use a laptop computer to take detailed notes.
Semi-annual progress reports (Instrument #2). CB will provide a template for the semi-annual progress reports, which will be submitted every six months. Narrative information can be entered directly into the template, or grantees can respond to the questions in other electronic file formats of their choosing. Grantee project directors will submit their reports to www.GrantSolutions.com.
Enrollment and service logs (Instrument #3). Intake workers will enter demographic characteristics, RPG enrollment and exit dates, and EBP enrollment and exit dates for each RPG case into this web-based log. Staff working on focal EBPs will enter individual service contact information on a weekly basis for the duration of participation in each specific focal EBP.
Staff survey (Instrument #4). This survey is web-based. The cross-site evaluation will obtain contact information for desired respondents from the RPG grantees and send advance notification of the survey with an email. Personalized links to the survey (along with both an email address and telephone number where any questions about the survey can be addressed) will then be distributed to each respondent via email. By clicking on the link or pasting it into their browser, they will be taken to the 30-minute survey. If they are unable to complete the survey in one sitting, they can return to the link as needed.
Partner survey (Instrument #5). This survey is web-based. The cross-site evaluation will obtain contact information for desired respondents from the RPG grantees and send advance notification of the survey with an email. Personalized links to the survey (along with both an email address and telephone number where any questions about the survey can be addressed) will then be distributed to each respondent via email. By clicking on the link or pasting it into their browser, they will be taken to the 30-minute survey. If they are unable to complete the survey in one sitting, they can return to the link as needed.
Information for the implementation and partnership study will be descriptive. In general, it will not involve formal hypothesis testing.
Outcomes study master instrument (Instrument #6). Each grantee is expected to maintain outcome data from the case-specific standardized instruments, a household roster and administrative records for all RPG participants in their project or agency database(s). Grantees will upload these data to the RPG Data Portal every six months, using file formats specified or provided by the cross-site evaluation. To maximize data quality, automatic data validation checks will occur during the upload and error messages will indicate any corrections needed before the submission can be accepted.
Information for the outcomes study will be descriptive. In general, it will not involve formal hypothesis testing.
Impact study master instrument (Instrument #7). Each of the seven grantees participating in the impact study is expected to maintain the case-specific outcome data for comparison group members from standardized instruments, a household roster, and administrative records in their project or agency database(s). They will upload these data to a data portal developed and maintained by the cross-site evaluation every six months using file formats specified or provided by the cross-site evaluation. To maximize data quality, automatic data validation checks will occur during the upload and error messages will indicate any corrections needed before the submission can be accepted.
Based on prior experience with similar grant projects and data collection efforts,the RPG Cross-Site Evaluation expects to obtain a high response rate of 80 percent or more for all instruments used. The grantee liaison will serve as a link to work with RPG grantees, if needed, to address nonresponse. Strategies for maximizing response in the data collection efforts are described below.
Conduct interviews with key grantee staff, administrators, supervisors, and staff during site visits. All interviews conducted with key grantee administrators, supervisors, and staff will occur during site visits. It is anticipated that all grantees will agree to participate in these visits. Our experience on many similar studies involving grantees indicates that participation rates of the desired interviewees are typically close to 100 percent. To help ensure high participation, we will coordinate with the grantees, supervisors, and staff to determine convenient dates and schedules for these visits.
Design staff and partner survey in a manner that minimizes respondent burden. To minimize burden on respondents, the surveys are brief, web-based, and structured such that respondents do not have to pay attention to routing and skip logic or view questions that do not apply to them.
Send advance and reminder emails to respondents. We will send advance emails to grantee and partner staff requesting their participation. If respondents have not completed the survey within a certain amount of time, we will send reminder emails requesting them to complete the surveys.
Solicit the help of grantees to encourage completion of the staff and partner surveys. If response rates for individual grantees lag, the cross-site evaluation team will work with lead grantee staff to identify additional strategies for increasing receipt of completed surveys without compromising respondent confidentiality. For instance, lead grantee staff may be asked to send an email to all the survey participants they had identified in their site, encouraging everyone’s response. This should assist with response rates, since lead grantee staff will have personal relationships with their staff and partners and can use their close proximity to encourage responses. This approach of combining follow-up requests from the evaluator to individuals who have not completed the survey with general requests from the grantee to all desired respondees has proved effective in multiple Mathematica projects that involved collecting similar data through web-based surveys, including in prior surveys performed for the CB.
Conduct telephone follow-up with non-respondents on the staff and partner survey. If email reminders and requests from the grantee prove ineffective, the cross-site evaluation team will deploy survey staff with expertise in obtaining responses to conduct one round of telephone follow-up with non-respondents. This approach of following up via telephone when email requests have not been effective has increased response rates in multiple Mathematica projects that involved collecting similar data through web-based surveys, including in prior surveys performed for the CB.
Provide an easy-to-use enrollment and services log. The design of the enrollment and service log component of the portal is based on web-based case management systems that Mathematica has developed and successfully implemented for multiple projects that collect these data from similar types of providers. The log can be accessed from any computer, allowing for ease of entry, while the data are housed on secure servers behind the cross-site evaluation contractor’s firewall, thereby maintaining data security.
Use multiple sources to check enrollment activity and completion of the service log. Information on the number of people enrolled in the RPG program every six months will be obtained in the semi-annual progress reports, and if the number does not reconcile with the number of new entries to the enrollment and service log, the cross-site evaluation will contact the grantee to reconcile the numbers and request that they add any missing enrollees to the log.
Use automatic prompts to enter service data. Once an RPG participant is enrolled in a focal EBP, the system will automatically prompt provider staff responsible for entering service data each week. If no services were provided during the week, provider staff members must so indicate.
Conduct regular data completion and quality checks. The cross-site evaluation contractor will examine each grantee’s enrollment and service log data at monthly intervals to identify any potential problems. If problems appear, contractor staff will notify the grantee and work with the grantee and providers as needed to obtain missing data or remedy other potential problems on a timely basis.
Design the master outcome and impact instruments in a manner that reduces burden. The outcome data that grantees are required to report comprises standardized instruments that often ask for similar information, such as demographic information about the respondent. To avoid such duplication, the master outcome instrument will exclude redundant items such that there is no duplication of items. This will reduce burden on grantee staff responsible for uploading this data to the RPG portal.
Develop a user-friendly, flexible upload process that has already been used successfully. The RPG Data Portal, which grantees will use to upload data, will provide easy access while maintaining the security of outcome data. The data portal has been designed with access by grantee staff in mind and is based on successful experience during prior studies collecting similar types of data from similar types of service providers. The outcome data management portion of the system, where grantees will upload data from the outcome instrument, is modeled on the data reporting system that was used for the first round of RPG grants from 2007 through 2011 (RPG1). Ten of the 17 RPG grantees also received RPG1 grants and are already familiar with using this type of system and have done so successfully. The outcome data management system will use updated features and improved technology to simplify the upload process compared to the former RPG1 system.
Provide training and technical assistance to grantee staff. We will provide documentation, training and technical assistance to grantees in collecting data from participants, uploading data to the RPG data portal, and using the web-based enrollment and service log.
Include data quality checks in the data portal. The data portal will also ensure data reliability by instituting automatic data quality checks. For example, if grantee staff enter out-of-range values in a particular field, the system will prompt users to check the value. For some fields, response values will be restricted; for others, grantee site staff will be able to override the check. We will also monitor the data entered by grantee sites and provide feedback to grantees on their data quality.
Optimizing the frequency of data collection. Our data collection strategy is designed to ensure the collection of accurate data. Grantees will upload outcome data once every six months, rather than waiting until their evaluation data collection is complete. This will enable the cross-site evaluation to identify and troubleshoot problems grantees experience in collecting data from respondents or uploading data.
Most of the instruments to be used in the RPG Cross-Site Evaluation build on existing measures and previous experience from other studies completed by the cross-site evaluation team.
Grantee and partner staff interview topic guide. The RPG grantee and partner staff interview topic guide has been modeled after interview guides used in similar studies such as the Early Head Start Enhanced Home Visiting Pilot Evaluation and the Evidence-Based Home Visiting (EBHV) cross-site evaluation. All site visitors will receive training in use of the topic guides. After the first site visit has been completed, the cross-site evaluation team will meet to discuss the use of the instruments in order to make any needed modifications to enhance data quality and reduce burden such as by eliminating or refining any questions that were unnecessary or redundant.
Staff and partner surveys. The staff survey adapted items used previously in other Mathematica projects including the EBHV cross-site evaluation, the Child Support Noncustodial Parent Employment Demonstration (CSPED), and Parents and Children Together (PACT). It also included several standardized scales. The partner survey was modeled after the ones used on the EBHV cross-site evaluation, the Integration Initiative cross-site survey, and the survey instrument (Collaborative Capacity Inventory) used in RPG1.
In September and October 2013, Mathematica conducted pretests of the surveys to assess the clarity and completeness of content and to estimate respondent burden. Three RPG grantees provided contact information for front-line staff and/or representatives from organizations in the RPG partnership, and Mathematica then recruited pretest participants. Each participant completed and returned (via email or fax) a hard-copy version of the questionnaire and provided feedback during a 30-minute telephone debriefing interview.
Six front-line staff from two grantees pretested the staff survey. On average, it took them 28 minutes to complete. Participants agreed the survey was comprehensive and generally easy to complete, and recommended a few changes to facilitate their understanding of the questions and survey goals. Based on pretest participant feedback Mathematica made the four substantive changes summarized below, along with other minor revisions. Burden is estimated to be 25 minutes on the final version of the survey.
The job titles in question A1 were revised. One pretest participant said that “family advocate” better captures her role, and another did not think that “child protective investigator” would be selected by any survey respondents. As a result, the former was added as a response option and the latter was deleted.
Pretest participants found question B3 to be very long. After consulting with the scale author, Phyllis Panzano, Mathematica replaced the 33-item scale with an abbreviated, 9-item version.
Questions B4, B5, and B6 will be asked only of supervisors. Several front-line staff said they did not know whether the evidence-based program (EBP) of interest was adapted, how it was adapted, or the reasons for the adaptations. They indicated they asked a supervisor to review their responses to these questions for accuracy. The grantee request to identify staff survey participants was similarly revised so that supervisors will be identified a priori.
Pretest participants wondered why the survey focused on one specific EBP, as some reported they work on multiple programs. The introduction text at the beginning of the survey and the frequently asked questions document were revised to explain that the evaluation focuses on specific EBPs and not every EBP a grantee may use.
Five representatives from partner agencies affiliated with three RPG grantees pretested the partner survey. On average, it took them approximately 20 minutes to complete it. As with the staff survey, partner survey pretest participants felt the survey was generally easy to understand and complete. Burden is estimated to be 20 minutes on the final survey. The major revisions are described below.
Several pretest participants had difficulty answering question 7 about the dollar amount of in-kind resources. Although the definition of “in-kind resources” was clear to them, they either felt it would require considerable effort to calculate an amount or did not have the information necessary to formulate an answer. The revised question now assesses the types of in-kind resources rather than their total value.
One participant interpreted question 8 to be factual in nature rather than subjective. The revised question is completely open-ended instead of requesting respondents to list up to three goals of the partnership.
Respondents indicated that collaborations in the partnership occurred at multiple organizational levels. They used different strategies to provide a single response to question 12. In order to encourage respondents to answer this question in a consistent fashion, the number of response categories was reduced to two and respondents are now instructed to answer based on the most common levels of collaboration.
At least one pretest participant had more difficulty answering the questions because in her advisory role she had less involvement in providing services to clients through RPG partnership. The grantee request to identify partner survey participants was revised to clarify the types of individuals who would be the most appropriate as survey respondents.
Prior to deployment of the staff and partner surveys on the web, Mathematica’s staff will conduct rigorous testing to ensure the accuracy of the web surveys compared to the hard-copy instruments. Web testers will check question wording and response option formats, and will use different scenarios to test the skip patterns.
RPG data portal. The outcome data management component of the RPG data portal is modeled after RPG1 and the service log component is modeled after service logs that we have built for other projects such as CSPED and PACT. All functions of the RPG data portal will be rigorously tested and evaluated by the development team to ensure proper functionality. Additionally, we will consult with grantees on the usability of the system and engage grantees in the testing phase.
Preliminary input on statistical methods was received from staff at Mathematica and Walter R. McDonald & Associates, and experts at other organizations, including:
Dr. Sarah Avellar
Mathematica Policy Research
1100 1st Street, NE, 12th Floor
Washington, DC 20002
Dr. Russell Cole
Mathematica Policy Research
P.O. Box 2393
Princeton, NJ 08543
Dr. John Deke
Mathematica Policy Research
P.O. Box 2393
Princeton, NJ 08543
Dr. Ying-Ying T. Yuan
Walter R. McDonald & Associates
12300 Twinbrook Parkway, Suite 310
Rockville, MD 20852
Dr. Allison Metz
University of North Carolina
Sheryl-Mar South, Room 142
Campus Box 8185
Chapel Hill, NC 27599-8185
Dr. Joseph Ryan
University of Michigan
Center for Political Studies
1080 South University Avenue
Ann Arbor, MI 48109-1106
Dr. Cheryl Smithgall
Chapin Hall at the University of Chicago
1313 East 60th Street
Chicago, IL 60637
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Autumn Parker |
| File Modified | 0000-00-00 |
| File Created | 2021-01-25 |