B. Collections of Information Employing Statistical Methods
The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When item 16 is checked "Yes," the following documentation should be included in the supporting statement to the extent that it applies to the methods proposed:
Respondent Universe and Sampling Methods
The universe of respondents is up to 351 representatives from 17 grantee organizations. Attachment G – Survey Respondents provides additional detail on how these numbers were calculated.
The grantee organizations have been funded as demonstration sites to develop and implement three types of child welfare projects, with seven grantees operating Family-finding and Family Group Decision-Making projects, five grantees operating Kinship Navigator-TANF projects, and five grantees operating Residential Family Treatment projects. Surveys have been developed for five respondent groups, represented by a cross-section of project leaders, service providers, local evaluators, and public child welfare and community partners.
Survey respondents will be a combination of key informants and a convenience sample, documented in Table 1. The following groups will be comprised of key informants known to the Children’s Bureau and the contractor. These groups include:
Project Leaders will consist of up to 79 grantee organization representatives (4 to 5 per grantee) from the following groups: project director, executive leadership (e.g., President, CEO), and other project leadership (e.g., Program or Project Manager, service provider Supervisor).
Evaluator Team members will consist of up to 51 lead evaluators/principal investigators and other evaluation team members (3 per grantee) providing local evaluation services to the grantee for the Family Connection project.
Public Child Welfare Partner representatives will be up to 34 representatives (2 per grantee) from one to two public child welfare agencies.
Community Partner representatives will be up to 34 representatives (2 per grantee) from one to two community organizations.
The contractor will work with the Family Connection project director and/or lead evaluator to identify a convenience sample of the following group. The contractor will establish an eligibility criteria of at least one year providing Family Connection services. Service Providers will be up to 153 representatives (9 per grantee) providing direct services to clients. Service providers may include, but not be limited to advocate, case manager, case worker, educator, facilitator, and mentor.
Family Connection grantee organizations are required to participate in cross-site evaluation activities, including data collection. The expected response rate is 90 to 100%. The contractor has developed relationships with project leaders, evaluation team members, and key service providers since they were funded in Fall 2012. The contractor has also met several public child welfare and community partners through grantee meetings and peer-to-peer networking events. Response rates of 100% are expected from project leaders, evaluation team members, and public child welfare and community partners. Response rates may be less from service providers due to potential turnover issues. While 100% response rate cannot be guaranteed, the expectation from CB that grantee representatives participate, the value of the survey, and availability of results to grantees are incentives for broad participation.
A similar collection was conducted in 2010-2011, 2011-2012, and 2013-2014 with the same type of cross-representation as described above via in-person and telephone discussions. Service providers tended to be a less reliable group in regard to availability and participation in discussions. All identified project leaders, evaluation team members, and public child welfare and community partners participated in earlier data collection efforts. The exceptions were representatives that left their positions prior to data collection. In those cases, the contractor made every effort to schedule the site visit or conduct the telephone interview prior to the participant’s departure.
Procedures for the Collection of Information
Survey respondents will be a combination of key informants and a convenience sample. (See Attachment G - Table 1. Survey Respondents). Key informants will be identified for the respondent groups of project leadership, evaluation team members, public child welfare partners, and community partners. Most identification has already occurred through the evaluation contractor’s work with each grantee organization since the beginning of their Federal funding cycle in the fall of 2012; the evaluation contractor will verify the list of key informants with each grantee. The majority of key informants will have been involved with their respective Family Connection projects since initial funding in the fall of 2012, and many will have been instrumental in developing the project’s application for Federal funding.
A convenience sample of service providers will be generated by the grantee organization’s project director or other leadership representative. Service providers will have provided Family Connection project services for at least one year in order to be included in the convenience sample. Should a grantee organization provide Family Connection services in more than one geographic site, service providers from each site will be chosen.
The combination of key informants and convenience sample is the most appropriate method to generate the five respondent groups as they will be asked detailed questions about the Family Connection project that cannot be answered from other existing data sources. These questions will address topics such as project planning, implementation and modifications; referral process, service flow, and service provision; collaboration with project partners, including the public child welfare agency and evaluation team; trends and effects of service use; project strengths and challenges; sustainability strategies; and evaluation processes and results. Survey results will be used to address key process evaluation questions that have been informed by implementation science research1, findings from 2009 and 2011-funded Family Connection projects, and key questions from the Children’s Bureau.
Key questions address intervention and innovation, implementation drivers, and influence factors. (See Attachment H – Cross-Site Process Evaluation Questions).
As this is an electronic survey, advance appointments will not be made. The contractor will disseminate a standard communication to all 17 grantee organization’s project directors and evaluation team members describing the purpose and process of the survey (See Attachment I - Communication 1). The evaluation contractor will follow up with each grantee organization individually to verify the key informant sample and generate the convenience sample of service providers. An additional communication to all survey respondents from the contractor with cc: to the project director, and the Children’s Bureau will be made prior to survey dissemination (See Attachment J - Communication 2).
Quality control procedures will be implemented by the contractor in regard to the following: 1) Accuracy and completeness of survey distribution lists, 2) Accuracy and completeness of electronic survey programming, including internal testing of all versions of electronic surveys, 3) Monitoring response rates and completeness of returned survey data, 4) Reminders to non-responders, 5) Quantitative analysis by respondent group, project group, grantee organizations, and other appropriate statistical procedures, 5) Completeness and accuracy of coding processes for qualitative responses to open-ended survey questions, and 6) Completeness and accuracy in all reporting in consideration of respondent confidentiality.
The evaluation contractor will not conduct processes to impute missing data.
Methods to Maximize Response Rates and Deal with Nonresponse
Response rates by respondent group, project group, and grantee will be calculated by dividing the number of surveys received by the number of surveys distributed. Family Connection grantee organizations are required to participate in cross-site evaluation activities, including data collection. The expected response rate is 90 to 100%. Response rates of 100% are expected from project leaders, evaluation team members, and public child welfare and community partners. Response rates may be less from service providers due to potential turnover issues.
As noted in B.1. Respondent Universe and Sampling Methods, all identified project leaders, evaluation team members, and public child welfare and community partners participated in earlier data collection efforts. The exceptions were representatives that left their positions prior to data collection. In those cases, the contractor made every effort to schedule the site visit or conduct the telephone interview prior to the participant’s departure. Of the five respondent groups, there were more challenges obtaining data from service providers, which was sometimes due to turnover in the organization or unavailability of the provider during the designated data collection time due to an unanticipated service commitment. We anticipate that the web-based, electronic survey, which can be done at the respondent’s convenience, will result in a relatively high response rate.
After communicating with all 17 grantee organization’s project directors and evaluation team members, and working with each grantee to generate the survey distribution list, the contractor will craft a communication to be distributed to all survey respondents. The communication will describe the purpose of the survey, survey dissemination processes, and steps to protect respondent confidentiality. The communication will be written in plain language and include contact information for the contractor should survey respondents have any questions. After initial survey dissemination, the evaluation contractor will send up to two reminders to grantee representatives who have not submitted a complete survey.
Test of Procedures or Methods to be Undertaken
There will be no tests of procedures or methods. The data collection instruments are based on semi-structured discussion protocols administered three times to a cross-section of project leaders, service providers, evaluation team members, public child welfare partners, and community partners from two earlier cohorts of Family Connection grantees. The current instruments were developed by the contractor, with response options informed by earlier codebooks used for qualitative analysis of discussion data. The contractor disseminated the survey to four Family Connection grantee staff representatives including a project leader and three evaluation team members (See Attachment D – Consultants). Overall feedback indicated that the questions were clear and understandable, and that response options were appropriate to each item. Grantee suggestions for improvement to style, comprehensiveness, and all other aspects of the survey, were thoroughly considered by the contractors with appropriate revisions to each instrument.
Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
The contractor was consulted on the statistical aspects of the design:
Dr. Jennifer Dewey; James Bell Associates, Inc.; (703) 247-2637
Additional consultation was provided by the following agency personnel:
Ms. Jan Shafer; Director, Division of Program Innovation; Children's Bureau/ACF/HHS; (202) 205-8172
Ms. Maria Woolverton; Office of Planning, Research and Administration/ACF/HHS; (202) 205-4039
The contractor will collect and analyze the information for the Children’s Bureau. The contractor has provided evaluation technical assistance to grantee organizations and conducted the cross-site evaluation of the Family Connection grant projects since the beginning of the project in the fall of 2009.
1 Fixsen, D., Naoom, S.F., Blasé, D.A., Friedman, R.M., and Wallace, F. (2005). Implementation Research: A synthesis of the literature. University of South Florida, Luis de la Parte Florida Mental Health Institute: The National Implementation Research Network (FMHI Publication #231).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Windows User |
| File Modified | 0000-00-00 |
| File Created | 2021-01-25 |