MRS Part B

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B1. Respondent Universe and Response Rates

The evaluation of Child Welfare Information Gateway is designed to reach the various types of customers using its services such as professionals, students, and customers with a personal situation or concern. Data will be collected by delivering surveys to customers receiving information services via the Web, e-mail, Live Chat, print, and telephone. Web surveys are scheduled to appear during randomly selected one-hour time slots throughout the year. Telephone, e-mail, and live chat customers are offered the survey on 52 randomly selected business days throughout the year. All versions of the Market Research Sub-Study Information Habits and Preferences Survey will be emailed directly to potential respondents, through listservs of partner agencies, organizations and associations. The universe of respondents for the evaluation of Child Welfare Information Gateway and the Market Research Sub-Study is up to 10,920.

Descriptions of the respondents to each instrument and calculation of the estimated response rates is shown in Exhibit B-1. Other details are provided by instrument following Exhibit B-1.

B-1. Calculation of Estimated Response Rates

Instrument	Respondent	# Respondents/ # Sampled	Expected Response Rate (%)
Child Welfare Information Gateway’s Targeted Survey	CW professionals (e.g., frontline staff, supervisors/directors)	2,660/ 88,667	3%
Child Welfare Information Gateway’s Event Survey	CW professionals (e.g., frontline staff, supervisors/directors)	900/ 3,000	30%
Child Welfare Information Gateway’s Focus Group Guide	CW professionals (e.g., frontline staff, supervisors/directors)	50/ 50	100%
Child Welfare Information Gateway’s General Customer Survey: Questions for Professionals	CW professionals (e.g., frontline staff, supervisors/directors)	960/ 32,000	3%
Child Welfare Information Gateway’s General Customer Survey: Questions for Students	undergraduate and graduate students	480/ 16,000	3%
Child Welfare Information Gateway’s General Customer Survey: Questions for Personal Customers	adopted person, foster/adoptive parent, prospective adoptive parent, birth parent, legal parent/guardian, general public	960/ 32,000	3%
Market Research Sub-Study: Information Habits and Preferences Survey (for child welfare professionals in state, county, and private agencies)	CW professionals in state, county, and private agencies	1,800/ 6,000	30%
Market Research Sub-Study: Information Habits and Preferences Survey (for child welfare professionals working with tribes)	CW professionals working with tribes	800/ 3,200	25%
Market Research Sub-Study: Information Habits and Preferences Survey (for legal professionals working in child welfare)	CW legal professionals	1,400/ 7,000	20%
Market Research Sub-Study: Information Habits and Preferences Survey (for students planning to enter the child welfare workforce)	CW undergraduate and graduate students	810/ 1,800	45%
Market Research Sub-Study: Focus Groups on Information Habits and Preferences	CW professionals in state, county, and private agencies	100/ 100	100%
TOTAL Sample of Respondents	10,920

Targeted Surveys will be delivered online for ten unique website sections (e.g., National Child Abuse Prevention Month, National Adoption Month, National Foster Care Month, and seven other website sections) that receive a high volume of website traffic (over 20,000 visits per quarter). A sample size of 266 for each survey will be desired to reach a 5% margin of error and 90% confidence level, which adds up to a total of 2,600 surveys for all ten website sections. Drawing upon our experiences using similar survey delivery methods (e.g., pop-up surveys) as ones that were used for other OMB approved data collection activities in FY 2017, we anticipate a response rate of 3%. To reach the desired number of respondents (2,660), we will send approximately 88,667 targeted survey invitations.

Event surveys will be delivered after three webinars and seven special CWIG presentations each year to gather feedback on the quality of information presented. We anticipate to have approximately 300 participants for each event, which adds up to a total of 3,000 participants for 10 events. Given an approximately 30% response rate, which is an estimate based on the similar data collection activities conducted in FY2017, we expect to receive about 900 completed Event Surveys each year.

Child Welfare Information Gateway will conduct five Focus Groups each year with a purposive sample of ten respondents per group for a total annual sample size of 50 respondents each year. Focus groups will be conducted with both known users and known non-users of Child Welfare Information Gateway as identified by responses to questions on the Market Research Sub-Study Information Habits and Preferences Survey (the survey will also request survey takers interested in participating in focus groups to provide their name and contact information).

Sample size estimates for Child Welfare Information Gateway’s General Customer Survey: Questions for Professionals (delivered via Web, phone, email, live chat, and print) are based on similar methods of data collection used in FY 2017. Sample sizes were developed to ensure that estimated percentages have required levels of precision. Specifically, estimated percentages for the professional subgroup should have a margin of error of 3% or less; i.e., 95% confidence intervals are within +/- 3%. We expect that that approximately 32,000 invitations to participate in Child Welfare Information Gateway’s General Customer Survey: Questions for Professionals will be made annually, with a 3% response rate, resulting in 960 completed surveys.

Sample size estimates for Child Welfare Information Gateway’s General Customer Survey: Questions for Students (delivered via Web, phone, email, live chat, and print) are based on similar methods of data collection used in FY 2017. Sample sizes were developed to ensure that estimated percentages have required levels of precision. Specifically, estimated percentages for the student subgroup should have a margin of error of 3% or less; i.e., 95% confidence intervals are within +/- 3%. We expect that that approximately 16,000 invitations to participate in Child Welfare Information Gateway’s General Customer Survey: Questions for Professionals will be made annually, with a 3% response rate, resulting in 480 completed surveys.

Sample size estimates for Child Welfare Information Gateway’s General Customer Survey: Questions for Personal Customers Students (delivered via Web, phone, email, live chat, and print) are based on similar methods of data collection used in FY 2017. Sample sizes were developed to ensure that estimated percentages have required levels of precision. Specifically, estimated percentages for the personal customer subgroup should have a margin of error of 3% or less; i.e., 95% confidence intervals are within +/- 3%. We expect that that approximately 32,000 invitations to participate in Child Welfare Information Gateway’s General Customer Survey: Questions for Professionals will be made annually, with a 3% response rate, resulting in 960 completed surveys.

Sample size estimates for the Market Research Sub-Study survey for CW professionals are based on background research about this respondent audience. We expect that that approximately 6,000 invitations to participate in the Market Research Sub-Study survey for CW professionals will be sent, with a 30% response rate, resulting in 1,800 completed surveys. Our estimated response rate is based on similar research that we have conducted in the past, as well as feedback collected from other researchers.

Sample size estimates for the Market Research Sub-Study survey for child welfare professionals working with Tribes are based on background research about this respondent audience. We expect that that approximately 3,200 invitations to participate in the Market Research Sub-Study survey for child welfare professionals working with Tribes will be sent, with a 25% response rate, resulting in 800 completed surveys. Our estimated response rate is based on similar research that we have conducted in the past, as well as feedback collected from other researchers.

Sample size estimates for the Market Research Sub-Study survey for CW legal professionals are based on background research about this respondent audience. We expect that that approximately 7,000 invitations to participate in the Market Research Sub-Study survey for CW legal professionals will be sent, with a 20% response rate, resulting in 1,400 completed surveys. Our estimated response rate is based on similar research that we have conducted in the past, as well as feedback collected from other researchers.

Sample size estimates for the Market Research Sub-Study survey for students planning to enter the child welfare workforce are based on background research about this respondent audience. We expect that that approximately 1,800 invitations to participate in the Market Research Sub-Study survey for students planning to enter the child welfare workforce will be sent, with a 45% response rate, resulting in 810 completed surveys. Our estimated response rate is based on similar research that we have conducted in the past, as well as feedback collected from other researchers.

The Market Research Sub-Study intends to conduct 10 Focus Groups with a purposive sample of 10 respondents per group for a total annual sample size of 100 respondents each year. Focus groups will be conducted to provide more context to the online survey data and better understand survey respondents’ information habits and preferences.

B2. Procedures for the Collection of Information

The specific procedures utilized for each data collection instrument are described below.

Child Welfare Information Gateway’s Targeted Survey. The goal of the targeted survey is to gather specific and actionable feedback on the quality of information and usability for specific Child Welfare Information Gateway website sections. The target population for targeted surveys includes child welfare professionals. Targeted surveys are voluntary and will be offered using a website “pop-up” methodology. The “pop-up” survey will be offered after a specified number of clicks (determined by the average page views for a particular website section). Once a user receives a “pop-up” survey invitation they will have the option to click “yes”, “no” or “ask me later”. (Appendix A).

Child Welfare Information Gateway’s Event Survey. The goal of Child Welfare Information Gateway’s Event survey is to assess customer’s level of satisfaction and intended use of information obtained from webinars and other presentations hosted by Child Welfare Information Gateway. The Event Surveys will be offered online at the conclusion of webinars or via hard copy at the conclusion of Information Gateway conference presentations. The target population for the Event Survey includes child welfare professionals. (Appendix B).

Child Welfare Information Gateway’s Focus Groups. Child Welfare Information Gateway will conduct five Focus Groups each year with a purposive sample of ten respondents per group for a total annual sample size of 50 respondents each year. Focus groups will be conducted with both known users and known non-users of Child Welfare Information Gateway as identified by responses to questions on the Market Research Sub-Study Information Habits and Preferences Survey (the survey will also request survey takers interested in participating in focus groups to provide their name and contact information). Audio recordings from focus groups will be transcribed and analyzed using Atlas.ti qualitative analysis software. All audio recordings will be destroyed once all transcriptions have been thematically coded and analyzed. (Appendix C).

Child Welfare Information Gateway’s General Customer Survey: Questions for Professionals, Students, and Personal Customers. The goal of Child Welfare Information Gateway’s General Customer Survey (e.g., including tailored questions for professionals, students, and personal customers) is to assess customer characteristics; the kind and quality of services customers are looking for; customer’s level of satisfaction with those services; and customers’ access to, use of, and need for Child Welfare Information Gateway services. The target population includes all customers (e.g., CW professionals, related professionals, undergraduate and graduate students, adopted persons, foster/adoptive parents, prospective adoptive parent, birth parent, legal parent/guardian, and the general public). The General Customer Survey will be offered using five different methods of delivery detailed below. (Appendix D).

• Pop-Up website survey delivery: Monday through Friday between 8:00 a.m. to 9:00 p.m. EST, website visitors will receive an invitation to complete an electronic survey after clicking on the website five times. The visitor will have the option to decline the survey, have the survey offered at a later time, or answer the survey.

• E-mail survey delivery: On all workdays throughout the year, customers who e-mail Child Welfare Information Gateway will be offered the opportunity to complete a customer survey. These customers will receive an invitation to complete the survey using a hyper-link in the response e-mail to their information request.

• Telephone survey delivery: On 52 randomly selected workdays throughout the year, customers who call Child Welfare Information Gateway will be offered the opportunity to complete a customer survey over the telephone. If they agree to complete the survey, they will be transferred to a separate phone surveyor who will read them the survey and record their answers.

• Mailed survey delivery: On 52 randomly selected workdays throughout the year, customers who call or e-mail Child Welfare Information Gateway to order publications or who order publications online will be offered the opportunity to complete a customer survey by mail. They will be sent a print version of the customer survey along with their requested publication and a pre-paid business reply envelope.

• Live Chat survey delivery: On all workdays (not including Federal government holidays) throughout the year, customers that solicit assistance via live chat will be offered the opportunity to complete a customer survey. They will be sent a hyperlink to complete the survey online at the conclusion of the live chat conversation.

Market Research Sub-Study: Information Habits and Preferences Survey (for child welfare professionals in state, county, and private agencies). The goal of the Market Research Sub-Study survey is to learn more about how child welfare professionals and students planning to enter the child welfare workforce access and consume work-related information. The target population for this version of the survey instrument is public and private child welfare professionals that work in six different states, as well as professionals that work for a large private company. The Market Research Sub-Study instrument will be emailed to potential respondents through distribution lists provided by the states and private company that are recruited for the study. The states and private company have not yet been determined, because they will depend on successful recruitment once OMB approval is finalized. (Appendix E).

Market Research Sub-Study: Information Habits and Preferences Survey (for child welfare professionals working with tribes). The target population for this version of the survey instrument is child welfare professionals that work with tribal populations. The Market Research Sub-Study instrument will be emailed to potential respondents through distribution lists provided by the tribes that are recruited for the study. These tribes have not yet been determined, because they will depend on successful recruitment once OMB approval is finalized. (Appendix F).

Market Research Sub-Study: Information Habits and Preferences Survey (for legal professionals working in child welfare). The target population for this version of the survey instrument is judges and attorneys that spend at least 10% of their professional work time on cases related to child welfare, as well as Court Improvement Program coordinators. The Market Research Sub-Study instrument will be emailed to potential respondents through distribution lists provided by organizations and professional associations that partner with the Gateway on this study. (Appendix G).

Market Research Sub-Study: Information Habits and Preferences Survey (for students planning to enter the child welfare workforce). The target population for this version of the survey instrument is current Bachelors and Masters of Social Work students that receive Title IV child welfare stipends. The Market Research Sub-Study instrument will be emailed to potential respondents through distribution lists provided by universities and other organizations that partner with the Gateway on this study. (Appendix H).

Market Research Sub-Study: Focus Groups on Information Habits and Preferences. Child Welfare Information Gateway will conduct ten Focus Groups each year with a purposive sample of ten respondents per group for a total annual sample size of 100 respondents each year. Focus groups will be conducted with respondents to the Market Research Sub-Study survey, in order to provide more context to the online survey data and better understand survey respondents’ information habits and preferences. Audio recordings from focus groups will be transcribed and analyzed using Atlas.ti qualitative analysis software. All audio recordings will be destroyed once all transcriptions have been thematically coded and analyzed. (Appendix I).

B3. Maximizing Response Rates and the Issue of Non-Response

Child Welfare Information Gateway has taken steps to maximize response rates. Specifically, we have made the instruments short and clear, as we understand that customers are unlikely to be willing to spend more than a few minutes on the survey. Customers receiving requests online to take Targeted Surveys and General Customer Surveys will be given an “Ask me later” option that he or she may use in order to complete the survey at a more convenient time. Finally, the random selection of survey time blocks for the General Customer Survey was weighted to increase the representation of business hours in order to increase the number of professional survey respondents.

For the Market Research Sub-Study instruments, other strategies will be used to maximize response rates including (a) sending periodic reminders to potential respondents; and (b) working with other professional organizations and associations that can encourage participation and help explain the value of the survey.

B4. Pretesting of Procedures or Methods

All instruments for this data collection effort were tested for usability by Child Welfare Information Gateway staff.

B5. Contact Information

Should you have any questions about the contents of this OMB submission package, please contact one of the following individuals:

• Brian Deakins, Children’s Bureau, 330 C Street, SW, Washington, DC 20201, telephone: 202.205.8769, email: Brian.Deakins@acf.hhs.gov

• Christine Leicht, ICF, 9300 Lee Highway Fairfax, VA 22031, telephone: 703.934.3000, email: Christine.Leicht@icf.com

List of Appendices

Appendix A: Child Welfare Information Gateway’s Targeted Survey

Appendix B: Child Welfare Information Gateway’s Event Survey

Appendix C: Child Welfare Information Gateway’s Focus Group Guide

Appendix D: Child Welfare Information Gateway’s General Customer Survey: Questions for Professionals; Students; and Personal Customers

Appendix E: Market Research Sub-Study: Information Habits and Preferences Survey (for child welfare professionals in state, county, and private agencies)

Appendix F: Market Research Sub-Study: Information Habits and Preferences Survey (for child welfare professionals working with tribes)

Appendix G: Market Research Sub-Study: Information Habits and Preferences Survey (for legal professionals working in child welfare)

Appendix H: Market Research Sub-Study: Information Habits and Preferences Survey (for students planning to enter the child welfare workforce)

Appendix I: Market Research Sub-Study: Focus Groups on Information Habits and Preferences

7