Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Supporting and Strengthening the Home Visiting Workforce (SAS-HV)
Pre-testing of Evaluation Data Collection Activities
0970 - 0355
Supporting Statement
Part B
August 2022
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officers:
Nicole Denmark
Shirley Adelstein
Part B
B1. Objectives
Study Objectives
The purpose of the Supporting and Strengthening the Home Visiting Workforce (SAS-HV) project is to advance understanding of how to support and strengthen the early childhood home visiting workforce. One focal area of the project is the use of reflective supervision in early childhood home visiting. The Maternal Infant Early Childhood Home Visiting (MIECHV) legislation mandates federally funded home visiting programs maintain high-quality supervision practices, but relatively little is known about specific supervision practices used in the field.
A prior phase of this project developed a conceptual model of reflective supervision and reviewed current research, measures, and practice to identify gaps in knowledge. In the current phase of the research, the project seeks to address a key gap by developing and testing a measure of reflective supervision that is practice-relevant and useful for research. As a first step in the measure development process, the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) is seeking approval to identify what end users of the measure view as key elements of reflective supervision for the home visiting context.
Generalizability of Results
This study is intended to present internally valid descriptions of how practitioners and researchers rate elements of reflective supervision in terms of their importance and their relevance to home visiting. The study is not intended to promote statistical generalization to other sites or service populations.
Appropriateness of Study Design and Methods for Planned Uses
The design includes concept mapping, which will include a web-based survey followed by a group interpretation meeting. The survey will include a list of elements obtained through a literature review and initial feedback from consultants and practitioners completed in earlier project phases1. The use of a web-based survey asking participants to sort and rate each element in terms of importance and relevance to home visiting is appropriate to better understand practitioner and researcher perspectives on key elements of reflective supervision in home visiting. The use of a group interpretation meeting is appropriate because it allows the study team to share findings with participants and allow for deeper probing of the findings. A limitation of the study design is that web-based survey and group interpretation meeting will not yield a representative sample; however, it will allow us to learn from a subset of practitioners and researchers to inform measure development.
As noted in Supporting Statement (SSA), this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.
B2. Methods and Design
Target Population
We propose to gather information from home visiting practitioners and researchers. We aim to obtain variation in participant characteristics, roles, length of time working in home visiting, home visiting program models implemented, and type of families served to understand a range of perspectives in relation to reflective supervision. Inclusion criteria for participants includes at least one year of experience with reflective supervision in the home visiting context. The universe of home visiting practitioners and researchers is not known, although estimates suggest more than 19,000 practitioners2 and more than 240 researchers3. However, only a portion of these practitioners and researchers have experience with reflective supervision required for study participation. We will select up to 40 total participants (30 practitioners and 10 researchers).
This sample size is consistent with that of other studies that have used concept mapping for measure development.4 The sample will include individuals who respond to a recruitment announcement sent to the Home Visiting Applied Research Collaborative (Appendix 1), complete a screening questionnaire (Instrument 1), and who are eligible for participation based on inclusion criteria. The sample will also include researchers recruited via individualized and targeted recruitment efforts (Appendix 2). Because participants will be purposively selected, they will not be representative of the population of practitioners and researchers within the home visiting field. Target practitioners will include supervisors and home visitors. Target researchers will include individuals with experience researching, evaluating, or developing measures of reflective supervision in the home visiting context.
Respondent recruitment
We will recruit practitioners through the Home Visiting Applied Research Collaborative’s (HARC) Practice-Based Research Network (PBRN). The purpose of HARC’s PBRN is to provide a national network for conducting collaborative, field-initiated studies with local home visiting programs, regardless of the model of home visiting being used. The PBRN is a voluntary network of over 240 home visiting researchers and hundreds of local home visiting programs, including individual sites and state and local networks. If more than 40 individuals respond, we will select practitioner participants purposively with the goal of achieving diverse and balanced representation across the categories listed below:
Race and ethnicity
Geographic location
Home visiting model(s) implemented
Types of communities served (urban, suburban, rural, frontier)
Number of families served by home visiting program
Primary language of families served by the home visiting program
If more respond than we can accommodate, we may select participants based on the level of interest they express in the project, or we may randomly select participants within a given category.
We will recruit researchers by contacting those who have conducted research related to reflective supervision, including those who have developed or tested measures of reflective supervision. Researchers will be identified from existing lists of measures of reflective supervision and seminal articles and work on reflective supervision in the home visiting context. We will engage in targeted and individualized recruitment of selected researchers.
B3. Design of Data Collection Instruments
Development of Data Collection Instruments
Data collection includes: (1) a web-based survey and (2) a group interpretation meeting. The survey instrument was guided by a systematic review of existing elements of reflective supervision and informed by the Study Team’s knowledge of existing research and literature on reflective supervision. As noted in SSA, elements of reflective supervision included in the survey were refined in collaboration with our technical and practitioner workgroup members. Specifically, technical and practitioner workgroup members were asked to provide input on the clarity of the elements and their definitions in virtual meetings. Careful attention was given to avoid overlap or duplication across survey questions and minimize participant burden. The facilitation guide for the group interpretation meeting was developed by reviewing existing research and best practices for facilitating concept mapping group interpretation meetings. We plan to work with members of our internal advisory group with expertise in concept mapping to refine and finalize the group interpretation meeting facilitation guide once findings from the structuring and rating tasks are available.
B4. Collection of Data and Quality Control
The Study Team will collect survey and group interpretation meeting data from individuals that consent to participate (see Appendix 3 for web-based survey consent and Appendix 4 for group interpretation meeting consent). ACF has contracted with James Bell Associates and partners Johns Hopkin University and University of Colorado Anschutz Medical Campus to conduct the study. James Bell Associates will lead all data collection and analysis activities in collaboration with Johns Hopkin University and University of Colorado Anschutz Medical Campus.
Recruitment of practitioners into the study will involve outreach to potential participants through a Home Visiting Applied Research Collaborative recruitment announcement (Appendix 1). Recruitment of researchers into the study will involve individualized, direct outreach to selected individuals—as described above (Appendix 2). Interested practitioner participants will complete a screening questionnaire (Instrument 1). The Study Team will then send a link via email to selected participants to complete the web-based survey (Instrument 2). The survey will be programmed into GroupWisdom web-based software.
For the group interpretation meeting guide (Instrument 3), data will be collected via virtual Zoom meetings, which will be recorded if participants agree to recording. If a participant does not consent to being recorded, a member of the project team will take notes during the meeting.
To monitor surveys for quality and consistency, the Study Team will review reports on survey completion rates and item completion rates periodically during the data collection period.
Prior to data collection, the Study Team will conduct initial trainings with meeting facilitators to cover general facilitation guidelines, provide tips on conducting successful meetings, and review the facilitation guide. During data collection phase for group interpretation meetings, weekly check-in calls with facilitators will be held to monitor data interpretation meetings for quality and consistency.
B5. Response Rates and Potential Nonresponse Bias
Response Rates
Based on experience recruiting participants via the Home Visiting Applied Research Collaborative, we anticipate being able to successfully recruit the desired number of participants described in section B2: Respondent Recruitment. We assume a minimum of 60% of the individuals who are invited to participate will complete the web-based survey. To account for this response rate, the study team will identify 15 alternate participants in addition to the 40 individuals selected for participation to reach a total of 40 participants and reach out to alternates as necessary. The surveys are not designed to produce statistically generalizable findings and participation is wholly at the participant’s discretion. Response rates will not be calculated or reported.
Non-response
As participants will not be randomly sampled and findings are not intended to be representative, nonresponse bias for the survey and group interpretation meetings will not be calculated. Participant demographics will be documented and reported in written materials associated with the data collection, however, we will not have demographic information on the universe of practitioners and researchers/evaluators within home visiting to compare with. All selected participants will be invited to participate in a group interpretation meeting.
B6. Production of Estimates and Projections
Data will not be used to generate population estimates, either for internal use or dissemination.
B7. Data Handling and Analysis
Data Handling
To minimize errors in survey data processing and analysis, the technical staff involved on the project have considerable experience working with survey data. Additionally, the GroupWisdom web-based software employs quality controls to ensure only valid responses are allowed and to minimize missing responses. We will also review all data during initial processing before data is approved and becomes part of the final data set. Finally, data processing is run directly in the GroupWisdom software using automated processes, which minimizes data handling errors.
All group interpretation meetings (upon participant agreement) will be audio recorded and transcribed before analysis is completed to mitigate errors. For participants who do not consent to being audio recorded, a note taker will be used to ensure responses are recorded.
Data Analysis
A mixture of descriptive and qualitative methods will be used to analyze the data. The descriptive and qualitative methods are described below.
Descriptive methods: We will analyze the data using software that has been developed specifically for concept mapping (GroupWisdom by Concept Systems Incorporated). Specifically, we will use this software to:
Create similarity matrices to show how frequently statements were sorted together and provide a visual presentation of which statements participants sorted together most often.
Conduct multidimensional scaling to provide a visualization of sorting data to identify clusters and emerging themes. This includes plotting statements frequently sorted together closer to one another and plotting statements further away from one another if they were not frequently sorted together. Hierarchical cluster analysis is then used to group statements that are close to one another to identify clusters, which represent emerging themes.
Create “go-zones” to provide a visual map of average ratings for the feasibility and importance of each statement. Average ratings provided to each individual element, for feasibility and importance, are mapped to visualize which elements are rated low in feasibility and importance, low in feasibility and high in importance, high in feasibility and low in importance, and high in feasibility and high in importance (the “go-zone”).
Conduct pattern matching to run results separately by participant role of characteristics. For example, comparing practitioner responses to those of researchers and evaluators.
Qualitative methods: During the group interpretation meetings, the Study Team will first present and explain findings using the visualizations described above. The facilitator will then invite participants to examine and name the emerging clusters. We will then ask participants to consider ratings for importance and feasibility. Finally, participants will discuss the implications for prioritizing items for the measurement tool. We will use an iterative approach for data reduction, analysis, and interpretation to enable deductive and emergent findings. Dedoose software will be used to check, clean, and code transcripts. All responses will be compiled by each question in an initial round of coding. During the second round of coding, we will use both a-priori codes that correspond with group interpretation meeting questions as well as free coding to capture themes emerging from the data.
Data Use
Data will be used to develop items for a measure of reflective supervision for home visiting. Once the measure is developed, we plan to submit another information collection to pilot test the measure. No dissemination products will be developed based on the study findings. In future publications regarding the measure, we will describe how the current information collection contributed to the draft measure.
B8. Contact Persons
The information for this study is being collected by James Bell Associates, Johns Hopkin University, and University of Colorado Anschutz Medical Campus on behalf of ACF. Principal Investigator Allison West (awest25@jhu.edu), Project Director Mariel Sparr (sparr@jbassoc.com) and Task Lead Nancy Whitesell (nancy.whitesell@ucdenver.edu) led development of the study design plan and data collection protocols and will oversee collection and analysis of data.
The agency responsible for receiving and approving contract deliverables is:
The Office of Planning, Research, and Evaluation (OPRE),
Administration for Children and Families (ACF)
U.S. Department of Health and Human Services
The Federal project officers for this project are Nicole Denmark and Shirley Adelstein.
Attachments
Appendix 1: Home Visiting Applied Research Collaborative Recruitment Announcement
Appendix 2: Recruitment Email for Researchers
Instrument 1: Screening Questionnaire
Instrument 2: Web-Based Survey
Instrument 3: Group Interpretation Meeting Guide
1 Since the same information was not requested of more than nine individuals, these activities were not subject to the Paperwork Reduction Act.
2 National Home Visiting Resource Center. (2021). 2021 Home Visiting Yearbook. James Bell Associates and the Urban Institute.
3 Home Visiting Applied Research Collaborative. Retrieved from: https://www.hvresearch.org/practice-based-research-network/current-members/
4 Rosas, R. & Ridings, J. (2017). The use of concept mapping in measure development and evaluation: Application and future directions. Evaluation and Program Planning, 60(2017), 265-276.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Mariel Sparr |
| File Modified | 0000-00-00 |
| File Created | 2024-07-25 |