SSB - Pretesting GenIC_HS2K

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Understanding Children’s Transitions from

Head Start to Kindergarten

Pre-testing of Evaluation Data Collection Activities

0970 - 0355

Supporting Statement

Part B

February 2022

Submitted By: 

Office of Planning, Research, and Evaluation 

Administration for Children and Families  

U.S. Department of Health and Human Services 

 

4^th Floor, Mary E. Switzer Building 

330 C Street, SW 

Washington, D.C. 20201 

 

Project Officer: Kathleen Dwyer, PhD 

Part B

B1. Objectives

Study Objectives

The Office of Planning, Research, and Evaluation (OPRE) at the Administration for Children and Families (ACF) under the U.S. Department of Health and Human Services (HHS) proposes to conduct cognitive interviews in an effort to refine a set of instruments that target the organizational strategies and approaches that support effective kindergarten transition practices at scale—including understanding relationships within and among organizations and systems. The primary objective of these novel surveys is to probe the practices, policies, and professional supports implemented at the administrative level in both the Head Start and K-12 systems, as well as perspectives about kindergarten transitions exhibited by administrators on both sides.

To that end, OPRE and its contractors (henceforth referred to as the research team) will conduct cognitive interviews with up to 40 Head Start and K-12 administrators in order to evaluate whether the instruments can successfully collect the desired data. Intended use of the cognitive interview findings is to evaluate and improve the quality of the instrument, which will inform future ACF data collections and testable hypotheses related to kindergarten transitions.

Generalizability of Results

As noted in Supporting Statement A, this study is intended to present an internally valid description of survey respondents’ understanding of instruments, not to promote statistical generalization to other populations. The findings from the cognitive interviews are not meant to be representative of other populations.

Appropriateness of Study Design and Methods for Planned Uses

Cognitive interviews are designed to assess whether potential respondents interpret the survey items in intended ways and to identify potentially problematic survey wording or item order. The items in the novel surveys ask questions about kindergarten transition activities at the organizational––rather than classroom, child, or family––level. No instrument currently exists to assess these aspects of the Head Start to kindergarten transition, and cognitive interview questions will focus on the extent to which respondents can understand and successfully answer questions.

This information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  

B2. Methods and Design

Target Population

The research team will sample and recruit up to 40 participants from two categories of respondents: Head Start program administrators (20) and K-12 local education agency (LEA) administrators (20). Cognitive interviews will require respondents in each category given that instruments are tailored to the organizational conditions and language is unique to each respondent category.

The research team will use non-probability, purposive sampling to identify potential respondents who can test and respond to questions on the instrument. Because participants will be purposively selected, they will not be representative of the population of Head Start or K-12 administrators. Instead, the study team aims to obtain variation in administrators’ experiences to understand their engagement of survey questions and response options.

Respondent Recruitment

The research team will intentionally recruit administrators, ensuring that at there are at least two sets of respondents (administrators from the Head Start program and the LEA) from each of the following categories, relying on U.S. Census data: urban, suburban, and rural areas. Researchers will aim for a sample of participants with a diverse set of experiences. For example, we may select participants who serve different student populations, whose schools have small versus large numbers of feeder programs into their respective LEAs, and who collectively represent diversity across race, ethnicity, and gender.

The overarching strategy for recruitment will begin with professional contacts from study team members and then snowball sampling. Identification of possible participants will build off existing relationships between study team members and Head Start and K-12 systems. The contacts for the Head Start and LEA administrators will be derived from previous research, technical assistance, or consulting that the study team members at NORC or its subcontractors have conducted. Additional potential respondents will be identified based on a snowball sampling procedure whereby participants will be asked to nominate new potential participants. The strategy for identifying Head Start and K-12 participants will focus on identifying the Head Start program or K-12 system (LEA) level administrators, because there are fewer existing survey items that target these administrators. This will support our goal of developing appropriate survey items to get at system-level implementation and alignment of kindergarten transition policies, perspectives, professional supports, and practices.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

The focus of the cognitive interviews is to understand how respondents understand and can respond to questions in an instrument about the Head Start to kindergarten transition. The instruments were developed by a team of researchers based on project team expertise, a literature review, the development of a new theory of change reflecting cross-systems transitions from Head Start to kindergarten, and a review of existing measures. The instruments were reviewed by the project officers and several experts. See Exhibit 1 in SSA Section A.8 for the list of experts.

The cognitive interview protocols (Instruments 1 – 6) incorporate the instrument questions and include appropriate framing of the activity (on the first page), along with prompts after questions that may help the participant reflect on how they understand particular survey items. Interview prompts focus on the extent to which different respondents are able to understand questions and response options about kindergarten transition perspectives, policies, professional supports, and practices. This will assist the team in refining survey questions and response options to ensure respondents understand and can answer them. Each survey (one for Head Start administrators and one for LEA administrators) was separated into three different cognitive interview protocols to minimize individual burden on any one participant given the length of the developed instruments. Each cognitive interview participant would respond to one protocol only.

B4. Collection of Data and Quality Control

Cognitive interviews will be conducted by research staff of NORC and subcontractors.

Training and oversight of research staff. All research staff will go through a training overseen by a senior researcher with expertise in cognitive interviewing methodology. Research staff will practice the technique on each other prior to working with study participants. All staff new to this methodology will be paired with a more seasoned researcher to observe at least one cognitive interview being led by the seasoned researcher before conducting one themselves. In addition, staff new to this methodology will be observed by a seasoned researcher when leading their first interview and the two researchers will debrief afterwards. If necessary, seasoned staff will continue to observe additional interviews led by researchers new to the cognitive interview methodology until competency in the method is achieved.

Recruitment. The cognitive interviewing will begin with recruitment of participants. The study team will use their contacts within both Head Start and K-12 systems to ask for volunteers to help test survey instruments. We aim to recruit administrators within both Head Start and K-12 systems at the program level (e.g., Head Start program, superintendent of an LEA). Since we plan to recruit from several different states or regions of the country, we will do outreach by email and follow up with phone calls.

Study team members will develop lists of potential participants and will categorize based on their geographic location and whether located in an urban, rural, or suburban area. Study team members (see Exhibit B.8) will have preliminary conversations with their contacts about the scope and rationale for the study, and the study team members will formally reach out to potential respondents via recruitment materials (see Attachment B). Screener questions, which will be part of the recruitment process, will also be completed to ensure that participants have the requisite experience (see Instrument 7).

Scheduling interviews. The study team will schedule interviews via email and at convenient times for the respondents, after they have been screened for participation, accommodating work schedules with late afternoon and evening availability. Interviews will last one and a half hours.

Conducting cognitive interviews. The interviews will be held virtually and be recorded—with respondent permission—to ensure all information is accurately captured. Consent to participate in the cognitive interview will be part of the introductory protocol for the cognitive interviews (see Attachment A for Consent Materials). Participants will be asked if they agree to participate in the cognitive interview and whether they agree to be recorded while completing the process. Respondents will be interviewed one at a time. Participants will complete the draft survey as they are being interviewed via screensharing; these data will be collected electronically as the participant “thinks aloud” and responds to interview prompts. A researcher will listen in on this process, take notes, and – where appropriate – probe on certain aspects of survey items according to the protocol prompts. At the end of the interview, respondents will be asked to provide any further feedback on the survey via probes. At the conclusion of the interview, the researchers will provide the respondent with their honoraria. The protocols are included as Instruments 1-6. Exhibit 1 (Supporting Statement A) provides the list of respondents and type of protocol they will be responding to.

Cognitive interviews will be conducted in “rounds” or groupings. After we receive feedback from several participants, modifications to the survey instrument (and, in turn, protocols) will be made to facilitate clarity and understanding. In the next round, new participants will respond to a modified version of the instrument with these clarified questions in the protocols. See section B7 for additional details about these rounds of interviews as related to the analyses and survey modification.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The interviews are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.

Non-Response

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent demographics will be documented and reported in the internal written materials associated with the data collection. The study team also will monitor the racial and ethnic backgrounds of study participants and the urbanicity/rurality of the location where participant Head Start programs or LEAs are located.

B6. Production of Estimates and Projections

The data will not be used to generate population estimates, either for internal use or for dissemination. The data collected will be used only to refine survey instruments for use in later research.

B7. Data Handling and Analysis

Data Handling

All interviews will be recorded and transcribed, reducing burden and chances of error by not needing to confirm responses post-data collection (which would require additional respondent time).

Data Analysis

The research team will write a summary immediately following each round of cognitive interviews, noting how many items were tested along with examples of issues identified. The interview team will hold periodic debriefing meetings to systematically identify issues for survey items, determine the source of the issue, and identify potential revisions to survey items. These debriefing meetings will occur after each round of interviews. Together, the team will decide if survey items either need modification in their wording, response categories, or order within the survey instrument, or if they need to be dropped completely. After modifications to the survey instrument are made following debriefing meetings, a new round of cognitive interviews will be conducted on the modified survey instrument. These steps will be repeated between two and eight times, until the maximum number of interviews are reached or until the research team reaches saturation on the survey items (i.e., no new issues are identified).

Data Use

The research team will not release any data from this collection to the public. The data collected will be for internal ACF use only.

The individual summaries for cognitive interview rounds will inform a final summary report provided to ACF that will include individual summaries of each round of cognitive interviewing and high-level findings alongside recommendations for final instruments. The summary report will include final formatted instruments as appendices. The research team will also develop a spreadsheet that documents the full extent of the measure development work, including original survey item wording and response options, findings from each iterative round of cognitive interviews, rationales for changes, and final item wording. ACF may use these internal reports and formatted instruments to inform future studies of Head Start to kindergarten transition practices.

B8. Contact Persons

Name	Affiliation	Email Address
Kathleen Dwyer, PhD*	Administration for Children and Families	kathleen.dwyer@acf.hhs.gov
Stacy Ehrlich, PhD*	NORC at the University of Chicago	ehrlich-stacy@norc.org
Kyle DeMeo Cook, PhD*	K Strategies, Inc.	kdemeocook@gmail.com
Marc Hernandez, PhD*	NORC at the University of Chicago	hernandez-marc@norc.org
Tamara Halle, PhD*	Child Trends	thalle@childtrends.org
Eileen Graf, PhD*	NORC at the University of Chicago	graf-eileen@norc.org
Kristie Kauerz, PhD	National P-3 Center	kristie.kauerz@ucdenver.edu
Sara Amadon*	Child Trends	samadon@childtrends.org
Sarah Her	Child Trends	sher@childtrends.org
Andrew Schaper, PhD*	National P-3 Center	andrew.schaper@ucdenver.edu
Mitch Barrows	NORC at the University of Chicago	barrows-mitchell@norc.org

*Can answer questions about the methodological aspects of the design and analyses.  

Attachments

Attachment A—Consent Language

Attachment B—Recruitment Materials

Instruments

Instrument 1—HS Program Director Protocol 1

Instrument 2—HS Program Director Protocol 2

Instrument 3—HS Program Director Protocol 3

Instrument 4—LEA Administrator Protocol 1

Instrument 5—LEA Administrator Protocol 2

Instrument 6—LEA Administrator Protocol 3

Instrument 7—Screener Questions

7