HBCC-NSAC Toolkit provider questionnaire matrix sampling
The initial version of the Home-Based Child Care Toolkit for Nurturing School-Age Children (HBCC-NSAC Toolkit) provider questionnaire, administered in the July 2022 pre-test, had 237 items and took participants between 40 minutes and almost 2 hours to complete (participants who took the longest also left open-ended comments, which will not be included in the pilot). The study team removed duplicative items after the pre-test, but the provider questionnaire is still too long to be administered to pilot study participants. In later use, we expect that providers will select specific domains – meaningful sections – that align with the provider’s interests, needs, and professional development goals. Once a provider has completed one section, they may choose to move on to another at their own pace. This may be decided individually or with the support of a coach or mentor.
To reduce burden on pilot study participants, replicate a more realistic administration of the provider questionnaire, and obtain information about more items, we will use matrix sampling and randomly assign HBCC providers to one of four shorter versions of the provider questionnaire. This approach has been used in national studies (for example, the mathematics and reading assessments for grades 3-5 of the Early Childhood longitudinal Study (1998 cohort), and the mathematics assessment for the Middle Grades Longitudinal Study 2017-2018) to develop assessments and will allow us to obtain information on a larger set of items without overwhelming participants. 1 This information on the larger set of items may allow us to create a briefer version of the instrument. This matrix sampling design will allow us to examine the dimensionality of the proposed sections and domains.
Exhibit 1 shows the domains and dimensions and number of items in each of the proposed four versions of the provider questionnaire. The added color-coding of the dimensions illustrates the domains being measured:
Color: Domain |
Blue: I.A-I.D Support for Social Development dimensions |
Green: II.A-II.C Support for Emotional Development dimensions |
Yellow: III.A-III.B Positive and Proactive Behavior Management |
Red: IV.A-IV.D Support for Learning dimensions |
Purple: V.A-V.D Support for Health and Physical Development dimensions |
To create each of the four versions, the study team divided the full list of items that make up the provider questionnaire to create shorter versions. Each version includes different combinations of domains and dimensions and no version exceeds 120 items, which will take approximately 25 minutes to complete. Based on responses to each of the four versions, the study team will be able to examine responses in relation to each combination of dimensions. Further, each dimension overlaps across at least two versions in order to obtain enough responses for items within a given dimension for analysis purposes and allow examination of each dimension with the other dimensions.
Exhibit 1. Domains and key dimensions in each provider questionnaire version for pilot study matrix sampling
Version 1: domains & dimensions |
N |
Version 2: domains & dimensions |
N |
Version 3: domains & dimensions |
N |
Version 4: domains & dimensions |
N |
I.A. Support for Social Development: Builds and strengthens a positive relationship with children |
11 |
I.B. Support for Social Development: Supports children’s perspective-taking and nonverbal communication |
14 |
I.D. Support for Social Development: Supports antibullying and antibias practices |
19 |
I.A. Support for Social Development: Builds and strengthens a positive relationship with children |
11 |
I.C. Support for Social Development: Supports children’s social skills |
15 |
I.C. Support for Social Development: Supports children’s social skills |
15 |
II.A. Support for Emotional Development: Helps children understand and regulate emotions |
12 |
I.B. Support for Social Development: Supports children’s perspective-taking and nonverbal communication |
14 |
I.D. Support for Social Development: Supports antibullying and antibias practices |
19 |
II.B. Support for Emotional Development: Supports a positive sense of belonging |
15 |
II.B. Support for Emotional Development: Supports a positive sense of belonging |
15 |
II.A. Support for Emotional Development: Helps children understand and regulate emotions |
12 |
III.A. Positive and Proactive Behavior Management: Uses predictable and responsive routines |
15 |
III.B. Positive and Proactive Behavior Management: Uses proactive and positive disciplinary practices |
15 |
II.C. Support for Emotional Development: Supports a positive self-identity (including racial and ethnic identity) |
15 |
II.C. Support for Emotional Development: Supports a positive self-identity (including racial and ethnic identity) |
15 |
IV.B. Support for Learning: Supports positive approaches to learning and a growth mindset |
10 |
IV.A. Support for Learning: Provides learning opportunities |
15 |
III.A. Positive and Proactive Behavior Management: Uses predictable and responsive routines |
15 |
IV.A. Support for Learning: Provides learning opportunities |
15 |
IV.C. Support for Learning: Scaffolds problem-solving |
9 |
IV.C. Support for Learning: Scaffolds problem-solving |
9 |
III.B. Positive and Proactive Behavior Management: Uses proactive and positive disciplinary practices |
15 |
V.A. Support for Health and Physical Development: Provides a variety of activities to support physical well-being |
13 |
IV.D. Support for Learning: Collaborates with families to strengthen learning connections with home |
8 |
IV.D. Support for Learning: Collaborates with families to strengthen learning connections with home |
8 |
IV.B. Support for Learning: Supports positive approaches to learning and a growth mindset |
10 |
V.B. Support for Health and Physical Development: Communicates with families about health and physical development |
7 |
V.A. Support for Health and Physical Development: Provides a variety of activities to support physical well-being |
13 |
V.B. Support for Health and Physical Development: Communicates with families about health and physical development |
7 |
V.C. Support for Health and Physical Development: Scaffolds physical activity |
11 |
V.C. Support for Health and Physical Development: Scaffolds physical activity |
11 |
V.C. Support for Health and Physical Development: Scaffolds physical activity |
11 |
V.D. Support for Health and Physical Development: Promotes health, safety, and nutrition |
19 |
|
|
V.D. Support for Health and Physical Development: Promotes health, safety, and nutrition |
19 |
|
|
|
|
|
|
|
|
Total number of items: |
111 |
|
117 |
|
112 |
|
117 |
The referenced collection of information is voluntary. Information
will be kept private. An agency may not conduct or sponsor, and a
person is not required to respond to, a collection of information
unless it displays a currently valid OMB control number. The OMB
control number for this collection is 0970-0355 and the expiration
date is 8/31/2024.
1 Pollack, J., Atkins-Burnett, S., Rock, D., and Weiss, M. (2005). Early Childhood Longitudinal Study, Kindergarten Class of 1998–99 (ECLS-K), Psychometric Report for the Third Grade (NCES 2005–062). U.S. Department of Education. Washington, DC: National Center for Education Statistics.
National Center for Education Statistics. “Middle Grades Longitudinal Study of 2017–18 (MGLS:2017)
2016 Item Validation Field Test (IVFT) Data Collection.” Supporting Statement Part A. Submitted to the Office of Management and Budget, 2015.
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
File Title | Mathematica Memo |
Subject | memo |
Author | Ann Li |
File Modified | 0000-00-00 |
File Created | 2024-07-23 |