Appendix E: HBCC-NSAC Toolkit provider questionnaire matrix sampling

Appendix E HBCC-NSAC Toolkit provider questionnaire matrix sampling_clean_12-2-22.docx

Pre-testing of Evaluation Data Collection Activities

Appendix E: HBCC-NSAC Toolkit provider questionnaire matrix sampling

OMB: 0970-0355

Document [docx]
Download: docx | pdf

Appendix E

HBCC-NSAC Toolkit provider questionnaire matrix sampling

HBCC-NSAC Toolkit provider questionnaire matrix sampling

Purpose

The initial version of the Home-Based Child Care Toolkit for Nurturing School-Age Children (HBCC-NSAC Toolkit) provider questionnaire, administered in the July 2022 pre-test, had 237 items and took participants between 40 minutes and almost 2 hours to complete (participants who took the longest also left open-ended comments, which will not be included in the pilot). The study team removed duplicative items after the pre-test, but the provider questionnaire is still too long to be administered to pilot study participants. In later use, we expect that providers will select specific domains – meaningful sections – that align with the provider’s interests, needs, and professional development goals. Once a provider has completed one section, they may choose to move on to another at their own pace. This may be decided individually or with the support of a coach or mentor.

To reduce burden on pilot study participants, replicate a more realistic administration of the provider questionnaire, and obtain information about more items, we will use matrix sampling and randomly assign HBCC providers to one of four shorter versions of the provider questionnaire. This approach has been used in national studies (for example, the mathematics and reading assessments for grades 3-5 of the Early Childhood longitudinal Study (1998 cohort), and the mathematics assessment for the Middle Grades Longitudinal Study 2017-2018) to develop assessments and will allow us to obtain information on a larger set of items without overwhelming participants. 1 This information on the larger set of items may allow us to create a briefer version of the instrument. This matrix sampling design will allow us to examine the dimensionality of the proposed sections and domains.

Four versions of the HBCC-NSAC Toolkit provider questionnaire for the pilot study

Exhibit 1 shows the domains and dimensions and number of items in each of the proposed four versions of the provider questionnaire. The added color-coding of the dimensions illustrates the domains being measured:

Color: Domain

Blue: I.A-I.D Support for Social Development dimensions

Green: II.A-II.C Support for Emotional Development dimensions

Yellow: III.A-III.B Positive and Proactive Behavior Management

Red: IV.A-IV.D Support for Learning dimensions

Purple: V.A-V.D Support for Health and Physical Development dimensions





To create each of the four versions, the study team divided the full list of items that make up the provider questionnaire to create shorter versions. Each version includes different combinations of domains and dimensions and no version exceeds 120 items, which will take approximately 25 minutes to complete. Based on responses to each of the four versions, the study team will be able to examine responses in relation to each combination of dimensions. Further, each dimension overlaps across at least two versions in order to obtain enough responses for items within a given dimension for analysis purposes and allow examination of each dimension with the other dimensions.



Shape1

Exhibit 1. Domains and key dimensions in each provider questionnaire version for pilot study matrix sampling

Version 1: domains & dimensions

N

Version 2: domains & dimensions

N

Version 3: domains & dimensions

N

Version 4: domains & dimensions

N

I.A. Support for Social Development: Builds and strengthens a positive relationship with children

11

I.B. Support for Social Development: Supports children’s perspective-taking and nonverbal communication

14

I.D. Support for Social Development: Supports antibullying and antibias practices

19

I.A. Support for Social Development: Builds and strengthens a positive relationship with children

11

I.C. Support for Social Development: Supports children’s social skills

15

I.C. Support for Social Development: Supports children’s social skills

15

II.A. Support for Emotional Development: Helps children understand and regulate emotions

12

I.B. Support for Social Development: Supports children’s perspective-taking and nonverbal communication

14

I.D. Support for Social Development: Supports antibullying and antibias practices

19

II.B. Support for Emotional Development: Supports a positive sense of belonging

15

II.B. Support for Emotional Development: Supports a positive sense of belonging

15

II.A. Support for Emotional Development: Helps children understand and regulate emotions

12

III.A. Positive and Proactive Behavior Management: Uses predictable and responsive routines

15

III.B. Positive and Proactive Behavior Management: Uses proactive and positive disciplinary practices

15

II.C. Support for Emotional Development: Supports a positive self-identity (including racial and ethnic identity)

15

II.C. Support for Emotional Development: Supports a positive self-identity (including racial and ethnic identity)

15

IV.B. Support for Learning: Supports positive approaches to learning and a growth mindset

10

IV.A. Support for Learning: Provides learning opportunities

15

III.A. Positive and Proactive Behavior Management: Uses predictable and responsive routines

15

IV.A. Support for Learning: Provides learning opportunities

15

IV.C. Support for Learning: Scaffolds problem-solving

9

IV.C. Support for Learning: Scaffolds problem-solving

9

III.B. Positive and Proactive Behavior Management: Uses proactive and positive disciplinary practices

15

V.A. Support for Health and Physical Development: Provides a variety of activities to support physical well-being

13

IV.D. Support for Learning: Collaborates with families to strengthen learning connections with home

8

IV.D. Support for Learning: Collaborates with families to strengthen learning connections with home

8

IV.B. Support for Learning: Supports positive approaches to learning and a growth mindset

10

V.B. Support for Health and Physical Development: Communicates with families about health and physical development

7

V.A. Support for Health and Physical Development: Provides a variety of activities to support physical well-being

13

V.B. Support for Health and Physical Development: Communicates with families about health and physical development

7

V.C. Support for Health and Physical Development: Scaffolds physical activity

11

V.C. Support for Health and Physical Development: Scaffolds physical activity

11

V.C. Support for Health and Physical Development: Scaffolds physical activity

11

V.D. Support for Health and Physical Development: Promotes health, safety, and nutrition

19



V.D. Support for Health and Physical Development: Promotes health, safety, and nutrition

19









Total number of items:

111


117


112


117

Shape2

The referenced collection of information is voluntary. Information will be kept private. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0355 and the expiration date is 8/31/2024.





1 Pollack, J., Atkins-Burnett, S., Rock, D., and Weiss, M. (2005). Early Childhood Longitudinal Study, Kindergarten Class of 1998–99 (ECLS-K), Psychometric Report for the Third Grade (NCES 2005–062). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

National Center for Education Statistics. “Middle Grades Longitudinal Study of 2017–18 (MGLS:2017)

2016 Item Validation Field Test (IVFT) Data Collection.” Supporting Statement Part A. Submitted to the Office of Management and Budget, 2015.

File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File TitleMathematica Memo
Subjectmemo
AuthorAnn Li
File Modified0000-00-00
File Created2024-07-23

© 2024 OMB.report | Privacy Policy