APPENDIX
X
FACES 2019 AND AIAN FACES 2019 NONRESPONSE BIAS ANALYSIS
SUMMARY MEMOS
This page has been left blank for double-sided copying.
APPENDIX
X.1
FACES 2019 NONRESPONSE BIAS ANALYSIS SUMMARY MEMO
This page has been left blank for double-sided copying.
To: Nina Philipsen Hetzner and Alysia Blandon
From: Barbara Lepidus Carlson, Ian Huff, and Cathy Lu
Date: 4/30/2021
Subject: FACES 2019–2020 Nonresponse Bias Analysis Report
We conducted a nonresponse bias analysis on the FACES 2019 data collected in fall 2019 and spring 2020 at the program, center, classroom, and child levels. Rather than doing a separate analysis for each data collection instrument, we looked at combinations of instrument completes that corresponded to our weighting definitions. Because these combinations revealed study participation or instrument completion rates that fell below 80 percent, these analyses were done to establish confidence in our weighted estimates, which we will present in this document. We first describe the response patterns for FACES 2019, then the purposes of nonresponse bias analysis, then the methodological approach we used. We then present the results of the analysis and our conclusions.
Impact of COVID pandemic. Historically, FACES has generally achieved high response rates at the staff, parent, and child levels. Spring 2020 data collection for FACES 2019 was scheduled to begin in mid-March, at around the same time that COVID-19 (for coronavirus disease 2019) was declared a pandemic by the World Health Organization and a public health emergency by the United States (Centers for Disease Control and Prevention 2020). In response to the COVID-19 pandemic, FACES cancelled the in-person data collection of child assessments and classroom observations in spring 2020. However, for programs newly entering the study in spring 2020, program recruitment and center and classroom sampling continued, and staff and parent surveys were still pursued in all programs. These surveys included the program director survey, the center director survey, the teacher survey, the Teacher Child Report (TCR), and the spring parent survey. The participation and response rates for these instruments were lower than they were in previous rounds because of the consequences of the COVID-19 pandemic.
Instead of recruiting an additional 120 sampled programs for the spring 2020 data collection in accordance with the study design target, we were able to get only 108 to agree to participate. Two of these 108 were ultimately treated as study nonparticipants because they provided no data, even though we were able to get through the center sampling stage. This left 106 programs of the 120 originally targeted. Two of the 106 programs provided a program director survey only, and so we treated those programs as program-level study participants but center-level nonparticipants. Another of the 106 programs had program and center director surveys completed but did not provide the information to complete classroom sampling. This program and its centers were considered study participants but classroom-level nonparticipants. A total of 165 programs participated in the study in spring 2020, including all 59 programs that participated in fall data collection and the 106 programs that joined the study in the spring.
Response rates. Unweighted and weighted response rates for the different survey instruments are in Table 1. The unweighted marginal response rate represents the unadjusted percentage of eligible respondents that completed the survey for the specific instrument. For example, the 76.4 percent unweighted response rate for the program director survey is the result of dividing the number of completed responses to the director survey (126) by the total number of programs participating in the study (165). The weighted cumulative response rate, on the other hand, represents the percentage of eligible respondents that completed the survey, weighted for the probability of selection and incorporating any prior sampling stages’ weights (if applicable). So, for example, although 61.9 percent of the teacher surveys we attempted (365 out of 590) were completed, we estimate that these completed surveys represent 50.1 percent of the population of study-eligible Head Start classrooms.
Nonresponse bias can occur when the survey responses of nonrespondents would have been different enough from those of respondents to change the overall results—that is, to bias them. Although a lower response rate does not necessarily indicate the presence of nonresponse bias, a higher response rate does lower the risk of nonresponse bias. Nonresponse bias itself can rarely be measured directly, as we generally do not know what the missing responses would have been. Instead, we examine variables that are available for both respondents and nonrespondents and that are presumably correlated with the survey items that are missing for some sample members. In a nonresponse bias analysis, we compare respondents to nonrespondents on the distributions or means of these characteristics and identify any potentially problematic differences, using statistical tests to indicate whether the differences are likely due to something other than sampling error. We adjust weights for differential response patterns with the goal of mitigating the risk of nonresponse bias, and then assess whether the adjusted weights appear to have diminished those differences without having introduced larger differences in other variables.
Weighting overview. We evaluated nonresponse bias for each FACES 2019 weight we produced for spring and fall-spring estimates. When we compare respondents to nonrespondents, we use the weights that account for sampling at the current stage, and for sampling and study participation at prior sampling stages. But, by definition, the nonrespondents do not have weights that account for nonresponse at the current stage, as they are assigned a value of zero. Only when we examine the fully weighted distributions and means for respondents do we use the final weights, which are adjusted for nonresponse at the stage in question.
There are two general methods we use to adjust weights for nonresponse. One divides the sample into weighting cells (based one or more characteristics available for all sample members and that are plausibly related to key outcomes and the likelihood of responding) and inflates the respondent sampling weights to account for the nonrespondents in each cell. The other method uses a response propensity logistic regression model to predict the likelihood of responding, generally using more characteristics, and multiplying the inverse of the resulting propensity scores to the respondent sampling weights to account for the nonrespondents. After using both methods to evaluate indicators for nonresponse bias correction for various FACES 2019 weights, we opted to use the weighting cell approach for all but two weights.
Table 1. Response rates
Respondent level |
Instrument |
Eligible |
Completed |
Unweighted marginal response rate (percent) |
Weighted cumulative response rate (percent) |
Program |
Program participation |
222 |
165 |
74.3 |
83.4 |
Program director survey |
165 |
126 |
76.4 |
66.2 |
|
Center |
Center participation |
326 |
318 |
97.5 |
81.1 |
Center director survey |
318 |
191 |
60.1 |
48.4 |
|
Classroom |
Teacher survey |
590 |
365 |
61.9 |
50.1 |
Child |
Fall child participation (among those still in sampled program at data collection) |
2494 |
2260 |
90.6 |
73.5 |
Fall child participation (among those in sampled program at time of sampling) |
2505 |
2260 |
90.2 |
73.2 |
|
Fall parent survey |
2260 |
1703 |
75.4 |
55.6 |
|
Fall Teacher Child Report |
2260 |
2090 |
92.5 |
68.0 |
|
Fall child assessment |
2260 |
2105 |
93.1 |
68.6 |
|
Spring child participation (among those estimated to still be in sampled program)a |
2352.75 |
2132 |
90.6 |
73.0 |
|
Spring child participation (among those estimated to still be receiving Head Start services)a |
2381.97 |
2132 |
89.5 |
72.5 |
|
Spring parent survey |
2132 |
1447 |
67.9 |
50.0 |
|
Spring Teacher Child Report |
2132 |
1485 |
69.7 |
51.3 |
aFor spring participation at the child level, we estimated that a certain proportion of the fall nonparticipating cases would have left Head Start and become ineligible.
At the program level, we start with the sampling weight for selecting programs within stratum with probability proportional to size. For program-level weights, we then adjust for study participation using a weighting cell approach, and then for response to the program director survey using a model-based approach. We also used the program-level, participation-adjusted weight as a building block for center-, classroom-, and child-level weights. For child-level weights, we constructed a separate program weight that adjusts for the subsampling of programs selected to participate in child-level data collection.
Similarly, at the center level, we start with the sampling weight for selecting centers within program with probability proportional to size. We then bring in the program-level building block weight, applying it to each sampled center. For center-level weights, we use a weighting cell approach to adjust this cumulative weight for study participation, and then use a model-based approach to adjust it for response to the center director survey. We also use the cumulative center-level, participation-adjusted weight as a building block for classroom- and child-level weights, using the appropriate program-level weight for each one (accounting for subsampling or not). We construct classroom-level weights in the same way, except that classrooms were selected within centers with equal probability, and we apply the cumulative center-level weight to the sampling weight for sampled classrooms before adjusting for nonresponse to the teacher survey.
Finally, at the child level, we start with the sampling weight for selecting children within classrooms with equal probability. We then bring in the classroom-level cumulative weight (the one that uses the program subsampling adjustment at the first stage), applying it to the children in the sample. Within center,1 we adjust this weight for whether the child had parental consent in the fall. In the spring, any children who are no longer being served by Head Start are ineligible for the study, so they are simply dropped from analyses; and any children who are known to be served by Head Start, but who left the sampled program, are adjusted for in the weights. We then create a series of weights that accounts for various combinations of instrument completes. This is described in detail below.
Covariates used in nonresponse bias analysis. To conduct a nonresponse bias analysis, we rely on variables (covariates) that are available for both respondents and nonrespondents and that are plausibly correlated with key survey outcomes. Although most covariates are categorical, there are a few continuous ones. For the continuous covariates, we created ordinal versions as well, often using the 33rd and 67th unweighted percentiles of the respondents and nonrespondents combined as the cut points, but sometimes dividing continuous variables into binary variables based on analytic reporting (for example, child age).
We analyzed categorical and continuous program-, center-, and child-level variables for indications of potential bias due to nonresponse. We evaluated program-level covariates (mostly from the Head Start Program Information Report) at the program, center, classrooms, and child levels. We evaluated center-level covariates (obtained from the Head Start programs as part of the center sampling process) at the center, classrooms, and child levels. We evaluated child-level covariates (obtained from the centers on the child sampling rosters or from parents on the consent form) at the child level only.2
For the program director survey weight, the final response propensity model to adjust for nonresponse included the following predictor variables (listed in bullets below): program sample cohort, categorized percent of children enrolled in the program with a disability, categorized percentage of program enrollees age 4 or older, and program sampling strata (collapsed). For the center director survey weight, the final response propensity model included the following predictor variables: categorized percentage of children enrolled in the program with a disability, categorized percentage of program enrollees are age 4 or older, and categorized percentage of program staff who left in the past year. For all other weights, we used the weighting cell methodology. For those weights, we did not use any of the following variables directly to adjust weights for nonresponse; however, recall that we did use the program as the weighting cell for center-level weights, and the center as the weighting cell for classroom- and child-level weights (sometimes using program or program stratum as weighting cells as needed).
The following variables were the program-level covariates we analyzed:
Program sample cohort: child level (fall-spring) vs. classroom/program level only (spring)3
Census region4 of the program or program sampling strata (census region, metropolitan statistical area, and Black/Hispanic enrollment category)
Whether the program zip code is in a metropolitan statistical area
Size of the program by total enrollment
Percentage of program enrollees with a disability
Percentage of program enrollees age 4 or older
Percentage of program staff who left
Percentage of program staff who were replaced
Percentage of program lead teachers who left
Program service type
Center-level covariates include:
Size of the center by total enrollment
Size of the center by number of classrooms
The child-level covariates we analyzed were:
Child’s age (less than 48 months versus 48 months or older)
Child’s sex
Language spoken at home (English versus not English)
Number of months child has been enrolled in Head Start
Whether the child participated in Early Head Start
Primary funding source (Head Start, state prekindergarten, and/or child subsidies and other sources)
Steps in the nonresponse bias analysis. For each covariate that is a categorical or ordinal variable, we run a design-adjusted chi-square test to compare the weighted distributions between the respondents and nonrespondents.5 We next check to see if the full sample distribution is within two standard errors of the final nonresponse-adjusted–weighted distribution for respondents only. Similarly, for each covariate that is a continuous variable, we compare the weighted means for respondents and nonrespondents, running a design-adjusted t-test. 6 We then check to see if the full sample mean is within two standard errors of the final nonresponse adjusted-weighted mean for respondents only.
We assess multiple weights for potential nonresponse bias at the program, center, classroom, and child levels. At the program level, we have the base weight, D_WT, which accounts for program participation, and the weight for the program director survey, D2_WT. Similarly, at the center level, we have the center base weight, C_WT, which accounts for center participation, and the center director survey weight, C2_WT.
At the classroom level, we do not analyze the base weight, CLS2_WT, for nonresponse bias because all sampled classrooms were study participants. We do assess the teacher survey weight at the classroom level, T2CLSWT. Although we sample classrooms, not teachers, we do construct a teacher-level version of this weight (T2TCHWT) that accounts for those who teach more than one classroom. This supports estimates at the teacher level, but we did not also assess this weight, as it differs only slightly from the classroom-level version.
Finally, we have a child-level base weight, CNST2WT, that accounts for parental consent and retention in the FACES study, for comparison against four child-level instrument-based weights. The first such weight has a non-zero value for children with a parent survey in fall and spring, P12WT, and the second weight has a non-zero value for children with a parent survey in fall or spring, P1_2WT. The other two weights are P21R2WT, for those with a parent survey in fall or spring as well as a TCR in spring, and PR12WT, for those with a parent survey in fall or spring as well as a TCR in fall and spring. Because all children with a non-zero value of PR12WT also had a completed teacher survey for their classroom, the PR12CWT weight is identical to the PR12WT, and so we do not assess PR12CWT separately for nonresponse bias. Table 2 shows the various nonresponse bias analyses carried out for this report.
Table 2. Weights associated with nonresponse bias analyses performed for FACES 2019
Level |
Time point(s) of information gathered |
Weight name |
Weight description |
Program |
Fall or spring |
D_WT |
Program participation base weight |
Spring only |
D2WT |
Program director survey weight |
|
Center |
Fall or spring |
C_WT |
Center participation base weight |
Spring only |
C2WT |
Center director survey weight |
|
Classroom |
Spring only |
T2CLSWT |
Teacher survey weight at classroom level |
Child |
Fall only |
CNST1WT |
Child participation (consent) base weight |
Fall and/or spring |
P12WT |
Weight for those with parent survey in fall and spring |
|
Fall and/or spring |
P1_2WT |
Weight for those with parent survey in fall or spring |
|
Fall and/or spring |
P21R2WT |
Weight for those with parent survey in fall or spring plus TCR in spring |
|
Fall and/or spring |
PR12WT |
Weight for those with parent survey in fall or spring plus TCR in fall and spring |
TCR = Teacher Child Report.
Tables 3–6.c compare weighted estimates for respondents and nonrespondents (before nonresponse weighting adjustments) in observed covariates for our various weight-defined respondent definitions. This makes it possible for us to assess the risk for nonresponse bias in estimates based only on respondents. Those estimates are followed by estimates of these same covariates that are based on a final weight adjusted for nonresponse.
Each table contains information on one or two respondent definitions and associated final weights applied to the respondents only. Column A in each table contains the variable name. Column B contains the values of each categorical or ordinal variable. Column C in each table shows the distribution among categorical variables and the mean of continuous variables for the full sample using its base weight (accounting for the probability of selection and any prior stages of sampling and participation). Column D shows the study participation rate or instrument response rate by subgroup (for categorical variables only). Column E shows the p-value associated with statistical tests comparing respondents and nonrespondents. (Due to space limitations, we do not present in the tables the respondent and nonrespondent weighted percentages and means being compared in these tests.) Column F again shows variable distributions and means, now for respondents only, and fully weighted for nonresponse, along with associated standard errors. The last three columns—participation/response rate, p-value, and final weighted distribution/mean—are repeated (in Columns G, H, and I) for an additional respondent definition in some tables. When diagnosing nonresponse bias, we use a significance level of 0.05 to suggest a potential for nonresponse bias and evaluate whether the weighting to mitigate bias has been successful by assessing whether the full sample value (Column C) is within two standard errors of the final weighted percentage or mean (Column F or Column I). Any estimates for which the full sample value falls outside two standard errors are indicated by an asterisk in the table.
Program level. Table 3 shows the nonresponse bias analysis at the program level for program participation, with corresponding weight D_WT; and for the program director survey, along with weight D2_WT. For D_WT, we observe significant differences between participants and nonparticipants for the percentage of children enrolled in the program who have a disability—although only as a categorical variable, not as a continuous one.7 For the program director survey, we observe significant differences between respondents and nonrespondents for the following: program sampling strata, whether the program was in a metropolitan statistical area, and percentage of children enrolled in the program with a disability as a continuous variable. After applying nonresponse-adjusted weights (D_WT and D2_WT, respectively), the differences between respondents and the full sample appear to be mitigated for both respondent definitions, as the full sample value is within two standard errors of the final weighted estimate for all covariates, indicating that any remaining differences are likely attributable to sampling error.
Center level. Table 4 shows the nonresponse bias analysis at the center level for center participation, with weight C_WT, and for the center director survey, with weight C2_WT. We see significant differences between participants and nonparticipants for the percentage of children enrolled in the program who are age 4 or older, percentage of lead teachers who left the program in the past year, and the program service type. In addition, we observe significant differences between participants and nonparticipants for the continuous (but not categorical) versions of size of the center by total enrollment and percentage of staff in the program who were replaced in the past year. The center director survey does not appear to have any variables indicative of nonresponse bias at our chosen significance level. For both weights (C_WT and C2_WT), the full sample value is within two standard errors of the final weighted estimate for all covariates, which indicates that any remaining differences are likely attributable to sampling error.
Classroom level. Table 5 shows our analysis at the classroom level for the teacher survey, and associated weight T2CLSWT. For program sample cohort, program census region, percentage of children enrolled in the program with a disability, and percentage of program staff who were replaced in the past year (only as categorical) there is a significant difference between respondents and nonrespondents. Differences due to nonresponse appear to be mitigated after applying weights, as all percentages and means for the full sample were within two standard errors of the responding sample estimate after weighting (T2CLSWT).
Child level. Tables 6.a, 6.b, and 6.c present results from the child-level nonresponse bias analysis. Table 6.a shows the analysis of child-level study participation (mostly a measure of parental consent) for fall 2019, and associated weight CNST1WT, where we observe significant differences between participants and nonparticipants for the following covariates: child’s sex, language spoken in child’s home, number of months the child has been enrolled in Head Start, the program sampling strata, and size of the program by total enrollment. Months the child has been enrolled in Head Start and size of the program by total enrollment do not appear to be indicate a risk for nonresponse bias as continuous variables, only as categorical ones. Again, when we compare the full sample value to the final weighted percentages and means (using CNST1WT), we can see that weighting appears to have mitigated the risk for bias for all covariates.
Table 6.b shows the nonresponse bias analysis for the fall and spring parent surveys for children who were still participating in the study in spring 2020. First, we look at the analysis for children whose parents responded to both the fall and spring surveys, and at the associated weight P12WT. We saw significant differences between respondents and nonrespondents for covariates that included language spoken at child’s home, primary funding source, percentage of program staff who left in the past year, and percentage of lead teachers in the program who left in the past year. Second, we look at the analysis for children whose parents responded to either the fall or spring parent surveys, and the associated weight P1_2WT, finding significant differences between respondents and nonrespondents for covariates that included language spoken in the child’s home, primary funding source, whether the program is in a metropolitan statistical area, and percentage of children enrolled in the program with a disability as a categorical variable. All full sample percentages and means were within two standard errors of the responding sample estimates after nonresponse weighting (P12WT and P1_2WT).
Table 6.c shows the nonresponse bias analysis for two different child-level survey combinations. The first combines the response to either wave of the parent surveys and to the spring TCR survey, and corresponding weight P21R2WT. Our analysis finds significant differences between respondents and nonrespondents for the following covariates: child age as a continuous variable, size of the program by total enrollment, percentage of children enrolled in the program who are age 4 or older, percentage of program staff who left in the past year, and percentage of program lead teachers who left in the past year. The latter two only point to potential bias as categorical variables. The second combines the response to either wave of the parent survey with both fall and spring TCRs, and corresponding weight PR12WT. We observe significant differences between respondents and nonrespondents for the same variables as for P21R2WT. Both survey combinations mitigate the risk for nonresponse bias for these variables, as evidenced by the full sample values being within two standard errors of the final estimate after weighting (P21R2WT and PR12WT). However, weighting also causes the full sample value to fall outside the estimates of primary funding source in both combinations by more than two standard errors, where this was not the case before weighting.8 Given the number of statistical comparisons being made in this analysis that use a Type I error rate of 0.05, we would expect 1 out of 20 differences to rise to the level of statistical significance even if no true difference existed. Therefore, statistical significance in this situation does not necessarily indicate a difference between the weighted estimate and the true population value.
We examined the potential for nonresponse bias for all FACES 2019 spring and longitudinal (fall-spring) weights at all levels of analysis (program, center, classroom, and child). Although we observed some statistically significant differences between respondents and nonrespondents, none remained after weighting the data. That is, where significant differences existed between respondents and nonrespondents before weighting adjustments, the full sample estimate fell within two standard errors of the final weighted estimates after nonresponse adjustments to the weights, indicating that any remaining differences were likely attributable to sampling error and not to unresolved nonresponse bias. Further, with only one exception (child’s primary funding source for weights P21R2WT and PR12WT), the nonresponse weighting adjustments did not introduce any new differences that caused the full sample value to fall more than two standard errors from the responding sample’s weighted estimate. Although researchers should feel free to control for any characteristics that appear to differ between respondents and nonrespondents (or more accurately, between respondents and the full sample) in their models, we think researchers should be reassured that the risk for nonresponse bias has been mitigated when using the appropriate weights to make estimates from the FACES 2019 study.
cc: Krystal Bichay-Awadalla, Jacquelyn Gross, Lizabeth Malone, Ashley Kopack Klein, Nikki Aikens, and Louisa Tarullo
Table 3. Nonresponse bias analysis at the program level
Variable |
Value |
Full sample estimate |
Study participation rate by subgroup |
Study participants vs. nonparticipants p-value |
Study participants’ estimate [standard error] |
Program director survey response rate by subgroup |
Program director survey respondents vs. nonrespondents p-value |
|
A |
B |
C |
D |
E |
F |
G |
H |
I |
Number of programs |
222 |
|
|
165 |
|
|
126 |
|
Weight |
Program base weight |
Program base weight |
|
D_WT |
D_WT |
|
D2_WT |
|
Categorical variables at program level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Program sample cohort |
Child-level (fall-spring) |
35.53 |
82.15 |
0.751 |
35.53 [3.37] |
75.57 |
0.329 |
35.53 [3.99] |
Classroom/program-level only (spring) |
64.47 |
84.04 |
64.47 [3.37] |
83.19 |
64.47 [3.99] |
|||
Program sampling strata |
Census Region A |
25.76 |
86.52 |
0.454 |
25.76 [3.04] |
91.44 |
0.009 |
25.76 [3.31] |
Census
Region B, < 40% |
9.78 |
91.40 |
9.78 [1.51] |
53.05 |
9.78 [1.44] |
|||
Census
Region B, > 40% |
10.69 |
85.56 |
10.69 [1.98] |
61.45 |
10.69 [1.77] |
|||
Census
Region C, < 40% |
11.87 |
69.60 |
11.87 [2.06] |
94.72 |
11.87 [2.23] |
|||
Census Region C, MSA, > 40% Hispanic or Black enrollment |
15.90 |
89.33 |
15.90 [2.81] |
74.82 |
15.90 [4.17] |
|||
Census Region C, Non-MSA, > 40% Hispanic or Black enrollment |
6.72 |
80.73 |
6.72 [1.65] |
97.26 |
6.72 [1.55] |
|||
Census Region D, MSA, > 40% Hispanic or Black enrollment |
10.77 |
73.12 |
10.77 [2.79] |
91.34 |
10.77 [2.87] |
|||
Census Region D, Other |
8.52 |
84.98 |
8.52 [1.37] |
66.50 |
8.52 [1.58] |
|||
MSAa |
Yes |
67.33 |
83.89 |
0.787 |
67.77 [3.75] |
74.91 |
0.039 |
68.05 [4.16] |
No |
32.67 |
82.29 |
32.23 [3.75] |
92.20 |
31.95 [4.16] |
|||
Program enrollment |
< 411 |
64.54 |
84.70 |
0.658 |
63.63 [4.40] |
83.30 |
0.452 |
61.17 [5.10] |
411-963 |
24.57 |
80.33 |
25.37 [3.84] |
75.52 |
25.36 [4.24] |
|||
> 963 |
10.88 |
82.33 |
11.01 [1.76] |
75.66 |
13.47 [2.22] |
|||
Percentage of children with a disability enrolled in program |
< 10.09 |
25.09 |
80.92 |
0.046 |
24.22 [4.33] |
74.25 |
0.640 |
25.42 [5.20] |
10.09-14.03 |
35.81 |
91.88 |
39.34 [5.16] |
81.12 |
38.16 [5.69] |
|||
> 14.03 |
39.10 |
77.14 |
36.44 [4.55] |
83.94 |
36.42 [5.09] |
|||
Percentage of children age 4 and older enrolled in program |
< 47.57 |
35.52 |
88.51 |
0.102 |
36.99 [5.28] |
80.45 |
0.380 |
36.28 [5.78] |
47.57-55.16 |
32.16 |
85.32 |
32.14 [4.81] |
87.26 |
33.94 [5.22] |
|||
> 55.16 |
32.32 |
75.78 |
30.87 [4.74] |
73.46 |
29.78 [5.14] |
|||
Percentage of staff who left |
< 8.70 |
27.17 |
81.03 |
0.643 |
28.45 [4.47] |
85.70 |
0.356 |
30.30 [5.25] |
8.70-14.39 |
40.14 |
86.41 |
40.21 [5.16] |
74.12 |
38.55 [5.85] |
|||
> 14.39 |
32.69 |
81.57 |
31.34 [4.70] |
83.91 |
31.16 [5.18] |
|||
Percentage of staff replaced |
< 64.10 |
34.86 |
86.35 |
0.712 |
35.76 [5.23] |
78.54 |
0.874 |
32.89 [5.48] |
64.10-95.95 |
30.77 |
82.30 |
29.17 [4.17] |
79.84 |
31.41 [4.59] |
|||
> 95.95 |
34.38 |
81.30 |
35.07 [5.17] |
83.01 |
35.70 [5.94] |
|||
Percentage of lead teachers who left |
< 8.82 |
29.52 |
82.78 |
0.658 |
30.39 [4.45] |
83.42 |
0.871 |
29.43 [5.00] |
8.82-19.44 |
24.55 |
79.84 |
24.70 [3.86] |
78.47 |
26.01 [4.29] |
|||
> 19.44 |
45.92 |
85.64 |
44.91 [5.38] |
79.60 |
44.56 [6.34] |
|||
Service type |
Center only |
84.50 |
82.23 |
0.246 |
82.39 [4.04] |
80.32 |
0.925 |
82.13 [5.03] |
Center + home |
15.50 |
89.56 |
17.61 [4.04] |
81.26 |
17.87 [5.03] |
|||
Continuous variables at program level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Program enrollment |
490.07 |
|
0.308 |
486.86 [38.43] |
|
0.439 |
548.38 [73.32] |
|
Proportion of children with a disability who are enrolled in program |
0.14 |
|
0.472 |
0.14 [0.006] |
|
0.038 |
0.14 [0.007] |
|
Proportion of children enrolled in program who are age 4+ |
0.52 |
|
0.056 |
0.51 [0.011] |
|
0.575 |
0.51 [0.013] |
|
Proportion of staff who left |
0.13 |
|
0.989 |
0.13 [0.010] |
|
0.784 |
0.13 [0.011] |
|
Proportion of staff replaced |
0.72 |
|
0.644 |
0.72 [0.032] |
|
0.387 |
0.75 [0.032] |
|
Proportion of lead teachers who left |
0.23 |
|
0.119 |
0.23 [0.030] |
|
0.871 |
0.23 [0.036] |
|
Note: Bolded p-values highlight values less than 0.05.
All continuous variables were also included as categorical (ordinal) variables, divided into tertiles (sometimes into binary variables) based on the full sample distribution.
a MSA refers to whether the program’s zip code was within a metropolitan statistical area.
Table 4. Nonresponse bias analysis at the center level
Variable |
Value |
Full sample estimate |
Study participation rate by subgroup |
Study participants vs. nonparticipants p-value |
Study participants’ estimate [standard error] |
Center director survey response rate by subgroup |
Center director survey respondents vs. nonrespondents p-value |
Center director survey respondents’ estimate [standard error] |
A |
B |
C |
D |
E |
F |
G |
H |
I |
Number of centers |
326 |
|
|
318 |
|
|
191 |
|
Weight |
Cumulative center base weight |
|
|
C_WT |
|
|
C2_WT |
|
Categorical variables at center level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Center’s child enrollment |
< 45 |
60.68 |
96.21 |
0.402 |
60.52 [3.61] |
57.51 |
0.663 |
58.76 [4.30] |
45-103 |
24.62 |
98.60 |
24.91 [2.99] |
60.25 |
25.31 [3.29] |
|||
> 103 |
14.70 |
98.06 |
14.57 [1.91] |
64.82 |
15.93 [2.52] |
|||
Center’s number of classrooms |
< 3 |
60.28 |
96.19 |
0.374 |
60.06 [3.68] |
57.94 |
0.829 |
58.76 [4.30] |
3-5 |
24.59 |
98.81 |
24.96 [3.06] |
60.47 |
24.89 [3.24] |
|||
> 5 |
15.13 |
97.76 |
14.98 [2.03] |
62.49 |
16.34 [2.71] |
|||
Categorical variables at program level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Program sample cohort |
Child-level (fall-spring) |
36.56 |
99.34 |
0.072 |
36.48 [2.65] |
60.16 |
0.863 |
36.48 [3.29] |
Classroom/program-level only (spring) |
63.44 |
95.76 |
63.52 [2.65] |
58.73 |
63.52 [3.29] |
|||
Program census region |
A |
22.70 |
93.30 |
|
22.70 [1.95] |
66.47 |
0.760 |
22.70 [2.75] |
B |
17.61 |
98.62 |
17.61 [2.57] |
58.89 |
17.61 [2.67] |
|||
C |
36.81 |
96.83 |
36.81 [2.26] |
55.48 |
36.81 [3.16] |
|||
D |
22.88 |
100.00 |
22.88 [1.74] |
58.45 |
22.88 [2.56] |
|||
MSAa |
Yes |
64.59 |
97.75 |
0.565 |
63.49 [3.32] |
59.57 |
0.920 |
64.69 [4.01] |
No |
35.41 |
95.83 |
36.51 [3.32] |
58.70 |
35.31 [4.01] |
|||
Program enrollment |
< 431 |
41.85 |
96.60 |
0.374 |
42.25 [4.70] |
58.99 |
0.367 |
41.13 [5.47] |
431-991 |
33.26 |
95.60 |
32.48 [4.81] |
53.79 |
30.45 [5.31] |
|||
> 991 |
24.89 |
99.83 |
25.27 [3.78] |
66.72 |
28.42 [4.99] |
|||
Percentage of children with a disability who are enrolled in program |
< 10.09 |
24.29 |
94.02 |
0.358 |
23.09 [3.50] |
61.46 |
0.198 |
23.91 [4.42] |
10.09-13.72 |
34.79 |
96.74 |
34.48 [4.50] |
50.73 |
31.16 [5.17] |
|||
> 13.72 |
40.92 |
99.16 |
42.44 [4.71] |
64.98 |
44.93 [5.54] |
|||
Percentage of children enrolled in program who are age 4+ |
< 50.9 |
45.70 |
99.91 |
<.0001 |
47.07 [4.76] |
57.59 |
0.682 |
46.53 [5.62] |
> = 50.9 |
54.30 |
94.68 |
52.93 [4.76] |
60.74 |
53.47 [5.62] |
|||
Percentage of staff who left |
< 10.67 |
49.48 |
97.32 |
0.880 |
49.21 [4.68] |
53.03 |
0.061 |
43.90 [4.87] |
> = 10.67 |
50.52 |
96.82 |
50.79 [4.68] |
65.29 |
56.10 [4.87] |
|||
Percentage of staff replaced |
< 64.10 |
33.84 |
99.84 |
0.245 |
34.93 [4.85] |
62.57 |
0.810 |
34.24 [5.56] |
64.10-94.12 |
31.50 |
95.21 |
31.16 [4.09] |
57.58 |
30.83 [4.84] |
|||
> 94.12 |
34.66 |
96.06 |
33.91 [4.63] |
57.38 |
34.93 [5.36] |
|||
Percentage of lead teachers who left |
< 11.76 |
48.54 |
94.07 |
<.0001 |
47.65 [4.66] |
57.10 |
0.575 |
47.49 [5.51] |
> = 11.76 |
51.46 |
99.90 |
52.35 [4.66] |
61.22 |
52.51 [5.51] |
|||
Service type |
Center only |
75.57 |
96.25 |
0.001 |
74.51 [4.73] |
60.73 |
0.548 |
77.87 [5.06] |
Center + home |
24.43 |
99.61 |
25.49 [4.73] |
54.93 |
22.13 [5.06] |
|||
Continuous variables at center level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Center’s child enrollment |
54.72 |
|
0.016 |
54.91 [3.26] |
|
0.557 |
57.01 [3.92] |
|
Center’s number of classrooms |
3.13 |
|
0.455 |
3.12 [0.18] |
|
0.800 |
3.21 [0.22] |
|
Continuous variables at program level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Program enrollment |
877.52 |
|
0.231 |
877.76 [115.82] |
|
0.379 |
942.54 [176.54] |
|
Proportion of children with a disability who are enrolled in program |
0.14 |
|
0.410 |
0.14 [0.01] |
|
0.547 |
0.14 [0.01] |
|
Proportion of children enrolled in program who are age 4+ |
0.52 |
|
0.000 |
0.52 [0.01] |
|
0.967 |
0.52 [0.01] |
|
Proportion of staff who left |
0.13 |
|
0.864 |
0.13 [0.01] |
|
0.239 |
0.13 [0.01] |
|
Proportion of staff replaced |
0.72 |
|
0.002 |
0.71 [0.03] |
|
0.916 |
0.73 [0.04] |
|
Proportion of lead teachers who left |
0.17 |
|
0.000 |
0.17 [0.02] |
|
0.448 |
0.18 [0.02] |
|
Note: Bolded p-values highlight values less than 0.05.
All continuous variables were also included as categorical (ordinal) variables, divided into tertiles (sometimes into binary variables) based on the full sample distribution.
aMSA refers to whether the program’s zip code was within a metropolitan statistical area.
Table 5. Nonresponse bias analysis at the classroom level
Variable |
Value |
Full sample estimate |
Teacher survey response rate by subgroup |
Teacher survey respondents vs. nonrespondents: p-value |
Teacher survey respondents’ estimate [standard error] |
A |
B |
C |
D |
E |
F |
Number of classrooms |
590 |
|
|
365 |
|
Weight |
Cumulative classroom sampling weight |
|
|
T2CLSWT |
|
Categorical variables at center level |
(Column percent) |
(Percent) |
|
(Column percent) |
|
Center’s child enrollment |
< 57 |
32.28 |
70.18 |
0.102 |
32.34 [3.65] |
57-111 |
30.17 |
56.03 |
29.96 [3.76] |
||
> 111 |
37.55 |
59.46 |
37.70 [4.51] |
||
Center’s number of classrooms |
1 |
62.81 |
57.15 |
0.111 |
62.48 [3.05] |
2 |
14.27 |
70.72 |
14.37 [2.97] |
||
> 2 |
22.92 |
69.38 |
23.15 [2.30] |
||
Categorical variables at program level |
(Column percent) |
(Percent) |
|
(Column percent) |
|
Program sample cohort |
Child-level (fall-spring) |
32.70 |
74.68 |
0.004 |
32.70 [1.61] |
Classroom/program-level only (spring) |
67.30 |
55.67 |
67.30 [1.61] |
||
Program census region |
A |
20.46 |
85.42 |
<.0001 |
20.46 [1.93] |
B |
18.12 |
68.67 |
18.12 [1.51] |
||
C |
39.98 |
49.38 |
39.98 [1.94] |
||
D |
21.44 |
57.00 |
21.44 [1.62] |
||
MSAa |
Yes |
74.13 |
62.75 |
0.618 |
74.72 [2.47] |
No |
25.87 |
59.42 |
25.28 [2.47] |
||
Program enrollment |
< 481 |
33.70 |
71.03 |
0.116 |
35.69 [4.33] |
481-997 |
32.41 |
54.91 |
31.21 [4.64] |
||
> 997 |
33.90 |
59.47 |
33.10 [4.61] |
||
Percentage of children with a disability who are enrolled in program |
< 10.03 |
28.79 |
51.81 |
0.023 |
28.27 [4.08] |
10.03-13.72 |
35.14 |
59.10 |
33.01 [4.69] |
||
> 13.72 |
36.07 |
72.65 |
38.72 [4.30] |
||
Percentage of children enrolled in program who are age 4+ |
< 45.98 |
32.16 |
58.49 |
0.103 |
33.60 [4.53] |
45.98-54.11 |
34.40 |
55.69 |
31.98 [4.55] |
||
> 54.11 |
33.44 |
71.53 |
34.42 [4.51] |
||
Percentage of staff who left |
< 8.75 |
30.64 |
59.90 |
0.592 |
30.08 [4.23] |
8.75-13.95 |
37.23 |
59.27 |
37.39 [4.66] |
||
> 13.95 |
32.14 |
66.81 |
32.53 [4.04] |
||
Percentage of staff replaced |
< 62.22 |
33.27 |
52.85 |
0.016 |
29.92 [4.27] |
62.22-94.05 |
33.14 |
73.72 |
35.10 [4.38] |
||
> 94.05 |
33.59 |
59.16 |
34.98 [4.52] |
||
Percentage of lead teachers who left |
< 8.82 |
37.49 |
65.09 |
0.634 |
36.48 [4.74] |
8.82-19.44 |
31.05 |
57.36 |
32.82 [4.35] |
||
> 19.44 |
31.46 |
62.53 |
30.70 [4.07] |
||
Service type |
Center only |
77.46 |
60.63 |
0.577 |
79.30 [4.09] |
Center + home |
22.54 |
66.19 |
20.70 [4.09] |
||
Continuous variables at enter level |
(Mean) |
|
|
(Mean) |
|
Center’s child enrollment |
107.45 |
|
0.955 |
108.73 [10.49] |
|
Center’s number of classrooms |
2.52 |
|
0.057 |
2.52 [0.20] |
|
Continuous variables at program level |
(Mean) |
|
|
(Mean) |
|
Program enrollment |
1101.2 |
|
0.285 |
1003.3 [111.12] |
|
Proportion of children with a disability who are enrolled in program |
0.13 |
|
0.001 |
0.13 [0.004] |
|
Proportion of children enrolled in program who are age 4+ |
0.50 |
|
0.099 |
0.50 [0.01] |
|
Proportion of staff who left |
0.13 |
|
0.159 |
0.13 [0.01] |
|
Proportion of staff replaced |
0.71 |
|
0.165 |
0.73 [0.03] |
|
Proportion of lead teachers who left |
0.17 |
|
0.525 |
0.17 [0.02] |
|
Note: Bolded p-values highlight values less than 0.05.
All continuous variables were also included as categorical (ordinal) variables, divided into tertiles (sometimes into binary variables) based on the full sample distribution.
aMSA refers to whether the program’s zip code was within a metropolitan statistical area.
Table 6.a. Nonresponse bias analysis at the child level: study participation
Variable |
Value |
Full sample estimate |
Fall participation rate by subgroup |
Fall study participants vs. nonparticipants p-value |
Fall participants’ estimate [standard error] |
A |
B |
C |
D |
E |
F |
Number of children |
2,482 |
|
|
2,260 |
|
Weight |
Cumulative child sampling weight |
|
|
CNST1WT |
|
Categorical variables at child level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Age group |
< 48 months |
49.42 |
90.62 |
0.835 |
49.27 [2.50] |
48+ months |
50.58 |
90.25 |
50.73 [2.50] |
||
Sex |
Female |
50.40 |
89.79 |
0.024 |
49.59 [1.11] |
Male |
49.60 |
93.02 |
50.41 [1.11] |
||
Language spoken at home |
English |
75.69 |
92.40 |
0.018 |
77.68 [4.20] |
Non-English |
24.31 |
84.30 |
22.32 [4.20] |
||
Months enrolled in Head Start |
< 2 |
42.52 |
87.29 |
0.002 |
42.21 [5.43] |
= 2 |
44.79 |
92.72 |
44.85 [4.04] |
||
> 2 |
11.80 |
93.59 |
12.07 [3.73] |
||
missing |
0.89 |
83.76 |
0.86 [0.30] |
||
Child participation in Early Head Start |
Don’t know |
13.47 |
89.68 |
0.496 |
13.41 [4.19] |
No |
71.13 |
90.92 |
71.30 [4.17] |
||
Yes |
15.40 |
88.83 |
15.29 [3.10] |
||
Primary funding source |
Child care subsidy or other |
2.80 |
92.36 |
0.559 |
2.80 [1.34] |
Head Start |
92.47 |
90.25 |
92.30 [3.18] |
||
State pre-K |
4.73 |
92.95 |
4.89 [2.87] |
||
Categorical variables at center level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Center’s child enrollment |
< 55 |
32.85 |
90.62 |
0.968 |
32.85 [6.63] |
55-117 |
33.16 |
90.55 |
33.16 [5.09] |
||
117 |
33.99 |
90.14 |
33.99 [5.02] |
||
Center’s number of classrooms |
< 4 |
34.35 |
90.91 |
0.861 |
34.35 [6.77] |
4-6 |
34.81 |
90.50 |
34.81 [5.34] |
||
> 6 |
30.85 |
89.82 |
30.85 [5.79] |
||
Categorical variables at program level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Program sampling strata |
Census Region A, MSA, < 40% Hispanic and < 40% Black enrollment |
5.33 |
82.91 |
0.020 |
5.33 [0.50] |
Census Region A, MSA, > 40% Hispanic or Black enrollment |
7.50 |
86.75 |
7.50 [1.39] |
||
Census Region A, Non-MSA |
4.35 |
92.55 |
4.35 [0.78] |
||
Census
Region B, < 40% Hispanic and < 40% |
10.32 |
84.57 |
10.32 [0.94] |
||
Census Region B, > 40% Hispanic or Black enrollment |
22.73 |
91.98 |
22.73 [1.80] |
||
Census
Region C, < 40% Hispanic and < 40% |
8.89 |
91.73 |
8.89 [1.20] |
||
Census Region C, MSA, > 40% Hispanic or Black enrollment |
20.83 |
92.75 |
20.83 [2.20] |
||
Census Region C, Non-MSA, > 40% Hispanic or Black enrollment |
6.31 |
97.22 |
6.31 [0.71] |
||
Census Region D, MSA, > 40% Hispanic or Black enrollment |
6.49 |
90.11 |
6.49 [0.75] |
||
Census Region D, Other |
7.25 |
88.12 |
7.25 [1.36] |
||
MSAa |
Yes |
84.69 |
89.87 |
0.169 |
84.69 [3.27] |
No |
15.31 |
93.57 |
15.31 [3.27] |
||
Program enrollment |
< 448 |
32.43 |
87.31 |
0.015 |
32.43 [6.35] |
448-1011 |
35.35 |
92.84 |
35.35 [8.21] |
||
> 1011 |
32.22 |
90.93 |
32.22 [7.89] |
||
Percentage of children with a disability who are enrolled in program |
< 10.02 |
29.00 |
90.60 |
0.691 |
29.00 [6.60] |
10.02-14.29 |
32.64 |
89.37 |
32.64 [7.51] |
||
> 14.29 |
38.35 |
91.21 |
38.35 [8.15] |
||
Percentage of children enrolled in program who are age 4+ |
< 43.72 |
31.58 |
91.36 |
0.691 |
31.58 [7.26] |
43.72-53.78 |
27.99 |
89.14 |
28.00 [5.79] |
||
> 53.78 |
40.43 |
90.61 |
40.42 [7.78] |
||
Percentage of staff who left |
< 9.09 |
29.83 |
91.36 |
0.782 |
29.83 [6.96] |
9.09-13.51 |
44.78 |
89.78 |
44.79 [7.34] |
||
> 13.51 |
25.39 |
90.50 |
25.38 [6.63] |
||
Percentage of staff replaced |
< 60 |
25.99 |
92.23 |
0.436 |
25.99 [5.16] |
60-96.55 |
40.55 |
89.26 |
40.57 [7.45] |
||
> 96.55 |
33.46 |
90.46 |
33.44 [7.50] |
||
Percentage of lead teachers who left |
< 8.82 |
35.29 |
89.58 |
0.684 |
35.29 [8.02] |
8.82-20.53 |
36.44 |
91.60 |
36.45 [7.91] |
||
> 20.53 |
28.27 |
90.01 |
28.26 [7.24] |
||
Service type |
Center only |
82.83 |
90.32 |
0.959 |
82.83 [6.34] |
Center + home |
17.17 |
90.17 |
17.17 [6.34] |
||
Continuous variables at child level |
(Mean) |
|
|
(Mean) |
|
Age in months |
47.83 |
|
0.736 |
47.86 [0.40] |
|
Months enrolled in Head Start |
2.82 |
|
0.060 |
2.86 [0.44] |
|
Continuous variables at center level |
(Mean) |
|
|
(Mean) |
|
Center’s child enrollment |
100.25 |
|
0.137 |
100.25 [8.93] |
|
Center’s number of classrooms |
5.65 |
|
0.179 |
5.65 [0.53] |
|
Continuous variables at program level |
(Mean) |
|
|
(Mean) |
|
Program enrollment |
1220.7 |
|
0.431 |
1220.7 [313.11] |
|
Proportion of children with a disability who are enrolled in program |
0.14 |
|
0.925 |
0.14 [0.01] |
|
Proportion of children enrolled in program who are age 4+ |
0.53 |
|
0.889 |
0.53 [0.03] |
|
Proportion of staff who left |
0.18 |
|
0.836 |
0.18 [0.04] |
|
Proportion of staff replaced |
0.72 |
|
0.170 |
0.72 [0.04] |
|
Proportion of lead teachers who left |
0.21 |
|
0.921 |
0.21 [0.04] |
|
Note: Bolded p-values highlight values less than 0.05.
All continuous variables were also included as categorical (ordinal) variables, divided into tertiles (sometimes into binary variables) based on the full sample distribution.
aMSA refers to whether the program’s zip code was within a metropolitan statistical area.
Table 6.b. Nonresponse bias analysis at the child level: parent surveys
Variable |
Value |
Full sample estimate |
Fall + Spring parent survey response rate by subgroup |
Fall + Spring parent survey respondents vs. nonrespondents p-value |
Fall + Spring parent survey respondents’ estimate [standard error] |
Fall or Spring parent survey response rate by subgroup |
Fall or Spring parent survey respondents vs. nonrespondents p-value |
Fall or Spring parent survey respondents’ estimate [standard error] |
A |
B |
C |
D |
E |
F |
G |
H |
I |
Number of children |
2,132 |
|
|
1,314 |
|
|
1,746 |
|
Weight |
CNST2WTa |
|
|
P12WT |
|
|
P1_2WT |
|
Categorical variables at child level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Age group |
< 48 months |
49.60 |
61.89 |
0.727 |
50.00 [3.17] |
81.18 |
0.917 |
49.21 [2.69] |
48+ months |
50.40 |
60.68 |
50.00 [3.17] |
81.37 |
50.79 [2.69] |
|||
Sex |
Female |
49.13 |
60.27 |
0.403 |
47.98 [1.65] |
80.78 |
0.609 |
48.78 [1.34] |
Male |
50.87 |
62.26 |
52.02 [1.65] |
81.75 |
51.22 [1.34] |
|||
Language spoken at home |
English |
77.29 |
58.53 |
0.006 |
76.43 [4.41] |
79.57 |
0.046 |
77.02 [4.29] |
Non-English |
22.71 |
70.64 |
23.57 [4.41] |
87.07 |
22.98 [4.29] |
|||
Months enrolled in Head Start |
< 2 |
42.99 |
57.87 |
0.457 |
43.33 [5.47] |
78.33 |
0.300 |
42.89 [5.46] |
= 2 |
44.87 |
63.12 |
44.48 [4.25] |
83.21 |
44.71 [4.20] |
|||
> 2 |
12.13 |
66.58 |
12.19 [3.72] |
84.56 |
12.40 [3.69] |
|||
Child participation in Early Head Start |
Don’t know |
13.12 |
59.35 |
0.834 |
13.07 [4.07] |
79.38 |
0.434 |
13.08 [4.07] |
No |
71.11 |
61.42 |
71.34 [4.34] |
81.10 |
71.34 [4.16] |
|||
Yes |
15.77 |
62.28 |
15.59 [3.31] |
83.63 |
15.58 [3.21] |
|||
Primary funding source |
Child care subsidy or Other |
2.71 |
55.30 |
0.031 |
2.59 [1.27] |
74.69 |
0.001 |
2.49 [1.23] |
Head Start |
92.42 |
61.82 |
92.79 [3.12] |
81.93 |
92.78 [3.12] |
|||
State pre-K |
4.88 |
54.46 |
4.61 [2.86] |
72.40 |
4.73 [2.87] |
|||
Categorical variables at center level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Center’s child enrollment |
< 55 |
32.55 |
61.64 |
0.981 |
32.55 [6.59] |
81.97 |
0.931 |
32.55 [6.59] |
55-117 |
34.04 |
61.39 |
34.04 [5.17] |
81.32 |
34.04 [5.17] |
|||
> 117 |
33.40 |
60.82 |
33.40 [5.14] |
80.55 |
33.40 [5.14] |
|||
Center’s number of classrooms |
< 4 |
33.95 |
61.34 |
0.923 |
33.95 [6.73] |
81.82 |
0.714 |
33.95 [6.73] |
4-6 |
34.93 |
62.01 |
34.93 [5.36] |
82.24 |
34.93 [5.36] |
|||
> 6 |
31.12 |
60.40 |
31.12 [5.76] |
79.59 |
31.12 [5.76] |
|||
Categorical variables at program level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Program census region |
A |
16.99 |
64.66 |
0.262 |
16.99 [1.69] |
85.26 |
0.091 |
16.99 [1.69] |
B |
32.82 |
56.61 |
32.82 [2.14] |
76.90 |
32.82 [2.14] |
|||
C |
36.10 |
62.46 |
36.10 [2.34] |
82.58 |
36.10 [2.34] |
|||
D |
14.09 |
65.07 |
14.09 [1.60] |
83.31 |
14.09 [1.60] |
|||
MSAb |
Yes |
84.76 |
60.87 |
0.456 |
84.76 [3.39] |
80.50 |
0.020 |
84.76 [3.39] |
No |
15.24 |
63.56 |
15.24 [3.39] |
85.59 |
15.24 [3.39] |
|||
Program enrollment |
< 449 |
36.42 |
63.70 |
0.476 |
36.42 [7.71] |
85.56 |
0.054 |
36.42 [7.71] |
449-1011 |
31.23 |
58.37 |
31.23 [8.05] |
77.89 |
31.23 [8.05] |
|||
> 1011 |
32.35 |
61.37 |
32.35 [7.91] |
79.71 |
32.35 [7.91] |
|||
Percentage of children with a disability who are enrolled in program |
< 10.11 |
30.11 |
66.46 |
0.074 |
30.11 [6.59] |
85.71 |
0.022 |
30.11 [6.59] |
10.11-14.28 |
31.46 |
58.83 |
31.46 [7.39] |
80.57 |
31.46 [7.39] |
|||
> 14.28 |
38.43 |
59.24 |
38.43 [8.22] |
78.38 |
38.43 [8.22] |
|||
Percentage of children enrolled in program who are age 4+ |
< 43.72 |
32.03 |
61.13 |
0.252 |
32.03 [7.35] |
81.15 |
0.958 |
32.03 [7.35] |
43.72-52.65 |
26.51 |
56.49 |
26.51 [5.59] |
82.08 |
26.51 [5.59] |
|||
> 52.65 |
41.46 |
64.46 |
41.46 [7.63] |
80.85 |
41.46 [7.63] |
|||
Percentage of staff who left |
< 8.82 |
32.07 |
64.63 |
0.001 |
32.07 [7.36] |
81.84 |
0.245 |
32.07 [7.36] |
8.82-12.12 |
41.30 |
64.05 |
41.30 [7.10] |
83.48 |
41.30 [7.10] |
|||
> 12.12 |
26.63 |
52.95 |
26.63 [7.34] |
77.16 |
26.63 [7.34] |
|||
Percentage of staff replaced |
< 57.58 |
28.59 |
62.55 |
0.056 |
28.59 [6.44] |
81.84 |
0.442 |
28.59 [6.44] |
57.58-94.05 |
37.96 |
56.84 |
37.96 [7.46] |
79.08 |
37.96 [7.46] |
|||
> 94.05 |
33.44 |
65.24 |
33.44 [7.51] |
83.28 |
33.44 [7.51] |
|||
Percentage of lead teachers who left |
< 8.51 |
35.86 |
66.73 |
0.018 |
35.86 [7.91] |
84.69 |
0.166 |
35.86 [7.91] |
8.51-15.68 |
36.39 |
57.32 |
36.39 [8.10] |
78.81 |
36.39 [8.10] |
|||
> 15.68 |
27.75 |
59.44 |
27.75 [7.10] |
80.09 |
27.75 [7.10] |
|||
Service type |
Center only |
84.13 |
62.14 |
0.119 |
84.13 [5.90] |
81.52 |
0.751 |
84.13 [5.90] |
Center + home |
15.87 |
56.73 |
15.87 [5.90] |
79.98 |
15.87 [5.90] |
|||
Continuous variables at child level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Age in months |
47.66 |
|
0.911 |
47.58 [0.43] |
|
0.310 |
47.73 [0.39] |
|
Months enrolled in Head Start |
2.86 |
|
0.718 |
2.79 [0.41] |
|
0.347 |
2.89 [0.42] |
|
Continuous variables at center level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Center’s child enrollment |
101.03 |
|
0.733 |
101.03 [8.97] |
|
0.914 |
101.03 [8.97] |
|
Center’s number of classrooms |
5.69 |
|
0.624 |
5.69 [0.53] |
|
0.893 |
5.69 [0.53] |
|
Continuous variables at program level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Program enrollment |
1208.6 |
|
0.673 |
1208.6 [302.83] |
|
0.434 |
1208.6 [302.83] |
|
Proportion of children with a disability who are enrolled in program |
0.14 |
|
0.498 |
0.14 [0.01] |
|
0.494 |
0.14 [0.01] |
|
Proportion of children enrolled in program who are age 4+ |
0.50 |
|
0.684 |
0.50 [0.02] |
|
0.594 |
0.50 [0.02] |
|
Proportion of staff who left |
0.11 |
|
0.004 |
0.11 [0.01] |
|
0.668 |
0.11 [0.01] |
|
Proportion of staff replaced |
0.70 |
|
0.241 |
0.70 [0.04] |
|
0.134 |
0.70 [0.04] |
|
Proportion of lead teachers who left |
0.14 |
|
0.112 |
0.14 [0.02] |
|
0.184 |
0.14 [0.02] |
|
Note: Bolded p-values highlight values less than 0.05.
All continuous variables were also included as categorical (ordinal) variables, divided into tertiles (sometimes into binary variables) based on the full sample distribution.
a Fall consent weight adjusted for spring participation in Head Start.
b MSA refers to whether the program’s zip code was within a metropolitan statistical area.
Table 6.c. Nonresponse bias analysis at the child level: survey combinations
Variable |
Value |
Full sample estimate |
Parent + Spring TCR response rate by subgroup |
Parent + Spring TCR respondents vs. nonrespondents p-value |
Parent + Spring TCR respondents’ estimate [standard error] |
Parent + Both Fall and Spring TCRa Response Rate by Subgroup |
Parent + Both Fall and Spring TCR respondents vs. nonrespondents p-value |
Parent + Both Fall and Spring TCR respondents’ estimate [standard error] |
A |
B |
C |
D |
E |
F |
G |
H |
I |
Number of children |
2,132 |
|
|
1,223 |
|
|
1,162 |
|
Weight |
CNST2WT |
|
|
P21R2WT |
|
|
PR12WT |
|
Categorical variables at child level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Age group |
< 48 months |
49.60 |
56.62 |
0.171 |
50.73 [3.48] |
52.65 |
0.178 |
52.22 [4.47] |
48+ months |
50.40 |
62.45 |
49.27 [3.48] |
58.15 |
47.78 [4.47] |
|||
Sex |
Female |
49.13 |
61.07 |
0.156 |
49.13 [1.12] |
57.68 |
0.058 |
49.13 [1.12] |
Male |
50.87 |
58.10 |
50.87 [1.12] |
53.24 |
50.87 [1.12] |
|||
Language spoken at home |
English |
77.29 |
56.95 |
0.136 |
76.85 [4.78] |
54.46 |
0.500 |
76.86 [4.70] |
Non-English |
22.71 |
68.43 |
23.15 [4.78] |
58.71 |
23.14 [4.70] |
|||
Months enrolled in Head Start |
< 2 |
42.99 |
57.73 |
0.466 |
43.03 [5.58] |
54.14 |
0.915 |
43.42 [5.31] |
= 2 |
44.87 |
57.77 |
43.99 [4.31] |
55.63 |
43.98 [4.30] |
|||
> 2 |
12.13 |
72.65 |
12.97 [3.89] |
59.18 |
12.61 [3.55] |
|||
Child participation in Early Head Start |
Don’t know |
13.12 |
52.03 |
0.624 |
14.54 [4.46] |
50.54 |
0.812 |
14.54 [4.46] |
No |
71.11 |
60.43 |
68.55 [4.26] |
55.96 |
68.16 [4.26] |
|||
Yes |
15.77 |
61.90 |
16.91 [3.27] |
57.03 |
17.30 [3.32] |
|||
Primary funding source |
Child care subsidy or other |
2.71 |
35.50 |
0.134 |
1.69 [0.89] |
35.50 |
0.253 |
1.69 [0.89] |
Head Start |
92.42 |
61.60 |
96.27 [1.48]c |
57.16 |
96.28 [1.48]c |
|||
State pre-K |
4.88 |
34.18 |
2.04 [1.04]c |
33.59 |
2.03 [1.03]c |
|||
Categorical variables at center level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Center’s child enrollment |
< 55 |
32.55 |
68.76 |
0.172 |
32.46 [6.66] |
65.12 |
0.057 |
32.60 [6.66] |
55-117 |
34.04 |
55.160 |
31.13 [5.73] |
48.86 |
31.13 [5.73] |
|||
> 117 |
33.40 |
55.08 |
36.42 [6.90] |
52.67 |
36.27 [6.91] |
|||
Center’s number of classrooms |
< 4 |
33.95 |
66.83 |
0.377 |
34.44 [6.82] |
63.34 |
0.211 |
34.59 [6.83] |
4-6 |
34.93 |
56.16 |
30.19 [5.67] |
49.89 |
30.19 [5.67] |
|||
> 6 |
31.12 |
55.43 |
35.37 [7.42] |
52.99 |
35.23 [7.42] |
|||
Categorical variables at program level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Program census region |
A |
16.99 |
76.52 |
0.328 |
16.99 [1.69] |
73.93 |
0.179 |
16.99 [1.69] |
B |
32.82 |
58.02 |
32.82 [2.15] |
49.98 |
32.82 [2.15] |
|||
C |
36.10 |
52.54 |
36.10 [2.35] |
50.75 |
36.10 [2.35] |
|||
D |
14.09 |
60.66 |
14.09 [1.68] |
57.76 |
14.09 [1.68] |
|||
MSAb |
Yes |
84.76 |
57.97 |
0.257 |
84.76 [3.39] |
53.61 |
0.162 |
84.76 [3.39] |
No |
15.24 |
68.38 |
15.24 [3.39] |
65.48 |
15.24 [3.39] |
|||
Program enrollment |
< 449 |
36.42 |
73.15 |
0.003 |
36.03 [7.73] |
65.46 |
0.024 |
36.03 [7.73] |
449-1011 |
31.23 |
57.81 |
30.81 [8.05] |
57.07 |
30.81 [8.05] |
|||
> 1011 |
32.35 |
45.95 |
33.16 [7.93] |
42.53 |
33.16 [7.93] |
|||
Percentage of children with a disability who are enrolled in program |
< 10.11 |
30.11 |
64.16 |
0.734 |
30.42 [6.62] |
55.00 |
0.987 |
30.42 [6.62] |
10.11-14.28 |
31.46 |
58.79 |
31.33 [7.42] |
56.37 |
31.33 [7.42] |
|||
> 14.28 |
38.43 |
56.58 |
38.25 [8.25] |
54.97 |
38.25 [8.25] |
|||
Percentage of children enrolled in program who are age 4+ |
< 43.72 |
32.03 |
44.87 |
0.006 |
32.47 [7.38] |
42.55 |
0.014 |
32.47 [7.38] |
43.72-52.65 |
26.51 |
55.39 |
25.94 [5.65] |
51.88 |
25.94 [5.65] |
|||
> 52.65 |
41.46 |
73.56 |
41.59 [7.66] |
67.63 |
41.59 [7.66] |
|||
Percentage of staff who left |
< 8.82 |
32.07 |
64.53 |
0.004 |
30.98 [7.35] |
63.88 |
0.005 |
30.98 [7.35] |
8.82-12.12 |
41.30 |
68.02 |
41.63 [7.15] |
60.58 |
41.63 [7.15] |
|||
> 12.12 |
26.63 |
40.45 |
27.38 [7.36] |
37.24 |
27.38 [7.36] |
|||
Percentage of staff replaced |
< 57.58 |
28.59 |
65.95 |
0.370 |
28.03 [6.43] |
63.61 |
0.341 |
28.03 [6.43] |
57.58-94.05% |
37.96 |
52.30 |
38.50 [7.50] |
49.10 |
38.50 [7.50] |
|||
> 94.05 |
33.44 |
62.33 |
33.47 [7.56] |
55.59 |
33.47 [7.56] |
|||
Percentage of lead teachers who left |
< 8.51 |
35.86 |
74.08 |
0.002 |
35.86 [7.91] |
67.39 |
0.015 |
35.86 [7.91] |
8.51-15.68 |
36.39 |
44.58 |
37.14 [8.12] |
42.19 |
37.14 [8.12] |
|||
> 15.68 |
27.75 |
60.43 |
27.00 [7.14] |
57.30 |
27.00 [7.14] |
|||
Service type |
Center only |
84.13 |
58.16 |
0.286 |
83.80 [5.95] |
53.51 |
0.102 |
83.80 [5.95] |
Center + home |
15.87 |
66.97 |
16.20 [5.95] |
65.58 |
16.20 [5.95] |
|||
Continuous variables at child level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Age in months |
47.66 |
|
0.027 |
47.59 [0.45] |
|
0.026 |
47.43 [0.58] |
|
Months enrolled in Head Start |
2.86 |
|
0.120 |
2.93 [0.44] |
|
0.471 |
2.89 [0.41] |
|
Continuous variables at center level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Center’s child enrollment |
101.03 |
|
0.636 |
103.72 [10.34] |
|
0.650 |
102.98 [10.33] |
|
Center’s number of classrooms |
5.69 |
|
0.584 |
5.86 [0.61] |
|
0.676 |
5.82 [0.61] |
|
Continuous variables at program level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Program enrollment |
1208.6 |
|
0.030 |
1216.2 [302.97] |
|
0.019 |
1216.2 [302.97] |
|
Proportion of children with a disability who are enrolled in program |
0.14 |
|
0.839 |
0.14 [0.01] |
|
0.481 |
0.14 [0.01] |
|
Proportion of children enrolled in program who are age 4+ |
0.50 |
|
0.013 |
0.50 [0.02] |
|
0.024 |
0.50 [0.02] |
|
Proportion of staff who left |
0.11 |
|
0.095 |
0.11 [0.01] |
|
0.069 |
0.11 [0.01] |
|
Proportion of staff replaced |
0.70 |
|
0.923 |
0.70 [0.04] |
|
0.515 |
0.70 [0.04] |
|
Proportion of lead teachers who left |
0.14 |
|
0.180 |
0.14 [0.02] |
|
0.228 |
0.14 [0.02] |
|
Note: Bolded p-values highlight values less than 0.05.
All continuous variables were also included as categorical (ordinal) variables, divided into tertiles (sometimes into binary variables) based on the full sample distribution.
aNote that this is equivalent to the weight for parent survey + both fall and spring TCR + teacher survey.
bMSA refers to whether the program’s zip code was within a metropolitan statistical area.
cThe values of this variable represent the only estimates for which the full sample value did not fall within two standard errors (for both weights presented in this table).
APPENDIX
X.2
AIAN FACES 2019 NONRESPONSE BIAS ANALYSIS SUMMARY MEMO
This page has been left blank for double-sided copying.
To: Meryl Barofsky and Laura Hoard
From: Barbara Lepidus Carlson, Ian Huff, and Cathy Lu
Subject: AIAN FACES 2019–2020 Nonresponse Bias Analysis Report
We conducted a nonresponse bias analysis on the AIAN FACES 2019 data collected in fall 2019 and spring 2020 at the program, center, classroom, and child levels. Rather than doing a separate analysis for each data collection instrument, we looked at combinations of instrument completes that corresponded to our weighting definitions. These combinations revealed study participation or instrument completion rates that fell below 80 percent. We conducted the analyses discussed in this document to establish confidence in our weighted estimates even though the participation or completion rates were below 80 percent. We describe the response patterns for AIAN FACES 2019, the purposes of nonresponse bias analysis, and the methodological approach we used. Finally, we present the results of the analysis and our conclusions.
Impact of the COVID pandemic. In the first round of AIAN FACES, the study achieved high response rates at the staff, parent, and child levels. Spring 2020 data collection for AIAN FACES 2019 began in early March, at around the same time that COVID-19 (for coronavirus disease 2019) was declared a pandemic by the World Health Organization and a public health emergency by the United States (Centers for Disease Control and Prevention 2020). In response to the COVID-19 pandemic, AIAN FACES suspended the in-person collection of child assessments and classroom observations in spring 2020. However, staff and parent surveys were still pursued in all programs. These surveys included the program director survey, the center director survey, the teacher survey, the Teacher Child Report (TCR), and the spring parent survey. The participation and response rates for these surveys were lower than they were in AIAN FACES 2015 due to the COVID-19 pandemic.
Response rates. Unweighted and weighted response rates for the different survey instruments are in Table 1. The unweighted marginal response rate is the unadjusted percentage of eligible respondents who completed the survey for the specific instrument. For example, the 81.8 percent unweighted response rate for the program director survey is the result of dividing the number of completed responses to the director survey (18) by the total number of programs participating in the study (22). The weighted cumulative response rate, on the other hand, is the percentage of eligible respondents who completed the survey, weighted for the probability of selection and incorporating any prior sampling stages’ weights (if applicable). So, for example, although we obtained 69.4 percent of the teacher surveys we attempted (59 out of 85), we estimate that these completed surveys represent 41.2 percent of the population of study-eligible Head Start classrooms in Region XI.
Table 1. Response rates
Respondent level |
Instrument |
Eligiblea |
Completed |
Unweighted marginal response rate (percent) |
Weighted cumulative response rate (percent) |
Program |
Program participation |
41 |
22 |
53.7 |
60.7 |
Program director survey |
22 |
18 |
81.8 |
51.7 |
|
Center |
Center participation |
41 |
40 |
97.6 |
59.9 |
Center director survey |
40 |
27 |
67.5 |
42.9 |
|
Classroom |
Teacher survey |
85 |
59 |
69.4 |
41.2 |
Child |
Fall child participation (among those still in sampled program at data collection) |
963 |
720 |
74.8 |
44.5 |
Fall child participation (among those in sampled program at time of sampling) |
981 |
720 |
73.4 |
43.7 |
|
Fall parent survey |
720 |
538 |
74.7 |
33.3 |
|
Fall Teacher Child Report |
720 |
634 |
88.1 |
38.9 |
|
Fall child assessment |
720 |
619 |
86.0 |
38.3 |
|
Spring child participation (among those estimated to still be in sampled program) |
917.5 |
686 |
74.8 |
44.5 |
|
Spring child participation (among those estimated to still be receiving Head Start services) |
940.1 |
686 |
73.0 |
44.2 |
|
Spring parent survey |
686 |
460 |
67.1 |
29.6 |
|
Spring Teacher Child Report |
686 |
481 |
70.1 |
32.0 |
a For spring participation at the child level, we estimated that a certain proportion of the fall’s nonparticipating cases would have left Head Start and become ineligible.
Nonresponse bias can occur when the survey responses of nonrespondents would have been different enough from those of respondents to change the overall results—that is, to bias them. Although a lower response rate does not necessarily indicate the presence of nonresponse bias, a higher response rate does lower the risk of nonresponse bias. Nonresponse bias itself can rarely be measured directly, because we generally do not know what the missing responses would have been. Instead, we examine variables that are available for both respondents and nonrespondents and that are presumably correlated with the survey items that are missing for some sample members. In a nonresponse bias analysis, we compare the distributions or means of these characteristics for respondents and nonrespondents, identifying any potentially problematic differences, and use statistical tests to indicate whether the differences are likely due to something other than sampling error. We adjust weights for differential response patterns with the goal of mitigating the risk of nonresponse bias, and then assess whether the adjusted weights appear to have diminished those differences without having introduced larger differences in other variables.
Weighting overview. We evaluated nonresponse bias for each AIAN FACES 2019 weight we produced for spring and fall-spring estimates. When we compare respondents to nonrespondents, we use the weights that account for sampling at the current stage, and for sampling and study participation at prior sampling stages. By definition, however, the nonrespondents do not have weights that account for nonresponse at the current stage, as they are assigned a value of zero. Only when we examine the fully weighted distributions and means for respondents do we use the final weights, which are adjusted for nonresponse at the stage in question.
There are two general methods we use to adjust weights for nonresponse. One divides the sample into weighting cells based on one or more characteristics that are (1) available for all sample members, (2) plausibly related to key outcomes, and (3) plausibly related to the likelihood of responding, and inflates the respondent sampling weights to account for the nonrespondents in each cell. The other method uses a logistic regression model to predict the likelihood of responding. This method generally uses more characteristics and multiplies the inverse of the resulting propensity scores to the respondent sampling weights to account for the nonrespondents. We opted to use weighting cells for all adjustments because that method had better indicators for nonresponse bias correction for AIAN FACES 2019 data than the model-based approach did.
Although we do not produce analysis weights for AIAN FACES 2019 at anything other than the child level, we do produce sampling weights accounting for study nonparticipation at the program, center, and classroom levels to use as building blocks for child-level weights. At the program level, we start with the sampling weight for selecting programs within stratum with probability proportional to size. For program-level weights, we then adjust for study participation. This program-level, participation-adjusted weight is used as a building block for center-, classroom-, and child-level weights.
Similarly, at the center level, we start with the sampling weight for selecting centers within program with probability proportional to size. We then bring in the program-level building block weight, applying it to each sampled center. For center-level weights, we then adjust this cumulative weight for study participation. We use the cumulative center-level participation-adjusted weight as a building block for classroom- and child-level weights. We construct classroom-level weights in the same way, except that classrooms were selected within centers with equal probability, and we apply the cumulative center-level weight to the classroom sampling weight and use this as the building block for the child-level weights.
Finally, at the child level, we start with the sampling weight for selecting children within classrooms with equal probability. We then bring in the classroom-level cumulative weight, applying it to the sampled children. Within center,9 we adjust this weight for parental consent in the fall. In the spring, any children who are no longer being served by Head Start are ineligible for the study so they are simply dropped from analyses, and any children who are known to be served by Head Start but who left the sampled program are adjusted for in the weights. We then create a series of weights that accounts for various combinations of instrument completes; combinations are described in detail below.
Covariates used in nonresponse bias analysis. To conduct a nonresponse bias analysis, we rely on variables (covariates) that are available for both respondents and nonrespondents and that are plausibly correlated with key survey outcomes. Although most covariates are categorical, there are a few continuous ones. For the continuous covariates, we created ordinal versions as well, often using the 33rd and 67th unweighted percentiles of the respondents and nonrespondents combined as the cut points, but sometimes dividing these continuous variables into binary variables based on analytic reporting (for example, child age).
We analyzed categorical and continuous program-, center-, and child-level variables for indications of potential bias due to nonresponse. We evaluated program-level covariates (mostly from the Head Start Program Information Report) at the program and child levels. We evaluated center-level covariates (obtained from the Head Start programs as part of the center sampling process) and child-level covariates (obtained from the centers on the child sampling rosters or from parents on the consent form) at the child level only.10 Although we did not use these variables directly to adjust weights for nonresponse, recall that we used the center—or center crossed with child’s sex—as the primary weighting cell (sometimes using program or program stratum as weighting cells as needed). We analyzed the following program-level covariates:
Geographic area of the program (masked here as regions A, B, C, D, and E to minimize the risk for data disclosure)
Whether the program zip code is in a metropolitan statistical area
Size of the program by total enrollment
Percentage of program enrollees who are American Indian or Alaska Native (AIAN)
Percentage of program enrollees with a disability
Percentage of program enrollees age 4 years or older
Percentage of program staff who left in the past year
Percentage of program staff replaced in the past year
Percentage of program lead teachers who left in the past year
Program service type
Size of the center by total enrollment
Size of the center by number of classrooms
Child age (less than 48 months versus 48 months or older)
Child’s sex
Language spoken at home (English, Tribal, or Other)
Number of months child has been enrolled in Head Start
Whether the child participated in Early Head Start
Primary funding source (Head Start, state prekindergarten, Tribal, and/or child subsidies and other sources)
Steps in the nonresponse bias analysis. For each covariate that is a categorical or ordinal variable, we compare the weighted distributions across categories for respondents and nonrespondents, running a design-adjusted chi-square test.11 We then check to see if the full sample percentage is within two standard errors of the final nonresponse-adjusted–weighted estimated percentage for respondents only. Similarly, for each covariate that is a continuous variable, we compare the weighted means for respondents and nonrespondents, running a design-adjusted t-test.12 We then check to see if the full sample mean is within two standard errors of the final nonresponse-adjusted–weighted mean for respondents only. One caution for AIAN FACES 2019, particularly for the program-level estimates, is that the standard errors around the final estimates can be relatively large due to smaller sample sizes, which reduces the power to detect instances in which the population estimate falls outside the two-standard-error range.
We assessed multiple weights for potential nonresponse bias at the program and child levels. At the program level, we have the base weight, D_WT, which accounts for program participation. Because AIAN FACES 2019 was not designed for estimates other than those at the child level, we assessed a series of child-level weights for instruments collected (in spring only) at the program, center, and classroom levels as well as those collected as the child level (collected in fall or spring).
We have a child-level base weight, CNST2WT, which accounts for parental consent and retention in the study, for comparison against several child-level instrument-based weights. The first set of child-level weights examine instruments collected at the staff level: PD_CHILDWT accounts for whether the child’s program has a complete program director survey; CD_CHILDWT accounts for whether the child’s center has a complete center director survey; TS_CHILDWT accounts for whether the child’s classroom has a complete teacher survey; and P21RC2WT accounts for whether the child’s classroom has a completed teacher survey in addition to the child having a completed parent survey (in fall or spring) and a completed TCR in spring. Next, we assess the weight that accounts for whether the child had parental consent (CNST1WT). We then assess a series of weights for children who were still in the study in spring 2020. The first such weight has a non-zero value for children with a parent survey in fall and spring, P12WT, and the second weight has a non-zero value for children with a parent survey in fall or spring, P1_2WT. The other three weights are P21R2WT, for those with a parent survey in fall or spring as well as a TCR in spring; PR12WT, for those with a parent survey in fall or spring as well as a TCR in fall and spring; and PR12CW, for those with a parent survey in fall or spring, a teacher survey in the spring, and a TCR in fall and spring. Table 2 shows the various nonresponse bias analyses carried out for this report.
Table 2. Weights associated with nonresponse bias analyses performed for AIAN FACES 2019
Level |
Time point(s) of information gathered |
Weight name |
Weight description |
Program |
Fall only |
D_WT |
Program participation base weight |
Child |
Fall only |
CNST1WT |
Child participation (consent) base weight |
Child |
Spring only |
PD_CHILDWT |
Program director survey weight |
Child |
Spring only |
CD_CHILDWT |
Center director survey weight |
Child |
Spring only |
TS_CHILDWT |
Teacher survey weight |
Child |
Fall and/or spring |
P21RC2WT |
Weight for those with parent survey in fall or spring plus teacher survey and TCR in spring |
Fall and/or spring |
P12WT |
Weight for those with parent survey in fall and spring |
|
Fall and/or spring |
P1_2WT |
Weight for those with parent survey in fall or spring |
|
Fall and/or spring |
P21R2WT |
Weight for those with parent survey in fall or spring plus TCR in spring |
|
Fall and/or spring |
PR12WT |
Weight for those with parent survey in fall or spring plus TCR in fall and spring |
|
Fall and/or spring |
PR12CW |
Weight for those with parent survey in fall or spring, plus TCR in fall and spring, plus teacher survey |
TCR = Teacher Child Report.
Tables 3–6.c compare weighted estimates for respondents and nonrespondents (before nonresponse weighting adjustments) in observed covariates for our various weight-defined respondent definitions. This makes it possible for us to assess the risk for nonresponse bias in estimates based only on respondents. Those estimates are followed by estimates of these same covariates that are based on respondents only with a final weight adjusted for nonresponse. Each table contains information on one, two, or three respondent definitions and the associated final weights.
Column A in each table contains the variable name. Column B contains the values of each categorical or ordinal variable. Column C in each table shows the distribution of categorical variable values and the mean of continuous variables for the full sample using the base weight (accounting for the probability of selection and any prior stages of sampling and participation). Column D shows the study participation rate or instrument response rate by subgroup (for categorical variables only). Column E shows the p-value associated with statistical tests comparing respondents and nonrespondents. (Due to space limitations, we do not present the weighted percentages and means that are being compared for respondents and nonrespondents in these tests.) Column F again shows variable distributions and means, this time for respondents only, and fully weighted for nonresponse, along with associated standard errors. The last three columns—participation/response rate, p-value, and final weighted distribution/mean—are repeated (in Columns G, H, and I) for an additional respondent definition in some tables.13 When diagnosing nonresponse bias, we use a significance level of 0.05 to suggest a potential for nonresponse bias and evaluate whether the weighting to mitigate bias has been successful by assessing whether the full sample value (Column C) is within two standard errors of the final weighted percentage or mean (Column F, Column I, or Column L). Any estimates for which the full sample value falls outside two standard errors are indicated by an asterisk in the relevant estimate-with-standard-error column.
Program level. Table 3 shows the nonresponse bias analysis at the program level for program participation, and corresponding weight D_WT. We observe significant differences between respondents and nonrespondents depending on whether the program was in a metropolitan statistical area. After applying nonresponse-adjusted weights (D_WT), the differences between respondents and the full sample appear to be mitigated, as the full sample value is within two standard errors of the final weighted estimate for all covariates, indicating that any remaining differences are likely attributable to sampling error. However, the overall sample size is small—only 22 of 41 programs responded—so the standard errors at the program level are relatively large, meaning the power to detect differences is reduced. Most differences (20 of 30 categories) are less than 5 percentage points in size, but eight differences are between 5 and 10 percentage points, and two differences are greater than 10.
Child-level analysis for director surveys. Table 4 shows the nonresponse bias analysis at the child level for the program and center director surveys (weights PD_CHILDWT and CD_CHILDWT, respectively).14 For the program director survey, we observe significant differences between respondents and nonrespondents for the following variables: number of months children have been enrolled in Head Start, child participation in Early Head Start, size of center by total enrollment, program geographic area, and the continuous versions of these variables: center’s number of classrooms, size of program by total enrollment, percentage of children enrolled in the program who are AIAN, percentage of children with a disability who are enrolled in the program, percentage of program’s staff who left in the past year, percentage of program’s staff who were replaced in the past year, and percentage of program’s lead teachers who left in the past year.15 For PD_CHILDWT, the full sample value is within two standard errors of the final weighted estimate for all covariates, which indicates that any remaining differences are likely attributable to sampling error.
For the center director survey, we observe significant differences between respondents and nonrespondents on the following variables: number of months children have been enrolled in Head Start, child participation in Early Head Start, size of center by total enrollment, one of the program’s geographic areas (whether or not in area “C’), percentage of children with a disability who are enrolled in the program, percentage of children enrolled in the program who are age 4 or older, and percentage of program’s staff who left in the past year. As in the program director survey, the differences between respondents and nonrespondents in the center director survey are significant for the following two continuous variables but not their categorical counterparts: child’s age in months and center’s number of classrooms. For CD_CHILDWT, we find that the potential for nonresponse bias remained associated with the size of center by total enrollment (categorical version), as the full sample percentages are more than two standard errors from the respondents’ weighted percentages.16 However, given the number of statistical comparisons being made in this analysis using a Type I error rate of 0.05, we would expect 1 out of 20 differences to rise to the level of statistical significance even if no true difference existed (and we had nearly 30 comparisons for CD_CHILDWT).
Child-level analysis for teacher surveys. Table 5 shows our analysis at the child level for the teacher survey (associated weight TS_CHILDWT) and for the combination of parent survey, teacher survey, and TCR (P21RC2WT). We observe significant differences between respondents and nonrespondents for the teacher survey on the variables of child’s sex, participation in Early Head Start, and continuous version of the variable for proportion of children enrolled in the program who are AIAN.17 For the combination of surveys, there are significant differences between respondents and nonrespondents for the following variables: child age group, participation in Early Head Start, and the percentage of children enrolled in the program who are AIAN. We also see differences between respondents and nonrespondents for the proportion of children with a disability who are enrolled in the program (as a continuous variable) and number of months children have been enrolled in Head Start (as a categorical variable), but not as a categorical or continuous ones, respectively. Differences due to nonresponse for both weights appear to be mitigated after applying weights, as all percentages and means for the full sample were within two standard errors of the responding sample estimate after weighting.
Child-level participation and instruments. Tables 6.a, 6.b, and 6.c show the results from the child-level nonresponse bias analysis for various combinations of participation and response. Table 6a shows the analysis of child-level study participation (mostly a measure of parental consent) for fall 2019, and the associated weight CNST1WT, where significant differences between respondents and nonrespondents are manifested in the following variables: child’s sex, program geographic area, the percentage of children enrolled in the program who are AIAN, and the percentage of children with a disability who are enrolled in the program (as a categorical variable but not as a continuous one). Again, when we compare the full sample value to the final weighted percentages and means (using CNST1WT), we can see that weighting appears to have mitigated bias for all covariates.
Table 6.b shows the nonresponse bias analysis for the fall and spring parent surveys for children who were still participating in the study in spring 2020. First, we look at the analysis for children whose parents responded to both the fall and spring surveys, and associated weight P12WT. We see significant differences between respondents and nonrespondents for the following variables: program geographic area, size of program by total enrollment, and program service type. There are also significant differences between respondents and nonrespondents in the percentage of children enrolled in the program who are AIAN, although only as a categorical variable.18 Second, we look at the analysis for children whose parents responded to either the fall or spring parent surveys, and associated weight P1_2WT, finding significant differences between respondents and nonrespondents for the following variables: program service type and categorical versions of size of program by total enrollment, percentage of children enrolled in the program who are AIAN, percentage of children with a disability who are enrolled in the program, and percentage of program’s staff who left in the past year. All full sample percentages and means were within two standard errors of the responding sample estimates after nonresponse weighting (P12WT and P1_2WT).
Table 6.c shows the nonresponse bias analysis for three different child-level survey combinations. The first combines the response to either wave of the parent survey and to the spring TCR survey, and associated weight P21R2WT. We find that respondents significantly differ from nonrespondents on the following variables: the number of months children have been enrolled in Head Start (categorical only), the child’s participation in Early Head Start, the percentage of children enrolled in the program who are AIAN, and the proportion of children with a disability who are enrolled in the program (continuous only). The second combines the response to either wave of the parent survey with both fall and spring TCRs, and with the associated weight PR12WT. We find significant differences between respondents and nonrespondents for the same set of variables as indicated for the first weight in this table, plus one additional covariate: program size by total enrollment (categorical only). The third combines the response to either wave of the parent surveys with both fall and spring TCRs and the teacher survey, and associated weight PR12CW. Respondents significantly differ from nonrespondents for the same set of variables as indicated for the first weight in this table, with two exceptions: the mean proportion of children with a disability who are enrolled in the program does not appear to indicate nonresponse bias for this survey combination, and program geographic area does. All three survey combinations mitigate the risk for nonresponse bias for these variables, evidenced by the full sample values being within two standard errors of the final weighted estimate.
For AIAN FACES 2019, we examined the potential for nonresponse bias in the study participation for programs and children study participation and for all spring and longitudinal (fall-spring) child-level weights for analysis of instruments collected at the program, center, classroom, and child levels. Although we observed some statistically significant differences between respondents and nonrespondents, only two differences (for one weight) remained after weighting. That is, after adjusting the weights for nonresponse, the full sample estimate almost always fell within two standard errors of the final weighted estimates where it did not before the weighting adjustments. There was one exception: for CD_CHILDWT, we found the potential for bias remained in association with the categorical version of center’s child enrollment. This indicates that the differences were likely attributable to sampling error and not to unresolved nonresponse bias. Further, the nonresponse weighting adjustments did not introduce any new differences that caused the full sample value to fall more than two standard errors from the responding sample weighted estimate.
Researchers should feel free to control for any characteristics of respondents and nonrespondents that appear to be different from each other (or more accurately, for differences between respondents and the full sample) in their model. Researchers who are using appropriate weights when making estimates from the AIAN FACES 2019 study should feel reassured that the risk of nonresponse bias has been mitigated.
cc: Allison Walker, Lizabeth Malone, and Sara Bernstein
Table 3. Nonresponse bias analysis at the program level
Variable |
Value |
Full sample estimate |
Study participation rate by subgroup |
Study participants vs. nonparticipants p-value |
Study participants’ estimate [standard error] |
A |
B |
C |
D |
E |
F |
Number of programs |
41 |
|
|
22 |
|
Weight |
Program base weight |
|
|
D_WT |
|
Categorical variables at program level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Program geographic area |
A |
4.84 |
81.82 |
0.806 |
4.84 [1.36] |
B |
33.62 |
50.81 |
31.29 [15.83] |
||
C |
33.21 |
57.93 |
35.92 [14.05] |
||
D |
8.29 |
87.79 |
8.29 [1.84] |
||
E |
20.04 |
65.45 |
19.66 [6.57] |
||
MSAa |
Yes |
23.96 |
84.47 |
0.039 |
29.23 [14.47] |
No |
76.04 |
53.18 |
70.77 [14.47] |
||
Program enrollment |
< 130 |
57.17 |
59.72 |
0.953 |
59.25 [9.31] |
130-227 |
26.58 |
59.26 |
24.25 [9.09] |
||
> 227 |
16.26 |
66.35 |
16.50 [6.20] |
||
Percentage of children enrolled in program who are AIAN |
< 84.21 |
34.10 |
63.64 |
0.752 |
34.74 [13.45] |
84.21-97.44 |
44.42 |
53.61 |
39.68 [16.15] |
||
> 97.44 |
21.47 |
70.58 |
25.58 [8.36] |
||
Percentage of children with a disability who are enrolled in program |
< 8 |
27.23 |
58.26 |
0.111 |
22.89 [8.63] |
8-15.38 |
34.41 |
38.43 |
33.63 [16.66] |
||
> 15.38 |
38.36 |
82.34 |
43.48 [16.35] |
||
Percentage of children enrolled in program who are age 4+ |
< 46.56 |
38.52 |
53.43 |
0.816 |
28.31 [10.03] |
46.56-53.43 |
28.65 |
66.45 |
37.57 [16.61] |
||
> 53.43 |
32.83 |
64.13 |
34.12 [15.33] |
||
Percentage of staff who left |
< 10 |
27.04 |
53.81 |
0.762 |
21.79 [7.34] |
10-16.67 |
46.62 |
58.78 |
37.74 [14.96] |
||
> 16.67 |
26.34 |
71.06 |
40.47 [15.37] |
||
Percentage of staff replaced |
< 50 |
21.79 |
60.91 |
0.760 |
20.14 [7.41] |
50-93.75 |
25.46 |
70.34 |
31.58 [14.58] |
||
> 93.75 |
52.75 |
55.91 |
48.29 [16.11] |
||
Percentage of lead teachers who left |
< 6.67 |
41.98 |
74.73 |
0.309 |
45.20 [14.39] |
6.67-16.67 |
18.64 |
65.44 |
22.06 [7.44] |
||
> 16.67 |
39.38 |
43.43 |
32.73 [15.77] |
||
Service type |
Center only |
90.41 |
59.28 |
0.490 |
90.93 [5.47] |
Center + home |
9.59 |
73.82 |
9.07 [5.47] |
||
Continuous variables at program level |
(Mean) |
|
|
(Mean) |
|
Program enrollment |
134.08 |
|
0.681 |
118.82 [15.52] |
|
Proportion of children enrolled in program who are AIAN |
0.83 |
|
0.895 |
0.84 [0.06] |
|
Proportion of children with a disability who are enrolled in program |
0.13 |
|
0.698 |
0.12 [0.02] |
|
Proportion of children enrolled in program who are age 4+ |
0.50 |
|
0.602 |
0.51 [0.03] |
|
Proportion of staff who left |
0.14 |
|
0.388 |
0.17 [0.04] |
|
Proportion of staff replaced |
0.72 |
|
0.871 |
0.71 [0.09] |
|
Proportion of lead teachers who left |
0.24 |
|
0.223 |
0.17 [0.08] |
|
Note: Bolded p-values highlight values less than 0.05.
All continuous variables were also included as categorical (ordinal) variables, divided into tertiles (sometimes into binary variables) based on the full sample distribution.
a MSA refers to whether the program’s zip code was within a metropolitan statistical area.
Table 4. Nonresponse bias analysis at the child level: program director and center director surveys
Variable |
Value |
Full sample estimate |
Program director survey response rate by subgroup |
Program director survey respondents vs. nonrespondents p-value |
Program director survey respondents’ estimate [standard error] |
Center director survey response rate by subgroup |
Center director survey respondents vs. nonrespondents p-value |
Center director survey respondents’ estimate [standard error] |
A |
B |
C |
D |
E |
F |
G |
H |
I |
Number of children |
686 |
|
|
592 |
|
|
485 |
|
Weight |
CNST2WT |
|
|
PD_CHILDWT |
|
|
CD_CHILDWT |
|
Categorical variables at child level |
(Column Percent) |
(Percent) |
|
(Column percent) |
(Percent) |
|
(Column percent) |
|
Age group |
< 48 months |
41.86 |
92.23 |
0.223 |
42.34 [2.59] |
66.75 |
N.C. |
36.58 [9.02] |
48+ months |
58.14 |
94.34 |
57.66 [2.59] |
86.60 |
63.42 [9.02] |
|||
Sex |
Female |
46.84 |
93.58 |
0.892 |
46.94 [2.96] |
77.56 |
0.594 |
47.38 [2.55] |
Male |
53.16 |
93.35 |
53.06 [2.96] |
78.94 |
52.62 [2.55] |
|||
Language spoken at home |
English |
93.83 |
93.03 |
N.C. |
93.23 [1.55] |
77.92 |
0.329 |
93.84 [2.13] |
Tribal |
3.48 |
100.00 |
3.90 [1.12] |
91.54 |
3.76 [1.57] |
|||
Others |
2.69 |
100.00 |
2.87 [1.12] |
74.24 |
2.41 [0.82] |
|||
Months enrolled in Head Start |
< 2 |
37.64 |
85.08 |
< .0001 |
35.23 [2.17] |
72.65 |
0.000 |
39.15 [13.84] |
= 2 |
17.19 |
99.03 |
18.57 [1.70] |
51.32 |
10.79 [5.12] |
|||
> 2 |
17.68 |
95.80 |
17.81 [1.65] |
83.88 |
18.52 [6.86] |
|||
missing |
27.49 |
99.93 |
28.39 [3.10] |
99.29 |
31.54 [18.26] |
|||
Child participation in Early Head Start |
Don’t know |
24.91 |
97.83 |
< .0001 |
24.42 [3.07] |
97.12 |
< .0001 |
27.79 [19.98] |
No |
57.00 |
89.99 |
56.30 [2.82] |
67.52 |
55.31 [16.22] |
|||
Yes |
18.09 |
98.35 |
19.28 [1.73] |
86.30 |
16.90 [5.12] |
|||
Primary funding source |
Child care subsidy or Other |
0.48 |
100.00 |
N.C. |
0.51 [0.33] |
100.00 |
N.C. |
0.54 [0.38] |
Head Start |
94.02 |
93.04 |
93.53 [1.12] |
76.91 |
84.15 [11.05] |
|||
State pre-K |
1.95 |
100.00 |
1.95 [0.70] |
100.00 |
1.95 [1.73] |
|||
Tribal |
3.54 |
100.00 |
4.00 [0.87] |
100.00 |
13.35 [10.63] |
|||
Categorical variables at center level |
(Column percent) |
(Percent) |
|
(Column percent) |
(Percent) |
|
(Column percent) |
|
Center’s child enrollment |
< 35 |
24.98 |
91.69 |
0.033 |
23.81 [1.67] |
85.17 |
0.000 |
34.91 [9.86] |
35-68 |
23.34 |
96.68 |
24.39 [1.72] |
38.70 |
9.49 [4.58]a |
|||
> 68 |
51.68 |
92.85 |
51.80 [2.40] |
92.85 |
55.59 [12.34] |
|||
Center’s number of classrooms |
< 4 |
44.10 |
93.54 |
0.929 |
43.45 [2.34] |
68.73 |
0.242 |
44.41 [12.34] |
4+ |
55.90 |
93.39 |
56.55 [2.34] |
85.84 |
55.59 [12.34] |
|||
Categorical variables at program level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Program geographic area: Cb |
Yes |
33.59 |
82.13 |
< .0001 |
30.74 [1.68] |
49.41 |
0.011 |
30.74 [11.83] |
No |
66.41 |
99.19 |
69.26 [1.68] |
92.90 |
69.26 [11.83] |
|||
Program geographic area: Eb |
Yes |
17.42 |
96.90 |
< .0001 |
19.29 [1.25] |
96.90 |
0.119 |
19.29 [8.66] |
No |
82.58 |
92.73 |
80.71 [1.25] |
74.37 |
80.71 [8.66] |
|||
MSAc |
Yes |
18.79 |
100.00 |
N.C. |
20.90 [1.58] |
97.55 |
0.090 |
20.97 [10.92] |
No |
81.21 |
91.94 |
79.10 [1.58] |
73.84 |
79.03 [10.92] |
|||
Program enrollment |
< 130 |
32.25 |
88.55 |
N.C. |
30.75 [1.90] |
87.12 |
0.667 |
30.75 [13.03] |
130-179 |
28.00 |
100.00 |
31.45 [2.04] |
66.43 |
32.81 [13.91] |
|||
> 179 |
39.74 |
92.83 |
37.80 [2.79] |
79.48 |
36.44 [19.10] |
|||
Percentage of children enrolled in program who are AIAN |
< 93.41 |
61.38 |
100.00 |
N.C. |
63.54 [2.05] |
83.94 |
0.486 |
67.39 [14.17] |
93.41+ |
38.62 |
83.06 |
36.46 [2.05] |
69.32 |
32.61 [14.17] |
|||
Percentage of children with a disability who are enrolled in program |
< 9.45 |
24.21 |
82.51 |
N.C. |
21.38 [1.65] |
41.78 |
0.013 |
22.80 [11.05] |
9.45-17.58 |
34.58 |
100.00 |
37.48 [2.15] |
84.66 |
31.52 [12.75] |
|||
> 17.58 |
41.21 |
94.40 |
41.14 [2.73] |
94.40 |
45.68 [18.38] |
|||
Percentage of children enrolled in program who are age 4+ |
< 44.96 |
20.72 |
82.18 |
N.C. |
18.17 [1.46] |
64.25 |
0.026 |
14.53 [9.35] |
44.96-50.56 |
26.27 |
100.00 |
29.21 [1.85] |
56.42 |
27.10 [13.72] |
|||
> 50.56 |
53.01 |
94.62 |
52.62 [2.63] |
94.62 |
58.37 [17.13] |
|||
Percentage of staff who left |
< 10 |
49.20 |
95.31 |
N.C. |
49.00 [2.49] |
95.31 |
0.045 |
53.54 [16.32] |
10-16.67 |
25.26 |
83.24 |
23.09 [1.82] |
68.53 |
19.38 [10.30] |
|||
> 16.67 |
25.55 |
100.00 |
27.91 [1.56] |
55.18 |
27.08 [12.36] |
|||
Percentage of staff replaced |
< 25 |
44.84 |
94.85 |
N.C. |
44.59 [2.72] |
94.85 |
0.115 |
49.13 [18.00] |
25-87.5 |
30.05 |
85.91 |
28.11 [1.95] |
65.46 |
33.19 [14.48] |
|||
> 87.5 |
25.12 |
100.00 |
27.29 [1.98] |
64.10 |
17.69 [8.94] |
|||
Percentage of lead teachers who left |
< 5 |
54.72 |
93.25 |
N.C. |
53.82 [2.44] |
92.41 |
0.109 |
57.46 [15.47] |
5-12.5 |
24.86 |
88.53 |
24.55 [1.68] |
65.66 |
30.52 [12.08] |
|||
> 12.5 |
20.43 |
100.00 |
21.63 [1.58] |
55.86 |
12.02 [5.36] |
|||
Service type |
Center only |
90.61 |
92.78 |
N.C. |
89.95 [1.20] |
86.00 |
N.C. |
99.63 [0.39] |
Center + home |
9.39 |
100.00 |
10.05 [1.20] |
3.98 |
0.37 [0.39] |
|||
Continuous variables at child level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Age in months |
48.45 |
|
0.377 |
48.38 [0.38] |
|
0.028 |
49.02 [1.25] |
|
Months enrolled in Head Start |
3.80 |
|
0.000 |
3.93 [0.30] |
|
0.162 |
3.94 [0.90] |
|
Continuous variables at center level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Center’s child enrollment |
80.85 |
|
0.000 |
83.92 [2.13] |
|
0.027 |
81.06 [13.98] |
|
Center’s number of classrooms |
4.94 |
|
0.000 |
5.20 [0.14] |
|
0.044 |
5.06 [0.94] |
|
Continuous variables at program level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Program enrollment |
273.54 |
|
0.000 |
273.97 [16.88] |
|
0.266 |
290.57 [115.33] |
|
Proportion of children enrolled in program who are AIAN |
0.81 |
|
0.000 |
0.81 [0.01] |
|
0.593 |
0.81 [0.05] |
|
Proportion of children with a disability who are enrolled in program |
0.15 |
|
0.000 |
0.16 [0.003] |
|
0.012 |
0.16 [0.02] |
|
Proportion of children enrolled in program who are age 4+ |
0.55 |
|
0.330 |
0.55 [0.01] |
|
0.061 |
0.57 [0.04] |
|
Proportion of staff who left |
0.11 |
|
0.004 |
0.11 [0.005] |
|
0.053 |
0.10 [0.03] |
|
Proportion of staff replaced |
0.43 |
|
0.000 |
0.45 [0.02] |
|
0.090 |
0.39 [0.14] |
|
Proportion of lead teachers who left |
0.10 |
|
0.000 |
0.11 [0.01] |
|
0.501 |
0.09 [0.04] |
|
Note: Bolded p-values highlight values less than 0.05.
All continuous variables were also included as categorical (ordinal) variables, divided into tertiles (sometimes into binary variables) based on the full sample distribution.
N.C. means the Rao-Scott chi-square statistic was not calculable for that variable (usually, but not exclusively, due to one or more categories with 100 percent response and 0 percent nonresponse).
a The value of this variable represents the only estimate for which the full sample value did not fall within two standard errors.
b Because there were some “zero cells” for program geographic area in this table, we have two binary indicator variables associated with two of the areas, instead of the five-category geographic area variable in some of the other tables.
c MSA refers to whether the program’s zip code was within a metropolitan statistical area.
Table 5. Nonresponse bias analysis at the child level: teacher survey
Variable |
Value |
Full sample estimate |
Teacher survey response rate by subgroup |
Teacher survey respondents vs. nonrespondents p-value |
Teacher survey respondents’ estimate [standard error] |
Parent + teacher survey + TCR response rate by subgroup |
Parent + teacher survey + TCR respondents vs. nonrespondents p-value |
Parent + teacher survey + TCR respondents’ estimate [standard error] |
A |
B |
C |
D |
E |
F |
G |
H |
I |
Number of children |
686 |
|
|
471 |
|
|
371 |
|
Weight |
CNST2WT |
|
|
TS_CHILDWT |
|
|
P21RC2WT |
|
Categorical variables at child level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Age group |
< 48 months |
41.86 |
70.22 |
0.239 |
40.33 [10.11] |
53.88 |
0.040
|
39.31 [10.28] |
48+ months |
58.14 |
80.69 |
59.67 [10.11] |
65.80 |
60.69 [10.28] |
|||
Sex |
Female |
46.74 |
72.18 |
0.003 |
45.66 [1.99] |
60.40 |
0.896
|
46.14 [3.07] |
Male |
53.26 |
79.94 |
54.44 [1.99] |
61.18 |
53.86 [3.07] |
|||
Language spoken at home |
English |
93.83 |
76.53 |
0.832 |
94.20 [1.81] |
60.68 |
0.891 |
92.46 [2.58] |
Tribal |
3.48 |
70.69 |
3.36 [1.26] |
66.35 |
4.83 [2.02] |
|||
Others |
2.69 |
75.82 |
2.44 [0.71] |
58.07 |
2.71 [1.13] |
|||
Months enrolled in Head Start |
< 2 |
37.64 |
63.18 |
N.C. |
36.56 [11.37] |
47.02 |
< .0001 |
35.23 [11.23] |
= 2 |
17.19 |
90.83 |
20.74 [8.93] |
76.50 |
23.29 [10.10] |
|||
> 2 |
17.68 |
53.42 |
15.14 [5.31] |
46.07 |
14.19 [5.49] |
|||
missing |
27.49 |
99.93 |
27.56 [17.16] |
79.36 |
27.30 [17.24] |
|||
Child participation in Early Head Start |
Don’t know |
24.91 |
97.83 |
0.000 |
24.54 [ 18.58] |
77.82 |
< .0001 |
24.19 [18.68] |
No |
57.00 |
67.10 |
58.17 [15.76] |
53.87 |
59.14 [16.03] |
|||
Yes |
18.09 |
75.68 |
17.29 [5.46] |
59.24 |
16.67 [5.34] |
|||
Primary funding source |
Child care subsidy or Other |
0.48 |
93.28 |
N.C. |
0.53 [0.36] |
93.28 |
0.423 |
0.62 [0.42] |
Head Start |
94.02 |
75.79 |
93.57 [3.93] |
59.83 |
93.37 [4.05] |
|||
State pre-K |
1.95 |
100.00 |
3.26 [2.35] |
87.11 |
3.18 [2.29] |
|||
Tribal |
3.54 |
74.64 |
2.64 [2.07] |
68.03 |
2.83 [2.20] |
|||
Categorical variables at center level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Center’s child enrollment |
< 35 |
24.98 |
59.96 |
0.303 |
22.91 [6.85] |
53.73 |
0.759 |
23.13 [7.08] |
35-68 |
23.34 |
85.71 |
30.35 [10.96] |
64.90 |
29.92 [11.64] |
|||
> 68 |
51.68 |
79.97 |
46.74 [12.79] |
62.39 |
46.95 [12.97] |
|||
Center’s number of classrooms |
< 3 |
27.10 |
63.09 |
0.460 |
25.20 [7.93] |
55.79 |
0.854 |
25.41 [8.15] |
3-4 |
24.91 |
82.50 |
31.76 [12.41] |
61.08 |
31.33 [13.00] |
|||
> 4 |
47.99 |
80.56 |
43.05 [14.17]
|
63.51 |
43.26 [14.31] |
|||
Categorical variables at program level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Program geographic area: Ba |
Yes |
18.99 |
60.34 |
0.343 |
18.99 [7.81] |
53.52 |
0.658 |
18.99 [7.82] |
No |
81.01 |
80.05 |
81.01 [7.81] |
62.52 |
81.01 [7.82] |
|||
Program geographic area: Ca |
Yes |
33.59 |
63.29 |
0.193 |
33.59 [10.96] |
48.87 |
0.143 |
33.59 [11.05] |
No |
66.41 |
82.90 |
66.41 [10.96] |
66.85 |
66.41 [11.05] |
|||
Program geographic area: Ea |
Yes |
17.42 |
79.88 |
0.767 |
17.42 [7.17] |
60.95 |
0.991 |
17.42 [7.18] |
No |
82.58 |
75.56 |
82.58 [7.17] |
60.78 |
82.58 [7.18] |
|||
MSAb |
Yes |
18.79 |
90.23 |
0.238 |
19.03 [9.47] |
70.63 |
0.442 |
19.03 [9.48] |
No |
81.21 |
73.09 |
80.97 [9.47] |
58.54 |
80.97 [9.48] |
|||
Program enrollment |
< 130 |
32.25 |
67.79 |
0.263 |
32.50 [11.74] |
53.97 |
0.324 |
32.50 [11.76] |
130-179 |
28.00 |
68.74 |
25.52 [12.95] |
55.12 |
26.93 [13.94] |
|||
> 179 |
39.74 |
88.56 |
41.98 [17.13] |
70.37 |
40.57 [17.93] |
|||
Percentage of children enrolled in program who are AIAN |
< 90.7 |
54.63 |
88.20 |
0.089 |
56.78 [13.04] |
72.18 |
0.003 |
58.20 [13.53] |
90.7-96 |
19.15 |
68.36 |
20.03 [9.78] |
59.61 |
20.45 [10.14] |
|||
> 96 |
26.21 |
57.33 |
23.19 [10.04] |
37.99 |
21.35 [10.74] |
|||
Percentage of children with a disability who are enrolled in program |
< 9.45 |
24.21 |
60.80 |
0.221 |
20.89 [11.76] |
45.53 |
0.117 |
22.30 [12.85] |
9.45-17.58 |
34.58 |
71.38 |
36.41 [13.37] |
56.88 |
37.55 [14.21] |
|||
> 17.58 |
41.21 |
89.56 |
42.70 [17.50] |
73.08 |
40.15 [17.91] |
|||
Percentage of children enrolled in program who are age 4+ |
< 44.96 |
20.72 |
79.78 |
0.947 |
21.02 [10.88] |
63.30 |
0.691 |
21.02 [10.89] |
44.96-50.56 |
26.27 |
74.27 |
30.53 [11.76] |
54.46 |
33.51 [11.31] |
|||
> 50.56 |
53.01 |
75.96 |
48.45 [17.11] |
62.99 |
45.47 [17.18] |
|||
Percentage of staff who left |
< 10 |
49.20 |
81.04 |
0.301 |
50.44 [15.80] |
64.46 |
0.243 |
47.89 [16.21] |
10-16.67 |
25.26 |
85.89 |
24.96 [11.31] |
70.26 |
24.96 [11.33] |
|||
> 16.67 |
25.55 |
57.73 |
24.60 [11.05] |
44.45 |
27.15 [11.31] |
|||
Percentage of staff replaced |
< 25 |
44.84 |
75.37 |
0.973 |
46.08 [17.34] |
59.29 |
0.973 |
43.53 [17.76] |
25-87.5 |
30.05 |
75.13 |
26.73 [12.70] |
61.19 |
28.14 [13.69] |
|||
> 87.5 |
25.12 |
79.39 |
27.19 [12.32] |
63.07 |
28.33 [13.21] |
|||
Percentage of lead teachers who left |
< 5 |
54.72 |
81.57 |
0.587 |
54.72 [15.11] |
65.36 |
0.617 |
54.72 [15.15] |
5-12.5 |
24.86 |
64.94 |
22.78 [10.68] |
52.79 |
21.64 [11.62] |
|||
> 12.5 |
20.43 |
76.07 |
22.51 [10.60] |
58.38 |
23.65 [11.59] |
|||
Service type |
Center only |
90.61 |
75.42 |
0.572 |
89.07 [8.45] |
61.12 |
0.794 |
87.93 [9.53] |
Center + home |
9.39 |
84.89 |
10.93 [8.45] |
57.80 |
12.07 [9.53] |
|||
Continuous variables at child level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Age in months |
48.45 |
|
0.284 |
48.84 [1.32] |
|
0.289 |
48.80 [1.31] |
|
Months enrolled in Head Start |
3.80 |
|
0.072 |
3.36 [0.63] |
|
0.391 |
3.22 [0.68] |
|
Continuous variables at center level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Center’s child enrollment |
80.85 |
|
0.192 |
80.42 [11.88] |
|
0.288 |
80.87 [11.88] |
|
Center’s number of classrooms |
4.94 |
|
0.248 |
4.89 [0.82] |
|
0.364 |
4.90 [0.82] |
|
Continuous variables at program level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Program enrollment |
273.54 |
|
0.207 |
273.55 [107.06] |
|
0.179 |
272.29 [107.49] |
|
Proportion of children enrolled in program who are AIAN |
0.81 |
|
0.031 |
0.81 [0.04] |
|
0.010 |
0.81 [0.04] |
|
Proportion of children with a disability who are enrolled in program |
0.15 |
|
0.197 |
0.16 [0.02] |
|
0.046 |
0.15 [0.02] |
|
Proportion of children enrolled in program who are age 4+ |
0.55 |
|
0.878 |
0.54 [0.04] |
|
0.797 |
0.53 [0.04] |
|
Proportion of staff who left |
0.11 |
|
0.336 |
0.10 [0.03] |
|
0.253 |
0.11 [0.03] |
|
Proportion of staff replaced |
0.43 |
|
0.842 |
0.44 [0.14] |
|
0.772 |
0.46 [0.15] |
|
Proportion of lead teachers who left |
0.10 |
|
0.299 |
0.11 [0.04] |
|
0.239 |
0.11 [0.04] |
|
Note: Bolded p-values highlight values less than 0.05.
All continuous variables were also included as categorical (ordinal) variables, divided into tertiles (sometimes into binary variables) based on the full sample distribution.
N.C. means the Rao-Scott chi-square statistic was not calculable for that variable (usually, but not exclusively, due to one or more categories with 100 percent response and 0 percent nonresponse).
a Because there were some “zero cells” for program geographic area in this table, we have three binary indicator variables associated with three of the areas, instead of the five-category geographic area variable in some of the other tables.
b MSA refers to whether the program’s zip code was within a metropolitan statistical area.
Table 6a. Nonresponse bias analysis at the child level: study participation
Variable |
Value |
Full sample estimate |
Fall participation rate by subgroup |
Fall study participants vs. nonparticipants p-value |
Fall participants’ estimate [standard error] |
A |
B |
C |
D |
E |
F |
Number of children |
981 |
|
|
720 |
|
Weight |
Cumulative child sampling weight |
|
|
CNST1WT |
|
Categorical variables at child level |
(Column percent) |
(Percent) |
|
(Column percent) |
|
Age group |
< 48 months |
41.72 |
72.53 |
0.303 |
42.52 [9.53] |
48+ months |
58.28 |
79.27 |
57.48 [9.53] |
||
Sex |
Female |
48.55 |
73.34 |
0.021 |
47.31 [1.85] |
Male |
51.45 |
79.82 |
52.69 [1.85] |
||
Language spoken at home |
English |
92.93 |
77.02 |
0.076 |
93.64 [1.68] |
Tribal |
2.80 |
100.00 |
3.68 [1.32] |
||
Others |
4.27 |
48.80 |
2.68 [0.58] |
||
Months enrolled in Head Start |
< 2 |
38.73 |
70.73 |
0.173 |
38.86 [11.41] |
= 2 |
17.03 |
81.32 |
17.32 [6.85] |
||
> 2 |
17.40 |
70.98 |
17.25 [6.07] |
||
missing |
26.84 |
85.17 |
26.57 [16.41] |
||
Child participation in Early Head Start |
Don’t know |
25.01 |
89.35 |
0.074 |
24.78 [18.49] |
No |
57.20 |
73.29 |
56.75 [15.30] |
||
Yes |
17.79 |
68.51 |
18.47 [5.73] |
||
Primary funding source |
Child care subsidy or Other |
0.68 |
47.31 |
0.648 |
0.46 [0.32] |
Head Start |
94.44 |
76.85 |
93.87 [3.70] |
||
State pre-K |
1.90 |
76.01 |
2.16 [1.93] |
||
Tribal |
2.97 |
71.04 |
3.51 [2.21] |
||
Categorical variables at center level |
(Column percent) |
(Percent) |
|
(Column Percent) |
|
Center’s child enrollment |
< 35 |
25.75 |
78.14 |
0.937 |
25.75 [7.80] |
35-70 |
27.54 |
74.39 |
27.54 [11.41] |
||
> 70 |
46.71 |
76.75 |
46.71 [14.10] |
||
Center’s number of classrooms |
< 3 |
27.87 |
79.80 |
0.778 |
27.87 [8.62] |
3-4 |
25.42 |
72.26 |
25.42 [11.11] |
||
> 4 |
46.71 |
76.75 |
46.71 [14.10] |
||
Categorical variables at program level |
(Column percent) |
(Percent) |
|
(Column percent) |
|
Program geographic area |
A |
1.11 |
90.02 |
0.001 |
1.11 [0.47] |
B |
18.73 |
64.55 |
18.73 [4.93] |
||
C |
34.38 |
71.27 |
34.38 [8.52] |
||
D |
28.66 |
90.54 |
28.66 [16.27] |
||
E |
17.12 |
75.45 |
17.12 [5.13] |
||
MSAa |
Yes |
18.90 |
73.77 |
0.696 |
18.90 [8.33] |
No |
81.10 |
77.08 |
81.10 [8.33] |
||
Program enrollment |
< 130 |
32.04 |
67.76 |
0.216 |
32.04 [11.61] |
130-179 |
27.37 |
78.19 |
27.37 [12.01] |
||
> 179 |
40.59 |
82.15 |
40.59 [17.29] |
||
Percentage of children enrolled in program who are AIAN |
< 90.70 |
54.59 |
82.07 |
0.012 |
54.59 [13.92] |
90.70-97.64 |
23.51 |
78.59 |
23.51 [10.19] |
||
> 97.64 |
21.90 |
60.16 |
21.90 [9.58] |
||
Percentage of children with a disability who are enrolled in program |
< 12.12 |
28.28 |
78.77 |
< .0001 |
28.28 [12.56] |
12.12-17.58 |
30.67 |
61.60 |
30.67 [11.51] |
||
> 17.58 |
41.04 |
85.96 |
41.04 [17.10] |
||
Percentage of children enrolled in program who are age 4+ |
< 46.56 |
26.86 |
75.53 |
0.914 |
26.86 [12.35] |
46.56-56.80 |
25.04 |
79.65 |
25.04 [10.30] |
||
> 56.80 |
48.09 |
75.31 |
48.09 [16.40] |
||
Percentage of staff who left |
< 9.26 |
46.90 |
76.93 |
0.965 |
46.90 [16.30] |
9.26-16.67 |
27.31 |
77.47 |
27.31 [12.12] |
||
> 16.67 |
25.79 |
74.52 |
25.79 [11.17] |
||
Percentage of staff replaced |
< 60 |
26.93 |
74.17 |
0.895 |
26.93 [9.74] |
60-93.75 |
16.86 |
75.88 |
16.86 [9.53] |
||
> 93.75 |
56.21 |
77.72 |
56.21 [13.34] |
||
Percentage of lead teachers who left |
< 5 |
54.47 |
75.97 |
0.886 |
54.47 [15.71] |
5-12.5 |
24.39 |
74.16 |
24.39 [10.69] |
||
> 12.5 |
21.13 |
80.36 |
21.13 [10.90] |
||
Service type |
Center only |
89.65 |
77.62 |
0.393 |
89.65 [7.95] |
Center + home |
10.35 |
66.38 |
10.35 [7.95] |
||
Continuous variables at child level |
(Mean) |
|
|
(Mean) |
|
Age in months |
48.52 |
|
0.540 |
48.35 [1.24] |
|
Months enrolled in Head Start |
3.73 |
|
0.121 |
3.71 [0.63] |
|
Continuous variables at center level |
(Mean) |
|
|
(Mean) |
|
Center’s child enrollment |
79.97 |
|
0.770 |
79.99 [13.10] |
|
Center’s number of classrooms |
4.90 |
|
0.433 |
4.90 [0.93] |
|
Continuous variables at program level |
(Mean) |
|
|
(Mean) |
|
Program enrollment |
273.66 |
|
0.183 |
273.66 [106.63] |
|
Proportion of children enrolled in program who are AIAN |
0.81 |
|
0.030 |
0.81 [0.04] |
|
Proportion of children with a disability who are enrolled in program |
0.15 |
|
0.758 |
0.15 [0.02] |
|
Proportion of children enrolled in program who are age 4+ |
0.55 |
|
0.521 |
0.55 [0.04] |
|
Proportion of staff who left |
0.11 |
|
0.712 |
0.11 [0.03] |
|
Proportion of staff replaced |
0.74 |
|
0.813 |
0.74 [0.08] |
|
Proportion of lead teachers who left |
0.11 |
|
0.417 |
0.11 [0.04] |
|
Note: Bolded p-values highlight values less than 0.05.
All continuous variables were also included as categorical (ordinal) variables, divided into tertiles (sometimes into binary variables) based on the full sample distribution.
a MSA refers to whether the program’s zip code was within a metropolitan statistical area.
Table 6.b. Nonresponse bias analysis at the child level: parent surveys
Variable |
Value |
Full sample estimate |
Fall + spring parent survey response rate by subgroup |
Fall + spring parent survey respondents vs. nonrespondents p-value |
Fall + spring parent survey respondents’ estimate [standard error] |
Fall or spring parent survey response rate by subgroup |
Fall or spring parent survey respondents vs. nonrespondents p-value |
Fall or spring parent survey respondents’ estimate [standard error] |
A |
B |
C |
D |
E |
F |
G |
H |
I |
Number of children |
686 |
|
|
410 |
|
|
568 |
|
Weight |
CNST2WT |
|
|
P12WT |
|
|
P1_2WT |
|
Categorical variables at child level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Age group |
< 48 months |
41.86 |
58.88 |
0.949 |
40.82 [10.12] |
81.93 |
0.900 |
41.82 [9.85] |
48+ months |
58.14 |
59.25 |
59.18 [10.12] |
82.44 |
58.18 [9.85] |
|||
Sex |
Female |
46.74 |
56.81 |
0.051 |
44.92 [1.91] |
82.79 |
0.842 |
47.08 [1.35] |
Male |
53.26 |
61.10 |
55.08 [1.91] |
81.73 |
52.92 [1.35] |
|||
Language spoken at home |
English |
93.83 |
58.37 |
0.289 |
92.53 [2.53] |
82.42 |
0.057 |
94.08 [2.02] |
Tribal |
3.48 |
82.88 |
4.90 [1.75] |
95.65 |
3.97 [1.47] |
|||
Others |
2.69 |
53.76 |
2.57 [1.25] |
58.07 |
1.95 [0.87] |
|||
Months enrolled in Head Start |
< 2 |
37.64 |
58.55 |
0.290 |
36.85 [10.92] |
82.71 |
0.580 |
37.63 [11.12] |
= 2 |
17.19 |
66.12 |
17.88 [7.25] |
85.97 |
17.60 [7.13] |
|||
> 2 |
17.68 |
60.64 |
17.98 [6.06] |
81.90 |
17.44 [5.57] |
|||
missing |
27.49 |
54.46 |
27.30 [17.22] |
79.43 |
27.32 [17.22] |
|||
Child participation in Early Head Start |
Don’t know |
24.91 |
55.90 |
0.231 |
24.63 [18.59] |
79.38 |
0.466 |
24.62 [18.59] |
No |
57.00 |
61.30 |
58.52 [15.90] |
83.17 |
57.79 [15.60] |
|||
Yes |
18.09 |
56.55 |
16.85 [5.37] |
83.17 |
17.59 [5.08] |
|||
Primary funding source |
Child care subsidy or Other |
0.48 |
75.74 |
0.320 |
0.55 [0.48] |
100.00 |
N.C. |
0.52 [0.37] |
Head Start |
94.02 |
59.20 |
94.59 [3.21] |
81.54 |
93.75 [3.86] |
|||
State pre-K |
1.95 |
41.61 |
1.09 [0.86] |
100.00 |
2.05 [1.80] |
|||
Tribal |
3.54 |
63.75 |
3.77 [2.44] |
88.32 |
3.67 [2.37] |
|||
Categorical variables at center level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Center’s child enrollment |
< 35 |
24.98 |
59.63 |
0.885 |
24.98 [6.38] |
82.86 |
0.760 |
24.98 [6.38] |
35-68 |
23.34 |
56.66 |
23.34 [9.86] |
79.29 |
23.34 [9.86] |
|||
> 68 |
51.68 |
59.94 |
51.68 [12.30] |
83.24 |
51.68 [12.30] |
|||
Center’s number of classrooms |
< 3 |
27.10 |
60.29 |
0.835 |
27.10 [7.25] |
82.64 |
0.808 |
27.10 [7.25] |
3-4 |
24.91 |
56.32 |
24.91 [11.04] |
79.86 |
24.91 [11.04] |
|||
> 4 |
47.99 |
59.87 |
47.99 [13.68] |
83.22 |
47.99 [13.68] |
|||
Categorical variables at program level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Program geographic area |
A |
1.16 |
55.58 |
0.031 |
1.16 [0.49] |
80.00 |
0.062 |
1.16 [0.49] |
B |
18.99 |
66.96 |
18.99 [7.80] |
88.65 |
18.99 [7.80] |
|||
C |
33.59 |
54.37 |
33.59 [10.71] |
79.98 |
33.59 [10.71] |
|||
D |
28.84 |
58.26 |
28.84 [16.28] |
79.67 |
28.84 [16.28] |
|||
E |
17.42 |
61.25 |
17.42 [7.57] |
83.93 |
17.42 [7.57] |
|||
MSAa |
Yes |
18.79 |
62.85 |
0.152 |
18.79 [9.23] |
86.76 |
0.056 |
18.79 [9.23] |
No |
81.21 |
58.23 |
81.21 [9.23] |
81.18 |
81.21 [9.23] |
|||
Program enrollment |
< 130 |
32.25 |
66.36 |
0.014 |
32.25 [11.44] |
87.66 |
0.009 |
32.25 [11.44] |
130-179 |
28.00 |
55.23 |
28.00 [12.45] |
76.49 |
28.00 [12.45] |
|||
> 179 |
39.74 |
55.92 |
39.74 [17.32] |
81.85 |
39.74 [17.32] |
|||
Percentage of children enrolled in program who are AIAN |
< 90.7 |
54.63 |
60.20 |
0.011 |
54.63 [12.70] |
82.60 |
0.026 |
54.63 [12.70] |
90.7-96 |
19.15 |
66.14 |
19.15 [9.23] |
88.11 |
19.15 [9.23] |
|||
> 96 |
26.21 |
51.65 |
26.21 [9.97] |
77.14 |
26.21 [9.97] |
|||
Percentage of children with a disability who are enrolled in program |
< 9.45 |
24.21 |
55.02 |
0.548 |
24.21 [11.53] |
74.72 |
0.015 |
24.21 [11.53] |
9.45-17.58 |
34.58 |
61.60 |
34.58 [12.35] |
86.61 |
34.58 [12.35] |
|||
> 17.58 |
41.21 |
59.39 |
41.21 [17.27] |
82.96 |
41.21 [17.27] |
|||
Percentage of children enrolled in program who are age 4+ |
< 44.96 |
20.72 |
61.53 |
0.775 |
20.72 [10.62] |
83.05 |
0.965 |
20.72 [10.62] |
44.96-50.56 |
26.27 |
59.18 |
26.27 [10.93] |
81.95 |
26.27 [10.93] |
|||
> 50.56 |
53.01 |
58.10 |
53.01 [16.53] |
82.04 |
53.01 [16.53] |
|||
Percentage of staff who left |
< 10 |
49.20 |
60.50 |
0.166 |
49.20 [15.64] |
83.87 |
0.044 |
49.20 [15.64] |
10-16.67 |
25.26 |
62.66 |
25.26 [11.11] |
86.01 |
25.26 [11.11] |
|||
> 16.67 |
25.55 |
52.88 |
25.55 [9.88] |
75.33 |
25.55 [9.88] |
|||
Percentage of staff replaced |
< 25 |
44.84 |
61.97 |
0.330 |
44.84 [17.24] |
83.57 |
0.782 |
44.84 [17.24] |
25-87.5 |
30.05 |
54.21 |
30.05 [12.53] |
80.46 |
30.05 [12.53] |
|||
> 87.5 |
25.12 |
59.82 |
25.12 [11.69] |
81.94 |
25.12 [11.69] |
|||
Percentage of lead teachers who left |
< 5 |
54.72 |
62.53 |
0.158 |
54.72 [15.06] |
84.87 |
0.319 |
54.72 [15.06] |
5-12.5 |
24.86 |
51.86 |
24.86 [10.25] |
78.98 |
24.86 [10.25] |
|||
> 12.5 |
20.43 |
58.72 |
20.43 [9.93] |
79.09 |
20.43 [9.93] |
|||
Service type |
Center only |
90.61 |
60.08 |
0.047 |
90.61 [7.19] |
83.16 |
0.001 |
90.61 [7.19] |
Center + home |
9.39 |
49.59 |
9.39 [7.19] |
73.16 |
9.39 [7.19] |
|||
Continuous variables at child level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Age in months |
48.45 |
|
0.757 |
48.33 [1.20] |
|
0.627 |
48.32 [1.26] |
|
Months enrolled in Head Start |
3.80 |
|
0.392 |
3.72 [0.71] |
|
0.595 |
3.73 [0.69] |
|
Continuous variables at center level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Center’s child enrollment |
80.85 |
|
0.998 |
80.88 [11.89] |
|
0.731 |
80.83 [11.89] |
|
Center’s number of classrooms |
4.94 |
|
0.858 |
4.94 [0.81] |
|
0.583 |
4.94 [0.81] |
|
Continuous variables at program level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Program enrollment |
273.54 |
|
0.049 |
273.54 [107.03] |
|
0.202 |
273.54 [107.03] |
|
Proportion of children enrolled in program who are AIAN |
0.81 |
|
0.434 |
0.81 [0.04] |
|
0.608 |
0.81 [0.04] |
|
Proportion of children with a disability who are enrolled in program |
0.15 |
|
0.593 |
0.15 [0.02] |
|
0.181 |
0.15 [0.02] |
|
Proportion of children enrolled in program who are age 4+ |
0.55 |
|
0.567 |
0.55 [0.04] |
|
0.751 |
0.55 [0.04] |
|
Proportion of staff who left |
0.11 |
|
0.356 |
0.11 [0.03] |
|
0.159 |
0.11 [0.03] |
|
Proportion of staff replaced |
0.43 |
|
0.333 |
0.43 [0.14] |
|
0.499 |
0.43 [0.14] |
|
Proportion of lead teachers who left |
0.10 |
|
0.871 |
0.10 [0.04] |
|
0.182 |
0.10 [0.04] |
|
Note: Bolded p-values highlight values less than 0.05.
All continuous variables were also included as categorical (ordinal) variables, divided into tertiles (sometimes into binary variables) based on the full sample distribution.
N.C. means the Rao-Scott chi-square statistic was not calculable for that variable (usually, but not exclusively, due to one or more categories with 100 percent response and 0 percent nonresponse).
a MSA refers to whether the program’s zip code was within a metropolitan statistical area.
Table 6.c. Nonresponse bias analysis at the child level: survey combinations
Variable |
Value |
Full sample estimate |
Parent + spring TCR response rate by subgroup |
Parent + spring TCR respondents vs. nonrespondents p-value |
Parent + spring TCR respondents’ estimate [standard error] |
Parent + both TCR response rate by subgroup |
Parent + both TCR respondents vs. nonrespondents p-value |
Parent + both TCR respondents’ estimate [standard error] |
Parent + teacher survey + both TCR response rate by subgroup |
Parent + teacher survey + both TCR respondents vs. nonrespondents p-value |
Parent + teacher survey + both TCR respondents’ estimate [standard error] |
A |
B |
C |
D |
E |
F |
G |
H |
I |
J |
K |
L |
Number of children |
686 |
|
|
404 |
|
|
358 |
|
|
330 |
|
Weight |
CNST2WT |
|
|
P21R2WT |
|
|
PR12WT |
|
|
PR12CW |
|
Categorical variables at child level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
||
Age group |
< 48 months |
41.86 |
58.24 |
0.146 |
39.83 [10.40] |
54.46 |
0.382 |
41.85 [11.07] |
50.10 |
0.178 |
41.32 [10.94] |
48+ months |
58.14 |
67.82 |
60.17 [10.40] |
61.17 |
58.15 [11.07] |
59.54 |
58.68 [10.94] |
||||
Sex |
Female |
46.74 |
64.16 |
0.909 |
48.26 [2.62] |
57.81 |
0.805 |
47.96 [2.31] |
54.19 |
0.542 |
46.02 [2.42] |
Male |
53.26 |
63.50 |
51.74 [2.62] |
58.84 |
52.04 [2.31] |
56.81 |
53.98 [2.42] |
||||
Language spoken at home |
English |
93.83 |
63.66 |
0.778 |
93.27 [2.58] |
58.11 |
0.661 |
92.90 [2.83] |
55.28 |
0.696 |
91.99 [2.81] |
Tribal |
3.48 |
72.31 |
4.22 [1.75] |
69.75 |
4.83 [2.28] |
66.35 |
5.55 [2.49] |
||||
Others |
2.69 |
58.07 |
2.51 [1.07] |
52.44 |
2.26 [0.97] |
52.44 |
2.46 [1.04] |
||||
Months enrolled in Head Start |
< 2 |
37.64 |
52.71 |
< .0001 |
37.71 [11.60] |
49.03 |
< .0001 |
37.57 [11.52] |
43.58 |
< .0001 |
35.05 [11.15] |
= 2 |
17.19 |
78.73 |
20.13 [8.65] |
74.34 |
21.16 [8.85] |
72.91 |
24.36 [10.28] |
||||
> 2 |
17.68 |
48.74 |
14.86 [5.05] |
41.89 |
13.97 [4.76] |
39.22 |
13.30 [5.23] |
||||
missing |
27.49 |
79.36 |
27.30 [17.20] |
71.73 |
27.30 [17.20] |
71.73 |
27.30 [17.24] |
||||
Child participation in Early Head Start |
Don’t know |
24.91 |
77.82 |
0.005 |
24.09 [18.66] |
73.92 |
0.001 |
24.10 [18.66] |
73.92 |
< .0001 |
24.19 [18.68] |
No |
57.00 |
58.27 |
59.49 [16.02] |
52.51 |
59.09 [15.92] |
48.52 |
58.74 [15.93] |
||||
Yes |
18.09 |
61.97 |
16.43 [5.26] |
55.35 |
16.81 [5.25] |
52.62 |
17.07 [5.34] |
||||
Primary funding source |
Child care subsidy or Other |
0.48 |
100.00 |
N.C. |
0.56 [0.39] |
100.00 |
N.C. |
0.64 [0.43] |
93.28 |
0.162 |
0.68 [0.45] |
Head Start |
94.02 |
62.98 |
93.43 [4.02] |
57.90 |
93.72 [3.86] |
54.99 |
93.68 [3.88] |
||||
State pre-K |
1.95 |
87.11 |
3.18 [2.29] |
87.11 |
3.18 [2.29] |
87.11 |
3.18 [2.29] |
||||
Tribal |
3.54 |
68.03 |
2.83 [2.19] |
49.04 |
2.46 [1.86] |
49.04 |
2.46 [1.86] |
||||
Categorical variables at center level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Center’s child enrollment |
< 35 |
24.98 |
55.82 |
0.588 |
28.02 [8.93] |
53.66 |
0.662 |
28.02 [8.92] |
51.57 |
0.842 |
23.13 [7.08] |
35-68 |
23.34 |
70.20 |
25.40 [10.74] |
66.69 |
25.40 [10.74] |
61.39 |
29.92 [11.64] |
||||
> 68 |
51.68 |
64.79 |
46.58 [12.68] |
56.87 |
46.58 [12.67] |
54.91 |
46.95 [12.97] |
||||
Center’s number of classrooms |
< 3 |
27.10 |
57.71 |
0.766 |
30.30 [9.85] |
55.41 |
0.888 |
30.30 [9.85] |
53.48 |
0.970 |
25.41 [8.15] |
3-4 |
24.91 |
66.04 |
26.81 [12.19] |
62.55 |
26.81 [12.19] |
57.59 |
31.33 [12.99] |
||||
> 4 |
47.99 |
66.09 |
42.88 [14.09] |
57.85 |
42.88 [14.08] |
55.74 |
43.26 [14.31] |
||||
Categorical variables at program level |
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
(Percent) |
|
(Column Percent) |
|
Program geographic area |
A |
1.16 |
54.76 |
0.283 |
1.16 [0.49] |
8.09 |
0.071 |
1.16 [0.27] |
8.09 |
0.037 |
1.16 [0.27] |
B |
18.99 |
53.52 |
18.99 [7.78] |
48.35 |
18.99 [7.78] |
48.35 |
18.99 [7.82] |
||||
C |
33.59 |
57.80 |
33.59 [10.65] |
55.45 |
33.59 [10.65] |
47.20 |
33.59 [11.04] |
||||
D |
28.84 |
79.67 |
28.84 [16.26] |
75.92 |
28.84 [16.26] |
75.92 |
28.84 [16.31] |
||||
E |
17.42 |
60.95 |
17.42 [7.14] |
49.15 |
17.42 [7.14] |
49.15 |
17.42 [7.18] |
||||
MSAa |
Yes |
18.79 |
70.63 |
0.569 |
19.03 [9.44] |
59.92 |
0.893 |
18.25 [9.33] |
59.92 |
0.725 |
18.25 [9.37] |
No |
81.21 |
62.23 |
80.97 [9.44] |
58.00 |
81.75 [9.33] |
54.59 |
81.75 [9.37] |
||||
Program enrollment |
< 130 |
32.25 |
53.97 |
0.096 |
32.50 [11.67] |
47.76 |
0.015 |
31.71 [11.54] |
47.76 |
0.127 |
31.71 [11.63] |
130-179 |
28.00 |
59.65 |
29.70 [13.12] |
52.76 |
29.70 [13.12] |
48.55 |
26.93 [13.94] |
||||
> 179 |
39.74 |
74.72 |
37.80 [17.81] |
70.91 |
38.58 [17.66] |
66.90 |
41.35 [17.79] |
||||
Percentage of children enrolled in program who are AIAN |
< 90.7 |
54.63 |
72.18 |
0.043 |
55.70 [13.07] |
69.39 |
0.001 |
54.91 [13.21] |
69.39 |
< .0001 |
57.42 [13.66] |
90.7-96 |
19.15 |
66.08 |
19.70 [9.63] |
55.68 |
20.48 [9.71] |
50.39 |
21.24 [10.24] |
||||
> 96 |
26.21 |
44.70 |
24.60 [10.18] |
37.34 |
24.60 [10.18] |
30.63 |
21.35 [10.74] |
||||
Percentage of children with a disability who are enrolled in program |
< 9.45 |
24.21 |
47.68 |
0.058 |
25.07 [11.88] |
44.44 |
0.110 |
25.07 [11.87] |
42.29 |
0.162 |
22.30 [12.85] |
9.45-17.58 |
34.58 |
60.46 |
35.54 [13.00] |
54.53 |
35.54 [13.00] |
50.96 |
37.55 [14.21] |
||||
> 17.58 |
41.21 |
76.09 |
39.39 [17.65] |
69.75 |
39.39 [17.65] |
67.29 |
40.15 [17.91] |
||||
Percentage of children enrolled in program who are age 4+ |
< 44.96 |
20.72 |
66.90 |
0.892 |
21.02 [10.85] |
61.50 |
0.926 |
21.80 [10.96] |
58.32 |
0.797 |
21.80 [11.00] |
44.96-50.56 |
26.27 |
61.04 |
28.24 [11.42] |
56.28 |
27.46 [11.31] |
50.23 |
32.73 [11.18] |
||||
> 50.56 |
53.01 |
63.97 |
50.74 [16.88] |
58.16 |
50.74 [16.88] |
57.18 |
45.47 [17.18] |
||||
Percentage of staff who left |
< 10 |
49.20 |
65.46 |
0.358 |
47.13 [15.82] |
60.01 |
0.423 |
47.13 [15.82] |
59.29 |
0.276 |
47.89 [16.21] |
10-16.67 |
25.26 |
73.22 |
24.96 [11.27] |
66.86 |
24.96 [11.27] |
64.26 |
24.96 [11.33] |
||||
> 16.67 |
25.55 |
51.33 |
27.91 ]10.13] |
46.78 |
27.91 [10.13] |
39.90 |
27.15 [11.30] |
||||
Percentage of staff replaced |
< 25 |
44.84 |
62.06 |
0.917 |
42.77 [17.47] |
57.94 |
0.967 |
42.77 [17.46] |
55.68 |
0.999 |
43.53 [17.76] |
25-87.5 |
30.05 |
62.92 |
30.91 [12.83] |
56.93 |
30.91 [12.83] |
55.20 |
28.14 [13.69] |
||||
> 87.5 |
25.12 |
67.99 |
26.32 [11.88] |
60.82 |
26.32 [11.88] |
55.90 |
28.33 [13.21] |
||||
Percentage of lead teachers who left |
< 5 |
54.72 |
66.73 |
0.721 |
54.72 [14.96] |
60.91 |
0.750 |
53.93 [15.10] |
59.71 |
0.667 |
53.93 [15.28] |
5-12.5 |
24.86 |
56.87 |
23.66 [10.48] |
51.79 |
23.66 [10.48] |
48.27 |
21.64 [11.62] |
||||
> 12.5 |
20.43 |
64.43 |
21.63 [10.08] |
59.53 |
22.41 [10.19] |
53.48 |
24.43 [11.69] |
||||
Service type |
Center only |
90.61 |
63.07 |
0.373 |
89.95 [7.78] |
57.62 |
0.421 |
89.17 [7.85] |
55.93 |
0.780 |
87.15 [9.60] |
Center + home |
9.39 |
70.97 |
10.05 [7.78] |
65.47 |
10.83 [7.85] |
52.31 |
12.85 [9.60] |
||||
Continuous variables at child level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Age in months |
48.45 |
|
0.504 |
48.50 [1.38] |
|
0.783 |
48.22 [1.42] |
|
0.484 |
48.53 [1.34] |
|
Months enrolled in Head Start |
3.80 |
|
0.302 |
3.31 [0.62] |
|
0.192 |
3.15 [0.57] |
|
0.229 |
3.06 [0.63] |
|
Continuous variables at center level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Center’s child enrollment |
80.85 |
|
0.186 |
78.41 [12.42] |
|
0.530 |
78.27 [12.44] |
|
0.612 |
80.73 [11.90] |
|
Center’s number of classrooms |
4.94 |
|
0.275 |
4.77 [0.83] |
|
0.810 |
4.77 [0.83] |
|
0.846 |
4.90 [0.82] |
|
Continuous variables at program level |
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
|
(Mean) |
|
Program enrollment |
273.54 |
|
0.153 |
271.63 [107.31] |
|
0.139 |
272.98 [107.01] |
|
0.160 |
273.64 [107.19] |
|
Proportion of children enrolled in program who are AIAN |
0.81 |
|
0.028 |
0.81 [0.04] |
|
0.010 |
0.81 [0.04] |
|
0.004 |
0.81 [0.04] |
|
Proportion of children with a disability who are enrolled in program |
0.15 |
|
0.011 |
0.15 [0.02] |
|
0.038 |
0.15 [0.02] |
|
0.089 |
0.15 [0.02] |
|
Proportion of children enrolled in program who are age 4+ |
0.55 |
|
0.962 |
0.54 [0.04] |
|
0.864 |
0.54 [0.04] |
|
0.914 |
0.53 [0.04] |
|
Proportion of staff who left |
0.11 |
|
0.438 |
0.11 [0.03] |
|
0.386 |
0.11 [0.03] |
|
0.211 |
0.11 [0.03] |
|
Proportion of staff replaced |
0.43 |
|
0.666 |
0.45 [0.14] |
|
0.818 |
0.44 [0.14] |
|
0.935 |
0.46 [0.15] |
|
Proportion of lead teachers who left |
0.10 |
|
0.354 |
0.11 [0.04] |
|
0.430 |
0.11 [0.04] |
|
0.281 |
||
Note: Bolded p-values highlight values less than 0.05.
All continuous variables were also included as categorical (ordinal) variables, divided into tertiles (sometimes into binary variables) based on the full sample distribution.
N.C. means the Rao-Scott chi-square statistic was not calculable for that variable (usually, but not exclusively, due to one or more categories with 100 percent response and 0 percent nonresponse).
a MSA refers to whether the program’s zip code was within a metropolitan statistical area.
1 If there were no child-level respondents within a center, the weighting cell was the program; if there were no respondents within a program, the weighting cell was the program stratum. For two weights (PR12WT and PR12CW), the weighting cell was center crossed with child’s sex (and, if needed, program by sex, or program stratum by sex).
2 We did not examine any classroom-level covariates because they were not available.
3 Used only for program, center, and classroom analyses. Not applicable for child analyses.
4 We have masked census region in the tables as regions A, B, C, and D to minimize the risk for data disclosure.
5 Using a Rao-Scott chi-square test in SAS SurveyFreq procedure.
6 Using a t-test in SAS SurveyMeans for continuous variables.
7 There are several instances in which the categorical version of the variable shows a significant difference between respondents and nonrespondents but the continuous version does not, or vice versa. We do not know what these scenarios signify other than the possibility of these achieving statistical significance by chance (Type I error), or because of the choice of using the 33rd and 67th percentiles as cut points.
8 In the full sample, 92.42 percent of the children have Head Start as their primary funding source; 4.88 percent have State Pre-K funding; and 2.71 funded by Childcare subsidy or something else. After nonresponse weighting, these percentages were 96.28, 2.03, and 1.69, respectively.
9 If there were no respondents at the child level within a center, the weighting cell was the program; if there were no respondents within a program, the weighting cell was the program stratum. For two weights (PR12WT and PR12CW), the weighting cell was center crossed with child’s sex (and, if needed, program by sex, or program stratum by sex).
10 We did not examine any classroom-level covariates because none were available.
11 Using a Rao-Scott Chi-square test in SAS SurveyFreq procedure.
12 Using a t-test in SAS SurveyMeans for continuous variables.
13 For Table 6.c, these repeat again for a third respondent definition in columns J, K, and L.
14 In Tables 4, 5, 6.b, and 6.c, some cells indicate “N.C.” instead of a p-value. This means that the Rao-Scott chi-square statistic was not calculable for that variable (usually, but not exclusively, due to one or more categories with 100 percent response and 0 percent nonresponse).
15 There are several instances in which the categorical version of the variable shows a significant difference between respondents and nonrespondents, but the continuous version does not, or vice versa. We do not know what these scenarios signify other than the possibility of these achieving statistical significance by chance (Type I error), or the choice of using the 33rd and 67th percentiles as cut points.
16 After weighting, 35 percent of children are estimated to be in centers with fewer than 35 children, 9 percent in centers with 35 to 68 children, and 56 percent in centers with more than 68 children. The percentages for the full sample of children are 25, 23, and 52 percent, respectively. (We note that the mean for the continuous version of this variable resolved to be almost identical to the target mean after weighting.) After weighting, 0.4 percent of children are estimated to be in programs with both center- and home-based services, whereas the percentage for the full sample of children is 9.4 percent.
17 We present percentage ranges for the categorical versions of continuous variables but present the mean proportions for the continuous versions of those variables.
18 It is worth noting that the variable for child’s sex closely approached our significance level with a p-value of 0.051 (associated with a difference of 1.8 percentage points). After weighting adjustments for nonresponse, the difference narrowed to 0.3 percentage points.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Subject | memo |
| Author | Barbara Carlson |
| File Modified | 0000-00-00 |
| File Created | 2022-03-28 |