2023-24 National Teacher and Principal Survey (NTPS 2023-24)
OMB# 1850-0598 v.45
Supporting Statement Part B
National Center for Education Statistics (NCES) U.S. Department of Education
January 2023 revised April 2023 revised December 2023 revised February 2024
|
|
Part B Collection of Information Employing Statistical Methods 3
B.1 Study Universe, Sample Design, and Estimation 3
B.1.1 Study Universe and Sample Design 3
B.1.2 Precision Requirements and Sample Sizes 4
B.2 Procedures for the Collection of Information 8
B.2.1 Preliminary Field Activities 8
B.2.1.1 Special Contact District Operation 8
B.2.1.2 School Precontact Notification 10
B.2.1.3 Screener Data Collection 10
B.2.2 School-level Data Collection Procedures 10
B.2.2.1 Schools without Publicly Available Teacher Data Available (Field Data Collection Path) 11
B.2.2.2 Schools with Publicly Available Teacher Data (Traditional Data Collection Path) 11
B.2.3 Teacher Data Collection 14
B.3 Methods to Secure Cooperation, Maximize Response Rates, and Deal with Nonresponse 15
B.3.1 Methods to Secure Cooperation and Maximize Response Rates 15
B.3.2 Methods to Minimize Nonresponse 15
B.3.2.1 Data Collection Strategies to Minimize Non-Response 16
B.3.2.2 Statistical Approaches to Nonresponse 19
B.4 Tests of Methods and Procedures 19
B.4.1 Tests Included in the Design of NTPS 2023-24 19
B.4.1.1 Testing at the School-level 20
B.4.1.2 Testing at the Teacher-level 24
B.5 Individuals Responsible for Study Design and Performance 29
This request is to conduct NTPS 2023-24 recruitment and data collection activities. Because of the overlap in time, this request also carries over the burden and materials for the NTPS 2023-24 preliminary activities (OMB# 1850-0598 v.41). Section B.1 of this document describes the study universe, sample design, and estimation details for NTPS 2023-24. Section B.2 describes the data collection procedures for NTPS 2023-24, including the preliminary field activities requested for approval in the earlier submission (OMB# 1850-0598 v.41). Section B.3 discusses methods to secure cooperation and mitigate nonresponse, focusing on methods used to improve response rates in NTPS 2017-18 and NTPS 2020-21 and the use of those methods in NTPS 2023-24. Section B.4 describes recent developments in a long history of tests of methods and procedures to improve data quality. Section B.5 lists the names of those involved in the design of the study and the development of these materials.
Section B.1.1 includes information on the study universe of interest and sample design planned for NTPS 2023-24, based on the NTPS 2020-21 sample design. Section B.1.2 describes the precision requirements and target sample sizes set out for the study.
The NTPS 2023-24 school sample will be drawn in April-May 2023, and sampled schools will begin to be contacted in June 2023.To be eligible for inclusion in the sample, schools must: provide classroom instruction to students; have one or more teachers who provide instruction; serve students in at least one of grades 1-12 or the ungraded equivalent; be located in one or more buildings; and be located in the United States.
The most recent final Common Core of Data (CCD) file available from NCES—the 2021-22 CCD—will be used to construct the public-school frame. The universe for charter schools will be identified as those public charter schools that meet the NTPS definition of an eligible school found on the CCD. The 2021-22 CCD includes 102,130 public, including public charter, schools. The study universe will be adjusted to remove kindergarten-terminal schools, which are not eligible for NTPS. Kindergarten-terminal schools are schools on the CCD that offer no level of instruction higher than kindergarten. Schools that only offer pre-kindergarten instruction or that provide prekindergarten and kindergarten instruction, but no higher grades, are examples of kindergarten-terminal schools. The public-school universe is expected to include about 93,400 eligible public schools.
The private school universe will be drawn from the 2021-22 Private School Survey (PSS) universe, which includes 38,790 active private schools. The study universe will be adjusted to remove preschools and schools with kindergarten as the highest grade offered. The private school universe is expected to include about 23,200 eligible active private schools.
Details of the first-stage sample design of schools are provided in section B.1.2.
Teachers will be randomly sampled within the second design stage from roster information provided by each participating sampled school or from publicly available teacher rosters. Teachers within the sampled school will be classified as ineligible for NTPS if they are a short-term substitute teacher, student teacher, or a teacher’s aide; or if they do not teach any of grades K-12 or comparable ungraded levels. The information that classifies teachers as ineligible will be based on responses to the Teacher Questionnaire.
This section details the approximate school sample sizes and precision requirements for the NTPS 2023-24 public and private school samples.
The final NTPS 2023-24 public school sample will include approximately:
10,100 schools and their school principals (8,730 traditional public schools/principals and 1,370 public charter schools/principals), with the goal of at least 6,700 completed interviews each for schools and principals; and
59,673 teachers (52,568 traditional public-school teachers and 7,105 public charter school teachers), with the goal of at least 34,722 completed interviews.
The final NTPS 2023-24 private school sample will include approximately:
3,120 schools and their principals, with the goal of at least 1,750 completed interviews each for schools and principals; and
8,316 teachers, with the goal of at least 4,500 completed interviews.
The NTPS 2023-24 oversampling stratification will be based primarily on the following domains:
Charter/Non-charter status;
School Level (elementary, middle, secondary/high, combined/other);
Urbanicity (city, suburb, town, rural);and
State.
The NCES standards for publication (“publishability” metrics) indicate that the coefficient of variation (CV) must be no larger than 50%, and, if the CV is between 30% and 50%, that the estimates be published with a caveat. For a population proportion of 20%, a CV of 30% corresponds to a standard error of 6%. To make sure that we meet the CV 30% threshold with the uncertainties about response rates and about the exact values of design effects, we set as a target a CV of 25% as an upper bound. This corresponds to an expected standard error of 5%. This considerably reduces the chance of not meeting the 30% threshold (e.g., if we set 30% itself as the upper bound, we would be above it half of the time). Our target goal then for each state is to make sure that the expected standard error is no larger than 5% for a population proportion of 20% (a CV of 25%), at both the school and teacher level. The teacher sample size for a sampled school should be proportional to the product of the final teacher multiplier (based on the expected completion rate adjustment factors), final school oversampling factor, and measure of size for the school (square root of the number of full-time teachers). The overall target number of completed interviews is 6,700, assuming the completion rate for the NTPS 2023-24 will be similar to the rate for NTPS 2020-21. The sample needs to include at least 10,096 schools and school principals to yield the target number of completed teacher surveys. For ease of reporting, the sample size for public schools and public-school principals will be 10,100.
Table 1 presents a portion of the analysis for public schools by the key domains of school type, school level, urbanicity, enrollment, and poverty status. Presented are the anticipated number of responding schools or principals for the NTPS design and the expected precision based on analyses using prior-cycle NTPS final response rates and CV of 25%.
Table 1. NTPS 2023-24 public school domain number of schools on the universe, expected completed interviews, expected standard error, and design effect, by school domain under the proposed oversampling scheme to achieve 25% CV or less
School domain |
Universe schools |
Expected completed interviews |
Expected standard error |
Design effect |
All |
93,388 |
6,700 |
0.61% |
1.57 |
Charter |
7,352 |
824 |
1.62% |
1.36 |
Non-charter |
86,036 |
5,876 |
0.65% |
1.55 |
Elementary |
51,444 |
3,104 |
0.86% |
1.44 |
Middle |
15,539 |
1,231 |
1.35% |
1.39 |
Secondary/High |
22,793 |
1,816 |
1.26% |
1.80 |
Combined/Other |
3,611 |
548 |
2.00% |
1.37 |
City |
26,105 |
1,940 |
1.14% |
1.57 |
Suburban |
30,279 |
1,995 |
1.09% |
1.49 |
Town |
12,248 |
1,042 |
1.57% |
1.60 |
Rural |
24,755 |
1,722 |
1.23% |
1.62 |
Enrollment < 100 |
8,113 |
337 |
3.05% |
1.96 |
100 <= Enrollment < 200 |
7,533 |
482 |
2.32% |
1.62 |
200 <= Enrollment < 500 |
37,049 |
2,448 |
0.97% |
1.44 |
500 <= Enrollment < 750 |
22,434 |
1,602 |
1.16% |
1.35 |
750 <= Enrollment < 1,000 |
8,915 |
776 |
1.66% |
1.33 |
1,000 <= Enrollment |
9,344 |
1,056 |
1.37% |
1.24 |
Percent FRPL < 35% |
27,201 |
2,091 |
1.12% |
1.63 |
35% <= Percent FRPL < 50% |
16,088 |
1,229 |
1.41% |
1.53 |
50% <= Percent FRPL < 75% |
25,315 |
1,755 |
1.19% |
1.55 |
75% <= Percent FRPL |
24,784 |
1,624 |
1.23% |
1.53 |
Note: Details may not sum to total due to rounding.
Table 2 presents the analogous precision analysis for public schools by state.
Table 2. NTPS 2023-24 public school number of schools on the universe, expected completed interviews, expected standard error, and design effect, by state under the proposed oversampling scheme to achieve 25% CV or less
State |
Universe schools |
Expected completed interviews |
Expected standard error |
Design effect |
All |
93,388 |
6,700 |
0.61% |
1.57 |
AK |
502 |
98 |
4.61% |
1.3 |
AL |
1,485 |
97 |
5.00% |
1.51 |
AR |
948 |
84 |
4.99% |
1.3 |
AZ |
2,355 |
142 |
4.34% |
1.67 |
CA |
10,257 |
544 |
2.12% |
1.53 |
CO |
1,744 |
104 |
4.74% |
1.46 |
CT |
1,003 |
100 |
4.46% |
1.24 |
DC |
213 |
60 |
4.94% |
0.91 |
DE |
214 |
64 |
4.96% |
0.99 |
FL |
4,094 |
288 |
2.97% |
1.59 |
GA |
2,279 |
159 |
3.60% |
1.29 |
HI |
293 |
69 |
4.96% |
1.05 |
IA |
1,152 |
83 |
4.95% |
1.28 |
ID |
715 |
83 |
4.97% |
1.28 |
IL |
4,019 |
223 |
3.19% |
1.42 |
IN |
1,880 |
113 |
4.33% |
1.32 |
KS |
1,234 |
85 |
4.97% |
1.32 |
KY |
1,459 |
92 |
4.97% |
1.43 |
LA |
1,353 |
93 |
4.98% |
1.43 |
MA |
1,771 |
107 |
4.56% |
1.39 |
MD |
1,400 |
86 |
4.95% |
1.32 |
ME |
5,96 |
77 |
4.96% |
1.19 |
MI |
3,355 |
188 |
3.61% |
1.53 |
MN |
2,103 |
121 |
4.82% |
1.76 |
MO |
2,017 |
113 |
4.40% |
1.36 |
MS |
1,055 |
87 |
4.96% |
1.19 |
MT |
554 |
96 |
4.93% |
1.46 |
NC |
2,684 |
172 |
3.51% |
1.33 |
ND |
421 |
78 |
4.94% |
1.19 |
NE |
851 |
85 |
4.98% |
1.31 |
NH |
454 |
78 |
4.95% |
2.19 |
NJ |
2,471 |
153 |
3.76% |
1.35 |
NM |
829 |
88 |
4.98% |
1.37 |
NV |
700 |
90 |
4.93% |
1.38 |
NY |
4,751 |
334 |
2.56% |
1.37 |
OH |
3,243 |
183 |
3.40% |
1.32 |
OK |
1,433 |
91 |
4.99% |
1.41 |
OR |
1,242 |
88 |
4.98% |
1.37 |
PA |
2,948 |
188 |
3.36% |
1.32 |
RI |
307 |
67 |
4.99% |
1.04 |
SC |
1,254 |
82 |
4.99% |
1.27 |
SD |
489 |
90 |
4.92% |
1.36 |
TN |
1,797 |
112 |
4.36% |
1.34 |
TX |
8,808 |
585 |
2.01% |
1.48 |
UT |
1,020 |
97 |
4.96% |
1.49 |
VA |
2,051 |
134 |
3.95% |
1.31 |
VT |
317 |
70 |
4.98% |
1.08 |
WA |
2,282 |
116 |
4.56% |
1.5 |
WI |
1,933 |
107 |
4.53% |
1.37 |
WV |
720 |
80 |
4.99% |
1.24 |
WY |
333 |
79 |
4.95% |
1.21 |
Note: Details may not sum to total due to rounding.
Table 3 provides the analogous precision analysis for public school teachers. The expected standard errors were calculated based on analyses using the NTPS 2020-21 final response rates and CV of 25%.
Table 3. NTPS 2023-24 major domain number of schools on the frame, expected public school teacher completed surveys, standard errors, and design effects with state oversampling to achieve 25% CV or less, by school domain
School domain |
Frame full-time equivalent teachers (in 1000s) |
Expected teacher completed surveys |
Expected standard error |
Design effect |
All |
3,132.8 |
34,722 |
0.37% |
2.97 |
Charter |
183.7 |
3,878 |
1.02% |
2.52 |
Non-charter |
2,949.1 |
30,844 |
0.39% |
2.90 |
Elementary |
1,484.5 |
13,819 |
0.55% |
2.61 |
Middle |
587.3 |
6,780 |
0.81% |
2.78 |
Secondary/high |
964.1 |
11,367 |
0.68% |
3.32 |
Combined/other |
97.0 |
2,757 |
1.36% |
3.17 |
City |
930.0 |
10,423 |
0.66% |
2.88 |
Suburban |
1,223.2 |
11,451 |
0.64% |
2.90 |
Town |
350.0 |
5,067 |
0.96% |
2.89 |
Rural |
629.5 |
7,781 |
0.78% |
2.97 |
Enrollment < 100 |
40.1 |
740 |
2.26% |
2.36 |
100 <= Enrollment < 300 |
91.7 |
1,511 |
1.65% |
2.56 |
300 <= Enrollment < 500 |
896.0 |
10,348 |
0.64% |
2.64 |
500 <= Enrollment < 750 |
838.4 |
8,447 |
0.72% |
2.72 |
750 <= Enrollment < 1,000 |
453.6 |
4,840 |
0.99% |
2.98 |
1,000 <= Enrollment |
812.9 |
8,836 |
0.80% |
3.50 |
Percent FRPL < 35% |
1,010.3 |
11,377 |
0.66% |
3.10 |
35% <= Percent FRPL < 50% |
549.7 |
6,401 |
0.88% |
3.08 |
50% <= Percent FRPL < 75% |
814.9 |
8,945 |
0.71% |
2.79 |
75% <= Percent FRPL |
757.8 |
8,000 |
0.76% |
2.91 |
Note: Details may not sum to total due to rounding.
The NTPS 2023-24 private school sample will be roughly the same as for NTPS 2020-21, which included 3,000 schools and school principals, with the goal of at least 1,750 completed interviews for each. The NTPS 2023-24 private school sample will need to include 3,120 schools and school principals to yield the target number of completed interviews of 1,750 for each group.
The sampling plan oversamples as follows:
Elementary/middle schools are sampled at a rate proportional to their measure of size;
Secondary/high schools are sampled at a rate proportional to 2.80 times their measure of size;
Combined/other schools are sampled at a rate proportional to 1.22 times their measure of size;
The oversampling rates for nonreligious schools are increased by an additional factor of 1.28 (e.g., the oversampling rate for nonreligious secondary/high schools is 2.80 ×1.28 = 3.58); and
The oversampling rates for Baptist schools are increased by an additional factor of 1.23 (e.g., the oversampling rate for Baptist combined/other schools is 1.22 ×1.23 = 1.50).
Under this design, all the precision goals are achieved and all the CVs for the public-school teacher domains are less than 25% except for the vocational/technical teacher domain. Note that the preceding statement has not yet been confirmed for NTPS 2023-24, but it is tentatively correct based upon NTPS 2020-21; any changes will be presented in a change request.
For private school teachers, the expected number of completed interviews is estimated to be proportional to the product of the final school oversampling factor and the number of full time equivalent (FTE) teachers over schools in the domain. The overall target number of completed interviews is 4,500. Assuming the completion rate for NTPS 2023-24 will be similar to the rate for NTPS 2020-21, the sample size needs to be 8,316 teachers in order to yield the target number of completed teacher interviews. The teacher sample size for a domain should be proportional to the product of the final teacher multiplier (based on the expected completion rate adjustment factors), final school oversampling factor, and the measure of size for the school (square root of the number of full-time teachers). Note that the sample size calculation may change based upon potential changes associated with teacher sampling and weighting for NTPS 2023-24; any changes will be presented in a change request.
Tables 4 and 5 show expected sample sizes, standard errors, and CVs for population percentages of 20% by the key private school domains of school type, grade level, and region. Table 4 presents a portion of the analysis for private schools by affiliation, grade level, and region.
Table 4. NTPS 2023-24 private school domain number of schools on the universe, expected completed interviews, standard error, and design effect, by school domain
School domain |
Universe schools |
Expected completed school interviews |
For 20% population percentage |
|||
Expected standard error |
95% CI half-width |
Design effect |
CV |
|||
All |
23,247 |
1,750 |
1.2% |
2.4% |
1.58 |
6.0% |
Catholic |
6,202 |
543 |
2.1% |
4.1% |
1.48 |
10.4% |
Other religious |
11,063 |
640 |
2.0% |
3.8% |
1.53 |
9.8% |
Nonsectarian |
5,982 |
567 |
2.0% |
3.9% |
1.44 |
10.1% |
Elementary/middle |
12,795 |
609 |
1.8% |
3.6% |
1.29 |
9.2% |
Secondary/high |
2,949 |
555 |
1.8% |
3.6% |
1.16 |
9.1% |
Combined/other |
7,503 |
587 |
1.9% |
3.7% |
1.29 |
9.4% |
Northeast |
5,493 |
461 |
2.5% |
5.0% |
1.87 |
12.7% |
Midwest |
5,832 |
374 |
2.6% |
5.1% |
1.57 |
12.9% |
South |
7,557 |
581 |
2.0% |
3.9% |
1.41 |
9.8% |
West |
4,365 |
334 |
2.7% |
5.3% |
1.51 |
13.5% |
Note: Details may not sum to total due to rounding.
Table 5 provides the analogous precision analysis for private school teachers.
Table 5. NTPS 2023-24 major domain expected private school teacher completed interviews, by school domain
School domain |
Frame teachers (FTE) |
Expected completed teacher interviews |
For 20% population percentage |
|||
Expected standard error |
Design effect |
CV |
|
|||
All |
420,420 |
4,500 |
1.07% |
3.21 |
5.34% |
|
Catholic |
140,907 |
1,532 |
1.86% |
3.31 |
9.29% |
|
Other religious |
160,558 |
1,460 |
1.75% |
2.78 |
8.73% |
|
Nonsectarian |
118,955 |
1,508 |
1.96% |
3.63 |
9.82% |
|
Elementary/middle |
167,029 |
1,230 |
1.81% |
2.51 |
9.03% |
|
Secondary/high |
76,741 |
1,602 |
1.69% |
2.87 |
8.47% |
|
Combined/other |
176,650 |
1,668 |
1.73% |
3.14 |
8.67% |
|
Northeast |
112,224 |
1,289 |
1.98% |
3.16 |
9.90% |
|
Midwest |
85,573 |
868 |
2.36% |
3.02 |
11.80% |
|
South |
148,988 |
1,511 |
1.86% |
3.25 |
9.28% |
|
West |
73,636 |
831 |
2.54% |
3.36 |
12.72% |
|
Note: Details may not sum to total due to rounding.
The NTPS 2023-24 will use implicit stratification based on the proposed systematic sampling sort order, which uses a hierarchy of domains which vary between public and private schools. For private schools, the domains are:
Three-level affiliation (Catholic, non-Catholic religious, nonreligious);
School level (elementary/middle, secondary/high, combined/other);
Four-level Census region (Northeast, Midwest, South, West);
Four-level urbanicity (city, suburb, town, rural);
Eleven-level affiliation;
Five-level school size (enrollment: <100, 100-199, 200-499, 500-749, 750+);
State;
Highest grade offered;
Twelve-level urbanicity (large city, medium-sized city, small city, etc.);
Zip code;
School enrollment; and
PIN number.
And for public schools, the domains are:
Collapsed state cells;
Charter status (charter vs. non-charter);
Collapsed school level (combined/other vs. all other);
Collapsed school enrollment size (enrollment: <100, 100-199, 200+);
School level (combined/other, elementary, middle, secondary/high);
Urbanicity (city, suburb, town, rural);
Collapsed Poverty based on FRPL (high, medium, medium low, and low);
School enrollment size (enrollment: <100, 100-199, 200-499, 500-749, 750-999, 1000+);
State (all 50 states and the District of Columbia); and
Number of FTE teachers.
Teachers in traditional public, public charter, and private schools will be sampled from roster information provided by each participating sampled school or from publicly available teacher data (when the school does not provide teacher information). Teachers that are added to verified prepopulated Teacher Listing Forms (TLFs) will be oversampled to increase the number of new teachers that are sampled. The target numbers of teacher completed interview per school are designed to be proportional to the square root of the number of full-time teachers for each school and assume a completion rate due to nonresponse.
For each sampled traditional public, public charter, and private school, the principal will be included in the survey as a result of their school being selected.
Schools, principals, and teachers will be weighted by the inverse of their probability of selection. The final weight will contain adjustments for nonresponse and any other sampling or field considerations that arise after the sample has been drawn.
We expect the NTPS 2023-24 response rates to fall somewhere between those of NTPS 2017-18 and NTPS 2020-21. Specifically, we expect them to fall lower than the NTPS 2017-18 response rates, given the long-term trend in declining response rates for federal surveys, but not quite as low as NTPS 2020-21, since that survey was administered during the height of the COVID-19 pandemic; it was a nontraditional school year due to widespread school closures and virtual learning environments. Table 6 provides the base-weighted response rates for NTPS 2017-18, and Table 7 provides the base-weighted response rates for NTPS 2020-21.
Table 6. Base-weighted response rates for NTPS 2017-18 by respondent and school type
School type |
Unit of observation |
||
Teacher |
Principal |
School |
|
Public |
76.9 |
70.2 |
72.5 |
Private |
75.9 |
62.6 |
64.5 |
Table 7. Base-weighted response rates for NTPS 2020-21 by respondent and school type
School type |
Unit of observation |
||
Teacher |
Principal |
School |
|
Public |
62.4 |
68.0 |
65.6 |
Private |
60.6 |
61.7 |
61.4 |
Section B.2.1 describes the operation for the preliminary field activities for NTPS 2023-24, with Section B.2.1.1 describing special districts operation and Section B.2.1.2 the school precontact notification. Section B.2.2 describes school-level data collection for the screener and the school-level questionnaires (i.e., Teacher Listing Form, School Questionnaire, and Principal Questionnaire), with Section B.2.2.1 describing the procedures to be used for schools without publicly available teacher data and Section B.2.2.2 for schools with publicly available teacher data. Section B.2.3 describes data collection procedures for the Teacher Questionnaire.
This section describes the preliminary field activities for the NTPS 2023-24, including the special contact district operations, school precontact notification, and screener data collection. These activities were included in OMB# 1850-0598 v.41 and are repeated here to give the reader context for where and how they occur relative to the main data collection procedures.
Special contact districts require that a research application be submitted to and reviewed by the district before it will allow schools under its jurisdiction to participate in a study. Districts are identified as “special contact districts” prior to data collection because they were flagged as such during previous cycles of SASS, NTPS, the School Survey on Crime and Safety (SSOCS), or other NCES studies (e.g., the School Pulse Panel (SPP)). Special contact districts are also newly identified during NTPS data collection when districts indicate that they will not complete the survey until a research application is submitted, reviewed, and approved.
Once a district is identified as a special contact district, basic information about the district is obtained from the NCES Common Core of Data (CCD). The basic information includes the NCES local education agency (LEA) ID number, district name, city, and state. The next step is to search the district’s website for a point of contact and any information available about the district’s requirements for conducting external research. Some districts identified as being a special contact district from the previous cycle may be incorrect and staff will verify whether a given district has requirements for conducting external research before proceeding.
The following are examples of the types of information that will be gathered from each district’s website to prepare a research application for submission to a given district:
Name and contact information for the district office or department that reviews applications to conduct external research, and the name and contact information of the person in charge of that office;
Information about district research application review schedules and submission deadlines;
Whether application fees are required, and if so, how much;
Whether a district sponsor is required;
Whether an online application is required, and if so, the link to the application if possible;
Information about research topics and/or agenda on which the district is focusing;
The web link to the main research department or office website; and
Research guidelines, instructions, application forms, District Action Plans, Strategic Plan or Goals, if any.
Recruitment staff will contact districts by phone and email to obtain key information not listed on the district’s website, (e.g., requirements for the research application, research application submission deadlines, etc.).
NTPS staff have developed a generic research application that covers the information typically requested in district research applications. Staff will customize this generic research application to each district’s specific requirements that need to be addressed or included in the research application (e.g., how the study addresses key district goals, or inclusion of a district study sponsor), or submit the generic application with minimal changes for districts that do not have specific application requirements.
Using information obtained from the district website or phone or email exchanges, a district research request packet will be prepared. Each research application will include the following documents, where applicable:
District research application cover letter;
Research application (district-specific or generic, as required by the district);
Study summary;
FAQ document;
Special contact district approval form;
Participant informed consent form (if required by the district);
NTPS Project Director’s resume;
Copy of questionnaires; and
Application fee (if required by the district).
Where required or requested, applications will include the draft NTPS 2023-24 questionnaires. The NTPS content that most closely matches the NTPS 2023-24 is the content fielded in 2017-18 (for questionnaires other than the Teacher Listing Forms, TLFs) and these are provided in Appendix B of this submission. The teacher and principal longitudinal components to the NTPS are referenced in the research applications, and, therefore, the NTPS 2021-22 Teacher and Principal Follow-up Survey questionnaires are also included in Appendix B of this submission and will be included in district research applications as needed. Additionally, the NTPS 2020-21 questionnaires will be provided to districts that request them. Other information about the study may be required by the district and will be included with the application or provided upon request.
Approximately one week after the application is submitted to the district (either electronically or in hard copy, as required by the district), NTPS district recruitment staff will contact the district’s research office to confirm receipt of the package and to ask when the district expects to review the research application and when a decision will be made. If additional information is requested by the district (e.g., the list of sampled schools), recruitment staff will follow up on such requests and will be available to answer any questions the district may have throughout the data collection period.
Some districts charge a fee (~$50-200) to process research application requests, which will be paid as necessary. Special district operations will begin in early 2023 by contacting up to 100 “certainty” special contact districts for which, due to their size, it is certain that at least one school from their jurisdiction will be randomly sampled. Other special contact districts will be contacted after the sample is drawn in the spring of 2023.
The school precontact notification includes mailing a two-sided, full-color postcard to sampled schools to introduce the survey, promote survey recognition, and verify the school mailing address (see Appendix A). In previous administrations of the NTPS, about 4% of all school addresses have been corrected by the U.S. Post Office in response to the precontact notification (previously a letter), saving time and effort during the main data collection period. Schools will be sent the precontact notification on June 20, 2023.
In July 2023, all schools will receive a letter addressed to the principal at the school address (see Appendix A for all school-level contact materials) which will include instructions for completing a brief screener interview online using the NTPS Screener internet instrument (see Appendix B for all questionnaires). Schools will be sent the screener mailout on July 20, 2023, and the initial screener email on July 24, 2023. The purpose of the Screener is to determine the school’s eligibility for NTPS and to ask the principal to establish a survey coordinator. The survey coordinator will be asked to facilitate the completion of NTPS questionnaires within their school, and materials will be mailed to them throughout data collection. A reminder email will be sent to non-responding school principals on August 2, 2023. Principals who do not self-screen will be contacted by telephone during August 9 – 25, 2023.
In September 2023, after the school precontact notification and the screener interview, main data collection will begin with sending all sampled schools an initial school mailout and an initial email. The mailout package will be addressed to the survey coordinator (when one was established during the screener) or the principal (if a coordinator was not established) at the school mailing address. The package includes a letter to the coordinator or principal introducing the survey and providing login credentials for completing the TLF electronically using the Respondent Portal instrument. It also includes two separate, sealed envelopes containing login credentials for completing the School Questionnaire and the Principal Questionnaire online via their respective survey instruments. The recipient is instructed to distribute these two sealed envelopes to the appropriate school staff and to follow up with them regarding their progress.
For NTPS 2023-24, the level of effort put forth in collecting a TLF from the school – which is needed to draw a sample for the Teacher Questionnaire – for each school will depend primarily on whether publicly available teacher data are available for the school. Publicly available teacher data will include vendor data, and data collected from school and district websites through web scraping and clerical research. Schools without publicly available teacher data do not have teacher roster data to fall back on for the purposes of selecting a teacher sample; therefore, it is important to put forth additional targeted efforts and resources into obtaining a TLF from these schools.
Schools without publicly available teacher data will follow a “field data collection path” and schools with publicly available teacher data will follow a “traditional data collection” path and will not receive field follow-up for their TLF. If schools with publicly available teacher data do not complete their TLF throughout the collection period, teachers will be sampled from the publicly available teacher data – that is, the publicly available teacher data will serve as a replacement TLF for nonresponding schools. Eliminating the field follow-up operation to collect or verify a TLF for a school with publicly available teacher data reduces the amount of time necessary for TLF data collection prior to selecting teachers from the publicly available teacher data. This will enable all teacher sampling from these schools to be completed in one teacher wave in mid-January 2024, allowing more time for these teachers to complete their Teacher Questionnaire prior to the end of data collection.
Following the initial school mailout and email, schools without publicly available teacher data will receive a TLF reminder email on September 21, 2023, and nonresponding schools will receive a Field operation email on October 2, 2023, informing the school that a Field Representative (FR) will be visiting their school for the collection of the school’s NTPS forms.
These nonresponding schools will then be included in up to three phases of Field data collection operations. During the first phase, in October 2022, an FR will visit schools that have not completed their TLF. They will request a printed roster of teachers, transcribe the information onto a labeled paper TLF, and drop off invitations to complete the Principal and School Questionnaires online. During the second phase, in November 2023, the same FR (when available) will follow up with school staff by phone to remind them to complete the School and Principal Questionnaire(s). Schools will continue to receive mailouts and emails but will not be included in telephone reminder operations conducted by the contact center staff. During the third phase in late February and March 2024, the same FR (when available) will visit schools with outstanding Teacher Questionnaires and will follow up on School and Principal Questionnaires when necessary.
Following the initial school mailout and email, schools with publicly available teacher data will be sent a second school mailout on October 23, 2023, containing invitations to complete the NTPS questionnaire(s) via the survey instruments, and a third school mailout on November 29, 2023, as needed. The third school mailout package will include a reminder letter, paper versions of the TLF, Principal and/or School Questionnaire(s), and postage-paid addressed return envelopes. Principal and survey coordinator email addresses will be used as a means of reminding nonresponding schools to complete their questionnaires throughout the fall data collection period, as well. Note that these mail and email operations will also be conducted for schools in the Field data collection path, alongside the field operations described above.
Following the third mailout, schools with publicly available teacher data who have not yet completed their TLF will be included in a telephone reminder operation. The main goal of this operation will be to follow up with survey coordinators and principals on the status of their TLF. Telephone interviewers will also check on the status of the Principal Questionnaire and School Questionnaire. This operation will take place between the Thanksgiving holiday and schools’ winter break periods. Following this operation, schools have not completed their TLF will have teachers sampled from the publicly available teacher data on January 10, 2024.
Schools with outstanding Principal and/or School Questionnaires will be sent a fourth school mailout on January 25, 2024. This package will be mailed to the principal at the school address and will include a reminder letter, paper versions of the Principal and/or School Questionnaire(s), and postage-paid addressed return envelopes. A fifth and final school mailout will be sent on February 21, 2024, to principals of schools with an outstanding Principal and/or School Questionnaire, as needed; the package contents will mirror that of the fourth mailout package. Principal and survey coordinator email addresses will be used as a means of reminding nonresponding schools and principals to complete their questionnaires from January through March 2024. Note that the mail and email operations described here will also be conducted for schools in the Field data collection path, alongside the field operations described above.
Beginning in late February 2024, schools that have not yet completed their School and/or Principal Questionnaires will be sent to a telephone reminder operation aimed at reminding the survey coordinator or school principal to complete their questionnaires.
Refer to Table 8, NTPS 2023-24 Data Collection – School-Level Mailouts and Emails, for the overall school collection schedule and Exhibit 1, NTPS 2023-24 School-Level Data Collection Operations, for additional details about the general flow of collection operations.
Table 8. NTPS 2023-24 Data Collection – School-Level Mailouts and Emails
School mailout/email operation |
Data collection path |
Date |
Precontact Mailout |
Both |
6/20/2023 |
Screener Mailout |
Both |
7/20/2023 |
Initial Screener Email |
Both |
7/24/2023 |
Screener Reminder Email |
Both |
8/2/2023 |
Initial School Mailout to Schools without Publicly Available Teacher Data |
Field |
9/7/2023 |
Initial School Mailout to Schools with Publicly Available Teacher Data |
Traditional |
9/18/2023 |
Initial School Emails (TLF/PQ/SQ) to Principal and/or Coordinator |
Field |
9/13/2023 |
Initial School Emails (TLF/PQ/SQ) to Principal and/or Coordinator |
Traditional |
9/21/2023 |
TLF Reminder Email to Principal or Coordinator |
Field |
9/21/2023 |
Field Operation Email |
Field |
10/2/2023 |
Second School Emails (TLF/PQ/SQ) to Principal and/or Coordinator |
Traditional |
10/3/2023 |
Second School Mailout |
Both |
10/23/2023 |
Third School Emails (PQ/SQ) to Principal and/or Coordinator |
Field |
10/24/2023 |
Third School Emails (TLF/PQ/SQ) to Principal and/or Coordinator |
Traditional |
11/6/2023 |
Third School Mailout |
Both |
11/29/2023 |
Fourth School Emails (TLF/PQ/SQ) to Principal and/or Coordinator |
Both |
12/12/2023 |
Fifth School Emails (PQ/SQ) to Principal and/or Coordinator |
Both |
1/9/2024 |
Fourth School Mailout |
Both |
1/25/2024 |
Sixth School Emails (PQ/SQ) to Principal and/or Coordinator |
Both |
2/13/2024 |
Fifth School Mailout |
Both |
2/21/2024 |
Seventh School Emails (PQ/SQ) to Principal and/or Coordinator |
Both |
3/7/2024 |
Exhibit 1:NTPS 2023-24 – School-Level Data Collection Operation
Teachers will be sampled weekly from completed TLFs throughout data collection. The teacher data collection strategy will vary for early- and mid-collection cycle sampled teachers versus teachers sampled later in data collection; this change will start with the teachers sampled from publicly available teacher data.
For the earlier waves of teachers (teachers sampled through the end of calendar year 2023), as teachers are sampled, they will be mailed an initial teacher package containing a letter that introduces the survey and provides login information for completing their survey online and a $5 cash incentive. Around the same time, teachers for whom an email address is available will also be sent an email including the hyperlink and User ID to complete their Teacher Questionnaire online. If the school has a survey coordinator established, the individually-sealed teacher packages will be sent to the survey coordinator, at the school address, with a cover letter requesting that the coordinator disseminate the teacher packages.
Teachers sampled from publicly available teacher data and all teachers sampled thereafter will be included in an experiment aimed at boosting teacher response by offering different levels and types of incentivizes. Teachers will be randomly assigned to a treatment group in which they receive either a $5 or $10 prepaid cash incentive. Teachers in both the $5 and $10 individual incentive groups may be randomly assigned to an experimental school-level incentive treatment in which their school will be offered a $200 promised school-level incentive if they achieve an overall teacher questionnaire response rate of at least 75 percent in their school. The remaining teachers in the $5 individual incentive treatment who are not assigned to the school-level incentive treatment will receive an additional promised $20 upon questionnaire completion. Note that only public schools are eligible for inclusion in the school-level incentive experiment.
Additionally, for schools where teachers were sampled from publicly available teacher data and schools where the teachers were sampled late, the survey coordinators identified in the screener and a subsample of principals (as proxy coordinators) from non-screened schools will receive a prepaid cash incentive of equivalent value to the teachers’ prepaid cash incentive in the first mailed package of teacher materials. The teachers in the remaining non-screened schools will be mailed their materials directly throughout data collection rather than through a proxy coordinator.
Additional details about the “Coordinator and Promised Incentive Experiment” are included in section B.4.1.2.
All nonresponding sampled teachers will receive as many as four mailed reminder packages to complete their Teacher Questionnaire, for a total of five mailed packages. The second mailout will include a reminder letter with login information to complete the questionnaire online. The paper teacher questionnaire will be introduced in the third mailout and be included in the fourth and fifth mailouts, and will also include a reminder letter and a return envelope. If the school has a survey coordinator established, the individually-sealed teacher packages for the second and third mailouts will be sent to the survey coordinator, at the school address, with a cover letter. All fourth and fifth mailout packages will be addressed directly to the sampled teachers at the school address, regardless of whether the school has a survey coordinator established.
Teachers with a valid email address will be sent emails containing the hyperlink to the online Teacher Questionnaire and their User ID throughout data collection, for a maximum of 12 possible emails.
Nonresponding teachers from schools assigned to the field data collection path will be contacted at their school via a personal visit from an FR from late February 2024 through late March 2024. When possible, this will be the same FR who visited the school during the previous fall.
From April through May 2024, teachers sampled from publicly available teacher data and from schools in low performing domains of interest will be sent to a field follow-up operation during which an FR will make a personal visit to the school to follow up on the incomplete teacher questionnaire(s). During the same time period, telephone interviewers will contact the remaining nonresponding teachers directly by phone and remind them to complete their outstanding questionnaire.
At the end of the school year, we will reach out to teachers who completed the NTPS Teacher Questionnaire. For teachers who submitted the web survey on or after April 15, we will thank them for their participation in the NTPS. For teachers who submitted a web survey before April 15 or who submitted a paper questionnaire at any time, we will ask them to answer one, additional survey item online: whether they believe they will return to the same school in the 2024-25 school year. Note that this question will be asked in the online version of the NTPS Teacher Questionnaire beginning on April 15. We will compare their responses to data collected in the 2024-25 Teacher Follow-up Survey (TFS) to determine whether, in future NTPS and TFS collections, this information can be used when developing the TFS sampling frame. For more information about the TFS, see OMB# 1850-0617.
This section describes the methods that NCES will use to secure cooperation, maximize response rates, and deal with nonresponse for NTPS 2023-24. Section B.3.1 details how NTPS plans to secure cooperation by leveraging its status as the primary source of information on K-12 schools and staffing in the United States. Section B.3.2 describes the methods that will be used to minimize nonresponse. The design is based on the results of NTPS 2020-21 and NTPS 2017-18, which employed a number of different contact strategies aimed at boosting response rates.
The entire survey process, starting with securing research cooperation from key public-school groups and individual sample members and continuing throughout the distribution and collection of individual questionnaires, is designed to increase survey response rates. In addition, the following elements of the data collection plan, in particular, will contribute to the overall success of the survey and will enhance the survey response rates.
Visible support from top-level Federal, State, and local education officials. Without the support of high-level officials in the U.S. Department of Education, State Education Agencies, and the sampled local school districts, surveys of public-school principals and teachers cannot be successfully implemented. Obtaining endorsements from these officials is a critical factor in the success of the data collection procedures. Top-level Education Department officials will fully support the data collection by endorsing the survey in writing and sending advance letters and notices to sampled districts that require prior research applications and to individual survey participants (principals and teachers) to encourage participation.
Endorsements from key public and private school groups and affiliations. The level of interest and cooperation demonstrated by key groups can often greatly influence the degree of participation of survey respondents. Endorsements are viewed as a critical factor in soliciting cooperation from state and local education officials. Endorsing groups will be listed on some of the recruitment materials sent to NTPS 2023-24 sample members.
NCES will seek endorsement for NTPS 2023-24 from key national public-school groups and key private school organizations and agencies. Additionally, for NTPS 2023-24, like its 2020-21 predecessor, NCES will seek endorsement from state organizations and agencies. Note that the number of state organizations and agencies are capped at two per state for efficiency of solicitation operations. The complete list of groups, agencies, and organizations that NCES will solicit endorsement from for the NTPS 2023-24 can be found at the end of this document as Exhibit 1.
Stressing the importance of the survey and the respondents' participation. Official letters will be used to motivate respondents to return surveys. NTPS 2023-24 respondent letters will be sent by the U.S. Census Bureau and signed by the NCES Commissioner. Communications in the form of both letters and emails will be personalized for the principals, and survey coordinators, and teachers whenever possible, which is expected to have positive effects on the survey response rates.
A major challenge in any survey is obtaining high response rates, and this is even more important today when response rates have been falling among federal surveys, including NTPS.
The main problem associated with nonresponse is the potential for nonresponse bias in the estimates produced using data collected from respondents. Bias can occur when respondents are systematically different from nonrespondents. Two approaches that will be used to reduce the potential for bias are designing the data collection procedures and methods wisely to reduce nonresponse (e.g., establishing survey coordinators) and using statistical methods of sampling and weighting to reduce the effect of nonresponse on the estimates. While the statistical approaches are important in controlling biases and costs, the data collection procedures and methods are at the heart of a successful study.
Methods selected to minimize nonresponse in NTPS 2023-24 will build upon those used in NTPS 2020-21 and NTPS 2017-18, including actions that were taken late in the data collection to boost principal and teacher response rates.
Minimize survey burden on schools. NTPS survey procedures are designed to minimize the burden on schools and sampled individuals (principals and teachers), and the survey instruments have been designed to be completed as quickly and easily as possible.
To reduce burden on schools, the TLF (both the electronic version in the NTPS Respondent Portal and the paper TLF) will once again be pre-populated with publicly available teacher data, and the school will be asked to verify the teacher information rather than provide it from scratch.
Questionnaire design techniques have been employed to minimize item nonresponse. Questionnaires from previous rounds of NTPS were carefully analyzed to determine which items had the highest levels of item nonresponse. This information guided NCES in reviewing the clarity of item wording, definitions, and instructions. Items that were not considered to be effective or useful were removed from the survey to streamline the questionnaires and reduce response burden.
A key design feature of NTPS is the ability to link to other NCES collections such as EDFacts and the Civil Rights Data Collection (CRDC). Information from these sources will be incorporated into final datasets to allow researchers and policymakers to analyze these data together. This will further reduce the need to collect data from schools that have already been collected from state or district education agencies.
Recruit survey coordinators. One of the main purposes of the NTPS Screener is to establish a “survey coordinator” at the school. Successive administrations of SASS and NTPS have shown that establishing a survey coordinator is an important procedure to help maximize response rates for the School, Principal, and Teacher Questionnaires, because the survey coordinator serves as a primary point of contact for NTPS staff throughout data collection. The use of a survey coordinator is expected to help keep response rates high, provide some minimal data quality checks, and simplify the follow-up process by having one point of contact per school.
Tailor data collection and nonresponse follow-up strategies for subdomains of schools. During previous cycles of NTPS, targeted data collection strategies were implemented for those schools with a potential large effect on nonresponse weighting adjustments and final estimates. This will be done, once again, for the NTPS 2023-24.
For NTPS 2023-24, the level of effort put forth in collecting a TLF– which is needed to draw a sample for the Teacher Questionnaire – for a school will depend primarily on whether there is publicly available teacher data for the school. Schools without publicly available teacher data do not have a teacher roster to fall back on for the purposes of selecting a teacher sample. Additionally, the assumption is that schools without publicly available teacher data are the harder to reach or less likely to respond schools and, therefore, it is important to put forth additional targeted effort and resources into obtaining a TLF from these schools.
For the NTPS 2023-24 data collection, schools without publicly available teacher data will receive the initial school mailout, the initial email, and a TLF reminder email. Non-responding schools will then be included in three Field-based data collection operations, known as the “Field Data Collection Path”. During the first phase, an FR will make a personal visit to schools that have not completed their TLF. The FR will request a printed roster of teachers, transcribe the information onto a labeled paper TLF, and drop off the schools sealed invitations to complete their Principal and School Questionnaires online. During the second phase, the same FR (when available) will follow up with school staff by phone to remind them to complete their outstanding School and Principal Questionnaires. These schools following the field data collection path will continue to receive mailouts and emails on the same schedule as schools on the traditional data collection path, but they will not be included in telephone reminder operations conducted by the contact center staff. During the third phase, the same FR (when available) will make personal visits to schools with outstanding Teacher Questionnaires and will follow up on School and Principal Questionnaires, as necessary.
NTPS focuses on obtaining cooperation and improving response rates at the school level for a number of reasons. Past administrations of SASS and NTPS have shown that when cooperation is obtained at the school level, teachers and principals are more likely to respond.
Additionally, evaluations of schools’ response propensities have shown that being in a special contact district is the primary driving force behind low response propensity. Special districts are those that require additional applications or documentation to collect data in their schools. Nearly 80% of the schools with high propensity for non-response reside in these special districts. For this reason, resources will continue to be allocated to focus on obtaining approvals from special contact districts to boost response rates for this group.
Use publicly available teacher data for teacher sampling. NTPS teacher-level response rates are calculated by multiplying response at the school level to the TLF by response at the teacher level. In the past, this has meant that if the school did not complete their TLF, teachers from that school could not be sampled, ultimately lowering the teacher response rate. Previous cycles of NTPS have aimed to improve the overall teacher response rate by allowing NTPS to sample teachers from schools that have not submitted a TLF; therefore, TLFs received from schools were supplemented with vendor-purchased teacher lists or clerically researched teacher lists when vendor data were not available. The vendor and clerically-researched lists have been evaluated during each cycle of the NTPS and have continued to show high levels of comparability to lists obtained directly from schools.
In NTPS 2023-24, TLFs will once again be pre-populated with publicly available teacher lists, that is, teacher data purchased from a vendor or, when vendor data are not available, obtained through web scraping or through a clerical look-up operation utilizing school and district websites. These schools will be asked to verify the listed teacher information rather than provide it from scratch. The approach of offering respondents pre-populated TLFs is expected to help improve the overall teacher response rate (by way of a higher TLF response rate) and allow teacher sampling in schools that have not submitted a TLF as a last-ditch effort to collect data in such schools.
As with past cycles of NTPS, the publicly available teacher data will be evaluated against school-reported data to assess the quality of the supplemental data.
Monitor publishability and bias measures. For NTPS 2017-18 and NTPS 2020-21, NCES monitored data collection progress throughout survey operations to identify and potentially minimize problems with nonresponse. The Census Bureau created weekly “publishability” reports from their data collection tracking system that showed whether key analysis cells were large enough to provide publishable estimates as of that point in time. By monitoring this publishability metric during NTPS 2017-18, NCES was able to identify populations of schools for which nonresponse could hamper reporting. As a result, a sub-population of teachers (working in city or charter schools) with outstanding questionnaires were sent to a final follow-up operation, which ultimately lead to meeting publishability standards for those subpopulations.
The results from monitoring the weekly publishability metric will be considered in designing the sample and nonresponse follow-up strategies for NTPS 2023-24. For example, schools in low-performing domains of interest, according to the publishability reports, will be included in a Field Follow-up operation, rather than an outbound telephone operation, to collect outstanding School Principal, and Teacher questionnaires during the spring of 2024. The goal is to boost response rates for these schools to meet publishability standards for these subpopulations.
During the prior two cycles of NTPS, NCES also monitored R-indicators, a measure of representativeness, or lack of bias in the respondent population, on a weekly basis. The closer the R-indicator is to 1, the more balanced is the respondent population. Towards the end of data collection, the R-indicator for the full sample indicated that the respondent population was fairly well balanced. NCES plans to continue to monitor these two indicators in NTPS 2023-24.
Personalize principal contact materials. As was done in previous NTPS collections, to maximize the chances that all mailed NTPS 2023-24 materials intended for the school principal successfully make it to the principal, all principal contact materials will be personalized with the principal’s name whenever possible. Principals’ names will be obtained from vendor-purchased school staff lists. If a principal’s name is not available from the vendor, the information will be obtained via web scraping or clerical research using school and/or district websites.
Use email to target principals, survey coordinators, and teachers. Previous NTPS collections demonstrated that email was an effective tool to drive participation in both the NTPS teacher and principal surveys. They proved that teacher email addresses could be effectively collected on the TLF, school websites, and from vendor lists of teachers; that principal email addresses could be effectively collected from school websites and from vendor purchased school data; and that survey coordinator email addresses could be effectively collected during the screener interview. Because personalized emails carry minimal cost and may help boost response, throughout the NTPS 2023-24 data collection, teachers, principals, and survey coordinators will be contacted via email. The emails will include login information to access the NTPS online survey instruments, in addition to text inviting and subsequently reminding these respondents to complete their survey online. Nonresponding teachers, specifically, will be sent as many as 12 possible emails throughout NTPS data collection, receiving an email approximately every ten days throughout data collection.
Offer additional incentives for late-sampled teachers. Results of previous administrations of the NTPS indicated that offering a prepaid teacher incentive significantly increased the final response rates for both public and private school teacher questionnaires. Due to these favorable results, NTPS 2023-24 will offer all teachers prepaid incentives at the first contact by mail.
Previous NTPS cycles showed that response rates for late-sample wave teachers in NTPS appeared to be lower than for earlier waves of teachers. This may have been a product of the timing of school testing and late-school year activities because late-sample wave teachers received an invitation to complete the survey during a period with a heavy school workload, while teachers sampled earlier may have completed their questionnaire in the fall. It may also have been because the late sampled teachers were in schools that were either late responders to the TLF or TLF non-respondents (in instances where teachers were sampled from a teacher roster obtained from clerical research or vendor data) and, therefore, may have had less support and encouragement from their principal and/or survey coordinator to complete their questionnaire.
Starting with the teacher sampling wave during which teachers are selected from publicly available teacher lists and all teacher sampling waves thereafter, teachers will be included in an experiment aimed at boosting teacher response for these historically low-responding teachers by offering different levels and types of incentivization.
All teachers will receive a prepaid cash incentive at the first contact by mail. The cash incentive amount will vary between $5 or $10 depending on the sampled teacher’s assigned treatment group. A portion of the teachers in the $5 group will be offered an additional promised monetary incentive ($20) upon completing their questionnaire. The remaining teachers in the $5 group and a portion of teachers in the $10 group will be offered a promised $200 school-level incentive if they complete their questionnaire and help their school achieve an overall teacher response rate of 75 percent or higher within their school.
Additionally, some schools’ survey coordinators identified in the screener and some principals (as proxy coordinators) will receive a prepaid cash incentive of equivalent value to the teachers’ prepaid cash incentive in the first mailed package of teacher materials for distribution.
Further information about the promised incentive experiment can be found below in section B.4.1.
Incorporate additional methods of minimizing nonresponse. NCES plans to incorporate additional methods of minimizing nonresponse to NTPS 2023-24, including the use of overprinted tailored statistics on the exterior of pressure-sealed mailers to teachers. In addition, the school precontact notification postcards will include a tailored statistic printed on their exterior. The goal is to engage respondents and encourage response by providing data from previous cycles of the survey. Further information about the experiments included in NTPS 2023-24 can be found in section B.4.1 of this document.
One of the methods employed to reduce the potential for nonresponse bias is adjustment of the sample weights to account for nonresponse. If schools or teachers with certain characteristics are systematically less likely than others to respond to a survey, the collected data may not accurately reflect the characteristics and experiences of the nonrespondents, which can lead to bias. To adjust for this, respondents are assigned weights that, when applied, result in them representing their own characteristics and experiences, as well as those of nonrespondents with similar attributes. The school weights are also raked to sampled-based control totals to maintain the background characteristics of the sample. This is another method used to reduce the potential for nonresponse bias in the estimates produced from the data.
Response rates will be computed for the TLF, the School Questionnaire, the Principal Questionnaire, and the Teacher Questionnaire. Data collected through any instrument with a response rate of less than 85 percent will be evaluated for nonresponse bias. In addition to comparing the characteristics of respondents and nonrespondents using data that are available from the sampling frames (for example, school type and school locale from the school frame), the NTPS 2023-24 estimates will be compared to estimates from previous rounds of NTPS. A methodology report covering NTPS 2023-24 will be developed and released, describing the methods and results of the nonresponse bias analysis.
The NTPS has a history of testing materials, methods, and procedures to improve the quality of its data. Appendix D describes those tests that have most influenced the NTPS design, beginning with the 2014-15 NTPS Pilot Test and continuing through NTPS 2020-21. Section B.4.1 describes the experiments proposed for NTPS 2023-24.
NCES is currently considering options for tests of methods, materials, and procedures to be conducted as part of NTPS 2023-24, with the goal to increase response. There are two sets of tests – school-level tests and teacher-level tests.
Three experiments aimed at increasing school-level response rates are planned for NTPS 2023-24, namely (1) testing the addition of an eye-catching data product to the screener mailout, (2) testing the mail packaging for the initial school mailout, and (3) testing various email subject lines for emails directed at principals and school coordinators. Each of these experiments is described briefly below.
Following data collection, each experiment will be evaluated using a series of metrics calculated for the control group and each treatment group of the experiment separately. These metrics will include but not be limited to:
Response rate;
Average number of contacts;
Days to respond; and
Data collection costs.
The response rates will be calculated for each treatment group and selected demographic domains and compared using significance tests for differences. To account for confounding variables, a model-based approach will also be calculated to determine what effect the experimental treatment had on a case’s likelihood of response, given that case’s unique characteristics.
The average number of contacts and average days to respond across the experimental groups will be used as a proxy for timeliness of response. A reduction in the average number of contacts could be used to justify the use of a particular treatment. For example, if cases within the experimental group using new mailed package contents respond in a timelier fashion, this could reduce the number of cases included in follow-up operations, allowing finite resources, such as Field, to be spread across fewer cases.
Using data collection costs associated with each mailout, any additional costs associated with a particular treatment group, and estimates for interviewer costs, an average cost-per-case and cost-per-response can be determined within each experimental group. A reduction in cost-per-case or cost-per-response could justify the use of a particular treatment to reduce the overall survey cost. Particular treatments may lead to more initial costs, such as the use of incentives, but if cases within a particular treatment group respond in a fewer number of contacts – specifically if they help to avoid more costly contacts, such as personal visits – this could lead to a reduction in cost at the end of data collection.
Exhibit 2 shows the breakdown of the school-level experiments and experimental groups within each experiment. The treatment flags for each experiment will be randomly assigned to schools at the time of sampling to ensure a similar distribution of schools will be included in each experimental group for each experiment. Figure 1 also includes additional school packaging information, such as TLF type and the initial mail-out timing, as the public availability of teacher data is a main determinant for which data collection path the school will follow during the 2023-24 data collection cycle.
Exhibit 2. School level experiments included in the NTPS 2023-24
Data availability |
Screener Instrument |
Screener data product |
Mail strategy experiment* |
Screener and Principal Email subject test |
Initial mail-out timing** |
TLF type*** |
|
||
Publicly available teacher data (“Traditional data collection path”)
|
Centurion |
Data product |
USPS |
Treatment 1, …, 6 |
Late |
Pre-populated |
|
||
UPS |
Treatment 1, …, 6 |
|
|||||||
FedEx |
Treatment 1, …, 6 |
|
|||||||
No data product |
USPS |
Treatment 1, …, 6 |
|
||||||
UPS |
Treatment 1, …, 6 |
|
|||||||
FedEx |
Treatment 1, …, 6 |
|
|||||||
Qualtrics |
Data product |
USPS |
Treatment 1, …, 6 |
|
|||||
UPS |
Treatment 1, …, 6 |
|
|||||||
FedEx |
Treatment 1, …, 6 |
|
|||||||
No data product |
USPS |
Treatment 1, …, 6 |
|
||||||
UPS |
Treatment 1, …, 6 |
|
|||||||
FedEx |
Treatment 1, …, 6 |
|
|||||||
No publicly available teacher data (“Field data collection path”) |
Centurion |
Data product |
USPS |
Treatment 1, …, 6 |
Early |
Blank |
|
||
No data product |
USPS |
Treatment 1, …, 6 |
|
||||||
Qualtrics |
Data product |
USPS |
Treatment 1, …, 6 |
|
|||||
No data product |
USPS |
Treatment 1, …, 6 |
|
||||||
Amish/ Mennonite |
n/a |
n/a |
USPS |
n/a |
Late |
Blank |
|
||
Late approving special district |
n/a |
n/a |
UPS |
Treatment 1 |
Flow |
Pre-populated/ Blank |
|
||
*Mail type is dependent on whether teacher data are publicly available and the experimental treatment group. **Mail timing is dependent on whether teacher data are publicly available, school type (Amish or Mennonite schools), and special district approval. Packages to non-Amish schools without publicly available teacher data will mail early, on 9/7/23. Packages to non-Amish schools with publicly available teacher data will mail later, on 9/18/23. Packages to schools in special districts that were “on hold” at the time of package preparation will be mailed on a flow basis as approval is received. |
|
|
|||||||
***TLF type is dependent on whether teacher data are publicly available. All schools with publicly available teacher data will receive an invitation to complete the prepopulated TLF via the NTPS Respondent Portal. Amish and Mennonite schools will receive the shortened paper TLF. |
|
|
|||||||
|
|
Testing the Addition of Eye-Catching Data Products to School Mailings (Screener Data Product Experiment). A randomized experiment in NTPS 2023-24 will compare the effects of including a data product (a colorful data wheel that displays state-level NTPS data from the 2020-21 collection) in the screener mailout to public schools, at which time the principal is invited to complete the Screener online using the Screener instrument. This experiment impacts only the screener mailout for public schools. There will be two experimental groups:
Control Group – Public schools do not receive a Data Product within the Screener mailout
Treatment Group – Public schools do receive a Data Product within the Screener mailout
Public schools will be randomly assigned to each experimental group at the time of public school sampling. Based on the estimated overall sample size of 10,100 public schools, approximately 5,050 public schools will be in the control group and 5,050 public schools will be in the treatment group that receives the data product. Given the sample sizes within the experimental groups, the minimum detectable difference in response rate between the groups is approximately 3.50% for the Screener Data Product Experiment.
Note that this experiment was included in OMB# 1850-0598 v.41, and the details are repeated here to give the reader the complete picture of tests included in NTPS 2023-24.
Testing the use of a new Screener instrument developed using Qualtrics. A randomized experiment in NTPS 2023-24 will determine the effects of using a more stream-lined instrument for completing the Screener. The Census Bureau has authorized the use of the Qualtrics software for survey data collection activities, where accessing the Qualtrics instrument will not require the need for a pin and security questions as is done in the Centurion instrument. Following the successful use of Qualtrics to collect data for the 2022 Principal Follow-up Survey (PFS), and given that the Screener is a relatively short questionnaire, it is ideal for testing the use of the Qualtrics instrument to determine if the efficiency of logging into the Qualtrics instrument encourages more response. This experiment impacts data collection for the screener only. There will be two experimental groups:
Control Group – Schools that receive the Centurion version of the instrument to complete their Screener
Treatment Group – Schools that receive the Qualtrics version of the instrument to complete their Screener
Public and private schools will be randomly assigned to each experimental group at the time of sampling. Based on the estimated overall sample size of 10,100 public schools and 3,055 private schools (approximately 65 Amish schools, which will not receive an electronic version of the Screener), approximately 5,050 public and 1,528 private schools will be in the control group and 5,050 public and 1,528 private schools will be in the treatment group that receives the data product. Given the sample sizes within the experimental groups, the minimum detectable difference in response rate between the groups is approximately 3.50% for public schools and 6.50% for private schools for the Screener Instrument Experiment.
Testing the Mail Strategy for the Initial School Mailout (Mail Strategy Experiment). NTPS 2023-24 will include an experiment aimed at testing the impact of the mail strategy on survey response for schools on the “traditional data collection” path. Schools will be randomly assigned to one of three experimental groups at the time of sampling:
Control - USPS flat, branded envelope
Treatment 1 – FedEx envelope
Treatment 2 – UPS envelope
The “traditional data collection” path includes all public and private schools with publicly available teacher data. Schools without publicly available teacher data will follow the “field data collection” path and receive the initial mailout package within the standard UPS envelope. Based on previous NTPS cycles, vendor coverage for public schools ranged from 84.63% in 2017-18 to 78.26% in 2020-21. Similarly, vendor coverage for private schools ranged from 55.53% in 2017-18 to 59.03% in 2020-21. Based on the expected vendor coverage for public and private schools, the expected sample sizes for this experiment and the minimal detectable differences in response rates between the experimental groups is shown in table 9.
Table 9. Expected sample sizes and minimum detectable differences for the NTPS 2023-24 Mail Strategy Experiment
|
Public schools– 10,100 |
Private schools – 3,120 |
||
2017-18 |
2020-21 |
2017-18 |
2020-21 |
|
Schools with vendor information available |
8,544 |
7,902 |
1,633 |
1,735 |
USPS (Control) |
2,848 |
2,634 |
544 |
578 |
FedEx (Treatment 1) |
2,848 |
2,634 |
545 |
579 |
UPS (Treatment 2) |
2,848 |
2,634 |
544 |
578 |
Minimum detectable difference between two groups |
4.66% |
4.84% |
10.36% |
10.68% |
Depending on the vendor coverage rate for the 2023-24 data collection cycle, the minimum detectable difference in response between each pair of experimental groups is approximately 4.75% for public schools and 10.5% for private schools.
All three mail strategies require hand assembly because of the inner envelope contents; however, there is a cost difference between the commercial carriers ($4.62 each) and USPS (estimated at $0.77 each, based on the current USPS rates). In addition, FedEx requires that packages be sent on the specified date; whereas UPS and USPS are more flexible.
This experiment impacts the initial school mailout and the initial mailout to teachers in early-responding schools with a survey coordinator. The results will be analyzed and used to inform the best mail strategy (FedEx or UPS) for the fourth school mailout.
Note that FedEx and UPS are unable to deliver mail to PO boxes; therefore, schools with a PO box as their mailing address will be sent their initial package (and subsequent packages) via USPS. They will be treated as though they are in the control group for data collection, but they will be excluded from analyses.
Testing Email Subject Lines for Screener and Principal Questionnaires (Screener and Principal Email Subject Test). NTPS 2023-24 will include an experiment designed to examine the effectiveness of various email subject lines on survey response. The experiment will be conducted on the initial email to principals regarding the Screener Questionnaire and the 3rd email to School Coordinators or Principals regarding the Principal Questionnaire.. The following metrics may be analyzed:
The email’s “Open” rate, as determined by the available paradata;
The email’s “Interview” rate, as determined by the email source URL from the Centurion instrument; and
The overall survey response rate.
The control group for this experiment will receive the traditional email subject lines and five additional treatment groups will be included in this experiment to test the tone and content of six different email subject lines. Table 10 shows the minimum detectable difference in the response rate based on the number of selected treatments.
Table 10. Expected sample sizes and minimum detectable differences for the NTPS 2023-24 Screener and Principal Email Subject Test
Experimental groups |
Public – 10,100 Schools |
Private – 3,055 Schools1 |
|||||
2020-21 PQ Response Rate at 3rd Email |
Group size |
MDD |
2020-21 PQ Response Rate at 3rd Email |
Group size |
MDD |
||
Screener – Control, 5 Treatments |
n/a |
1,683 |
6.06% |
n/a |
491 |
11.25% |
|
Principal Questionnaire - Control, 5 Treatments |
31.69% |
1,150 |
7.33% |
29.34% |
347 |
13.39% |
1 Adjusted from total private school sample of 3,120 for expected number of Amish or Mennonite Private Schools
Three experiments aimed at increasing teacher-level response rates are planned for NTPS 2023-24, namely (1) testing tailored contact materials, (2) testing promised teacher incentives and coordinator proxies, and (3) testing various email subject lines for emails directed to teachers. Each of these experiments is described briefly below.
Following data collection, analyses will be conducted at the treatment level for each experiment. The analyses examined will include:
Response rate;
R-indicators;
Average number of contacts;
Days to response; and
Data collection costs.
The response rates will be calculated for each treatment group and selected demographic domains and compared using significance tests for differences. To account for confounding variables, a model-based approach will also be calculated to determine what effect the incentive had on a case’s likelihood to respond, given that case’s unique characteristics.
R-indicators will be used to determine the overall balance of the respondent population, as well as within each experimental group. R-indicators will be calculated for the full sample, as well as variable-level and category-level partial R-indicators to determine which characteristics specifically are contributing to imbalance within the respondent population.
The average number of contacts and average days to respond across the experimental groups will be used as a proxy for timeliness of response. A reduction in the average number of contacts could be used to justify the use of one type of treatment over another. For example, if cases within the experimental group that receive the prepaid $5 teacher incentive and the $20 promised incentive respond in a timelier fashion, this could reduce the number of cases that need to be included in follow-up operations, allowing finite resources, such as Field, to be spread across fewer cases.
Using data collection costs associated with each mailout, the value of the incentives or overprint, and estimates for interviewer costs, an average cost-per-case can be determined within each experimental group. A reduction in cost-per-case could justify the use of incentives to reduce the overall survey cost or cost-per-response. While using incentives leads to more initial costs, if cases respond in a fewer number of contacts – specifically if incentives help NTPS to reduce the need for more costly contacts such as personal visits – this could lead to a reduction in overall cost at the end of data collection.
Exhibit 3 shows the breakdown of the teacher-level experiments and experimental groups within each experiment. The treatment flags for each experiment will be randomly assigned at the school-level, so every teacher within a school receives the same treatment, at the time of sampling to ensure a similar distribution of schools will be included in each experimental group for each experiment.
Exhibit 3. Teacher level experiments included in the NTPS 2023-24
Treatment path |
TQ sample timing |
Overprint experiment |
Screener status |
Email subject line experiment |
Coordinator/ principal contact experiment |
Coordinator incentive? |
Incentive(s) available |
1-6 |
Early and mid-sampled teachers |
Overprint |
Screened |
Treatment 1,…, 6 |
n/a - mail to coordinator |
No |
n/a |
7-12 |
Not screened |
Treatment 1,…, 6 |
n/a - mail to teacher |
n/a |
|||
13-18 |
No overprint |
Screened |
Treatment 1,…, 6 |
n/a - mail to coordinator |
No |
||
19-24 |
Not screened |
Treatment 1,…, 6 |
n/a - mail to teacher |
n/a |
|||
25 |
Vendor-sampled and beyond |
n/a |
Screened |
n/a |
Mail
to coordinator |
Yes1 |
1A. Prepaid ($10) |
26 |
n/a |
n/a |
Yes1 |
1B. Prepaid ($10) + Promised (School) |
|||
27 |
n/a |
n/a |
Yes1 |
1C. Prepaid ($5) + Promised (Teacher $20) |
|||
28 |
n/a |
n/a |
Yes1 |
1D. Prepaid ($5) + Promised (School) |
|||
29 |
n/a |
Not screened |
n/a |
Mail
to principal |
Yes |
2A. Prepaid ($10) |
|
30 |
n/a |
n/a |
Yes |
2B. Prepaid ($10) + Promised (School) |
|||
31 |
n/a |
n/a |
Yes |
2C. Prepaid ($5) + Promised (Teacher $20) |
|||
32 |
n/a |
n/a |
Yes |
2D. Prepaid ($5) + Promised (School) |
|||
33 |
n/a |
n/a |
Mail
to teacher |
n/a |
3A. Prepaid ($10) |
||
34 |
n/a |
n/a |
n/a |
3C. Prepaid ($5) + Promised (Teacher $20) |
Testing Tailored Contact Materials at the Teacher Level (“Teacher Overprint Experiment”). NTPS 2023-24 will include an experiment in which tailored statistics will be overprinted on the exterior of a pressure-sealed mailer to non-responding early and mid-sampled teachers in the second teacher mailings. Teachers in the treatment group will receive their reminder letter with login information in a business-size envelope with overprinted information printed on the exterior and teachers within the control group will receive their reminder letter in the traditional business-size envelope without the additional overprint. Teachers in schools that have not been screened will be sent their reminder letter directly; letters to teachers in screened schools will be sent to the survey coordinator for distribution.
This experiment will follow a split-panel design where all teachers within one school will be assigned to the same treatment group and schools will be randomly assigned to each experimental group at the time of school sampling. During the NTPS 2020-21 data collection cycle, approximately 34.96% of public-school teachers and 37.43% of private school teachers were sampled in the early- to mid-cycle time period. In addition, because this treatment is applied to the second teacher mailing, using a conservative estimate of 20% teacher response by the second mailing means that roughly 80% of teachers will be eligible to receive the overprint treatment. Given these assumptions and the estimated overall sample sizes of 59,673 for public school teachers and 8,316 for private school teachers, an estimated 13,774 public school teachers and 1,886 private school teachers will be included in this experiment. Given the sample sizes of 6,887 public school teachers and 943 private school teachers in each experimental group, the minimal detectable difference in response rate between the groups is approximately 3.96% for public school teachers and 11.05% for private school teachers.
Testing Various Incentive Strategies for Teachers sampled from publicly available teacher data (“Coordinator and Promised Incentive Experiment”). NTPS 2023-24 will once again include the use of monetary incentives for teachers. Early and mid-sampled teachers will all receive a prepaid, $5 cash incentive at the first contact by mail. Beginning with the teacher wave selected from publicly available teacher data, and all teacher waves thereafter, all teachers will receive a prepaid cash incentive at the first contact by mail.
Half will receive $5 cash at the first contact by mail while the other half will receive $10 cash at the first contact by mail. A portion of the of the public-school teachers and all of the private school teachers in the $5 group will be offered an additional promised cash monetary incentive ($20) upon questionnaire completion. The remaining public-school teachers in the $5 group and a portion of the public school teachers in the $10 group will be offered a promised school-level incentive if they complete their questionnaire and help their school achieve an overall response rate of at least 75 percent in their school. In summary, teachers sampled from vendor TLFs and all teachers sampled thereafter will be eligible for inclusion in a teacher incentive experiment with the following four treatments:
Treatment A: Teachers receive a prepaid cash ($10) incentive at the first contact by mail.
Treatment B: Teachers receive a prepaid cash ($10) incentive, and their school will receive a school-level promised incentive if their school reaches at least 75 percent in overall teacher response. This treatment is restricted to public school teachers only.
Treatment C: Teachers receive a prepaid cash ($5) incentive and will receive an additional $20 promised incentive upon completion of their teacher listing form.
Treatment D: Teachers receive a prepaid cash ($5) incentive, and their school will receive a school-level promised incentive if their school reaches at least 75 percent in overall teacher response. This treatment is restricted to public school teachers only.
In addition, for the teacher wave selected from publicly available teacher data and beyond, survey coordinators identified in the screener interview (treatment 1) and a subsample of principals (as proxy coordinators) from non-screened schools (treatment 2) will receive a prepaid cash incentive of the same amount as the teachers’ prepaid cash incentive ($10 for the A and B treatments and $5 for the C and D treatments) in the first mailed package of teacher materials for distribution. This nonexperimental prepaid incentive for school coordinators is based on research from the 2017-18 NTPS. The teachers in the remaining non-screened schools will be mailed their materials directly throughout data collection rather than through a proxy coordinator (Treatment 3). As a result, there are ten treatment groups for the Coordinator and Promised Incentive Experiment:
Treatment 1: Schools with a survey coordinator – the coordinator will be sent a package of individually-sealed teacher survey invitations and asked to distribute them to the teachers.
Treatment 1a: Each teacher will receive a half-sheet-sized envelope containing a letter and a prepaid cash ($10) incentive. Survey coordinators will receive a prepaid cash ($10) incentive.
Treatment 1b: Each teacher will receive a half-sheet-sized envelope containing a letter and a prepaid cash ($10) incentive. Survey coordinators will receive a prepaid cash ($10) incentive. The school will receive a school-level promised incentive if the school reaches at least 75% in overall teacher response, and this will be communicated to sampled teachers and the survey coordinator. This treatment is restricted to public school teachers only.
Treatment 1c: Each teacher will receive a half-sheet-sized envelope containing a letter and a prepaid cash ($5) incentive. The letter will offer a promised cash incentive ($20) upon the teacher’s survey completion. Survey coordinators will receive a prepaid cash ($5) incentive.
Treatment 1d: Each teacher will receive a half-sheet-sized envelope containing a letter and a prepaid cash ($5) incentive. Survey coordinators will receive a prepaid cash ($5) incentive. The school will receive a school-level promised incentive if the school reaches at least 75% in overall teacher response, and this will be communicated to sampled teachers and the survey coordinator. This treatment is restricted to public school teachers only.
Treatment 2: Schools without a survey coordinator, principal coordinator treatment – the principal will be treated as a proxy coordinator and will be sent a package of individually-sealed teacher survey invitations and asked to distribute them to the teachers.
Treatment 2a: Each teacher will receive a half-sheet-sized envelope containing a letter and a prepaid cash ($10) incentive. Principals will receive a prepaid cash ($10) incentive.
Treatment 2b: Each teacher will receive a half-sheet-sized envelope containing a letter and a prepaid cash ($10) incentive. Principals will receive a prepaid cash ($10) incentive. The school will receive a school-level promised incentive if the school reaches at least 75% in overall teacher response, and this will be communicated to sampled teachers and the principal. This treatment is restricted to public school teachers only.
Treatment 2c: Each teacher will receive a half-sheet-sized envelope containing a letter and a prepaid cash ($5) incentive. The letter will offer a promised cash incentive ($20) upon the teacher’s survey completion. Principals will receive a prepaid cash ($5) incentive.
Treatment 2d: Each teacher will receive a half-sheet-sized envelope containing a letter and a prepaid cash ($5) incentive. Principals will receive a prepaid cash ($5) incentive. The school will receive a school-level promised incentive if the school reaches at least 75% in overall teacher response, and this will be communicated to sampled teachers and the principal. This treatment is restricted to public school teachers only.
Treatment 3: Schools without a survey coordinator, no coordinator treatment - The teacher packages will be mailed directly to the teachers.
Treatment 3a: Each teacher will receive a half-sheet-sized envelope containing a letter and a prepaid cash ($10) incentive.
Treatment 3c: Each teacher will receive a half-sheet-sized envelope containing a letter and a prepaid cash ($5) incentive. The letter will offer a promised cash incentive ($20) upon the teacher’s survey completion.
Treatments 1a, 1b, 1c, and 1d will be evaluated against one another to determine the most effective incentive strategy for screened schools. Based on the previous cycle of NTPS, during the vendor-sampled and beyond phase of teacher sampling, roughly 24% of public-school teachers and 35% of private school teachers are screened.
Based on the overall sample sizes of 59,673 public school teachers and 8,316 private school teachers, where 65% and 63% are sampled during the vendor-sampled and beyond phase, respectively, each of the four treatment groups will have approximately 1,909 public school teachers and treatment groups 1a and 1c will have 688 private school teachers. Significant differences in response will be detected if the difference in response between two groups exceeds 7.51% for public school teachers and 12.94% for private school teachers.
Treatments 2a, 2b, 2c, 2d, 3a, and 3b will be evaluated against one another to determine the most effective incentive strategy for schools that are not screened. Based on the previous cycle of NTPS, during the vendor-sampled and beyond phase of teacher sampling, roughly 76% of public-school teachers and 65% of private school teachers are not screened.
Based on the overall sample sizes of 59,673 public school teachers and 8,316 private school teachers, where 65% and 63% are sampled during the vendor-sampled and beyond phase, respectively, each treatment group will have approximately 4,066 public school teachers and 641 private school teachers. Significant differences in response will be detected if the difference in response between two groups exceeds 5.15% for public school teachers and 13.41% for private school teachers.
Note that the cash incentives will be adhered to a piece of yellow cardstock using removable sticky glue to be inserted with the letter. The cardstock will be half-sheet-sized rather than a full sheet and will include text thanking respondents for their participation in the study. Continued use of this contact material item will (1) increase the weight of the envelope, making it feel more “substantial” and important; (2) prevent the money from free-floating inside the envelope; and (3) help ensure that the respondent notices the cash.
This experiment impacts all teacher mailings, since the teachers in the treatments with promised incentives (B, C, and D treatments) will be reminded about the promised incentive throughout data collection.
Note that schools with only one teacher sampled that are assigned to treatments B or D will not receive the additional promised school incentive; instead, the teacher will receive the prepaid cash ($10) incentive at the first contact by mail (equivalent to treatment A).
Testing Email Subject Lines for the Teacher Questionnaire (Teacher Email Subject Test). NTPS 2023-24 will include an experiment designed to examine the effectiveness of various email subject lines on survey response. The experiment will be conducted primarily on emails to teachers regarding the Teacher Questionnaire. The following metrics may be analyzed:
The email’s “Open” rate, as determined by the available paradata;
The email’s “Interview” rate, as determined by the email source URL from the Centurion instrument
The overall survey response rate.
To test the effectiveness of the tone and content of different email subject lines, a variety of email subject lines will be tested at three different points during teacher data collection. For early- and mid-sampled teachers, six different email subject lines will be tested on the initial email to teachers and two email subject lines will be tested on the fourth email to teachers. For vendor- or late-sampled teachers, three different email subject lines will be tested on the third email to teachers. Table 11 shows the minimum detectable difference in response based on the number of selected treatments.
Table 11. Expected sample sizes and minimal detectable differences for the NTPS 2023-24 Teacher Email Subject Test
Timing |
Experimental groups |
Public – Teachers |
Private – Teachers |
||||
2020-21 Response Rate at 3rd/4th Email |
Group size |
MDD |
2020-21 Response Rate at 3rd/4th Email |
Group size |
MDD |
||
Early- or mid-sampled Teachers |
Initial Email – 6 Treatments |
n/a |
2,870 |
6.13% |
n/a |
393 |
17.13% |
4th Email – 2 Treatments |
47.20% |
4,545 |
4.87% |
43.84% |
662 |
13.19% |
|
Vendor- or late-sampled Teachers |
3rd Email – 3 Treatments |
14.13% |
9,168 |
3.43% |
14.42% |
1,125 |
10.12% |
The following individuals are responsible for the NTPS 2023-24 study design, data collection and analysis: Maura Spiegelman, Julia Merlin, and Andy Zukerberg at NCES; Shawna Cox, Walter Holmes, Teresa Thomas, Allison Zotti, Svetlana Mosina, and Michael Foss at the U.S. Census Bureau; and Rebecca Medway, Jana Kemp, Carol Wan, and Noelle Poirier at the American Institutes for Research (AIR).
Exhibit 1. Endorsing groups and organizations
Endorsements from key public-school groups. The level of interest and cooperation demonstrated by key groups can often greatly influence the degree of participation of survey respondents. Endorsements are viewed as a critical factor in soliciting cooperation from state and local education officials and endorsing groups will be listed on recruitment materials sent to NTPS 2023-24 sample members. NCES will seek endorsements for NTPS 2023-24 from the following national organizations and agencies:
American Association of School Administrators
American Counseling Association
American Association of School Librarians
American Federation of Teachers
American Montessori Society
American School Counselors Association
Association for Middle Level Education (formerly National Middle School Association)
Association for Supervision and Curriculum Development
Association of American Educators
Council of Chief State School Officers
Council of the Great City Schools
National Association of Elementary School Principals
National Association of Secondary School Principals
National Education Association
National Parent Teacher Association
The School Superintendents Association
Endorsements from key state public-school groups. In addition, for NTPS 2023-24, like its 2020-21 predecessor, NCES will seek endorsement from the following state organizations and agencies (the number of state organizations and agencies are capped at two per state for efficiency of solicitation operations):
Alabama
Birmingham Federation of Teachers
Alabama Education Association
Alaska
Alaska Public Employees Association
Alaska Education Association
Arizona
American Federation of Teachers - Arizona
Arizona Education Association
Arkansas
Arkansas Education Association
California
American Federation of Teachers - California
California Teachers Association
Colorado
American Federation of Teachers - Colorado
Colorado Education Association
Connecticut
American Federation of Teachers - Connecticut
Connecticut Education Association
Delaware
Delaware State Education Association
District of Columbia
Washington Teachers' Union
Florida
Florida Education Association
Georgia
American Federation of Teachers - Georgia
Georgia Association of Educators
Hawaii
Hawaii State Teachers Association
Idaho
Idaho Education Association
Illinois
American Federation of Teachers - Illinois
Illinois Education Association
Indiana
American Federation of Teachers - Indiana
Indiana State Teachers Association
Iowa
Iowa State Education Association
Kansas
American Federation of Teachers - Kansas
Kansas National Education Association
Kentucky
Kentucky Education Association
Louisiana
American Federation of Teachers - Louisiana
Louisiana Association of Educators
Maine
Maine State Employee Association
Maine Education Association
Maryland
American Federation of Teachers - Maryland
Maryland State Education Association
Massachusetts
American Federation of Teachers - Massachusetts
Massachusetts Teachers Association
Michigan
American Federation of Teachers - Michigan
Michigan Education Association
Minnesota
Education Minnesota
Mississippi
American Federation of Teachers - Mississippi
Mississippi Association of Educators
Missouri
American Federation of Teachers - Missouri
Missouri Education Association
Montana
Montana Federation of Public Employees
Nebraska
Nebraska State Education Association
Nevada
Nevada State Education Association
New Hampshire
American Federation of Teachers - New Hampshire
New Hampshire Education Association
New Jersey
American Federation of Teachers - New Jersey
New Jersey Education Association
New Mexico
American Federation of Teachers - New Mexico
New Mexico Education Association
New York
New York State United Teachers
North Carolina
North Carolina Association of Educators
North Dakota
North Dakota United
Ohio
Ohio Federation of Teachers
Ohio Education Association
Oklahoma
American Federation of Teachers - Oklahoma
Oklahoma Education Association
Oregon
American Federation of Teachers - Oregon
Oregon Education Association
Pennsylvania
American Federation of Teachers - Pennsylvania
Pennsylvania State Education Association
Rhode Island
Rhode Island Federation of Teacher and Healthcare Professionals
Rhode Island Education Association
South Carolina
South Carolina Education Association
South Dakota
South Dakota Education Association
Tennessee
Tennessee Education Association
Texas
American Federation of Teachers - Texas
Texas State Teachers Association
Utah
American Federation of Teachers - Utah
Utah Education Association
Vermont
American Federation of Teachers - Vermont
Vermont Education Association
Virginia
Fairfax County Federation of Teachers
Virginia Education Association
Washington
American Federation of Teachers - Washington
Washington Education Association
West Virginia
American Federation of Teachers - West Virginia
West Virginia Education Association
Wisconsin
American Federation of Teachers - Wisconsin
Wisconsin Education Association Council
Wyoming
Wyoming Education Association
Endorsements from key private-school groups. In addition to the endorsements from key public-school organizations and agencies, NCES will also seek endorsements for NTPS 2023-24 from the following private school organizations:
Agudath Israel of America/Lefkowitz Leadership Initiative
American Association of Christian Schools
American Association of School Librarians
American Counseling Association
American Federation of Teachers
American Montessori Society
American School Counselors Association
Association for Middle Level Education
Association for Supervision and Curriculum Development
Association Montessori International
Association of American Educators
Association of Christian Schools International
Association of Christian Teachers and Schools
Association of Classical Christian Schools
Association of Military Colleges and Schools
Association of Waldorf Schools of North America
Christian Schools International
Council for American Private Education
Council of Chief State School Officers
Council of Islamic Schools of North America
Council of the Great City Schools
Evangelical Lutheran Church in America
Friends Council on Education
General Conference of Seventh-Day Adventists
Islamic School League of America
Jesuit Schools Network
Jewish Education Services of North America
Lutheran Church-Missouri Synod
National Association of Elementary School Principals
National Association of Episcopal Schools
National Association of Independent Schools
National Association of Private Special Education Centers
National Association of Secondary School Principals
National Catholic Educational Association
National Christian School Association
National Coalition of Girls’ Schools
National Council for Private School Accreditation
National Education Association
National Independent Private School Association
National Parent Teacher Association
Office of Education, General Conference of Seventh Day Adventists
Oral Roberts University Educational Fellowship
Prizmah: Center for Jewish Day Schools
RAVSAK: Jewish Community Day School Network
Southern Baptist Association of Christian Schools
The Association of Boarding Schools
The Jewish Education Project
The School Superintendents Association
Torah Umesorah National Society for Hebrew Day Schools
United States Conference of Catholic Bishops
Wisconsin Evangelical Lutheran Synod
1School coordinator is only given an incentive in schools where the teachers are sampled from vendor purchased or clerically searched Teacher Listing Forms (TLFs).
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
File Modified | 0000-00-00 |
File Created | 2024-07-21 |