To: Lauren Supplee, Head Start CARES Project Officer
From: Head Start CARES Project Team
Date: March 29, 2012
Subject: Head Start CARES Incentive Study
This memo describes the results of a study built into the data collection process of the Head Start CARES project by OMB request. The study assesses whether the payment amount provided to teachers for completion of the teacher self survey affects teacher response rates.
Background
Teacher self surveys were collected in the Head Start CARES study in order to collect information about teachers’ demographics, psychosocial variables, and background characteristics. Teacher self surveys were collected twice in the CARES project: 1) at baseline, during the spring prior to the school year that the enhancements were implemented, and 2) at follow-up during the spring of the implementation year. At baseline, classroom observations were also being collected, and observers dropped the teacher surveys off to the teachers when they came to the classroom for the observation, and teachers were asked to mail the surveys back to the survey firm. Incentives were mailed to teachers after surveys were received.
At follow-up, the project was also collecting child assessments, and assessors dropped off the surveys during their first day of assessments and collected them from the teacher on their last day of assessments. Any teachers who did not return the surveys to the assessor were asked to mail them to the survey firm, and incentives were again mailed to teachers after surveys were received.
The CARES project was approved by OMB with a $15 incentive for the completion of the teacher self survey for a subset of sites (OMB Control Number: 0970-0364). This amount was consistent with MDRC’s and the survey firm’s prior experience interviewing similar populations. For a second subset of sites, OMB required including a planned variation study. This memo describes the results of that variation study.
Research Question
The study addresses the following research question: Does the amount of payment provided to teachers for completion of the teacher self survey affect response rates? Specifically, we assess whether a $5 difference in incentive payment amount affects response rates for the teacher self survey.
Description of the Study
The study includes 12 grantees in eight regional hubs. We randomly assigned each hub to one of the following two groups:
Group A: Teachers receive a $15 payment if they complete the teacher self survey at baseline and a $15 payment if they complete the survey at follow-up.
Group B: Teachers receive a $10 payment if they complete the teacher self survey at baseline and a $20 payment if they complete the survey at follow-up.
Teachers in both groups are eligible to receive the same total incentive amount over the two data collection periods ($30). Four hubs (containing six grantees) were randomly assigned to Group A, and four hubs (containing six grantees) were randomly assigned to Group B. Random assignment results are shown in Table 1.
To assess whether a $5 difference in incentive payments makes a difference for response rates, we compare grantee-level response rates of Groups A and B at baseline ($10 vs. $15) and the response rates of the two groups at follow-up ($15 vs. $20).
Results
Tables 2 and 3 show the response rates in each grantee at baseline and follow-up, respectively. There was only a difference between the two groups at baseline. At baseline, Group A, who received $15, had an overall response rate of 100%, and Group B, who received $10, had an overall response rate of 93%. At follow-up, both the $15 group and the $20 group had response rates of 100%.
Conclusion
The seven percentage point difference at baseline suggests that, at an incentive level of $10, a $5 increase may make a difference for improving response rates. However, because we randomly assigned eight hubs, there is not sufficient statistical power to perform a t-test to formally test whether this seven percentage point difference is statistically significant. Our results do not show evidence that a $5 increase in payment amount at the $15 level improves response rates.
When interpreting the results of these comparisons and considering incentive levels for other projects, it is useful to keep in mind two factors specific to the CARES project that may largely explain the very high response rates overall:
The project’s data collection process allowed for multiple contacts with the teachers in the study, both in person and over the phone. In fact, the higher response rates at follow-up compared to baseline may be in part due to the fact that assessors came to the classrooms at follow-up to pick up the surveys in person, rather than having the teachers send them back by mail as was done at baseline.
The project’s survey firm had a strong commitment to achieving high response rates and had the flexibility to invest additional resources in pursuing non-respondents when needed.
Table 1. Random Assignment Results |
|
|
|
|
||
|
|
|
|
|
||
Hub # |
Hub |
Name of Grantee/Delegate Agency |
Location |
Spring Baseline Incentive |
Spring Follow up Incentive |
Total |
|
|
|
|
|
|
|
5 |
Winona, MS |
Central Mississippi, Inc. |
Winona, MS |
$10 |
$20 |
$30 |
6 |
San Diego, CA |
Episcopal Community Services |
Chula Vista, CA |
$10 |
$20 |
$30 |
7 |
East Texas |
Region 7 Education Service Center |
Kilgore, TX |
$10 |
$20 |
$30 |
Tyler ISD |
Tyler, TX |
$10 |
$20 |
$30 |
||
8 |
Chicago, IL |
Chicago Youth Centers |
Chicago, IL |
$15 |
$15 |
$30 |
9 |
Central Ohio |
Child Development Council of Franklin County |
Columbus, OH |
$15 |
$15 |
$30 |
LEADS Head Start |
Newark, OH |
$15 |
$15 |
$30 |
||
WSOS Community Action Commission, Inc. |
Fremont, OH* |
$15 |
$15 |
$30 |
||
10 |
Bay Area, CA |
Santa Clara County Office of Education |
San Jose, CA |
$10 |
$20 |
$30 |
Berkeley-Albany YMCA Head Start |
Berkeley, CA |
$10 |
$20 |
$30 |
||
11 |
Los Angeles, CA |
Pacific Asian Consortium In Employment |
Los Angeles, CA |
$15 |
$15 |
$30 |
12 |
Denver, CO |
Rocky Mountain SER Head Start |
Denver, CO |
$15 |
$15 |
$30 |
Table 2. Baseline Teacher Self-Survey Response Rates |
|
|
||
|
|
|
|
|
|
Total Teachers |
Total Completed |
% Complete |
|
|
|
|
|
|
|
Group A: $15 incentive |
|||
|
|
|
|
|
Central Ohio |
|
|
|
|
Child Dev. Council of Franklin County (Columbus) |
16 |
16 |
100% |
|
LEADS Head Start (Newark) |
8 |
8 |
100% |
|
WSOS Comm. Action Commission, Inc. (Fremont) |
14 |
14 |
100% |
|
|
|
|
|
|
Los Angeles, CA |
|
|
|
|
Pacific Asian Consortium in Employment (LA) |
40 |
40 |
100% |
|
|
|
|
|
|
Denver, CO |
|
|
|
|
Rocky Mountain SER Head Start (Denver) |
12 |
12 |
100% |
|
|
|
|
|
|
Chicago, IL |
|
|
|
|
Chicago Youth Centers (CYC) |
17 |
17 |
100% |
|
|
|
|
|
|
Total |
107 |
107 |
100% |
|
|
|
|
|
|
|
Group B: $10 incentive |
|||
|
|
|
|
|
Winona, MS |
|
|
|
|
Central Mississippi, Inc. (Winona) |
38 |
36 |
95% |
|
|
|
|
|
|
San Diego, CA |
|
|
|
|
Episcopal Community Services (Chula Vista) |
23 |
23 |
100% |
|
|
|
|
|
|
East Texas |
|
|
|
|
Region 7 Edu. Service Center (Kilgore) |
31 |
26 |
84% |
|
Tyler ISD (Tyler) |
11 |
11 |
100% |
|
|
|
|
|
|
Bay Area, CA |
|
|
|
|
Santa Clara County Office of Education (San Jose) |
10 |
9 |
90% |
|
Berkeley-Albany YMCA Head Start (Berkeley) |
8 |
8 |
100% |
|
|
|
|
|
|
Total |
121 |
113 |
93% |
|
Table 3. Follow-Up Teacher Self-Survey Response Rates |
||||
|
|
|
|
|
|
Total Teachers |
Total Completed |
% Complete |
|
|
|
|
|
|
|
Group A: $15 incentive |
|||
|
|
|
|
|
Central Ohio |
|
|
|
|
Child Dev. Council of Franklin County (Columbus) |
15 |
15 |
100% |
|
LEADS Head Start (Newark) |
8 |
8 |
100% |
|
WSOS Comm. Action Commission, Inc. (Fremont) |
14 |
14 |
100% |
|
|
|
|
|
|
Los Angeles, CA |
|
|
|
|
Pacific Asian Consortium in Employment (LA) |
39 |
39 |
100% |
|
|
|
|
|
|
Denver, CO |
|
|
|
|
Rocky Mountain SER Head Start (Denver) |
11 |
11 |
100% |
|
|
|
|
|
|
Chicago, IL |
|
|
|
|
Chicago Youth Centers (CYC) |
16 |
16 |
100% |
|
|
|
|
|
|
Total |
103 |
103 |
100% |
|
|
|
|
|
|
|
Group B: $20 incentive |
|||
|
|
|
|
|
Winona, MS |
|
|
|
|
Central Mississippi, Inc. (Winona) |
38 |
38 |
100% |
|
|
|
|
|
|
San Diego, CA |
|
|
|
|
Episcopal Community Services (Chula Vista) |
21 |
21 |
100% |
|
|
|
|
|
|
East Texas |
|
|
|
|
Region 7 Edu. Service Center (Kilgore) |
30 |
30 |
100% |
|
Tyler ISD (Tyler) |
8 |
8 |
100% |
|
|
|
|
|
|
Bay Area, CA |
|
|
|
|
Santa Clara County Office of Education (San Jose) |
10 |
10 |
100% |
|
Berkeley-Albany YMCA Head Start (Berkeley) |
8 |
8 |
100% |
|
|
|
|
|
|
Total |
115 |
115 |
100% |
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
File Title | MDRC Memo Template |
Author | Nellie Ng |
File Modified | 0000-00-00 |
File Created | 2021-01-30 |