Appendix G: Incentive Study

OMB CARES 2nd Package_Appendix G_Incentive Study 4 4 12.docx

HHS/ACF/OPRE Head Start Classroom-based Approaches and Resources for Emotion and Social skill promotion (CARES) project: Impact and Implementation Studies

Appendix G: Incentive Study

OMB: 0970-0364

Document [docx]
Download: docx | pdf


To: Lauren Supplee, Head Start CARES Project Officer      

From: Head Start CARES Project Team     

Date: March 29, 2012     

Subject: Head Start CARES Incentive Study



This memo describes the results of a study built into the data collection process of the Head Start CARES project by OMB request. The study assesses whether the payment amount provided to teachers for completion of the teacher self survey affects teacher response rates.


Background

Teacher self surveys were collected in the Head Start CARES study in order to collect information about teachers’ demographics, psychosocial variables, and background characteristics. Teacher self surveys were collected twice in the CARES project: 1) at baseline, during the spring prior to the school year that the enhancements were implemented, and 2) at follow-up during the spring of the implementation year. At baseline, classroom observations were also being collected, and observers dropped the teacher surveys off to the teachers when they came to the classroom for the observation, and teachers were asked to mail the surveys back to the survey firm. Incentives were mailed to teachers after surveys were received. 

At follow-up, the project was also collecting child assessments, and assessors dropped off the surveys during their first day of assessments and collected them from the teacher on their last day of assessments. Any teachers who did not return the surveys to the assessor were asked to mail them to the survey firm, and incentives were again mailed to teachers after surveys were received. 


The CARES project was approved by OMB with a $15 incentive for the completion of the teacher self survey for a subset of sites (OMB Control Number: 0970-0364). This amount was consistent with MDRC’s and the survey firm’s prior experience interviewing similar populations. For a second subset of sites, OMB required including a planned variation study. This memo describes the results of that variation study.


Research Question

The study addresses the following research question: Does the amount of payment provided to teachers for completion of the teacher self survey affect response rates? Specifically, we assess whether a $5 difference in incentive payment amount affects response rates for the teacher self survey.


Description of the Study

The study includes 12 grantees in eight regional hubs. We randomly assigned each hub to one of the following two groups:

  • Group A: Teachers receive a $15 payment if they complete the teacher self survey at baseline and a $15 payment if they complete the survey at follow-up.

  • Group B: Teachers receive a $10 payment if they complete the teacher self survey at baseline and a $20 payment if they complete the survey at follow-up.


Teachers in both groups are eligible to receive the same total incentive amount over the two data collection periods ($30). Four hubs (containing six grantees) were randomly assigned to Group A, and four hubs (containing six grantees) were randomly assigned to Group B. Random assignment results are shown in Table 1.


To assess whether a $5 difference in incentive payments makes a difference for response rates, we compare grantee-level response rates of Groups A and B at baseline ($10 vs. $15) and the response rates of the two groups at follow-up ($15 vs. $20).


Results

Tables 2 and 3 show the response rates in each grantee at baseline and follow-up, respectively. There was only a difference between the two groups at baseline. At baseline, Group A, who received $15, had an overall response rate of 100%, and Group B, who received $10, had an overall response rate of 93%. At follow-up, both the $15 group and the $20 group had response rates of 100%.


Conclusion

The seven percentage point difference at baseline suggests that, at an incentive level of $10, a $5 increase may make a difference for improving response rates. However, because we randomly assigned eight hubs, there is not sufficient statistical power to perform a t-test to formally test whether this seven percentage point difference is statistically significant. Our results do not show evidence that a $5 increase in payment amount at the $15 level improves response rates.


When interpreting the results of these comparisons and considering incentive levels for other projects, it is useful to keep in mind two factors specific to the CARES project that may largely explain the very high response rates overall:

  • The project’s data collection process allowed for multiple contacts with the teachers in the study, both in person and over the phone. In fact, the higher response rates at follow-up compared to baseline may be in part due to the fact that assessors came to the classrooms at follow-up to pick up the surveys in person, rather than having the teachers send them back by mail as was done at baseline.

  • The project’s survey firm had a strong commitment to achieving high response rates and had the flexibility to invest additional resources in pursuing non-respondents when needed.

Table 1. Random Assignment Results










Hub #

Hub

Name of Grantee/Delegate Agency

Location

Spring Baseline Incentive

Spring Follow up Incentive

Total








5

Winona, MS

Central Mississippi, Inc.

Winona, MS

$10

$20

$30

6

San Diego, CA

Episcopal Community Services

Chula Vista, CA

$10

$20

$30

7

East Texas

Region 7 Education Service Center 

Kilgore, TX

$10

$20

$30

Tyler ISD

Tyler, TX

$10

$20

$30

8

Chicago, IL

Chicago Youth Centers 

Chicago, IL

$15

$15

$30

9

Central Ohio

Child Development Council of Franklin County

Columbus, OH

$15

$15

$30

LEADS Head Start

Newark, OH

$15

$15

$30

WSOS Community Action Commission, Inc.

Fremont, OH*

$15

$15

$30

10

Bay Area, CA

Santa Clara County Office of Education

San Jose, CA

$10

$20

$30

Berkeley-Albany YMCA Head Start 

Berkeley, CA

$10

$20

$30

11

Los Angeles, CA

Pacific Asian Consortium In Employment 

Los Angeles, CA

$15

$15

$30

12

Denver, CO

Rocky Mountain SER Head Start

Denver, CO

$15

$15

$30










Table 2. Baseline Teacher Self-Survey Response Rates







 

Total Teachers

Total Completed

% Complete

 

 


 

 

Group A: $15 incentive

 

 


 

Central Ohio

 


 

Child Dev. Council of Franklin County (Columbus)

16

16

100%

LEADS Head Start (Newark)

8

8

100%

WSOS Comm. Action Commission, Inc. (Fremont)

14

14

100%

 

 


 

Los Angeles, CA

 


 

Pacific Asian Consortium in Employment (LA)

40

40

100%

 

 


 

Denver, CO

 


 

Rocky Mountain SER Head Start (Denver)

12

12

100%

 

 


 

Chicago, IL

 


 

Chicago Youth Centers (CYC)

17

17

100%

 

 

 

 

Total

107

107

100%

 

 


 

 

Group B: $10 incentive

 

 


 

Winona, MS

 


 

Central Mississippi, Inc. (Winona)

38

36

95%

 

 


 

San Diego, CA

 


 

Episcopal Community Services (Chula Vista)

23

23

100%

 

 


 

East Texas

 


 

Region 7 Edu. Service Center (Kilgore)

31

26

84%

Tyler ISD (Tyler)

11

11

100%

 

 


 

Bay Area, CA

 


 

Santa Clara County Office of Education (San Jose)

10

9

90%

Berkeley-Albany YMCA Head Start (Berkeley)

8

8

100%

 

 


 

Total

121

113

93%






Table 3. Follow-Up Teacher Self-Survey Response Rates





 

Total Teachers

Total Completed

% Complete

 

 


 

 

Group A: $15 incentive

 

 


 

Central Ohio

 


 

Child Dev. Council of Franklin County (Columbus)

15

15

100%

LEADS Head Start (Newark)

8

8

100%

WSOS Comm. Action Commission, Inc. (Fremont)

14

14

100%

 

 


 

Los Angeles, CA

 


 

Pacific Asian Consortium in Employment (LA)

39

39

100%

 

 


 

Denver, CO

 


 

Rocky Mountain SER Head Start (Denver)

11

11

100%

 

 


 

Chicago, IL

 


 

Chicago Youth Centers (CYC)

16

16

100%

 

 


 

Total

103

103

100%

 

 


 

 

Group B: $20 incentive

 

 


 

Winona, MS

 


 

Central Mississippi, Inc. (Winona)

38

38

100%

 

 


 

San Diego, CA

 


 

Episcopal Community Services (Chula Vista)

21

21

100%

 

 


 

East Texas

 


 

Region 7 Edu. Service Center (Kilgore)

30

30

100%

Tyler ISD (Tyler)

8

8

100%

 

 


 

Bay Area, CA

 


 

Santa Clara County Office of Education (San Jose)

10

10

100%

Berkeley-Albany YMCA Head Start (Berkeley)

8

8

100%

 

 


 

Total

115

115

100%




File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File TitleMDRC Memo Template
AuthorNellie Ng
File Modified0000-00-00
File Created2021-01-30

© 2024 OMB.report | Privacy Policy