Attachment A - Incentives/Outlets Test Overview

Incentives/Outlets Field Test – Test Overview

1. Study Overview

In 2009, the Bureau of Labor Statistics (BLS) Consumer Expenditure Survey (CE) initiated the multi-year Gemini Project for the purpose of researching, developing, and implementing an improved survey design. In addition to reducing measurement error, a secondary objective of the redesign is to address survey response rates, which have been trending downward over the past 20 years. In the survey literature, as well as findings from the Gemini Incentive Structure Review: Summary of Incentive Experiences (To, 2014), previous CE incentives operational summaries, as well as the final CE Interview Incentives Test (Goldenberg et al. 2009) and CE Diary Incentives Test (McGrath et al. 2007) reports, research has shown that providing incentives to respondents is an effective method of increasing respondent cooperation and response rates (Eyerman et al., 2002).

Incentives can be cash or non-cash, and distributed as token, promised, conditional, or performance-based. For the majority of federal household survey incentive tests reviewed in To (2014), incentives were more often distributed to compliant respondents after survey completion; however, some studies did distribute incentives prior to the administration of the survey (either the full incentive or a token incentive) to all potential respondents. Past research has also shown that monetary incentives perform better than non-cash incentives under most conditions, but their effectiveness varies depending on the amount of the incentive, how it is administered, and respondent characteristics (Singer et al., 1999). To’s 2014 research also showed that on average, incentives being tested or in use in Federal surveys hover around the $50 level, with some in excess of $100 per sample unit.

CE plans to test the effect different incentive delivery procedures and incentive amounts have on survey costs, response rates, and data quality for the CE Interview Survey (CEQ). The results of the FY2015 Incentives Field test will be used to inform the Large Scale Feasibility test (to be fielded in 2018) as well as the overall Gemini Redesign project. CE plans to implement incentives into production if to-be-determined cost-benefit trade-offs warrant.

This portion of the study will attempt to answer the following research questions:

1. Cost Effectiveness. What is the cost effectiveness of using incentives to gain cooperation, reduce contact attempts, and encourage engagement (use of infobook, use of records, FR report of respondent engagement, etc.) through the interview period, and for subsequent interview waves?

2. Feasibility. What are the operational issues related to implementing incentives for CEQ data collection?

   1. Debrief with field representatives (FRs) to determine:

1. What issues if any did respondents have with the materials provided?

2. What issues with using the debit card, if any, were raised by the respondents?

3. How should respondent materials be revised?

1. Debrief with respondents to determine:

   1. What did they like about the incentive they were offered?

   2. What did they dislike about the incentive they were offered?

   3. What extra information would have been helpful to them?

2. Debrief with the Regional Office (RO) and Demographic Survey Divisions (DSD) staff that handle respondent help requests to determine:

1. What kinds of questions did respondents ask?

2. How should respondent materials be revised?

3. What were the number of contact attempts necessary to obtain a completed interview?

4. What was the per completed interview cost of administering the survey with the incentive?

3. Data Quality. How do high-level expenditure reporting rates and data quality (or data quality indicators) differ by test group and control?

1. Doorstep concerns

2. Item-missingness

3. Number of combined expenses

4. Record use

5. Information booklet usage

6. Mode of subsequent interview waves

7. Endorsement of filter questions

8. Expenditure reports

4. Response Rates. How do response rates differ by test group and control?

1. Wave 1 Response Rates (Debit Cards)

2. Wave 2 Response Rates (Debit Cards)

In addition, CE and the Consumer Price Index (CPI) plan to test integrating outlet questions into the CEQ survey. Outlet data are currently collected by the Telephone Point of Purchase Survey (TPOPS). The results of the integration of outlet questions into the CEQ survey will be used to inform future CPI initiatives.

This portion of the study will attempt to answer the following research questions:

1. Expenditure Entries. How do different outlet designs affect a respondent’s entries and understanding of CE data requirements?

2. Design. Which outlet entry designs are preferred by respondents?

3. Burden. Which outlet entry designs are least likely to burden respondents?

2. Study Design

1. Field period

The field period for the monetary incentive portion of the test is scheduled for July 2016 through December 2016 with debriefing questions asked starting in October 2016 through March 2017. The field period for the outlets portion of the test is scheduled for July 2016 through September 2016.

2. Respondents

The Production sample (July 2016 through March 2017) will serve as the test group as well as the control group.

3. Methodology

The incentives portion of the Incentives/Outlets Field Test will consist of three test groups that will be offered an incentive option. Respondents will be randomly assigned to one of the test groups or the control group. All monetary incentives, except for the token cash incentive, will be offered in the form of a debit card.

1. Incentive plus Record Only (Debit Card distributed with Advance Letter mentioning incentive) – The CU will receive a monetary survey incentive of $40, in the form of a debit card, with the Advance Letter. This monetary incentive is conditional on interview completion and the incentive will be activated at the completion of the Wave 1 interview. In addition, the CU will receive a record use monetary debit card incentive of $20, distributed by mail from NPC after the completion of the interview, if they have satisfied the record use requirement. Record use will be defined as the use of at least one receipt, paper record, or electronic record.

2. Incentive plus Token plus Record (Debit Card and token distributed with Advance Letter mentioning incentive) – The CU will receive a monetary survey incentive of $40, in the form of a debit card, with the Advance Letter as well as a $5 token cash incentive, also sent with the Advance Letter. This monetary incentive is conditional on interview completion and the incentive will be activated at the completion of the Wave 1 interview; however, the token incentive is unconditional. In addition, the CU will receive a record use monetary debit card incentive of $20, distributed by mail from NPC after the completion of the interview, if they have satisfied the record use requirement. Record use will be defined as the use of at least one receipt, paper record, or electronic record.

3. Incentive plus Token (Debit Card and token distributed with Advance Letter mentioning incentive) – The CU will receive a monetary survey incentive of $40, in the form of a debit card, with the Advance Letter as well as a $5 token cash incentive, also sent with the Advance Letter. This monetary incentive is conditional on interview completion and the incentive will be activated at the completion of the Wave 1 interview; however, the token incentive is unconditional.

The test groups will be given the monetary incentive offerings during Wave 1 of the interview. All test groups will be asked the appropriate CAPI questions during the visit as if they were a part of a normal production interview. However, monetary incentives test groups will be asked debriefing questions regarding the effectiveness of the cash incentive as well as reminded that items purchased using the monetary incentive should not be included as part of the reported expenditures during Wave 2.

The outlets portion of the Incentives/Outlets Field Test will consist of one test group that will be asked outlets questions requested by CPI.

1. Outlets – Wave 4 CUs will receive questions from an alternate questionnaire to specify where an item was purchased and where the outlet is located. Additional outlet questions will be added to five sections (4C, 6B, 9A, 12C, and 17) in the instrument.

4. CAPI Instrument Changes

Census will modify the CAPI instrument to include (See Attached CAPI Specifications – attachment B):

1. Questions applicable to the Incentive Only (Debit Card distributed with Advance Letter), Incentive plus Token (Debit Card and token distributed with Advance Letter) to determine receipt of debit card or token incentive to the Wave 1 interview.

2. Outlet questions to sections 4C, 6B, 9A, 12C, and 17 to the Wave 4 interview (See Appendix A).

3. Debriefing questions for respondents in the monetary incentives test groups to the Wave 2 interview.

4. Questions for FRs in the monetary incentives test groups to the Wave 1 interview including records use questions.

5. A reminder to the respondents that items purchased using the monetary incentive should not be included as part of the reported expenditures during Wave 2.

5. Sampling and Sample Size

A portion of the production sample will serve as the test group (up to approximately 4,050 Wave 1 cases during the fielding period with the remaining Wave 1 cases serving as the control group and 2,000 Wave 4 cases during the first three months of the fielding period with the remaining Wave 4 cases serving as the Control Group) and will be randomly selected from PSUs within the six regional office regions. The control group will come from the production sample for the quarters that correspond to the fielding of this test.

Incentives

Incentives Test Groups	Starting Sample
Test Group 1: Incentive + Record (Advance Letter)	1,350
Test Group 2: Incentive + Token + Record (Advance Letter)	1,350
Test Group 3: Incentive + Token (Advance Letter)	1,350
Starting Sample	4,050
Completed Interviews1	2,430
Control Group	1,950

The power calculations source data is the 2014 Quarter 1 CE Phase III Interview Data. The calculations assume that the control group’s sample size n_c equals the entire Interview sample n.

The sample size n_t of usable interviews is calculated as for a 10% statistical significance level.

Response rate for CEQ in 2014 is equal to 66.43%. The response rates are calculated for eligible households for interviews two through five only.

The number of HHs needed to achieve statistical significance = (n_t / 0.66) is shown in the table below for the chosen Z-VARIABLES.

Table of z-variables: Interview expenditure totals for collection year 2014 quarter 1 (n = 6,483)

VARIABLE	MEAN	SE	# usable interviews	# HHs
ZALCBEVS	89.55	3.75	13,415	20,326
ZAPPAREL	324.50	7.32	1,577	2,389
ZCASHCTB	551.35	41.50	∞	∞
ZEDUCATN	251.66	23.52	∞	∞
ZENTRMNT	572.49	11.58	1,209	1,832
ZFOODAWY	559.99	9.56	817	1,238
ZFOODHOM	1,259.50	13.30	290	439
ZFOODTOT	1,819.50	19.12	287	435
ZHEALTH	963.17	21.53	1,540	2,333
ZHOUSING	4,087.80	51.41	419	635
ZMISCELS	117.50	11.72	∞	∞
ZPERCARE	73.52	1.48	1,190	1,803
ZPERLINS	1,179.50	38.54	4,508	6,830
ZREADING	18.28	0.94	∞	∞
ZSHELTER	2,496.60	39.10	674	1,021
ZTOBACCO	76.43	3.64	44,175	66,932
ZTOTAL	12,302.00	151.90	403	611
ZTRANPRT	2,178.20	71.53	4,585	6,947

*∞= statistical significance cannot be attained with any sample size.

Outlets Questions

Outlets Test Group	Starting Sample
Test Group: Outlets Questions	2,000
Starting Sample	2,000
Completed Interviews2	1,040

6. Available Materials

The following materials are available to OMB for consideration of this project:

1. CAPI Instrument Specifications

2. Test Group 1: Incentive + Record - Advance Letter

3. Test Group 2: Incentive + Token + Record - Advance Letter

4. Test Group 3: Incentive + Token - Advance Letter

5. Debit Card Sleeve

6. Pin Flyer

3. Burden Hour Estimate

BLS estimates that this field study will require a total of 370.2 additional burden hours, 349.95 of which will be in CY2016 and 20.25 of which will be in CY2017.

	Mins.	Starting Sample	Completed Interviews	Total Hours
Estimated time to complete additional questions including debriefing: Test Group 1 (July – September Wave 1 Sample)	2	675	405	13.5
Estimated time to complete additional questions including debriefing: Test Group 2 (July – September Wave 1 Sample)	2	675	405	13.5
Estimated time to complete additional questions including debriefing: Test Group 3 (July – September Wave 1 Sample)	2	675	405	13.5
Estimated time to complete additional questions: Test Group 1 (October – December Wave 1 Sample)	1	675	405	6.75
Estimated time to complete additional questions: Test Group 2 (October – December Wave 1 Sample)	1	675	405	6.75
Estimated time to complete additional questions: Test Group 3 (October – December Wave 1 Sample)	1	675	405	6.75
Estimated time to complete records incentive debriefing questions in CY2017: Test Group 1 (October – December Wave 1 Sample)	1	675	405	6.75
Estimated time to complete records incentive debriefing questions in CY2017: Test Group 1 (October – December Wave 1 Sample)	1	675	405	6.75
Estimated time to complete records incentive debriefing questions in CY2017: Test Group 1 (October – December Wave 1 Sample)	1	675	405	6.75
Estimated time to complete additional questions: Outlets Test Group	5	2,000	1,040	86.7
Additional time for Incentives protocols during placement	5	4,050	2,430	202.5
TOTAL				370.2

4. References

1. To, N. (2014). Gemini Incentive Structure Review: Summary of Incentive Experiences, Internal Bureau of Labor Statistics Document.

2. McGrath, D., B. Chopova, C. Gomes, and N. McDermott, (2007) The Effects of Incentives on the Consumer Expenditure Diary, Internal Bureau of Labor Statistics Document.

3. Goldenberg, K., L. Tan, and D. McGrath, (2009) The Effects of Incentives on the Consumer Expenditure Interview Survey, Internal Bureau of Labor Statistics Document.

4. CE Incentives Team (2006) Wrap-up of Procedures During the Test, Internal Bureau of Labor Statistics Document.

5. Ruffin, J. (2006) Incentives Wrap-up Report, Internal Census Bureau/Bureau of Labor Statistics Document.

6. CE Demographic Surveys Division (2006) DSD’s Report on the CE Incentive Test, Internal Bureau of Labor Statistics Document.

7. Gfroerer, J., J. Eyerman, and J. Chromy, (2002). Redesigning an Ongoing National Household Survey: Methodological Issues, Substance Abuse and Mental Health Services Administration, Office of Applied Studies.

8. Singer, E., J. Van Hoewyk, N. Gebler, T. Raghunathan, and K. McGonagle, (1999). The Effect of Incentives on Response Rates in Interviewer-mediated Surveys. Journal of Official Statistics.

Appendix A. Outlet Questions

Section 4C (1 additional question)

1. What is the name of the company providing the internet service?

Section 6B (3 additional questions)

1. Where did you purchase this television?

2. Was this purchased online or in-person?

   1. Online

   2. In person

3. Where is [STORE] located?

Section 9A (3 additional questions)

1. Where did you purchase this [Coats, jackets, furs, sport coats, tailored jackets, or blazers]?

2. Was this purchased online or in-person?

   1. Online

   2. In person

3. Where is [STORE] located?

Section 12C (1 additional questions)

1. From reference month through last month, where did you last/most frequently purchase gas?

Section 17 (total 3 questions)

1. Where did you purchase these tickets?

2. Was this purchased online or in-person?

   1. Online

   2. In person

3. Where is [STORE] located?

1 Completed Interviews is defined as the expected number of completed interviews (Starting Sample multiplied by expected response rate)

2 Completed Interviews is defined as the expected number of completed interviews (Starting Sample multiplied by expected response rate)

11