Gemini Proof-of-Concept Test Overview
The Proof-of-Concept test is designed to test a redesign plan for the Bureau of Labor Statistics (BLS) Consumer Expenditure Survey (CE).
The current Consumer Expenditure Survey consists of two independent samples – (1) a quarterly interview sample, conducted via a Computer Assisted Personal Interview (CAPI) collecting income, demographics and expenditures based on a 3 month recall, and (2) a diary sample, with demographics and income collected via CAPI and a paper-and-pencil Diary kept by one household member recording all expenditures made during two consecutive one-week periods. Each of these surveys have faced declining response rates in recent years and there has been evidence of measurement error1.
In 2009, the CE initiated the multi-year Gemini Project for the purpose of researching, developing, and implementing an improved survey design. The objective of the redesign is to improve the quality of the survey estimates through a verifiable reduction in measurement error while improving or maintaining response rates and with no increase in budget. On July 1, 2013, a redesign proposal developed to meet this objective was released, using as a major source of input a set of recommendations from a convened panel of the National Academies’ Committee on National Statistics (CNSTAT) charged with evaluating and proposing design options for the CE2 and is available online at http://www.bls.gov/cex/ce_gemini_redesign.pdf.
The planned design includes two waves of data collection set 12 months apart. Each wave contains the same interview structure consisting of two visits and individual one-week diaries. Visit 1 is an in-person interview made up of several parts. The first part identifies the roster of the household, while the second is a recall interview that collects large, easily-recalled household expenditures. Lastly, Visit 1 will also incorporate instructions to collect relevant expenditure records for the Visit 2, records-based interview, as well as training for and placement of the individual diaries. Following Visit 1, an electronic, web-based diary (accessible via PC, smartphone, or other mobile device) is maintained for one week by all household members 15 years old and older with a paper diary available by request. Visit 2 is an in-person, records-based expenditure interview on household expenditures that can reasonably be found in records such as receipts, utility bills, and bank statements. Diaries will also be reviewed during this second visit.
As recommended by CNSTAT, performance-based incentives were included as part of the new design. The proposed incentive structure for the design includes a $2 prepaid cash incentive per household sent with an advance letter, a $20 household incentive (debit card) provided after completion of Visit 1, a $20 individual incentive (debit card) for each member who completes the diary, and a $20 or $30 household incentive (debit card) after completion of Visit 2, with the higher amount given to respondents who use at least one record.
Further details on the full redesign plan and background can be found on CE’s Gemini website: http://www.bls.gov/cex/geminiproject.htm.
Prior to moving forward with further research regarding the components of the design plan, a proof-of-concept test is needed to ensure that the basic underlying structure and components of the new design are feasible. The proof-of-concept test is designed to mirror the proposed design to the fullest extent possible. It will include the administration of one wave of the proposed design, consisting of a Visit 1 recall interview, electronic individual diaries for eligible household members with a paper back-up, and a Visit 2 records-focused interview. Lessons learned from previous tests, including the Web Diary Feasibility test (fielded in 2013) and the Individual Diary Feasibility Test (fielded in 2014), were leveraged to refine procedures, field materials, and test protocols for the diary portion of the proof-of-concept test. Additionally, the design will use incentives in order to motivate respondents with an expected reduction in the number of required contact attempts.
A research sample frame will be randomly selected from several areas known to have a high prevalence of English-speaking households and a high technology penetration rate in the Census Regions of New York, Atlanta, Chicago, and Denver. The target number of completed interviews is 450 – with a complete interview defined as a completed Visit 1 interview, at least one completed Diary in the Consumer Unit, and a completed Visit 2 interview. BLS estimates a starting sample size of 1200 needed cases to obtain the desired 450 complete cases3.
The selected initial sample will receive an advance letter via priority mail, including a token cash incentive of $2. The advance letter will provide information that is currently included in the CE interview survey (CEQ) advance letter, modified to streamline the language and include an FAQ of the proof-of-concept test interview structure and promised incentives.
The remaining incentives are performance-based for the completion of each component of the survey. Upon completion of the Visit 1 interview, the respondent will receive a $20 debit card, handed to him/her by the field representative (FR). Each eligible household member that completes a diary will receive a $20 debit card. A completed Diary will be defined as a diarist recording at least one expenditure or checking a box to indicate no expenditures for the diary-keeping week. Upon completion of the Visit 2 interview the respondent will receive a $20 incentive. If the respondent used at least one record for the record portion of the interview, then he/she will receive an additional $20 incentive4. The debit cards for the diary week and/or Visit 2 interview will be mailed to each recipient after the completion of the Visit 2 interview, or once it’s determined that a Visit 2 interview will not be obtained.
The Visit 1 interview will include 3 parts: (1) collection of the household roster and household recall expenses collected through a Blaise CAPI instrument, (2) Instruction on collection of records for the Visit 2 interview, and (3) Introduction to the Diary-keeping week and training of how individuals should complete the web and/or paper diaries.
Collection of roster and household recall expenses. The Visit 1 CAPI will include the administration of 5 sections of the current CE Interview. As in the production CEQ, the recall period will be for the 3 months prior to the Interview month. The recall interview portion is estimated to take on average 20 minutes, based on timing data for the roster and selected sections from the production 2012 CAPI interview. The current interview sections to be administered are as follows:
Interview Sections |
Home Furnishings and Related Household Items |
Appliances, Household Equipment, and Other Selected Items |
Subscriptions, Memberships, Books and Entertainment Expenses |
Trips and Vacations |
Miscellaneous Expenses |
The FR will instruct the respondent on what types of records to collect from the household for the Visit 2 interview. The targeted records include, but are not limited to utility bills, mortgage statements, cancelled checks or bank statements, credit card statements, loan statements, insurance records, and pay stubs. The respondent will be instructed to gather records for the previous 3 months (the same reference period as the recall expenditure items collected in Part 1 of the visit). The FR will leave a checklist showing the records to collect, and remind the respondent of the additional promised incentive for the use of records during the second visit.
The third and final part of the Visit 1 interview will be to explain the diary-keeping week procedure to the respondent and any additional eligible household members that are present. Eligible members are age 15 and older. Eligible members will be asked to keep a one week diary of all expenditures made by that individual. Additionally, the main respondent will be asked to keep track of expenditures that he/she makes in addition to expenditures of ineligible members, members unwilling to participate, or joint household expenditures. Diary keepers that have in-home internet access and access the internet at least a few times a week (by self-report) will be given the choice to collect the data online through a secure Census server using a computer and/or mobile device or via a paper and pencil diary. (The paper diary is the same as the current CE Diary instrument). All other diary keepers will be given a paper and pencil diary. Members that keep a web-based Diary will be provided with login-credentials and instructive materials for the web-based version of the individual diary. For members not present during the training, the main respondent will select the mode of diary response for each absent eligible household member and receive the login credentials or individual paper diary on their behalf. The FR will review the diary-keeping procedures for both the web-based and paper version (if applicable) with present members and provide materials and information that they will need during the diary-keeping week. The FR will also instruct the respondent to train non-present members of the household on how to keep the diary. Lastly, the FR will review the requirements for receiving the promised individual incentives. During the mode selection process, the FR will ask for each diary keeper’s email address and phone number for contacting each diary keeper mid-week. For members that will keep an electronic diary and provide an email, the FR will send an email with links to the diary, the user guide, and video tutorials.
At the conclusion of the interview, the FR will provide the respondent with a $20 debit card for completing the recall interview. Before leaving the respondent’s home, the interviewer will schedule the Visit 2 interview with the respondent in one week’s time.
Each eligible member will keep the diary beginning the day following the Visit 1 interview. During the diary-keeping week, paradata will be collected on the logins for the web-based diary. A report will be generated for FRs that will identify the number of successful and unsuccessful logins by respondents and summary information of expenditures entered in the diary. FRs will use this information to contact the diarist and encourage him/her to record their expenditures or to solve any technical issues. Regardless of logins, the FR will contact the main respondent mid-week to further encourage the diary keeping task and to remind him/her of the records collection task. The FR will use their discretion to contact other household members, based on the member’s willingness to share contact information during Visit 1, log in history, or other similar factors.
For respondents, a Help Desk will be available during their diary keeping week to handle technical inquiries regarding the web-based diary.
The Visit 2 interview will include three parts: (1) review of the individual diary expenditures collected (2) collection of household records expenses through the reduced Blaise CAPI instrument (3) a respondent debriefing of the Visit 1, Diary week, and Visit 2 processes. (See Attachment A10 - POC Respondent Debriefing Questions)
Review of individual diary expenditures. Visit 2 will begin with a review of the Diary week expenditures and, dependent on when the visit 2 interview is conducted, recall of weekly expenditures that were not recorded. The FR will have guidelines for picking up the diaries, similar to current production. The FR will have Electronic Diary Expenditure Log to view respondent entries in the electronic diary for correct expenditure categorization and ensure that each household respondent accessed and made entries in the electronic diary. For any available present members that kept a paper diary, the FR will review expenditures entered by each respondent to ensure there is not missing information. The FR will also collect any additional expenditures that may have been missed during the diary-keeping week through a scripted recall process. The date of pickup (e.g. 8 days after placement) will determine the level of review and follow-up recall questions that will be asked. If the date of pickup is 8-9 days after the placement of the diary, then the available respondents will be asked all recall questions (regardless of whether expenditures had been recorded in the diary or not). If the date of pickup is between 10-12 days after the placement of the diary, then the available respondents will be asked recall questions if they made some entries in the diary or if they did not make any entries, but have receipts of the diary week’s expenses. If the date of the pickup is between 13-18 days after the placement date, then the available respondents will only be asked about any missed expenditure if they had entered some expenditures during the diary week (i.e., total recall of the diary week will not be conducted). If the date is 19 days or later after the initial diary placement, then the FR will not collect any additional information on the diary expenses and will move on to the records portion of the interview. See the following table for a summary of these procedures.
Summary of diary pick up procedures:
Diary Day |
Allowed |
0 |
Diary placement. |
1 - 7 |
Diary keeping. |
8 - 9 |
Review of expenditures, recall of additional expenditures, including total recall if no expenditures recorded. |
10 - 12 |
Review of expenditures, recall of additional expenditures, including total recall if no expenditures recorded, but receipts were kept. |
13 - 18 |
Review of expenditures, recall of additional expenditures, but no total recall. |
19 – closeout |
Conduct Visit 2 interviews, diaries accepted, but not reviewed and no recall completed. |
Collection of household records expenses. The modified Visit 2 CAPI will include the administration of an additional 8 sections of the CEQ. This interview will heavily rely on the records that the FR instructed the respondent to gather during the Visit 1 interview. If the respondent does not have records, then the sections will be completed via recall. The records interview portion is estimated to take on average 36 minutes, based on timing data for these sections from the 2012 production CAPI interview looking at respondents that used records more than 50 percent of the time5. However, the exact impact of record use on timing is currently unknown. The expenditure sections that will be collected during the Visit 2 interview include:
Records |
General Housing Characteristics |
Rented Living Quarters |
Owned Living Quarters and Other Owned Real Estate |
Utilities and Fuels |
Rented and Leased Vehicles |
Owned Vehicles |
Insurance Other than Health |
Health Insurance |
Medical and Health Expenses |
Occupations, Work Experience and Income |
Assets and Liabilities |
Upon completion of the interview, if the respondent used records at least one record, then the respondent will receive two $20 debit cards, otherwise, the respondent will receive a single $20 debit card. Each diarist that logged at least one expenditure entry during the week or checked a box to indicate that no expenses were made during the week will also receive a $20 debit card. These debit cards will be mailed to each individual recipient after the Visit 2 interview.
Debriefing of the Visit 1, Diary week, and Visit 2 process. After the Visit 2 interview is completed, the respondent will be asked a series of debriefing questions. The questions will cover the topics including:
perceptions of burden,
time spent completing tasks outside of the interviews,
helpfulness of the instructions and materials
reactions to advance letter (including token incentive),
reactions to the performance based incentives, and
perceived task difficulty.
A combination of respondent materials from the production CEQ and materials used in the Individual Diary Feasibility test will be modified as appropriate for the proof-of-concept test:
2015 Modified CAPI Instrument – Modifications include that not all CEQ sections are included, the section order is different, respondents are asked about whether any records were used at the end of every section, placement and pick-up questions are modified; and questions on recall were added. (See Attachment A1 – POC Instrument Requirements and Attachment A3 - Diary 2015 Inst Reqts - COVERAGE.)
Advance letters describing the test and the incentive structure (See Attachment A4 – Advance Letter and
2015 production Survey Infobook (no modifications from production)
Electronic, web-based diary (See Attachment A2 –Screenshots Electronic Diary.)
Paper diary (no modifications from production)
Respondent User guide for the web-based diary (Attachment A8 – Respondent User Guide.)
Four video tutorials for the web-based diary (Attachment A9 – Video Tutorials Scripts.)
An email template to respondents keeping a web-based diary (See Attachment A7 – Email Template.)
Records checklist (See Attachment A6 – Records Checklist.)
Debit card sleeves with instructions to the respondent with incentives (See Attachment A5 - Debit Sleeves and Respondent Instructions.)
The field test is planned to begin data collection in July 2015. Data will be collected for a period of three months.
The analysis of the data will include overall response rates, quantity and quality of data collected, demographic subgroup comparisons, effectiveness and cost of incentives, and operational issues. (See Attachment A11- Analysis Plan.)
BLS estimates that this study will require 1,313 burden hours. We base this estimate upon the following assumptions:
|
Mins. |
Sample |
Total Hours |
Estimated time to complete Visit 1 interview (incl. diary placement and records training) |
45 |
450 |
338 |
Estimated time to complete diary (one week) per member |
406 |
9007 |
600 |
Estimated time to complete Visit 2 interview (incl. diary review and respondent debriefing) |
50 |
450 |
375 |
TOTAL |
|
|
1,313 |
1 Tourangeau, R., Kopp, B., and Tan, L. 2013. Report on the State of Knowledge on Measurement Error in the CE. http://www.bls.gov/cex/Tourangeau_CEMeasureError.pdf
2 Dillman, D. and House, C., Editors. 2012. Measuring what we spend: Towards a new Consumer Expenditure Survey. Panel on Redesigning the BLS Consumer Expenditure Surveys; Committee on National Statistics; Division of Behavioral and Social Sciences and Education; National Research Council.
3 Estimates based on results from the Web Diary Feasibility Test with expected improvements to the eligibility and response rates.
4 The additional incentive is $10 more than the original design suggested in order to be a multiple of $20 to allow easier ATM withdrawal.
5 The data were subset to look at respondents that completed were eligible to complete all sections, including the income sections that are only asked during the second and fifth interview. The data were further subset to identify respondents that used records 50 percent or more of the time as observed by the FR.
6 Using preliminary results of the Individual Diary Feasibility test (n=402 diarists), the average amount of time spent keeping a diary for two weeks was roughly 26 minutes. Because the POC diary will consist of only 1 week of diary keeping, this time will be reduced. However, because we’re offering a choice of mode and an incentive for keeping the diary, we expect respondents to be more motivated to fill out their diary and willing to take more time. Additionally, a portion of the participants will be keeping a paper diary, which will take longer. As a results, we’ve estimated an average of 40 minutes per person to keep the diary for a single week.
7 BLS estimates that the average number of eligible diarists (household members age 15 and older) in a household is 2 based on 2012 CE interview survey member data for the identified PSUs. This estimate includes the time for an average of two CU member to complete the diary for one week.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Erhard, Laura - BLS |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |