Attachment A - Individual Diary Feasibility Test Overview

Individual Diaries Feasibility Test – Feasibility Test

1. Study Overview

The current Consumer Expenditure Diary Survey (CED) at the Bureau of Labor Statistics (BLS) uses a pencil-and-paper instrument (PAPI) to collect expenditure information from respondents. PAPI diary data collection has a number of inherent drawbacks, such as limiting entry to a single individual in a single location, and requiring that the main diary keeper carry the diary with them throughout the day. Additionally, having one person in the consumer unit (CU) maintain the diary for all members of the CU has drawbacks (Edgar et al., 2006).

In addition, the Consumer Expenditure Survey (CE) program would like to leverage data collection technologies and present CUs with more timely data entry options, such as a mobile-optimized web option for Smartphones or Tablets and a web diary. These modes of data collection have been recommended by several researchers (Westat, 2011; Mathiowetz et al., 2011). The technologies have the potential to improve unit and item response rates as well as measurement error. Additionally, a web component and a mobile-optimized web option component have potential cost savings over PAPI due to reduced or eliminated materials, scanning, and data entry expenses.

This study will test the feasibility and impact of using both individual diaries and multiple modes to collect CE diary expenditures. CU members in the research sample would be offered sequentially a mobile-optimized web option and then a web survey. These modes would be available for each member of the CU to complete individually.

The results of this study and the work done leading up to its fielding will be used to as inputs into a separate feasibility study, the Gemini Proof-of-Concept (POC) test, that is designed to field data collection efforts that mirror what is proposed in the Gemini Redesign Report.

This study will attempt to answer the following research questions:

1. What are the operational issues related to implementing a mobile-optimized web option for CED data collection versus web mode with an individual log-in?

   1. Debrief with field representatives (FRs) to determine:

1. What issues if any did respondents have with the materials provided?

2. What technical issues if any were raised by the respondents?

3. How should respondent materials be revised?

1. Debrief with respondents to determine:

   1. What did they like about the mode they were offered?

   2. What did they dislike about the mode they were offered?

   3. What extra information would have been helpful to them?

2. Debrief with the Regional Office (R.O.) and Demographic Survey Divisions (DSD) staff that handle respondent help requests to determine:

1. What kinds of questions did respondents ask?

2. How should respondent materials be revised?

4. Debrief with the National Processing Center (NPC) staff to determine:

   1. How would they change the data keying and coding processes to improve efficiency if the modes were put into Production?

2. How do high-level expenditure reporting rates and data quality differ by research group and control?

   1. Expenditures

   2. Item-missingness

3. How do acceptance and completion rates differ by research group and control?

   1. Week 1 acceptance rate

   2. Week 1 completion rates

   3. Week 2 completion rates

4. What are web respondents’ and mobile-optimized web option respondents’ specific data entry patterns?

   1. Number of log-ins per case

   2. Start/stop time stamps by diary day (to determine multiple times per day versus multiple days per week versus only data entry at the end of the week)

2. Study Design

1. Field period

The field period is scheduled for August 2014 through December 2014. July 2014 test group data will be “dummy” data comprised of August 2014 test group data to facilitate CE Production and Control processing.

2. Respondents

The Research sample will serve as the test group. The control group will come from the Production sample from August 2014 through December 2014.

3. Modes and diary changes

The control group will use the current CED methodology. The test group will use multiple modes: an in-person first visit with some CAPI data collection, a telephone reminder instead of visit 2, completion of individual diaries via web or mobile-optimized web, and an in-person third visit with some CAPI data collection.

Changes to the web diary and mobile-optimized web diary formats and content were based on one round of usability tests conducted by the Office of Survey Methods Research (OSMR) at the Bureau of Labor Statistics¹.

4. CAPI changes

The CAPI survey instrument will have to be modified to add three questions for both the Research and Production samples: internet access, mobile access, and how accessed. The first question will allow for matching between Research and Production samples since households/consumer units with internet access are different from those without internet access.²

There will also be questions added into CAPI to de-brief both the respondents and FRs. These will gauge respondents’ general comments, use of records, and burden as well as FRs’ assessments of the respondents and their general comments.

5. Methodology

The control group will consist of production sample from the study’s field period, and will be administered the current diary using standard procedures. The control group will be matched on socio-demographic characteristics to the test group after the field period concludes, as mentioned above.

Test Group

The test group will consist of research sample cases, and will be administered the web diary and/or mobile-optimized web diary using modified procedures. The contact schedule for the Test Group would be as follows:

1. Visit 1 - All respondents randomly assigned to the treatment group will be asked the internet access screening question and a follow-up questions on how the internet is accessed. The web diaries and/or mobile-optimized web diaries would be placed in person. If eligible respondents are not available at placement, the head-of-household will determine which instrument the unavailable respondent will receive. Respondents would be given materials describing the web diary and/or mobile-optimized web diary and instructions on how to log in and complete the diary. If the CU does not have any internet connection, then the CU is screened out. Any CAPI questions would be asked as currently scheduled:

   1. Front section: http://www.bls.gov/cex/ced/2013/csxfront.htm

   2. Household Characteristics: http://www.bls.gov/cex/ced/2013/csxsection1.htm

   3. CU characteristics: http://www.bls.gov/cex/ced/2013/csxsection2.htm

   4. Back section (not asked, but FR-completed): http://www.bls.gov/cex/ced/2013/csxback.htm

Non-English households in the Research group will be screened out as Type C-language barrier (“WD Language Barrier”). (Type Cs are ineligibles for the FRs response rates.) There will not be a Spanish web survey or mobile-optimized web survey, but this could be revisited if the web survey and/or mobile-optimized web survey are put into Production.

Spawned cases (multiple CUs found in one address) will also be coded out as Type Cs.

2. Visit 2 - The second-week visit will not occur; instead, CUs will be reminded about the diary via telephone. The FRs will not have access to the respondents’ web diaries or mobile-optimized web diaries. Therefore, the question for the FRs will be modified such that it will be directly asked of respondents: “Did you or anyone in your household record any expenses in the Web Diaries or Mobile-Optimized Web Dairies during the first week?”

FRs would not visit week 1 non-contacts. The following sections of CAPI questions would be asked as currently scheduled:

1. Coverage (not asked, but FR-completed): http://www.bls.gov/cex/ced/2013/csxcoverage.htm

2. Back section (not asked, but FR-completed): http://www.bls.gov/cex/ced/2013/csxback

3. Visit 3 - The CUs would be visited in-person at the end of the diary period to thank them for their participation. Any CAPI questions would be asked as currently scheduled:

   1. Work experience and income: http://www.bls.gov/cex/ced/2013/csxsection4a.htm

   2. Work experience and income for the CU: http://www.bls.gov/cex/ced/2013/csxsection4b.htm

   3. Coverage (not asked, but FR-completed): http://www.bls.gov/cex/ced/2013/csxcoverage.htm

   4. Back section (not asked, but FR-completed): http://www.bls.gov/cex/ced/2013/csxback.htm

FRs will also collect receipts if provided by respondents. FRs would enter any receipts into a modified CAPI instrument. (The CAPI instrument will be modified to include a new tab that mimics the web survey instrument. FRs won’t have access to respondents’ web or mobile-optimized web surveys. FRs will be trained on the new web survey procedures and follow the documented procedures.)

At the end of the last visit, respondents will be debriefed, as well as FRs (both using the CAPI instrument).

There will be no reinterviews for Research cases.

6. Sampling and sample size

The Production sample will serve as the control group (meaning not offered the web diary or mobile-optimized web diary options). The research sample will be randomly selected from a starting sample of 1,200 addresses.

Table 1: Estimated sample sizes.

Starting sample addresses	1,200
Occupied housing units (occupancy rate = 80%)	960
Eligible CUs (screening rate = 68%)	653
Completed diaries for week 1 (CED response rate = 70%)	457
Completed diaries for week 2 (CED attrition rate = 0%)	457
Total completed diaries (weeks 1 and 2)	914

7. Available Materials

The following materials are available to OMB for consideration of this project:

1. OSMR’s Round 2 Usability Report

2. Analysis Plan

3. CAPI Change Document – Respondent Debriefing Questions p. 5

4. Respondent User Guides

5. IDFT Change Request Spreadsheet

2. Study Modifications

The following modification were made to the feasibility study due to recommendations coming from the second round of Usability Testing and the Individual Diaries Pilot Test.

1. Simplify accessing and logging into the instrument

In order to simplify the logging process while remaining within Census Security Requirements, a hyphen was added to the username breaking up the character sequence and making the username easier to recall, modified training to recommend that the FRs offer assistance in logging in the first time to the respondents, a QR code was added to the respondent user guides, and creating a trigger for a dialog box to open letting the respondent know that they successfully changed their password. Due to time and budget restraints the simplified URL was not implemented.

2. Consolidate and create additional instructional materials

To reduce the amount of materials and the perceived burden that is associated, the FR Talking Points, Respondent User guides, and shortcut handouts have been consolidated into a single document for the respondent. In addition, instructional videos have been created and will be uploaded to YouTube, instructions to create a shortcut for the web diary have been added to the user guides, a helpdesk email is being setup, additional visuals have been added to user guides and a test site for FRs to train on is being implemented.

3. In-instrument ease of navigation

Ease of navigation within the instrument has been increased by ensuring correct instrument scaling regardless of Smartphone, bringing up the appropriate mobile keyboard for ease of entry, changing the double gear icon to the more ubiquitous single gear icon, by developing a protocol to acknowledge voice-to-text capabilities, and updating the user guides to include explicit directions on how to correctly enter Food Away from Home expenditures.

IV. Burden Hour Estimate

BLS estimates that this feasibility study will require a total of 2,135.29 burden hours. This is approximately a 13 percent (304.66³ burden hours) reduction in burden hours when compared to CED production estimates for a similar sample size. We base this estimate upon the following assumptions:

	Mins.	Sample	Total Hours
Estimated time to place diary	24	960	384
Estimated time to be screened for Research sample	2	960	32
Estimated time to complete electronic diary (first week)	854	457	647.42
Estimated time to complete electronic diary (second week)	85	457	647.42
Estimated time at diary pick-up	24	653	261.20
Additional time for IDFT research questions during pick-up	15	653	163.25
TOTAL			2,135.29

5. References

1. Edgar, J., Davis, J., Spell, S., Verlander, L., and Wetzel, G. (2006). Individual Diary Feasibility Test. BLS, Division of Consumer Expenditure Survey internal report.

2. Westat. (2011). Data Capture Technologies and Financial Software for Collecting Consumer Expenditure Data. BLS, Division of Consumer Expenditure Survey internal report.

3. Mathiowetz, N., Olson, K., and Kennedy, C. (2011). Redesign Options for the Consumer Expenditure Survey. Prepared for National Academies of Sciences. BLS, Division of Consumer Expenditure Survey internal report.

1 Usability testing by OSMR was covered by their generic clearance and they submitted their own clearance packages.

2 Consumer units will also be matched by other variables, such as geography and income.

3 CED production estimates are based on time estimates stated in Censes CED Advance Letter to respondents

4 BLS estimates that the web diary and/or mobile-optimized web diary will take 85 minutes to complete per week per consumer unit, based on results from the Individual Diaries Pilot Test where, on average, it took each respondent 40 minutes per week at 2.13 respondents per consumer unit.

9