Residential Energy Consumption Survey (National Pilot)

Department of Energy Washington, DC 20585 August 26, 2015

Department of Energy Desk Officer

Office of Information and Regulatory Affairs

Office of Management and Budget

Washington, DC 20503

SUBJECT: Use of Generic Clearance for U.S. Energy Information Administration (EIA-882T(61), approved under OMB number 1905-0186), to conduct second pilot test of Web and mail modes of collection for form EIA-457A, “Residential Energy Consumption Survey.”

Dear Desk Officer:

The Residential Energy Consumption Survey (RECS) is a periodic series of surveys of households and their energy suppliers with the purpose of collecting and reporting energy characteristics, consumption, and expenditures data of homes in the United States. The data are widely used throughout the government, academia and the private sector to understand current and future energy demand. Results are released via data tables, public-use data files, and analysis articles.

Background: The Household Survey phase of EIA’s RECS program (Form 457A) has been conducted primarily using in-person interviews since its inception in the late 1970s. Since 1997, EIA and its contractors have used computer-assisted personal interviewing (CAPI) as the means to collect household and structural characteristics data related to energy use in the residential sector. With the rising cost of in-person interviews and the time lapse between each RECS cycle, EIA seeks to test the feasibility, cost effectiveness, time efficiency, and response validity of conducting the RECS Household Survey in self-administered (Web and mail) modes. If proven successful, these new modes of collection will make data collection more sustainable, flexible, and cost effective for residential demand analyses. Using Web and mail modes of data collection may also allow EIA to extend program capacities to cover new topics, close data gaps, and develop estimates for smaller geographic areas.

In spring 2015, EIA conducted a small pilot test of Web and mail modes of collection in five U.S. cities (the “Cities Pilot”). EIA now proposes a nationwide test of these new modes. This pilot will allow for a direct comparison of results using the new methods against the official in-person interviews from the 2015 RECS benchmark. The 2015 RECS will be conducted concurrently with the proposed National Pilot test.

Proposed Methods: EIA will conduct a National Pilot test of the Household Survey using Web and mail data collection modes. This pilot test will incorporate key design elements and lessons learned from the Cities Pilot in order to compare these results against our benchmark RECS CAPI collection. 

• The Cities Pilot tested open-ended response options for appliance usage questions. Results showed that response quality was sufficient to support their use in the National Pilot. Response options for these items were also updated in the official 2015 RECS survey collected via CAPI.

• In one experiment embedded in the Cities Pilot, we studied the impact of survey length on response rates and bias. We randomized respondents to receive either a long or short self-administered questionnaire. The long form, at 30 minutes, included the full battery of RECS questions; the short form, at 20 minutes, was a reduced set of questions. The impact on response rates was negligible (a 2% difference), so we will field the full, long form only in the National Pilot test against the RECS CAPI. This permits the fullest evaluation between the in-person and web/mail approaches.

• To optimize web response, the Cities Pilot also tested an Internet propensity model that assigned the initial data collection mode to households based on their likelihood of having internet access. Although the propensity model was not sufficient in predicting mode choice, we will run additional mode-related experiments in the National Pilot to identify the best contact protocol for a self-administered RECS.

Thus, the National Pilot will include experiments that vary two design components: response incentives and contact protocols. Sampled housing units will be distributed equally across the treatment groups. Results from these experiments will inform future applications of a self-administered survey approach within a benchmark residential survey data program to achieve optimal validity, precision and accuracy of estimates of U.S. home energy use within occupied housing units.

Incentive treatments (Factor A):

A1. A $5 prepaid incentive will be included in the first questionnaire mailing. $10 will be promised for response under the response protocol specified by Factor B.

A2. A $5 prepaid incentive will be included in the first questionnaire mailing. $20 will be promised for response under the response protocol specified by Factor B.

Contact protocol treatments (Factor B):

B1. Web Only protocol - only the web response option is offered for all survey response invitations.

B2. Choice protocol – both paper and web questionnaires are offered for all survey response invitations and reminders

B3. Choice-Plus protocol – both paper and web questionnaires are offered for all survey response invitations and reminders. However, a $10 promised bonus incentive is offered in addition to the incentives specified by Factor A if the respondent chooses to respond by the web option rather than by paper.

B4. Web then Paper protocol (control) – the web response option is offered in the first invitation and first nonresponse invitation; both web and paper are offered in all subsequent reminders.

A sub-sample of nonrespondents will be drawn and selected households will be mailed a short paper questionnaire consisting of 22 items that can be used to evaluate response bias.

Results of the National Pilot should inform EIA on the feasibility and cost to produce key statistical measures of household energy use of quality that is comparable to the 2015 RECS benchmark collected via CAPI. EIA will evaluate response propensity and data quality by Web and mail respondents to determine optimal mode mix, contact strategies, and statistical methodologies for future RECS studies.

The National Pilot sample will be drawn from a housing unit frame constructed using the U.S. Postal Services’ Computerized Delivery Sequence (CDS) file. EIA estimates a total sample size of 9,650 housing units will yield 4,000 completed questionnaires. Average response time is expected to be 30 minutes per complete. EIA estimates 200 completes for the short, nonresponse follow-up paper questionnaire. Average response time for the short questionnaire is expected to be 5 minutes per complete. The total respondent burden to conduct the National Pilot is estimated to be 2,017 hours.

The questionnaire used for the National Pilot is based on the Cities Pilot long form, which includes both minor updates and some additional items to ensure the questionnaire is comparable to the RECS CAPI instrument. This includes the addition of energy assistance questions used in past rounds. Updates were also based on results from cognitive testing conducted in June 2015 and informed by lessons learned from the Cities Pilot collection.

The questionnaire burden estimate per response is based on the recorded Web response times for the Cities Pilot “long form” questionnaire and accounts for burden of the additional questions.

The estimated cost to the Federal government is $1,500,000. This includes the entire cost of modifying the Web and Paper instruments, sample design, data collection, respondent incentives, post collection statistical activities, analysis, documentation, and reporting.

The cost of burden hours to household respondents is estimated to be $145,163 (2017 burden hours times $71.97 per hour.) Other than the cost of burden hours, EIA estimates that there are no additional costs to respondents for generating, maintaining and providing the information.

EIA will provide OMB with the results of these tests in the annual summary of generic clearances. Attached are the draft survey questions to be used for the National Pilot test.

For questions on these methods, please contact Jacob Bournazian (202) 586-5562.

Sincerely,

Nanda Srinivasan, Director

Office of Survey Development and Statistical Integration

U.S. Energy Information Administration

3