WAP ARRA Evaluation - Supporting Statement Part A Revised2

Supporting Statement

Weatherization Assistance Program ARRA-Period Evaluation

OMB Control Number: 1910-NEW

A. Justification

1. Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the information collection.

The American Recovery and Reinvestment Act of 2009 (ARRA or Recovery Act) provided $5 billion to the Weatherization Assistance Program, and the Department of Energy required grantees to expend the funds within three years. At approximately $1.7 billion per year, this DOE funding is approximately eight times larger than historic federal appropriations levels. The Recovery Act also changed some key program guidelines, raising the average expenditure per home from $2,500 to $6,500 and raising the poverty guidelines from families at 150% of poverty to a limit of 200% of poverty. In light of these historic changes, this evaluation is necessary to determine the effectiveness of WAP during the ARRA-Period, also known as Program Years 2009, 2010, and 2011.

The collection of this information will verify the cost-effective performance of authorized tasks by the Department of Energy’s Weatherization Assistance Program, highlight best practices in reaching program objectives, and identify ways in which program performance can be improved. It will build upon the WAP Retrospective Evaluation, which covers Program Years 2007 and 2008 and was previously approved by the Office of Management and Budget (OMB) with OMB Control Number 1910-5151.

As previously described when requesting approval for the WAP Retrospective Evaluation, OMB found in its Program Assessment Rating Tool (PART) rating for WAP that:

“The program lacks an assessment of performance that is current, comprehensive, and independent. The program reports a favorable benefit-cost ratio through internal assessments. However, these assessments rely in part on old data…”

The ARRA-Period and Retrospective evaluations together will allow a current, comprehensive, and independent evaluation of WAP. This evaluation will also include a Randomized Control Trial (RCT) study for a portion of weatherization service recipients to lay the groundwork for using academic best practices. In addition, studies will determine effectiveness of the Weatherization Innovation Pilot Program, Sustainable Energy Resources for Consumers grants, bulk fuels energy conservation, indoor air quality, and non-energy benefits.

Authority to conduct survey and evaluation activity can be found in 10 CFR 440, Section 440.23(a) mandating that the Secretary of Energy shall “…monitor and evaluate the operation of projects carried out by CAA’s [community action agencies] receiving financial assistance under this part through on-site inspections, or through other means, in order to ensure the effective provision of weatherization assistance for the dwelling units of low-income persons.”

2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

The information that will be gathered and analyzed through this collection will serve multiple audiences. It will be used to inform Congress, the Department, and the Administration of the program performance during the Recovery Act and the effectiveness of the $5 billion investment into WAP. Statistics will be used to update and improve PART and Government Performance and Results Act (GPRA) assessments. Results of the study will also be used to identify strengths and weaknesses of program performance in order to target ways in which this can be improved at federal, state, and local implementation levels.

As an example, findings from the previous evaluation (in 1993, covering program years 1989-1990) that program performance in hot-climate regions was not on par with the national average created the impetus for a hot-climate initiative, which has targeted that region with additional training, technical assistance, and unique weatherization techniques in order to improve performance. It is anticipated that findings from the current evaluation will result in similar adjustments and improvements to the Weatherization Assistance Program.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses.

It is expected that over 90% of the data collection effort for this study will involve some form of electronic exchange, either through the use of online survey instruments or the transfer of utility billing records or other relevant household data via electronic mail and attached files.

Requesting utility bills and other relevant household data in electronic format will ease the reporting burden of the utilities and local agencies providing the information and will yield information in a form that is easier for the evaluation contractor to process. This approach is expected to reduce the time required by all parties.

4. Describe efforts to identify duplication.

There is no similar information available on a national basis. There is no low-income energy efficiency program comparable in scale or scope to the Weatherization Assistance Program and only a comprehensive national evaluation can now provide an accurate picture of this program’s operation and performance. This ARRA Period evaluation is necessary in addition to the Retrospective evaluation because of the large increase in funding and rule changes described in response to question 1 above.

5. If the collection of information impacts small businesses or other small entities, describe any methods used to minimize burden.

The information collection is not expected to have a significant economic impact on small businesses or other small entities.

6. Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

If the data collection is not allowed to proceed, the details of program performance with $5 billion of Recovery Act funds will not be known. Most important, managers at the federal, state, and local levels will be deprived of essential information regarding program operations and areas in need of improvement. This information ranges from appropriate client education techniques, best audit methods, and crew training to energy efficiency measures selection and appropriate health and safety techniques for clients and crews. In short, the information is vital to insure that the program delivers effective services to low-income households with hundreds of millions of federal dollars per annum after such a historic change in the program which was originated by the Recovery Act.

7. Explain any special circumstances that require the collection to be conducted in a manner inconsistent with OMB guidelines. (a) requiring respondents to report information to the agency more often than quarterly; (b) requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it; (c) requiring respondents to submit more than an original and two copies of any document; (d) requiring respondents to retain records, other than health, medical government contract, grant-in-aid, or tax records, for more than three years; (e) in connection with a statistical survey, that is not designed to product valid and reliable results that can be generalized to the universe of study; (f) requiring the use of statistical data classification that has not been reviewed and approved by OMB; (g) that includes a pledge of confidentially that is not supported by authority established in stature of regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; (h) requiring respondents to submit proprietary trade secrets, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.

This information collection will not involve any of the special circumstances itemized in the Specific Instructions for this Supporting Statement.

8. If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency’s notice, required by 5CFR 320.8(d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken in response to the comments. Specifically address comments received on cost and hour burden. Describe efforts to consult with persons outside DOE to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or report.

A 60-Day notice was published on March 11, 2011, (Vol. 76, No. 48, p 13398) and was the first public notice on this topic. No comments from the public were received. A 30-Day notice was published in June 16, 2011 (Vol. 76, No. 116, p 35199) and was the second public notice on this topic. No comments from the public were received.

9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

No payments will be made to state and local agency respondents for information collection instruments used in this study. Similarly, no payments will be made to those local agencies and utility companies who will be asked for information related to billing records, metering data, and subject identification. Incentives will be provided to households and weatherization staff to participate in this evaluation. A one-time incentive of $20 will be provided to households who agree to complete all or portions of the Occupant Survey as part of these studies: Weatherization Innovation Pilot Program; Sustainable Energy Resources for Consumers Program; Deferral Study; Under-performers Study; and Territories Study. A one-time incentive of $20 will be paid to each weatherization staff who agrees to complete the Weatherization Staff Survey as part of these studies: Weatherization Staff Study; and Weatherization Innovation Pilot Program. A one-time incentive of $50 will be paid to households who allow the inspections of the work done in their homes as part of these studies: Weatherization Innovation Pilot Program; Sustainable Energy Resources for Consumers Program; Persistence Study; and Under-performers Study. The incentive will be delivered in the form of a personal check made out to the most appropriate household member and weatherization staff.

10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.

The information provided by respondents to the surveys and household data requests will be reported only in the aggregate and a subject’s name, agency, or other identifying information will not be reported in association with the individual answers. The evaluation contractor selected to perform this study will be required to take appropriate steps to protect the privacy of those agencies and individuals from whom data are collected. Personally identifying information will not be delivered by the evaluation contractor to Oak Ridge National Laboratory or DOE. A key component of this is identifying and addressing potential vulnerabilities in the processes by which electronic data are collected and stored. Oak Ridge National Laboratory has prepared a Privacy Needs Assessment and a Privacy Impact Analysis at the request of the U.S. Department of Energy’s Oak Ridge Operations Office. The PNA/PIA determined that no PII is being collected by this project and that procedures in place to protect the collected data are reasonable.

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why DOE considers the questions necessary, the specific uses to be made of the information., the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

No questions of a sensitive nature are included in any of the surveys used in the Weatherization Assistance Program ARRA-Period Evaluation.

12. Provide estimates of the hour burden of the collection of information. The statement should indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. Unless directed to do so, DOE should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample fewer than 10 potential respondents is desirable.

An attachment shows the burden estimate for each element of the data collection effort as well as the total burden. Eight surveys will be administered to representatives of various populations (states, local agencies, utilities, weatherization workers, and occupants). In addition, a series of five data requests will be made to local agencies and utility companies related to the collection of billing records, and the identification of study subjects. The above-mentioned survey instruments and household data request forms are attached to this submission for OMB review. Data will be collected from 58 WAP Grantees, 16 Weatherization Innovation grantees, 34 Weatherization Training Centers, 1,000 local weatherization agencies, 1,200 utility companies, 1,240 occupants, and 3,056 weatherization staff, for a total of 6604 unique respondents. This information collection request does not entail any surveys of any households that would take place during the Census embargo of such activities.

In May 2011, it was estimated that the hour burden for all respondents combined will total 83,527 hours over the entire study period. Since that time, several decisions have been made to significantly reduce this burden to the public. These decisions are as follows:

• The WAP national evaluation team has decided to consolidate its utility data requests across its retrospective and WAP ARRA period evaluations so that only two utility data requests are being made instead of four. Data for most PY2009 is being requested along with data for PY2007 and PY2008 as part of the retrospective evaluation, which is currently on-going (OMB Control Number 1910-5151). Data for the remainder of PY2009 and PY2010 will entail the second request. The national evaluation will not attempt to estimate energy savings for PY2011, which would have entailed an additional utility data request. This decision reduced the estimated burden hours by 28385 hours.

• Information on measures installed in a national sample of weatherized homes will be requested from the subgrantees as part of the WAP ARRA period evaluation. This is done through our data forms DF2 and DF3. These are lengthy forms. For agencies that are asked for information on more than 75 units, a shortened version for DF2 and DF3 is used for all the additional units. Both the long-form and short-form versions will be submitted as part of this ICR. Because this method had already been tested during the retrospective evaluation, reductions in burden were already incorporated into the original ARRA period burden estimate. It is estimated that this decision reduces burden by 3944 hours.

• The evaluation team will extract from PAGE grantee reported information on expenditures and units weatherized to pre-populate the two main program information surveys to be sent to the grantees and subgrantees, S1 - State (grantee) Program Information Survey, and S2 - Agency (subgrantee) Program Information Survey. The impacts of this decision had already been incorporated into the original ARRA period burden. It is estimated that this decision reduces respondent burden by 17348 hours.

• The WAP national evaluation team is contacting all states/grantees to ascertain whether they have state-level weatherization databases. As of Nov. 3, 2011, the team has confirmed that at least 26 states have such databases. The team has already received permission to use about a dozen of the databases. The databases will allow the team in many cases to eliminate a request to subgrantees to provide lists of homes weatherized during the ARRA period (see DF4a); our current estimate is that 75% of subgrantees will not be burdened with this request. This decision reduces estimated burden by 3000 hours.

• Additionally, the databases will provide some measure installation information, which will be used to pre-populate DF2 and DF3 for some subgrantees. Project team efforts to utilized state weatherization data bases are estimated to reduce respondent burden by 819 hours.

In summary, given these changes, the burden on the public was reduced to 51,323 hours, a reduction of 32,204 hours (38%) from the original estimate.

Two surveys have been dropped from this evaluationand a new data form has been added. Because of these changes, the estimated burden is now 50,539 hours, broken into the various surveys and data forms as shown in the table below:

Survey Name	# respondents	# responses	# burden hours per respondent	# burden hours
S1 All States Program Information Survey	58	1	24	1392
S2 All Agencies Program Information Survey	1000	1	24	24000
S3 Subset of Agencies Detailed Program Information Survey	450	1	8	3600
S4 FULL Occupant Survey	474	1	1.75	829.5
S4 PARTIAL Occupant Survey	766	1	0.5	383
S5 FOLLOW-UP Weatherization Staff Survey	1813	1	0.5	906.5
S5 FULL Weatherization Staff Survey	243	1	1	243
S8 Weatherization Training Centers Survey	34	1	1	34
DF2/3 FULL ONLY Housing Unit/Building Information Data Form	403	1	18	7254
DF 2/3 FULL + SHORT Housing Unit/Building Information Data Form	66	1	32	2112
DF4 Electric/NG Billing History Information from Agencies Data Form	469	1	10	4690
DF5 Electric/NG Bills from Utilities Data Form	1000	1	5	5000
DF9 Contact Information from Agencies for Occupant Survey Data Form	66	1	2	20
DF11 Weatherization Staff Survey Data Form	2	1	1.1	7206
DF12 Deferral Data Form	10	1	1	2

The data collection effort will span three years, from Summer 2011 to Summer 2014 but each of the individual components shown in the attachment will require less than the entire two year period. Hour burden estimates were prepared separately for each individual data collection effort based on the nature of the information requested, the instrument used, experience with similar data collection efforts in the previous National Weatherization Assistance Program Evaluations, and feedback provided by survey pre-testers.

The estimate of burden hours of the information collection is as follows:

Total number of unduplicated respondents: 5404

Reports filed per person: 1.46

Total annual responses: 7912

Total annual burden hours: 50,539 hours

Average Burden Per Collection: 3,610 hours

Per Applicant: 9.35 hours

The estimated cost for collecting the required information is $781,164. As noted above, burden will be placed on states, local agencies, utilities, households, and weatherization staff. Office workers employed by the states, local agencies, and utilities will mostly likely be assigned to meeting the information collection requests. In most cases, no particular professional skills will be needed to complete the surveys and it is expected that most of the work will be done by employees who fall into the secretaries and administrative staff occupational category. The most recent mean hourly wage rate for this occupational category published by the Bureau of Labor Statistics in May 2010 is $15.38/hour (http://www.bls.gov/oes/current/oes436014.htm). 43,177 of the burden hours fall into this category. The state and local agency costs are $664,062. The total utility costs (which include but privately-owned and municipally-owned utilities are $76,900.

A burden of 1212 hours will be placed on households. A reasonable estimate for the hourly wage rate of a low-income person is $10.50/hour.¹ Thus, the total cost to households is $12,726. This cost is off-set by $24,800 in incentives to participate in the surveys. The average wage for a weatherization worker is $14/hour (http://www.ehow.com/info_7763558_average-weatherization-workers-salary.html). A burden of 1149 hours will be placed on this group of respondents. The total cost for weatherization staff to respond to the surveys is $16,086. This cost is off-set by $41, 120 in incentives to participate in the surveys.

There will be no start-up or capital costs for the respondents.

13. Provide an estimate for the total annual cost burden to respondents or recordkeepers resulting from the collection of information.

No costs are anticipated for (a) capital and start-up costs or (b) an operation and maintenance and purchase of services component.

14. Provide estimates of annualized cost to the Federal government.

The additional cost for this data collection is $9.3 million for an independent subcontractor to Oak Ridge National Laboratory to collect the data. The average annual cost for this three-year project is $3.1 million. Costs by survey and study are as follows:

SURVEY/STUDY	COST
S1 All States Program Information Survey	150,000
S2 All Agencies Program Information Survey	200,000
S3 Subset of Agencies Detailed Program Information Survey	200,000
S4 Occupant Survey	300,000
S5 Weatherization Staff Survey	250,000
S8 Weatherization Training Centers Survey	30,000
Program Characterization	200,000
DF 2/3 Housing Unit/Building Information Data Form	400,000
DF4 Electric/NG Billing History Information from Agencies Data Form	200,000
DF5 Electric/NG Bills from Utilities Data Form	500,000
DF9 Contact Information from Agencies for Occupant Survey Data Form	20,000
DF11 Weatherization Staff Survey Data Form	20,000
DF12 Deferral Data Form	20,000
Utility Bills Analysis	1,000,000
Persistence Study	400,000
Cost Effectiveness Study	600,000
Non-Energy Benefits Study	330,000
Social Network Study	300,000
Deferral Study	250,000
Under-Performers Study	405,000
Territories Case Studies	175,000
Indoor Air Quality Study	100,000
Weatherization Innovation Pilot Project Study	1,000,000
Sustainable Energy Resources for Consumers Study	750,000
Synthesis and Special Reports	1,000,000
Other	500,000
Total	9,300,000

15. Explain the reasons for any program changes or adjustments reported in Items 13 (or 14) of OMB Form 83-I.

This request would cover evaluation of WAP for Program Years 2009, 2010, and 2011, including the $5 billion funding for WAP from the Recovery Act.

16. For collections whose results will be published, outline the plans for tabulation and publication.

Once the survey information and instrumented/metered data have been collected and analyzed, a series of 12 reports will be written to present the survey information and analysis results. These reports will be published as follows (the dates of the reports are still to-be-determined but range from 2012 to 2014):

• Program characterization–A report describing the characteristics of the Program and how it is implemented.

• Program energy savings–A report documenting the billing data energy analyses and the energy savings achieved in the housing units weatherized by the Program in PY’s 2009, 2010, and 2011.

• Non-energy impacts–A report documenting the non-energy impacts that can be attributable to the Program.

• Impact assessment–A report that draws together the Program characterization, energy savings, and non-energy impact studies along with additional analysis on Program cost effectiveness, explanatory factors, and attribution of energy savings to the Program.

• Process assessment–A report that documents findings on how well the Program is being implemented, with a special focus on audits, client education, training, and monitoring.

• Synthesis–A report that draws together all the findings presented in the previous reports and provides recommendations for Program improvement will be published in December 2012.

• Weatherization Innovation Pilot Program–A report with the evaluation results of the $30 million Innovation program with16 grantees and their effectiveness.

• SERC–A report on the 27 grantees, approximately 100 local agencies, and the 45 different alternative technologies being implemented with $90 million of SERC funding.

• Under-performing and Multi-Family Buildings Study–This report will document research into identifying why some homes and large multi-family buildings save substantially less energy than homes and multi-family buildings of similar size that received similar weatherization measures.

• Territories–A report of at least one territory that received funding through the Recovery Act and is establishing a robust weatherization program for the first time ever.

• Persistence of Energy Savings study–A report on how well energy savings persist over time, including factors such as measure degradation, replacement, or retention and changes in occupancy or condition (such as an addition to the home) of the home

• Deferral study–This report will present results of a study that tracks what happens to homes after weatherization auditors determine that the homes cannot be weatherized due to various conditions in the homes.

17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons why display would be inappropriate.

Not seeking approval not to display the expiration date for OMB. Expiration date display is not inappropriate.

18. Explain each exception to the certification statement identified in Item 19 of OMB Form 83-I.

No exceptions are being requested to the certification requirements.

1 ASPE Research Brief, February 2009.