Memo on violations_04302014

TO: Jennifer Park,

Statistical and Science Policy,

Office of Information and Regulatory Affairs,

Office of Management and Budget

FROM: Timothy Wojan DATE: 04/30/2014

Economic Research Service

SUBJECT: Reporting two oversights that occurred during ERS Rural Establishment Innovation Survey (REIS) (OMB Control No. 0536-0071) pilot study data fielding.

This memo describes two changes in the ERS Rural Establishment Innovation Survey (REIS) (OMB Control No. 0536-0071) pilot study data fielding that did not conform to OMB prior approval and steps to be taken by ERS to insure errors of this type do not happen in the future.

Changes to the REIS Pilot Study

The first change affected the composition of the sample. The sample for the pilot study proposed in the Supporting Statement was composed of 3 different groups: 1) roughly 2600 respondents from the 1996 Rural Manufacturing Survey (RMS), 2) roughly 90% of the 1400 needed to meet 4,000 cases from the BLS Business Registry, and 3) roughly 10% of the 1400 from a proprietary vendor to cover the states not granting access to the BLS Business Registry. Since the 140 cases from the proprietary vendor would not be able to provide reliable information on differences in response rates across strata the decision was made to source all 1400 cases from the BLS Registry. This change should have been submitted to OMB for approval but this was not as an oversight.

The second change affected the total number of cases released that exceeded the 4,000 cases originally approved by OMB. Since we were uncertain as to how many of the respondents from the 1996 RMS would still be in business we decided to draw roughly 2,800 cases from the BLS Business Registry, include 1,400 in the initial pilot sample and then release replicates from the remaining 1400 cases as needed to meet the 4,000 case specification. The cause for the error here—the release of the full 2,800 cases—was due to the significant time lag between when the sample was drawn and when the sample was prepared for release. The sample was drawn in late May. OMB clearance was granted in July but pre-screening did not begin until October due to problems getting timely Office of Personnel Management clearance for interviewers. When it came time to prepare the sample for release the original plan to release 1,400 initially was a faint memory. There is no paper trail of how the release of 2,800 was arrived at but its availability and the instinctual response that more sample is preferred to less sample undoubtedly contributed to this error. The source of the error, though not a justification, was the hybrid sample. If the sample had been drawn solely from the Business Registry a draw of 4,000 cases would have been made.

The implications of this larger than intended sample for the pilot study are relatively benign given the central concern of respondent burden. Because observed mode share skewed heavily toward more time efficient data collection (web and mail) than anticipated in the Supporting Statement, total respondent burden for the pilot of 5,208 cases was roughly 1,200 hours (Table 1). Part of this reduction was also due to a higher nonresponse rate and a much lower time burden for nonrespondents than assumed in the original burden estimate. This is substantially less than the 1,808 hours for 4,000 cases estimated in the original Supporting Statement where phone was assumed to be the predominant survey mode and where repeated phone contact was assumed to be a productive strategy for increasing response rates (Table 2).

Preventive Steps to Be Taken by ERS

Steps to be taken to insure errors of this type do not happen in the future include:

1. Project managers involved in ICRs will receive an overview of PRA requirements at the initiation of their project.

2. Encourage special training about OMB PRA clearance and survey management for future ERS survey project managers (e.g. via in-house special PRA training or related JPSM short courses).

3. Prior to the release of sample for either a pilot or main study the Supporting Statement will be reviewed to ensure that all of the elements related to burden, sample size, design and purpose approved by OMB are adhered to. And project managers will revisit these elements with their field staff periodically during data fielding.

4. Weekly update reports from survey research cooperators or contractors administering the survey will be produced that provide a breakdown of cases to be worked, cases worked, and cases completed.

5. Maintaining regular conversation/communication with field staff.

6. Communicating closely and regularly with PRA clearance officer throughout the project.

 

Table 1. Estimated respondents’ burden hours incurred during fielding based on results from the REIS pilot study

PILOT STUDY 30 MINUTE INTERVIEW
		Number of Respondents	Responses Annually per Respondent	Total Annual Reponses	Est Ave Number of Hrs per Resp*	Est Total Annual Hrs Resp Burden	Number of Completes
Business Phone Screener initial sample		5,208*
Completed screener interviews	60.0%	3,125	1	3,125	0.07	219	3,125
Attempted interviews (number not completing)	40.0%	2,083	1	2,083	0.04	83.328	2,083
Main Survey:
Phase 1: Business Phone Interview to All respondents
Completed interviews	17.0%	531	1	531	0.37	197	531
Mail Short Form for Tel. Refusals
Completed 1 pg questionnaire	4.0%	125	1	125	0.1	12.4992	125
Phase 2: Business 1st Follow-up Mail Survey to nonrespondents
Completed questionnaires	12.0%	375	1	375	0.37	139	375
Phase 3: Business 2nd Follow-up Mail Survey to nonrespondents
Completed interviews	12.3%	384	1	384	0.37	142	384
Phase 4: Business Web Questionnaire
Completed questionnaires	2.0%	63	1	63	0.5	31.5	63
Phase 1 to 4
Attempted Interviews/contacts	52.8%	1,650	1	1,650	0.23	380
Total responding burden						1,203**	1,478

Note:

*Including 1,208 samples that exceeded the total number of respondents approved for the REIS pilot study by OMB. Total number of burden hours associated with this group of respondents is 279 hours (=1,203 *1,208/5,208).

**Because observed mode share skewed heavily toward more time efficient data collection (web and mail) than anticipated in the Supporting Statement, total respondent burden for the pilot of 5,208 cases was roughly 1,200 hours.  This is substantially less than the 1,808 hours for 4,000 cases estimated in the original Supporting Statement where phone was assumed to be the predominant survey mode.   

Table 2. Anticipated respondents’ burden hours submitted in A.12, Supporting Statement for REIS ICR (and approved by the OMB)

PILOT STUDY 30 MINUTE INTERVIEW
		Estimated Number of Respondents	Responses Annually per Respondent	Total Annual Reponses	Est Ave Number of Hrs per Resp*	Est Total Annual Hrs Resp Burden	Anticipated Completes
Business Phone Screener initial sample		4,000
Completed screener interviews	80%	3,200	1	3,200	0.07	224	3,200
Attempted interviews (number not completing)	20%	800	1	800	0.04	32
Main Survey:
Phase 1: Business Phone Interview to All respondents						0
Completed interviews	47%	1,504	1	1,504	0.5	752	1,504
Mail Short Form for Tel. Refusals
Completed 1 pg questionnaire	2%	64	1	64	0.1	6.4	64
Phase 2: Business 1st Follow-up Mail Survey to nonrespondents
Completed questionnaires	9%	288	1	288	0.5	144	288
Phase 3: Business 2nd Follow-up Mail Survey to nonrespondents
Completed interviews	6%	192	1	192	0.5	96	192
Phase 4: Business Web Questionnaire
Completed questionnaires	7%	224	1	224	0.5	112	224
Phase 1 to 4
Attempted Interviews/contacts	29%	928	1	928	0.476	442
Total responding burden						1,808	2,272

Note: * Estimates are rounded.