CS-08-095 - Supporting Statement

OMB SUPPORTING STATEMENT

STUDY TO MEASURE CUSTOMER SATISFACTION

IRS APPEALS TAX CUSTOMERS

May 2008

A. Introduction

Background/Overview

IRS has replaced the traditional measures of accomplishment with a balanced measurement system consisting of business results, customer satisfaction, and employee satisfaction. As an important customer interface the Appeals Division will need feedback from customers to continuously improve its operations. This initiative is part of the service-wide effort to establish a system of balanced organizational performance measures mandated by the IRS Restructuring and Reform Act of 1998. This is also a result of Executive Order 12862, which requires all government agencies to survey their customers and incorporate customer preferences in their process improvement efforts.

Objectives of Data Collection

The primary goals of the survey are to survey Appeals external customers on an ongoing basis regarding their expectations, track customer satisfaction progress over time nationwide, and identify operational improvements.

B. Methodology

Sample Design

Appeals will draw a census of closed case records in two waves (First Half FY2008 and Second Half FY2008) and provide a dataset to the survey contractor, Macro International Inc. (Calverton, MD). Macro will select a random sample of 750, stratified by category (e.g., CDP, OIC) and area. This yields a sampling error of +/-3.6% at a 95% confidence level + minimal design effect depending on the final stratification.

Data to be Collected

Appeals will draw a sample from approximately 110,000 closed cases (not including exclusions) from FY 2008 in 2 waves and send all cases to Macro by October 2008. Macro will select a random stratified sample of 750 cases. The numbers below are based on FY2007.

	Population	Sample	95% +/-
Total	107118	750	5.7%
ADR	213	70	9.8%

Category	Population	Sample	95% +/-
CDP	35202	120	10.2%
OICC	11289	100	10.5%
INNSP	3085	85	11.1%
POSTPEN	10180	95	12.0%
CIC	519	70	11.4%
IC	975	70	12.0%
EXM/TEG	37602	125	11.8%
OTHER	8249	85	11.2%
TOTAL	107101	750	5.7%

Area	Population	Sample	95% +/-
1	11452	101	14.0%
2	17758	97	13.3%
3	17838	128	15.1%
4	9762	103	17.5%
7	14532	96	16.3%
8	16587	104	12.9%
9	18565	121	14.7%
TOTAL	106464	750	5.7%

How Data is Collected and Used

Once the Final Survey Instrument has been approved, Macro’s designated survey programmer will transform the survey document into an instrument for Computer-Assisted Telephone Interviewing (CATI) administration. When the survey programming is complete, several rounds of testing are conducted by Macro and Appeals staff. Once the testing is complete and the interviewers have been trained, Macro will conduct a “slow start,” consisting of interviews with 10 potential respondents with intensive monitoring from project management staff, quality assurance staff, and supervisors. If changes are necessary based on the “slow start,” Macro will make modifications until the survey is correct; if no changes are necessary these survey responses will be maintained as valid responses.

Macro will summarize the quantitative ratings and produce up to two national reports showing customer satisfaction scores on all Appeals survey items. The vendor will include any relevant database variables in the analysis and will weight the survey responses as necessary to accurately reflect the entire customer base.

For the analysis reports, the vendor will use basic and advanced statistical techniques including:

• Survey counts and overall response rates

• The overall level of customer satisfaction with services provided

• The frequencies and mean for all survey items

• The difference in satisfaction ratings and attitudes across customer segments

• Analysis of the relationship between survey responses

• Analysis of the open-ended questions for improvement suggestions

Macro will hold the identities of respondents confidential. Macro will not provide the IRS with data or status updates that are linked to individual respondents. Upon completion of fielding, Macro will provide anonymous survey data to the IRS, without any individually identifying information such as name, address, or taxpayer identification number.

Dates Collection Begin and End

Assuming OMB approval, the first wave of the survey is scheduled to field by July 2008, with the second wave fielding to begin by October 15, 2008. Each fielding period is estimated to take 5-6 weeks.

Who is Conducting the Research and Where

Macro will be responsible for pulling the sample, administering the survey, and conducting data analysis. Specifically, Macro’s Burlington, VT office will be responsible for pulling the sample and administering the survey. The survey data will be provided in a secure manner to the staff at the Macro Calverton, MD location for analysis and reporting.

Cost of Study

Macro contract: $80,997.

There will be no stipends paid to participants.

Recruitment Efforts

Sample will be provided by the IRS.

Location-Region/City and Facility

Data extract: IRS

Sampling and data collection: Macro, Burlington, VT and Calverton, MD

Data analysis: Macro, Calverton, MD

Expected Response Rate

The expected response rate is 70-80%.

Methods to Maximize Response Rate

Standard procedures will be used in order to obtain the highest response rate possible for the telephone survey. These will include:

1) Incremental release of sample. The vendor will pull a small sample, begin administration, and then add sample batches.

2) Attempt protocol. The vendor will make up to 10 calls.

3) Call rotation. The CATI system automatically schedules calls to rotate among various times across weekdays and spacing calls at appropriate intervals.

4) Refusal conversion. The vendor will work with Appeals to determine appropriate levels of refusal conversion.

5) Flexibility. Respondents will have the option to call in to complete or continue a survey over multiple sittings. The vendor can also schedule appointments, so respondents can participate at a time convenient to them.

6) High quality interviewing. CATI instruments have clear instructions with conversational scripts. Interviewers are also highly trained and closely monitored.

Further, Macro will take two concrete steps to help understand the extent of any non-response bias.

A) Analyze population and respondent demographics to determine face validity

B) Analyze late responders as a proxy for non-responders

Testing and Survey Structure/Design

The Appeals Customer Satisfaction Survey is attached and has been tested previously. The survey asks respondents to evaluate various aspects of their experience and to provide an overall summary evaluation. The results should facilitate more effective management of Appeals by providing insight from the customer’s perspective about possible improvements and providing useful input for program evaluation.

The survey includes several rating questions evaluating service delivery as well as one demographic question. In addition, there is one open-end question to allow for suggestions for improvement. Survey scoring for this contract will be based on the Customer Satisfaction Survey Score response average to the key question—“Regardless of the outcome of your appeal, please rate your overall satisfaction with the Appeals process.” Questions will utilize a 5-point rating scale, with 1 being very dissatisfied and 5 being very satisfied.

Efforts Not to Duplicate Research

This is the only study conducted by the IRS on this specific population. The vendor will make reasonable efforts to ensure that customers are only surveyed once within a 12-month period.

Participants Criteria

The population frame consists of all individuals with closed cases during FY2008 using files provided by the IRS. All listed participants will be eligible except cases identified during the survey process as having an incorrect telephone number.

C. Privacy, Disclosure, Confidentiality, Security Issues

All participants will be subject to the provisions of the Taxpayer Bill of Rights II during this study and the vendor will ensure that all participants are treated fairly and appropriately.

The security of the data used in this project and the privacy of taxpayers will be carefully safeguarded at all times.  Security requirements are based on the Computer Security Act of 1987 and Office of Management and Budget Circular A-130, Appendices A & B.  Physical security measures include a locked, secure office.  Notes are stored in locked cabinets or shredded.  Data security at the C-2 level is accomplished via the Windows NT operating system.  Systems are password protected, users profiled for authorized use, and individual audit trails generated and reviewed periodically. 

The IRS will apply and meet fair information and record-keeping practices to ensure privacy protection of all taxpayers.  This includes criteria for disclosure—laid out in the Privacy Act of 1974, the Freedom of Information Act, and Section 6103 of the Internal Revenue Code—all of which provide for the protection of taxpayer information as well as its release to authorized recipients.

The survey will not contain tax return or taxpayer information. Survey participants will not be identified in any of the documents or files used for this project.  We will limit and control the amount of information we collect to those items that are necessary to accomplish the research questions.  We will carefully safeguard the security of data utilized as well as the privacy of the survey respondents.  We will apply the fair information and record-keeping practices to ensure protection of all survey respondents. The criterion for disclosure laid out in the Privacy Act, the Freedom of Information Act, and section 6103 of the Internal Revenue Code provides for the protection of information as well as its releases to authorized recipients.

PRA Statement and OMB Control Number

The OMB Control Number and required information will be provided on the survey itself.

D. Burden Hours

The Appeals Survey has been designed to minimize burden on the taxpayer. The time that a respondent takes to complete the survey has been carefully considered and only the most important areas are being surveyed. The average time of survey completion is expected to be 11 minutes. This is based on the questionnaire consisting of several screener questions, 18 satisfaction questions, and one open-ended question. The questions are generally one sentence in structure and on an elementary concept level.

The total estimated burden in hours is estimated to be:

750 surveys x 11 minutes = 137.5 hours

E. Statistical Contact

For questions regarding the study or questionnaire design or statistical methodology, contact:

Mr. Larry Luskin

Vice-President

Macro International Inc.

11785 Beltsville Drive, Suite 300

Calverton, MD 20705

Telephone: 301-572-0334

E-Mail: Larry.Luskin@macrointernational.com

F. Attachments

Final Appeals Survey

6