VOV_Supporting Statement A_6 25 12

Supporting Statement A

For

Veterans Benefits Administration (VBA) Voice of the Veteran (VOV) Surveys

A. JUSTIFICATION

1. Explain the circumstances that make the collection of information necessary. Identify legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

In 2008, VBA recognized a need to develop and design an integrated, comprehensive Voice of the Veteran (VOV) measurement program for its lines of business: Compensation Service (CS), Pension Service (PS) Service, Education (EDU) Service, Loan Guaranty (LGY) Service and Vocational Rehabilitation and Employment (VR&E) Service. This continuous measurement program will help VBA understand what is important to Veterans relative to VBA benefits and services and will provide VA/VBA leadership with actionable and timely Veteran feedback on how VBA is performing against those metrics. Insights will help identify opportunities for improvement and measure the impact of improvement initiatives.  

Based on interviews conducted, J.D. Power and Associates (JDPA) has separated the Veterans experience with the VBA into two categories – Enrollment in a Benefit and Servicing of a Benefit. Stakeholders expressed the need to understand the different elements that make up the overall experience with VBA for the Veteran. The following outlines how that will be approached with each of the lines of business.

J.D. Power will be testing three survey instruments for CS & PS lines of business. Based on the numerous interviews conducted, there will be one survey instrument for the Enrollment category that will be used for both compensation and pension claimants; while compensation beneficiaries; and pension beneficiaries will receive separate Servicing instruments. The survey pool for the benchmark study CS & PS Enrollment questionnaire will include individuals who have received a decision on a compensation or pension benefit claim within 30 days. Specifically, the sample will be stratified as follows: compensation enrollment will include individuals who have received a decision in the past 30 days (includes those who were found eligible on a new or subsequent claim and those who have been denied and are not appealing the decision) and pension enrollment will include individuals who have received a decision in the past 30 days (includes those who were found eligible on a new or subsequent claim and those who have been denied and are not appealing the decision).

The survey pool for the benchmark study for the Compensation Servicing questionnaire will include individuals who received a decision or were receiving benefit payments between 6-18 months ago. The survey pool for the benchmark study for the Pension Servicing questionnaire will include individuals who have currently been receiving benefits for at least 6 months or individuals who received a decision 12 months ago.

J.D. Power will be testing two survey instruments for Education Service. The survey pool for the benchmark study for the Education Enrollment questionnaire will include individuals who have received a decision on their education benefit application within 90 days (i.e., the original end-product has been cleared within the past 90 days) prior to the fielding period. The sample will be stratified as follows: (1) accepted and enrolled, (2) accepted and not enrolled. The survey pool for the benchmark study Education Servicing questionnaire will include beneficiaries who have been enrolled and receiving benefit payments for at least 2 consecutive school terms.

J.D. Power will be testing two survey instruments for Loan Guaranty Service. The survey pool for the benchmark study for the LGY Enrollment questionnaire will include individuals who closed on a VA home loan in the 90 days. The sample will be stratified as follows: (1) those who closed on purchase loans, (2) those who received loans for interest rate reductions, and (3) those who obtained cash out or other refinancing. The survey pool for the benchmark study for the SAH Servicing questionnaire will include individuals who are eligible for a specially adapted housing grant and in the past 12 months have: (1) received an approval on their grant and are currently somewhere in post-approval, (2 have had all their funds dispersed and final accounting is not yet complete, and (3) have had all of their funds dispersed and final accounting is complete.

J.D. Power will be testing three survey instruments for VR&E Service. The survey pool for the benchmark study for the VR&E Enrollment questionnaire will include individuals who had an initial meeting with their VR&E counselor and were granted a decision regarding their entitlement in the past 60 days prior to the fielding period. The sample will be stratified as follows: (1) those who apply/do not show up for initial appointment/never receive an entitlement decision, (2) those who apply/show up for initial appointment/entitled to program and pursue, (3) those who apply/show up for initial appointment/entitled to program and do not pursue, (4) those who apply/show up for initial appointment/not entitled to program). The survey pool for the benchmark study for the VR&E Servicing questionnaire will include individuals who have entered and been enrolled in one of the five tracks for at least 60 days prior to the fielding period. The sample will be stratified as follows: (1) Veterans who have been rehabilitated, (2) Veterans who did not fully complete program (negative closures), and (3) Veterans who have reached maximum rehabilitation gain and could not proceed in program. The VR&E Escaped Beneficiary survey will explore why eligible individuals did not take advantage of their benefit entitlement. The survey pool for the benchmark study for the VR&E Escaped Beneficiary questionnaire will include individuals who dropped out of the program prior to completing a rehabilitation plan. Included in this survey pool are individuals who applied for the benefit but did not appear for their initial meeting. The sample will be stratified as follows: (1) applicants who never attended the initial meeting with a counselor, (2) applicants who were determined to be entitled and did not complete a rehabilitation plan, and (3) applicants who started, but did not complete rehabilitation (i.e., negative closures).

This information collection request is for benchmark study data collection for the VOV program. The results of the benchmark study will be used to examine the effectiveness and reliability of the survey instruments, including an evaluation of the levels of non-response across questions.

These surveys will be conducted per the legislative requirement set forth in the Executive Order 12862 and section 5701 of Title 38 of the United States Code. The Order mandates that government agencies set standards and seek to measure agency performance against such standards. These surveys are also being conducted in concurrence with the Government Performance Results Act (GPRA) and its requirements for improving customer service.

Specifically, VBA requires customer satisfaction information from Veterans in order to benchmark the effectiveness and usability of the benefits offered, as well as to assess their level of satisfaction with the level of service provided by VBA. The data collected with this survey will be used by VBA to make improvements to the benefit application and enrollment process, which in turn, will enable VBA to maximize the efficiency and quality of service provided to its nation’s Veterans and their dependents.

2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from current collection.

The information to be collected with the planned surveys will be used by VBA leadership and the lines of business: Compensation Service (CS), Pension Service (PS), Education (EDU) Service, Loan Guaranty (LGY) Service, and Vocational Rehabilitation and Employment (VR&E) Service to determine Veteran satisfaction with benefits and services. The major use of this data will be three-fold: 1) to provide Veterans an opportunity to comment on their experience with benefits and services, 2) to formulate program and policy changes in the enrollment and servicing process to ensure the most effective delivery of benefits and services, and 3) to identify the most influential elements of Veterans’ experience on their satisfaction and establish performance benchmarks to gauge the effectiveness of program and policy changes in improving Veterans’ experience with benefits and services. Currently, there are no other means of evaluating Veteran satisfaction with VBA’s C&P, EDU, LGY, and VR&E benefits and services.

This is a new data collection; no current collections exist.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g. permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.

In conjunction with a contractor, VBA staff will develop, administer, and analyze the Compensation and Pension Enrollment surveys, Compensation Servicing survey, Pension Servicing survey, Education Enrollment and Servicing surveys, the LGY Enrollment and SAH Servicing surveys, and the VR&E Enrollment, Servicing, and Escaped Beneficiary surveys.

To provide flexibility to the Veteran, the surveys will be self-administered in both paper form and online using web-based data collection. This will maximize the timeliness, efficiency, and response rate of data collection from Veterans. Based on previous experience, it is expected that the online web-based data collection will yield a 10% completion rate. Once the benchmark study effort has been completed, completion rates will be adjusted based on the actual responses returned. VA has selected a contractor that has strong capabilities in programming and hosting surveys on the worldwide web, and maintains effective security and privacy procedures (e.g., unique passwords for respondents, data encryption) when designing and programming web surveys. The web address (URL) on which the surveys will be hosted and accessed by respondents will be included in the materials sent to Veterans advertising the surveys. Veterans can either connect to the appropriate web page and complete the survey or fill out the survey included in the materials packet.

4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.

As noted, VBA has not recently conducted customer satisfaction surveys specifically geared toward Veteran satisfaction with CS, PS, EDU, LGY or VR&E benefits and services.

Our review of available data suggests that no sources outside of VBA’s VOV survey program are available that provide a reliable, representative sample of the targeted audience who have used CS, PS, EDU, LGY or VR&E services and from which detailed customer satisfaction indicators can be derived.

5. If the collection of information impacts small businesses or other small entities, describe any methods used to minimize burden.

No small businesses or other small entities are impacted by this information collection.

6. Describe the consequences to Federal program or policy activities if the collection is not conducted or is conducted less frequently as well as any technical or legal obstacles to reducing burden.

If the 2012 Voice of the Veteran surveys are not conducted, VBA will not be able to fulfill its performance measurement requirements as articulated by the requirements of P.L. 103-62, the Government Performance and Results Act of 1993 and Title 38 USC, §527, Evaluation and Data Collection. Furthermore, VBA would miss an opportunity to learn from America’s Veterans about how to improve the CS, PS, EDU, LGY or VR&E benefit enrollment and servicing processes to best serve their needs. VBA would also lose an opportunity to track and document improvements or declines in Veteran satisfaction and service delivery over time.

The design and administration of the suite of VOV surveys incorporates significant measures to minimize burden on respondents. (These specific measures are discussed in more detail in section 12). There are currently no technical or legal obstacles to reducing burden using the planned methods.

7. Explain any special circumstances that would cause an information collection to be conducted more often than quarterly or require respondents to prepare written responses to a collection of information in fewer than 30 days after receipt of it; submit more than an original and two copies of any document; retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years; in connection with a statistical survey that is not designed to produce valid and reliable results that can be generalized to the universe of study and require the use of a statistical data classification that has not been reviewed and approved by OMB.

There are no special circumstances that would require respondents to prepare or submit the documents outlined above, or respond in fewer than 30 days. The surveys will be designed and carried out with appropriate scientific rigor, and are intended to produce valid and reliable results that can be generalized to the universe of study.

8. Part A: If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the sponsor’s notice, required by 5 CFR 1320.8(d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the sponsor in responses to these comments. Specifically address comments received on cost and hour burden.

The 60-day Federal Register (FR) notice was published on January 24, 2011, pages 4152-4154. There were no comments received in response to this notice.

8. Part B: Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, clarity of instructions and record keeping, disclosure or reporting format, and on the data elements to be recorded, disclosed or reported. Explain any circumstances which preclude consultation every three years with representatives of those from whom information is to be obtained.

VBA is not aware of any benchmark surveys in private or government sectors.

9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

No payment or gift shall be provided to respondents.

10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statue, regulation, or agency policy.

An assurance that the information will be kept private to the extent of the law is made in the introductory letter respondents receive with the survey as well as written prologue appearing at the beginning of each survey, regardless of the mode of administration. Respondents are assured that answers given will be kept private to the extent of the law and will not affect their current or future entitlement to benefits and services. The information that respondents supply is protected by law (the Privacy Act of 1974, 5 U.S.C. 522a and section 5701 of Title 38 of the United States Code).

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private; include specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

None of the survey instruments contain any questions of a sensitive nature.

12. Estimate of the hour burden of the collection of information.

TABLE 1: ESTIMATED ANNUALIZED TIME BURDEN, BY RESPONDENT GROUP
Population surveyed	Number of respondents	Number of responses per respondent	Average burden per response (in hours)	Total burden hours
Compensation claimants	3000	1	0.25	750
Pension claimants	3000	1	0.25	750
Compensation beneficiaries	3000	1	0.25	750
Pension beneficiaries	3000	1	0.25	750
Education claimants	3000	1	0.25	750
Education beneficiaries	3000	1	0.25	750
Loan Guaranty claimants	3000	1	0.25	750
SAH beneficiaries	700	1	0.45	315
VR&E claimants	3000	1	0.25	750
VR&E beneficiaries	3000	1	0.25	750
VR&E Escaped beneficiaries	1500	1	0.25	375
Totals	30,000	NA	NA	7,500

In order to understand the different elements that make up the overall experience with VBA for a Veteran, questions were added to the surveys that result in a variation of length in the questionnaires. Questionnaires range from 10 minutes to 25 minutes depending on the benefit line, method of contact and Veteran experience.

13. Provide an estimate of the total annual cost burden to respondents or record keepers resulting from the collection of information. (Do not include the cost of any hour burden shown in Items 12 and 14).

a. There is no capital, start-up, operation, or maintenance costs.

b. Cost estimates are not expected to vary widely. The only true cost is that for the time of the respondent (average of 15 minutes per respondent). Using the Federal minimum wage of $7.25 an hour and the estimated 15 minute per respondent time burden, the total monetized estimate of time is $54,375.00

c. There are no anticipated capital start-up cost components or requests to provide information.

14. Provide estimates of annual cost to the Federal Government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operation expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information. Agencies also may aggregate cost estimates from Items 12, 13, and 14 in a single table.

The total cost to the Federal Government for the benchmark study program is estimated at $1,001,885. Table 2 below presents the labor and contracting costs for conducting the surveys. Operational costs will be outsourced to a contractor and will be included in the contractor’s total cost. The cost is based on the cost per hour x number of respondent’s x minutes -divided by sixty.

The VA Labor cost was estimated using a composite average salary and benefits figure of $45.51 per hour for 750 hours of a GS 13, step 3 and $55.45 per hour for 250 hours of a GS14, step 4.¹ The amount paid to the contractor for the survey effort includes as its major components the survey of recent claimants or existing beneficiaries for total cost of $953,890. For the benchmark program, these costs include development of the instruments, development of the sampling plan, review of the instrument, locating of respondents, programming of the questionnaire for Web administration, questionnaire pretest, validation, data processing, providing a clean data file, project management and analysis, and reporting.

15. Explain the reason for any changes reported in Items 13 or 14 above.

This is a new information collection.

16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

Based on previous survey experience, Section 16A below documents the proposed project plan for the tabulation and publication of survey data, including the analytical techniques and database management strategies to be employed. Following this, Section 16B provides an overview of the project timeline.

16A: TABULATION AND PUBLICATION

• Scan Surveys and Monitor Response Rates

During the survey field-period, the surveys will be scanned as they are received to ensure the currency of the name and address data file. During the scanning process, VA’s contractor will inspect and remove duplicate surveys in case any individuals inadvertently received a second survey after completing a survey from the first mailing. Data from damaged surveys could potentially be lost unless appropriate actions are rendered to make the data usable. Staples, paper clips, and notes will be removed and stray pencil marks that would interfere with the scanning process will be erased.

Our contractor will host a secure password-protected FTP site that will provide response rates from the survey. Throughout the period that the surveys are being fielded, response rate data will be updated at a minimum on a weekly basis for the survey. CS, PS, EDU, LGY and VR&E staff will be provided with passwords to access the site at any time.

• Develop Prototype Reports

Prototype reports will be developed for the survey. VBA will determine the appropriate weighting methods (if any) for the report. The selection of analyses and report exhibits in the prototype reports will rely on the knowledge and experience of CS, PS, EDU, LGY and VR&E staff.

The prototype report will be developed with the end user in mind, providing a clear explanation of findings. The report will consist of the following sections:

• Summary of key score matrices

• Key attributes

• Key diagnostics and demographics

The national level report for each business line will contain a summary of business line specific results, as well as a summary comparison of all business lines for measures that are common across business lines.

VBA’s contractor will pretest automation routines for new designs in the prototype report to ensure that the report exhibits, graphics, and data tables that can be reliably and accurately programmed for each report type. Development of the automated report system will consist of several concurrent tasks, including a data crosswalk of the survey iterations to map identical variables and response sets, SPSS syntax routines, and development of prototype report templates using Microsoft Excel with Visual Basic for Applications.

• Revise Prototype Reports

Based on discussions between CS, PS, EDU, LGY and VR&E staff and the contractor, the contractor will incorporate suggestions and changes/edits to the report’s content and layout and will produce a revised report.

• Submit “Live” Draft Reports

The production of the draft report requires “quick” turnaround of data cleaning and automated report production. This section details the steps to be taken to produce draft reports for each of the CS, PS, EDU, LGY and VR&E surveys using actual data from the survey administration. There are two significant subtasks involved in producing the draft report:

• Subtask 1: Clean and analyze survey data

• Subtask 2: Develop prototype report

• Subtask 3: Upload partial data into prototype report

• Subtask 4: Upload final data into prototype report

Each of these subtasks is described below.

Subtask 1—Clean and analyze survey data. Shortly after the survey scanning process commences, the contractor will examine the respondent data file to ensure that the scanning program is reading the surveys properly and that all of the variables are correctly placed and formatted. The sample data file will also be used to test the data cleaning programs. This process will identify any errors in the survey scanning process.

Once the final surveys have been scanned and a raw ASCII data file has been produced, the process of creating SPSS data files will begin. An SPSS syntax program will be run to convert the ASCII data into separate SPSS data files for the national file. A copy of the original ASCII data file will be archived separately as a quality control measure. The contractor will analyze the SPSS data files, including conducting frequencies, cross-tabulations, and quadrant analyses. The analyses will be geared toward providing CS, PS, EDU, LGY and VR&E and other VBA staff with user-oriented results.

Using the survey results, an Index Model will be created for CS, PS, EDU, LGY and VR&E. The Index model will identify the factors (key categories of satisfaction drivers) and attributes (specific elements of the experience within each factor) that contribute to a customer’s Overall Satisfaction score, along with their relative impact on the overall score. While the primary purpose of the benchmark study is to test the survey instruments, VBA will be able to identify how it is performing on overall customer satisfaction within each included line of service. The results will be evaluated according to several criteria. Using a statistically valid index model, we can provide insights into customer satisfaction and establish a benchmark for the ongoing program which can measure how each specific aspect of the experience is contributing to or detracting from the overall performance. Ultimately refinement to the surveys for the ongoing program will be based on the validity of the information gathered during the benchmark study.

(Note: The selected contractor, J.D. Power and Associates, uses an index model approach to all studies – the index model identifies Key Performance Indicators with the Veteran/beneficiary experience.)

Subtask 2—Develop prototype report. A prototype report will be produced based on discussions between CS, PS, EDU, LGY, VR&E and the contractor. The prototype report will be programmed to automatically populate data into a report format based on predetermined specifications. The reports will be programmed for creation at the National and Regional Office levels for VBA. Production of a prototype report provides CS, PS, EDU, LGY and VR&E staff an opportunity to assess the utility of the report structure and content prior to production of the draft report. Additionally, the automated report generation system enhances the quality and accuracy of data reporting.

Subtask 3—Develop draft report. A draft report will be produced based on data from the national file. Using the automated report generation system, the data will be analyzed and key findings will be identified and summarized in the executive summary. Production of a draft report provides CS, PS, EDU, LGY and VR&E staff an opportunity to assess the accuracy, quality, and utility of the report structure and content prior to production of the final report.

• Conduct Final Briefing

The contractor will develop data tabulations on survey highlights from the survey and present the results to CS, PS, EDU, LGY ,VR&E and VBA staff. The contractor will work closely with the VBA to determine the most salient findings to present in the briefing. The contractor will deliver bound color copies of the briefing presentation to CS, PS, EDU, LGY and VR&E.

• Submit Final Reports

The final report will address the efficiency of the benchmark study program as well as draw preliminary conclusions about the Veteran’s experience with the VBA and what is important to his/her satisfaction. Additionally, the results will be used to improve and refine the ongoing survey instruments where applicable. The contractor will submit all reports in final form, incorporating suggestions from CS, PS, EDU, LGY and VR&E staff where agreed. The contractor will deliver bound color copies of the final survey reports to CS, PS, EDU, LGY and VR&E.

• Deliver CD of Final Products

The contractor will submit the final deliverable of CDs containing electronic files as specified in the SOW. Documents and materials will include:

The contractor shall provide the following files (in MS Excel format) on CD or secure FTP site: a) the names and addresses of beneficiaries who were selected for the survey, but who were determined through the course of the survey to have moved, or have undeliverable addresses; (b) the names, and addresses of any veterans who were reported as deceased, or who were reported to have ‘undeliverable’ addresses, delineated by type of survey (c) the names and addresses of all veterans who were sent a questionnaire during this survey iteration; (d) the names and addresses of all non-respondents, broken out by type of survey they were selected for.

• Raw survey data on CD in SPSS format

• Copies of the reports in electronic format on CD

• Copies of the questionnaire and survey mailing materials

• Survey respondent verbatim, grouped by Regional Office/Line of Business

• Raw respondent data for undeliverable addresses, deceased respondents, non-responders, and a list of all respondents who were selected for mailing

These CDs of electronic files will provide CS, PS, EDU, LGY and VR&E with all materials used during the project, along with the final reports.

16B: PROJECT TIMELINE

The major activities for the VBA VOV survey project are structured by task, and are outlined below.

• Task 1: Conduct kickoff meeting and develop the project management plan (PMP)

• Task 2: Discuss recommendations for the survey instruments and methodology

• Task 3: Draft benchmark surveys and mailing materials

• Task 4: Revise benchmark surveys and mailing materials

• Task 5: Conduct cognitive labs and summarize results

• Task 6: Revise and finalize surveys and mailing materials

• Task 7: Create OMB Supporting documentation

• Task 8: Develop sample plan for benchmark study

• Task 9: Mail surveys and related materials

• Task 10: Host toll-free telephone and web helpdesk assistance and corresponding call/inquiry logs

• Task 11: Host live survey websites

• Task 12: Monitor response rates and provide response rate reports

• Task 13: Develop prototype reports

• Task 14: Upload draft reports

• Task 15: Upload and submit final reports

• Task 16: Conduct briefing on survey results

• Task 17: Submit final reports, raw data, and respondent information on CD

TIMELINe – VOICE OF THE VETERAN SURVEYS

Field Surveys	September – December 2012
Prototype Report	October 2012
Analyze Data	January 2013
Draft Report	January 2013
Finalize Report	February 2013

17. If seeking approval to omit the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

These forms do not display an expiration date, and if required to do so would result in unnecessary waste of existing stocks. VA is seeking an exemption that waives the display of the expiration date on the forms.

18. Explain each exception to the certification statement identified in Item 19, “Certification for Paperwork Reduction Act Submissions,” of OMB 83-I.

There are no exceptions.

1 FY 2010 Budget Estimates

11