Why Conduct the Surveys?
1. The 2006 VBA Loan Guaranty (LGY) Surveys project is representative of the VBA’s continuing commitment to develop and utilize customer satisfaction surveys in each of its business lines. A collection of three customer satisfaction surveys will comprise the LGY Surveys project. Each of these surveys and the customers to be surveyed include:
Lender Survey of the VA Home Loan Guaranty Program: Lending institutions that participate in the VBA LGY program
Survey of Veterans Satisfaction with the VA Home Loan Guaranty Program: veterans who have received a home loan through the VBA LGY program
Survey of Specially Adapted Housing Program: Disabled veterans who participate in VBA’s Specially Adapted Housing (SAH) grant program
Results of the surveys will be used by VBA leadership to determine priorities for program policy and procedure, to improve customer service to program participants, to comply with GPRA requirements, and to gather data for the reporting of program performance measures to the Department and OMB. Past customer satisfaction surveys have been extremely useful tools in gauging how we can improve customer service, and how program and policy changes and modifications, and new information technology initiatives have impacted the satisfaction levels of our program participants and stakeholders. The 2006 LGY surveys will provide VBA with a much-needed opportunity to obtain data for important program performance measures.
2. The three LGY surveys will be conducted per the legislative requirement set forth the in Executive Order 12862. The Order mandates that government agencies set standards and seek to measure agency performance against such standards. These surveys are also being conducted in concurrence with the Government Performance Results Act (GPRA) and its requirements for improving customer service. The information collected from these surveys will be used by the VBA Loan Guaranty program to determine the satisfaction of lending institutions and veterans with the LGY program, and the satisfaction of severely disabled veteran’s satisfaction with the Specially Adapted Housing grant program in addressing their adaptive housing needs. These data, in turn, will be utilized to formulate potential program policy and procedural changes within the SAH and LGY programs.
3. VBA has not formally collected satisfaction survey data from participants of the SAH program since 2004. Customer satisfaction survey data from the LGY program have not been systematically collected from veterans or lending institutions since 2003 and 2002, respectively. The three planned LGY surveys for 2006 are needed to collect current information on these VBA programs, particularly since these programs have undergone both legislative and programmatic changes in recent years, including several major enhancements in benefits, and the rollout of new web-based data collection and processing systems. The 2006 surveys will retain approximately 80% of the survey content from previous iterations of these surveys. The remaining 20% of survey content will be in the form of new survey items and revised (often reworded) survey stems and response sets.
Historically, the first full administrations of the Veteran and Lender Surveys were in 2000, with the most recent iterations being conducted in FY03. The inaugural administration of the SAH survey was completed in FY04. Previous iterations of these surveys took into account data gathered from focus groups, pilot implementations of the survey instruments, and input gathered from the LGY field and central office managers.
Who Is Going to Be Surveyed?
4. Lender Survey of the VA Home Loan Guaranty Program: The population for the lender survey to be surveyed will consist of mortgage lenders that have used the VA home loan program to serve their customers. This population is comprised of lenders’ home offices and branches that have made more than 12 VA loans in the past fiscal year. The population of current lending institutions is approximately 5,500 across the nine Regional Loan Centers (RLC); therefore a census will be included in the survey.
Survey of Veterans Satisfaction with the VA Home Loan Guaranty Program: For the veteran survey, the survey population is determined by the sample stratification and the LGY national workload for FY06. LGY’s national workload for FY2006 was 142,723 loans guaranteed. The veteran population to be surveyed will include two types: original and refinanced loan customers. Therefore, the Veteran survey population is 142,723.
Survey of Specially Adapted Housing Program: For the SAH survey, the primary population to be surveyed includes all FY06 SAH grant recipients, and is estimated to be 600. A secondary target population includes those individuals who are eligible to receive this grant, but have not taken the necessary actions to become grant recipients. This population is estimated to be 1,000.
5. For the Lender survey, lenders who have typically processed at least 12 VA loans in the past fiscal year will be selected from the existing list of lender participants. The size of this universe will be 5,500 from which a 60% response rate is expected. Since the survey is a census, the results will be statistically significant and representative of lender presence at the Regional Loan Center (RLC) level, if the response rates are high enough (see question 16 below).
For the Veteran Survey, the sample will include 12,600 veterans from the total veteran population. We will compile 1,400 names and addresses of veterans for each of the nine RLCs, to obtain this sample (1,400 x 9 = 12,600 total).
For the SAH survey, the population to be surveyed will consist of the total number of SAH grant recipients, which is estimated to be 600. The VA requires that a full census of this population be used, in order to achieve the desired level of accuracy in sampling at the national level. Additionally, the secondary population of 1000 SAH-eligible, non-applicants will be surveyed to inform LGY program officials about potential barriers (perceived or otherwise) to SAH grants.
6. For the Lender survey, specific lenders will be selected from the existing list of program participants, based on the quantity of VA loans processed by the lending institution. For the Veteran survey, participating veterans will be selected from the existing list of veterans from each RLC who are either original or refinanced loan customers in the past fiscal year. The primary sample for the SAH survey will include the complete list of individuals who are SAH grant recipients. The source for the secondary sampling will be a sample from the complete list of individuals who are eligible, but have not applied for the SAH grant.
7. The sample for the Lender survey will be a census. The sample for the veteran survey will be stratified by creating a sample of 1,400 names and addresses for each of the nine RLCs (1,400 x 9 = 12,600 total). In creating the stratified samples for the Veteran Survey, We will ensure that the appropriate number of names is available for each RLC. The sample for the SAH survey will not be stratified. Upon drawing the samples for all three surveys and following de-duplication procedures to remove any duplicate names both within and between the RLC files, all names and addresses will undergo National Change of Address (NCOA) processing. Non-essential demographic fields or other identifying fields will then be stripped from the resulting survey response database.
8. This set of surveys will be administered by an outside contractor.
9. The survey administrator will be:
Caliber, an ICF International Company
9300 Lee Highway
Fairfax, VA 22031-1207
(703) 934-3000
Contract documents are available upon request.
10. In collaboration with the LGY program office, Caliber/ICF will develop, administer, and analyze this set of surveys. The development of all three surveys will reflect the comments, suggestions, and results from previous iterations of each survey. Caliber/ICF has 20+ years experience in survey design and administration. Caliber also serves Department of Defense, other VA business lines and other numerous Federal government agencies in the same capacity.
11. The Lender instrument will be administered exclusively as a Web-based survey. Respondents will be contacted by mail and informed of the URL address. Lenders will then connect to the appropriate Web page, create a password, and complete the survey. Data from the Internet survey responses will be collected by the contractor, analyzed, and organized into report form. The Lender Survey will be administered online to maximize the timeliness, efficiency, and response rate of data collection. The Veteran survey will be administered both as a Web and paper survey, at the preference of the respondent. As with the Lender survey, the URL for the Web survey will be included in the mailings indicated below. Data from the survey responses will be collected by the contractor, analyzed, and organized into report form. This survey will utilize both online and self-administered surveys to maximize the timeliness, efficiency, and response rate of data collection. The SAH survey will be administered via mail survey. The details of survey distribution and administration are presented below.
Specifications: Design and Distribution of Mailing Materials by Survey and Each of Four Mailings |
||||
Mailing |
Mailing Material |
Veteran Survey |
Lender Survey |
SAH Survey |
#1 |
Notification/cover letter Notification/cover letter w/ URL & password Paper survey Business Reply Envelope |
|
|
|
#2 |
1st reminder card w/ URL & password 1st reminder card |
|
|
|
#3 |
Second notification/cover letter Second notification/cover letter w/ URL &password Paper survey Business Reply Envelope |
|
|
|
#4 |
2nd reminder card w/ URL & password 2nd reminder card |
|
|
|
12. Interviewers will not be utilized in any of these surveys.
13. As stated in #11 (above), both the Lender and Veteran surveys will take advantage of Internet technology by having participants respond online. The Web-based Lender and Veteran surveys will be developed with the end user in mind, with the goal of providing a user-friendly Web site in which to complete the survey.
14. Caliber/ICF will conduct cognitive labs and pretests with as many as three potential participants for each of these surveys (Lender, Veteran, and SAH) and per medium (2 methods – paper and Web), to determine whether respondents understand the survey questions and answer choices, as intended. Working closely with the VBA, Caliber/ICF will draw a small pool of names from potential participants in each of the three surveys for inclusion in the cognitive labs. Cognitive lab participants will be drawn from the same population that will be used for the main study. Caliber/ICF will submit the list of potential participants to the VBA for review and approval. Once identified, Caliber/ICF will contact potential participants by telephone and ask them to participate. Cognitive lab sessions will take place in the metropolitan Washington DC area. Once the participants have been selected, we will conduct cognitive lab sessions aimed at identifying needed additions or refinements to the questionnaire. Caliber/ICF will prepare a summary report of the cognitive testing session for paper and Web versions of the customer satisfaction surveys. The results of the cognitive labs will be taken into account when revising and finalizing the survey questionnaires.
15. The nature of the guaranteed loan and Specially Adapted Housing (SAH) grant programs are such that the questionnaires solicit specific information on loans and grants that respondents have received. For example, SAH grant recipients are asked to identify which specific adaptations they have received. While some respondents may consider such information sensitive or private, it is necessary to gather this data so that the LGY program is able to identify specific aspects of the program which may require attention.
16. The response rate is anticipated to be approximately 60% for each of the three surveys.
17. For survey notification and reminders, Caliber/ICF will use a 4-step survey and follow-up process to administer the three surveys. An increase in the overall response rate is the major advantage of using this process. The use of a reminder post card as a follow-up tends to increase the response rate by between 5 and 8 percentage points. The use of both reminder post cards and a second survey mailing almost double the response rate. The improved response rate and reliability of the data more than offset the increase in the cost of this survey administration process. The survey administration process for the Veteran and SAH Surveys will consist of four mailings to veterans and the process for the Lender survey will consist of three mailings to lenders. The specifications of the mailings by survey are presented in the table below.
Our on-line survey technology incorporates several features to maximize response rates and respondent usability. These include a password system, which prevents any one person from completing more than one survey and allows respondents to begin the survey, then come back at a later point to finish it (i.e., this is particularly useful for long surveys). Other features include user-friendly drop-down boxes, internal links to the directions throughout the survey, and internal links to key terms and definitions.
Specifications of Mailings by Survey |
|||
Mailing Process |
Veteran Survey |
Lender Survey |
SAH Survey |
Mailing # 1: notification cover letter with URL and password, survey, and BRE (Qty = 12,600)
Mailing #2: reminder card one with URL and password (Qty 12,600)
Mailing #3: second survey with cover letter and BRE (Qty = 9,450)
Mailing #4: second reminder card (Qty = 9,450) |
Mailing #1: notification letter with URL and password (Qty = 5,500)
Mailing #2: reminder card one with URL and password (Qty = 5,500)
Mailing #3: reminder card two with URL and password (Qty = 5,500)
|
Mailing # 1: notification cover letter, survey, and BRE (Qty = 600)
Mailing #2: reminder card one (Qty = 600)
Mailing #3: second survey with cover letter and BRE (Qty = 500)
Mailing #4: second reminder card (Qty = 500)
|
|
Additionally, during the period that the surveys are in the field, Caliber/ICF will provide and maintain a toll free telephone line to answer any questions respondents and regional office points of contact (POCs) may have about the survey (e.g., how to interpret questions and response items, the purpose of the survey, how to get another survey if their copy has been lost/damaged). Project staff will be available to answer telephone calls during regular business hours (8:30 a.m.-6 p.m. ET). A voice messaging system will be available to receive messages after regular business hours so after-hours calls can be responded to within 24 hours.
18. The project team at Caliber/ICF will be responsible for all data coding, tabulation, analysis and publication. [See #9 (above) for contact information].
19. Please refer to VA Form 20-0278, which includes the following information on Public Burden. The overall Public Burden is 3,115 hours, spread equally among 12,460 respondents. For the Lender survey, approximately 3,300 lenders (5,500 lenders at 60% response rate) are expected to reply. One response per person is required. Each survey will take an average of 15 minutes to complete. Therefore, approximately 825 annual burden hours will be required for this survey. For the Veteran survey, approximately 7,560 veterans are expected to reply. One response per person is required. Each survey will take an average of 15 minutes to complete. Therefore, approximately 1,890 annual burden hours will be required for this survey. For the SAH survey, approximately, 600 SAH grant recipients are expected to reply. One response per person is required. Each survey will take an average of 15 minutes to complete. In addition, approximately 1,000 SAH grant eligible, non-recipient veterans are expected to reply. Therefore, approximately 400 annual burden hours (150 for grant recipients, 250 for eligible, non-recipients) will be required for this survey.
20. A safeguard to ensure that only one response per respondent is obtained for each survey will be utilized. For all three surveys, upon drawing the samples and following de-duplication procedures to remove any duplicate names both within and between the RLC files, all names and addresses will undergo National Change of Address (NCOA) processing. Non-essential demographic fields or other identifying fields will then be stripped from the resulting survey response database. For the Lender and Veteran surveys, a password system will be used, which prevents any one person from completing more than one survey and allows respondents to begin the survey, then come back at a later point to finish it (i.e., this is particularly useful for long surveys).
21. Survey respondents for all three surveys will be assured that their personal anonymity will be maintained. All hard copy questionnaires (i.e., Veteran and SAH surveys) will be scannable, and consist of approximately six printed pages printed back to back with a numeric Litho-Code on the front and back cover tied to RLC rather than the individual. On the Veteran and SAH questionnaires, the respective loan information (e.g., date loan closed or date grant made; original or refinanced loan for the Veteran Survey) will be overprinted in a box on the front cover. For the Lender survey, each response will be identified by its corresponding ‘Lender ID number’; thus, each response will be attributed to a specific lender, not an individual. Respondents will be informed of this fact in the initial notification letter. Additionally, Caliber/ICF has a secure network infrastructure that will protect the integrity of the databases, the survey application, and all associated server resources. Our servers are protected by a strong firewall system and our operations center is in a secure temperature-controlled environment with video surveillance, where network services are continually monitored by automated real-time programs to ensure the integrity and availability of all critical components. All key servers are supported by a backup power supply that can continue to run the systems in the event of a power outage. Additionally, our engineers are immediately alerted if critical monitor thresholds are exceeded, so that they can proactively respond before outages occur.
22. The survey response files for each of the three surveys will be stripped of any identifying information or link to the original administrative mailing file. These “de-identified” response files will be used for analyses but survey responses will only be reported in the aggregate at the Regional Loan Center level (9) and nationally. For all three surveys, since no response will be connected with an individual, confidentiality of respondents’ information will not be an issue. Additional protections for confidentiality of data and privacy of survey respondents will be assured through Caliber/ICF’s Institutional Review Board.
23. There will be no linkages between the VBA databases for the Lender survey, Veteran and SAH surveys.
24. All costs for survey development, administration, and reporting for this set of LGY surveys will be covered under Contract 101-Y67266 with Caliber/ICF in the amount of $324,983. The government’s independent cost estimate is attached. The contractor’s proposal outlining the cost categories and subcategories is available upon request.
| File Type | application/msword |
| File Title | Supporting Statement for VBA Generic Customer Survey Clearance |
| Author | VBA |
| Last Modified By | Denise McLamb |
| File Modified | 2007-03-01 |
| File Created | 2007-03-01 |