B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
The current survey pool stratifications represent mutually-exclusive and exhaustive categories for each line of business based upon the level/degree of experience a respondent has with a particular benefit. This stratification method was chosen because the level/degree of experience with a benefit or line of business can influence their level of satisfaction with their experience. The number and types of questions answered will vary slightly based on the experiences of the targeted population (e.g., whether or not the respondent has had contact or not with the VBA regarding their claim). The tables for the lines of business describe the universe of respondents for the survey, and the sampling plan that will be employed to achieve a representative sample for the survey. Each table displays the sampling frame, the expected response rates and the actual response rate from FY2014 and FY2015.
1) Compensation Enrollment
The targeted population for the Compensation Enrollment questionnaire tracking study will include individuals who have received a decision on a compensation benefit claim within 30 days prior to the fielding period (includes those who were found eligible on a new or subsequent claim and those who have been denied and are not appealing the decision).
TABLE 1: EXPECTED RESPONSE RATE AND SURVEY YIELD
|
|||
Number of Claimants |
Expected Response Rate |
FY2014FY2014 Response Rate |
FY2015 Response Rate |
160,000 |
30% |
19.41% |
21.96% |
2) Compensation Servicing
The targeted population for the Compensation Servicing questionnaire tracking study will include individuals who received a decision or are receiving benefit payments.
TABLE 2: EXPECTED RESPONSE RATE AND SURVEY YIELD
|
|||
Number of Claimants |
Expected Response Rate |
FY2014 Response Rate |
FY2015 Response Rate |
60,000 |
30% |
21.39% |
25.03% |
3) Pension Enrollment
The targeted population for the Pension questionnaire tracking study will include individuals who have received a decision on a pension benefit claim within 30 days prior to the fielding period (includes those who were found eligible on a new or subsequent claim and those who have been denied and are not appealing the decision).
TABLE 3: EXPECTED RESPONSE RATE AND SURVEY YIELD
|
||||
Number of Veterans |
Expected Response Rate |
FY2014FY2014 Response Rate |
FY2015 Response Rate |
|
10,000 |
30% |
2424.15% |
26.31% |
|
4) Pension Servicing
The targeted population for the Pension Servicing questionnaire tracking study will include individuals who have currently been receiving benefits for at least 6 months.
TABLE 4: EXPECTED RESPONSE RATE AND SURVEY YIELD
|
|||
Number of Claimants |
Expected Response Rate |
FY2014FY2014 Response Rate |
FY2015 Response Rate |
10,000 |
30% |
2121.05% |
18.49% |
5) Education Enrollment
The targeted population for the Education Enrollment questionnaire tracking study will include individuals who have received a decision on their education benefit application within 90 days (i.e., the original end-product has been cleared within the past 90 days) prior to the fielding period. The sample will be stratified as follows: (1) accepted and enrolled, (2) accepted and not enrolled.
TABLE 5: EXPECTED RESPONSE RATE AND SURVEY YIELD
|
|||
Number of Claimants |
Expected Response Rate |
FY1014FY1014 Response Rate |
FY2015 Response Rate |
16,000 |
30% |
66.23% |
4.91% |
6) Education Servicing
The targeted population for the Education Servicing questionnaire tracking study will include beneficiaries who have been enrolled and receiving education benefit payments for at least 2 consecutive school terms prior to the fielding period.
TABLE 6: EXPECTED RESPONSE RATE AND SURVEY YIELD
|
||||
Number of Beneficiaries |
Expected Response Rate |
FY2014FY2014 Response Rate |
FY2015 Response Rate |
|
10,000 |
30% |
8.48% |
5.86% |
|
7) Loan Guaranty Enrollment
The targeted population for the Loan Guaranty Enrollment questionnaire tracking study will include individuals who closed a VA home loan in the 30 days prior to the fielding period. The sample will be stratified as follows: (1) those who closed on purchase loans, (2) those who received loans for interest rate reductions, and (3) those who obtained cash out or other refinancing.
TABLE 7: EXPECTED RESPONSE RATE AND SURVEY YIELD
|
|||
Number of Beneficiaries |
Expected Response Rate |
FY2014FY2014 Response Rate |
FY2015 Response Rate |
40,000 |
30% |
66.46% |
9.69% |
8) Specially Adapted Housing Servicing
The targeted population for the Specially Adapted Housing Servicing questionnaire tracking study will include individuals who are eligible for a specially adapted housing grant and in the past 12 months have: (1) received an approval on their grant and are currently somewhere in post-approval, (2) have had all their funds dispersed and final accounting is not yet complete, and (3) have had all of their funds dispersed and final accounting is complete. This survey was not fielding during FY15 due to IC revisions and changes to the benefit program.
TABLE 8: EXPECTED RESPONSE RATE AND SURVEY YIELD
|
|||
Number of Beneficiaries |
Expected Response Rate |
FY2014FY2014 Response Rate |
FY2015 Response Rate |
5,000 |
30% |
88.06% |
Not Fielded |
9) Vocational Rehabilitation and Employment Enrollment
The targeted population for the Vocational Rehabilitation and Employment (VR&E)Enrollment questionnaire tracking study will include individuals who had an initial meeting with their VR&E counselor and were granted a decision regarding their entitlement in the past 1212 months prior to the fielding period. The sample will be stratified as follows by those who applied/applied and show up for initial appointment and: (1) were entitled to program and pursued, (2) not entitled to program.
TABLE 9: EXPECTED RESPONSE RATE AND SURVEY YIELD
|
|||
Number of Claimants |
Expected Response Rate |
FY2014 Response Rate |
FY2015 Response Rate |
1414,000 |
30% |
1111.89% |
9.62% |
10) Vocational Rehabilitation and Employment Servicing
The targeted population for the VR&E Servicing questionnaire tracking study will include individuals who have entered and been enrolled in one of the five tracks for more than 12 months prior to the fielding period. The sample will be stratified as follows: (1) Veterans who have been rehabilitated and (2) Veterans who have reached maximum rehabilitation gain and could not proceed in program.
TABLE 10: EXPECTED RESPONSE RATE AND SURVEY YIELD
|
|||
Number of Beneficiaries |
Expected Response Rate |
FY2014FY2014 Response Rate |
FY2015 Response Rate |
60,000 |
30% |
99.35% |
13.78% |
11) Vocational Rehabilitation and Employment Non-Participant Beneficiary
The targeted population for the VR&E Non-Participant questionnaire tracking study will include individuals who dropped out of the program prior to completing a rehabilitation plan in the past 12 months. Included in this survey pool are individuals who applied for the benefit but did not appear for their initial meeting. The sample will be stratified as follows: (1) applicants who never attended the initial meeting with a counselor and (2) applicants who started, but did not complete a rehabilitation plan (i.e., negative closures).
TABLE 11: EXPECTED RESPONSE RATE AND SURVEY YIELD
|
|||
Number of Non-Participant Beneficiaries |
Expected Response Rate |
FY2014 Response Rate |
FY2015 Response Rate |
5,000 |
30% |
5.76% |
5.60% |
2. Describe the procedures for the collection of information, including: Statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose in the proposed justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.
Please note that all volumes outlined below for each program line are per 12-month period.
The Voice of the Veteran Compensation and Pension surveys will entail stratified random sampling of 240,000 Veterans for each of the four population segments outlined in Section 1. VBA is using a 95% confidence interval for categorical variables for all surveys. There are no unusual procedures that will be required to draw a representative sample meeting these criteria.
The Voice of the Veteran Education surveys will entail stratified random sampling of 26,000 Veterans for the each of the population segments outlined in Section 1. VBA is using a 95% confidence interval for categorical variables for all surveys. There are no unusual procedures that will be required to draw a representative sample meeting these criteria.
The Voice of the Veteran Loan Guaranty Enrollment survey will entail stratified random sampling of 40,000 Veterans and the Specially Adapted Housing Servicing survey will entail stratified random sampling of 5,000 Veterans for the population segment outlined in Section 1. VBA is using a 95% confidence interval for categorical variables for all surveys. There are no unusual procedures that will be required to draw a representative sample meeting these criteria.
The Voice of the Veteran Vocational Rehabilitation and Employment Enrollment and Servicing surveys will entail stratified random sampling of 7474,000 Veterans for the population segments outlined in Section 1. The Voice of the Veteran Vocational Rehabilitation and Employment Non-Participant survey will entail stratified random sampling of 5,000 Veterans for the population segment outlined in Section 1. VBA is using a 95% confidence interval for categorical variables for all surveys. There are no unusual procedures that will be required to draw a representative sample meeting these criteria.
VBA has obtained the services of a contractor to develop, administer, and analyze this set of surveys.
Strategies to Maximize Response Rates
VBA employs methods to minimize respondent burden and to maximize survey response rates. This section identifies the strategies to be employed to reach these objectives. Each strategy is outlined below.
Strategy # 1 to Maximize Response Rates: Using Web Technologies for Ease of Response
For most of the survey instruments, Veterans will have the option to complete the surveys via paper or electronic submission. Exceptions include Pension Enrollment, Pension Servicing, VR&E Servicing, VR&E Non-Participant, and SAH Servicing which are only available in mail format. The survey website address is included in the mailed notification.
Respondents are only invited to participate in one survey during the Tracking study. The sample is de-duplicated and Veterans that appear in the sample list multiple times and/or across multiple lines of business will be directed to the survey with the lowest number of possible respondents.
Both the paper and web-based surveys were developed with the end user in mind; the form is designed to be clear, easily reviewed, and maximize response rates.
The online survey technology incorporates several features to maximize response rates and respondent usability. These include a password system, which prevents any one person from completing more than one survey and allows respondents to begin the survey then come back at a later point to finish it.
Strategy # 2 to Maximize Response Rates: Using Advance and Follow-Up Mailing to Publicize the Surveys and Encourage Response
VBA will use a 2-step survey and follow-up process to administer the surveys (see Table 13 below). An increase in the overall response rate is the major advantage of using this process. The combined use of a mailed survey package and postcards is anticipated to increase the response rate by between 15 and 18 percentage points. The order of the mailings is flexible to allow adjustment due to holidays and scheduling.
Mailing |
Table 13: Mailing Material |
A
|
Mail Surveys: Notification cover letter and paper survey (mail only surveys)
Mixed Methodology Surveys: Notification cover letter w/ URL & password, and
|
B
|
Mail Surveys: Notification cover letter w/ paper survey (mail only surveys)
Mixed Methodology Surveys: Notification postcard w/ URL & password (mail and web-based surveys) |
Strategy # 3 to Maximize Response Rates: Maintaining a Toll-Free Survey Hotline
During the period that the survey is in the field, the contractor provides and maintains a toll-free telephone line and dedicated e-mail address to answer any questions respondents may have about the survey (e.g., how to interpret questions and response items, the purpose of the survey, how to get another survey if their copy has been lost/damaged). Project staff is available to answer telephone calls or emails during regular business hours (8:00 a.m.-5:00 p.m. PST). A voice messaging system will be available to receive phone messages after regular business hours; after-hours calls are responded to within 24 hours. Respondents who utilize the e-mail option are sent an initial automatically generated response email informing them that their e-mail was received and that they will receive a response within 24 hours.
Strategy # 4 to Maximize Response Rates: Excluding Questions of a “Sensitive” Nature
None of the questions included in the surveys are sensitive, or private in nature, which will encourage compliance.
Strategy # 5 to Maximize Response Rates: Assuring and Maintaining Confidentiality
Survey respondents for all surveys will be assured that their personal anonymity will be maintained. All hard copy questionnaires will be scannable, and consist of approximately six-eight printed pages, printed back to back with a numeric Litho-Code on the front and back cover. Veterans will be provided unique passwords that will allow the contractor to identify when a respondent has completed the survey and exclude them from further reminder letter or postcards.
Strategy #6 to Maximize Response Rates: Survey Instrument Revisions
All VBA business lines have refined the survey instruments. Unnecessary response options and responses that were provided information contained in the VBA System of Record were removed.
Strategy # 7 to Maximize Response Rates: Secure Networks and Systems
The contractor has a secure network infrastructure that protects the integrity of the databases, the survey application, and all associated server resources. The servers are protected by a strong firewall system and the operations center is in a secure temperature-controlled environment with video surveillance, where network services are continually monitored by automated real-time programs to ensure the integrity and availability of all critical components. All key servers are supported by a backup power supply that can continue to run the systems in the event of a power outage. Additionally, the contractors will be immediately alerted if critical monitor thresholds are exceeded, so that they can proactively respond before outages occur.
Strategy #87 to Maximize Response Rates: Maintaining 508 Compliance
The contractor will ensure the web-based surveys are accessible to people with disabilities by maintaining 508 compliant standards. These standards will enable Veterans with disabilities to complete the survey by keyboard navigation and through the use of assistive technology such as screen readers and refreshable Braille displays. In addition to technical 508 compliance standards, the web-based surveys will enable customization for font size, color, and style to make the survey more visible to a range of respondents. The web-based surveys will also include the number of desired answer choices listed next to each question (e.g. Mark only one, Mark all that apply, etc.) to help respondents understand how to answer each question. These features will ensure that Veterans with disabilities have equal access to the Voice of the Veteran surveys and enable their participation.
Approach to Examine Non-Response Bias
Non-response bias refers to the error expected in estimating a population characteristic based on a sample of survey data that under-represents certain types of respondents. Stated more technically, non-response bias is the difference between a survey estimate and the actual population value. Non-response bias associated with an estimate consists of two components – the amount of non-response and the difference in the estimate between the respondents and non-respondents. While high response rates are always desirable in surveys, they do not guarantee low response bias in cases where the respondents and non-respondents are very different. Two types of non-response can affect the interpretation and generalization of survey data: item non-response and unit non-response. Item non-response occurs when one or more survey items are left blank in an otherwise completed, returned questionnaire. Unit non-response is non-participation by an individual that was intended to be included in the survey sample. Unit non-response – the failure to return a questionnaire – is what is generally recognized as survey non-response bias.
There are two approaches to tackling the effects of non-response. One is to minimize the chances of non-response at the data collection stage. This may involve introducing measures, which aim to maximize the response rate. The other approach is to make statistical adjustments at a survey follow-up stage when all the data is collected. Both approaches are described in the next paragraphs of this section.
Since it is not always possible to measure the actual bias due to unit non-response, there are strategies for reducing non-response bias by maximizing response rates across all types of respondents. In the face of a long-standing trend of declining response rates in survey research (Steeh, 1981; Smith, 1995; Bradburn, 1992; De Leeuw & Heer, 2002; Curtin & Presser, 2005), these strategies include:
Use of notification letters, duplicate survey mailings, reminder letters and postcards.
Use of novelty in correspondence such as reminder postcards designed in eye-catching colors.
Use of an extended survey field-period to afford opportunities to respond for subgroups having a propensity to respond late (e.g., males, young, full-time employed).
Use of well-designed questionnaires and the promise of confidentiality.
Providing a contact name and telephone number for inquiries.
Employing these strategies to the administration of these surveys will be crucial for maximizing high response rates across all respondent types (see section on maximizing response rates above).
Since FY2014 and FY2015 response rates for the Voice of the Veteran program are below 80%, the contractor conducted non-response analyses for each of the surveys to identify potential sources of bias and reassure VBA of the quality of the data collected. The approach to examining the presence of non-response bias was conducted as follows:
Compare the Demographics of Respondents from the Voice of the Veteran surveys to the Demographics of Non-Respondents from the Voice of the Veteran surveys. To examine the presence of non-response bias, the contractor compared the demographics of responders (i.e., those who responded to each Voice of the Veteran Survey) to the non-responders (i.e., those who did not respond to each Voice of the Veteran Survey). The comparison between responders and non-responders was based on the sample variables included in the sample files from VBA, which includes the following variables:
Region – it is possible that participants from a certain part of the country (i.e., region) may respond to the survey at a higher rate than those who are from another part of the country.
Age – it is possible that participants from a certain age group (i.e., over 30 years old) may respond at a higher rate than those who are members of another age group.
Gender – it is possible that participants from a certain gender (i.e., male) may respond at a higher rate than their counterpart.
Based on the steps discussed above, the contractor determined that six of the eleven surveys were not impacted by any statistical differences among the demographic groups analyzed, and thus, were not impacted by nonresponse bias. The only statistical difference found among the remaining four surveys was age. Older Veterans were more likely to respond to the survey and provided higher ratings for Overall Satisfaction compared to younger Veterans. Results from these analyses indicate Overall Satisfaction scores for these four surveys would likely be lower if the sample was more representative of the populations. In the non-response bias reports submitted to VBA, the contractor included recommended approaches to mitigating the effects of nonresponse bias and strategies to correct non-response bias such as data imputation and weighting. VBA, in conjunction with the contractor, are in the process of finalizing the action items from the non-response bias analyses.
The contractor has conducted cognitive labs with three or more test users for each survey instrument to determine whether respondents understand the survey questions and answer choices, as intended. Working closely with VBA, the contractor drew a small pool of names from potential participants in each of the surveys for inclusion in the cognitive labs.
Cognitive lab participants were drawn from the same population that will be used for the main study. The contractor submitted the list of potential participants to VBA for review and approval. Once identified, the contractor contacted potential participants by telephone and asked them to participate. Cognitive lab sessions took place in the metropolitan Washington, DC area or via conference call depending upon participant availability.
Once the participants were selected, VBA and the contractor conducted cognitive lab sessions aimed at identifying needed additions or refinements to the questionnaire. Cognitive labs are one-on-one or group sessions with potential survey participants, in which respondents are asked to complete the questionnaire while thinking aloud. The primary purpose of these sessions was to gather feedback on survey questions and answer choices to ensure they are easily understood and correctly interpreted.
Outcomes of cognitive labs included, but were not limited to: addition or omission of specific questions, changes to wording of questions, clarification of question response options, addition of response options, and changes to ordering of questions.
The cognitive labs have been completed and changes to the questionnaires have been made based on those findings.
5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
The VBA Benefits Assistance Service VOV Program contact persons are:
Pamela Liverman, Pamela.Liverman@va.gov, 202-530-9439
Dawna Quick, Dawna.Quick@va.gov, 202-530-9397
The VBA Compensation Service contact persons are:
Christi Greenwell, Christi.Greenwell@va.gov
Voncelle James, Voncelle.james@va.gov
Heather Osborne, Heather.Osborne@va.gov
The VBA Pension and Fiduciary Service contact persons are:
Robert Pomarico, Robert.Pomarico@va.gov
Tashonda Henry-Pugh, Tashonda.henry-pugh@va.gov
The VBA EDU contact persons are:
Christopher Sutherland, Christopher.Sutherland@va.gov
Danita Cohen, Danita.cohen@va.gov
The VBA LGY contact persons are:
Carleton Sea, Carleton.Sea@va.gov
Elysium Drumm, Elysium.Drum@va.gov
The VBA VR&E contact persons are:
Nelson Foster, Nelson.Foster@va.gov
Lisa Jong Thomas, Lisa.thomas4@va.gov
Kevin P. Scott, Kevin.Scott2@va.gov
VBA has contracted the services of JD Power & Associates (JDPA) to administer the survey. JDPA contacts are as follows:
Greg Truex, 805-418-8078, Greg.Truex@jdpa.com
Tara PorterPorter, 202-383-3707, Tara.porter@jdpa.com
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Hee-Seung L. Seu |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |