Evaluation of the Department of Veterans Affairs Mental Health Services
Supporting Material for OMB Clearance
OMB No. 2900-XXXX
Respondent Universe and Sampling Methods
A two-phase sample design will be employed for the survey of OEF/OIF/OND veterans. In the first phase of sampling, the VA will select a random sample of approximately 25 percent of all OEF/OIF/OND veterans, yielding a sample of approximately 150,000 veterans. Through a Data Use Agreement, the VA will provide Westat with a data file containing a non-personally identifiable information (PII) identifier for each veteran in the first-phase sample, along with other non-PII variables—such as, age, gender, military-service characteristics, and utilization of VA health care services—which Westat will use to stratify the first-phase sample and then select a stratified second-phase sample. All of Westat’s processing to select the second-phase sample will be conducted in Westat’s secure FISMA High Enclave (FHE) enclave with access given only to Westat project staff. The identifiers for the second-phase sample will be provided to VA, who will provide back to Westat the identities and contact information for the veterans in the second-phase sample. Westat will use the contact information to conduct data collection. The total targeted sample size will be 8,900 completed cases, which, assuming a response rate of 46 percent, will require a fielded sample size of 19,400.
Attachment 8, Sampling Overview, provides a simplified graphic overview of the sampling plan. The survey instrument includes validated questions that will identify veterans with current mental health needs. The target number of completed surveys is 4,000 such veterans—2,000 veterans who need mental health services and are using VA mental health services and 2,000 veterans who also need mental health services but have not elected to use the mental health services provided by VA.
Procedures for the Collection of Information
Tracing: VA will provide Westat with administrative data files that will include all the information needed for sampling and fielding the survey, including contact information of sample members. For all postal non-deliverables (PNDs) received, we will conduct individual-level tracing using various sources, including LexisNexis and the National Change of Address database maintained by the U.S. Postal Service.
Administration: The survey will use a multi-mode data collection approach that includes the administration of a web survey to sampled veterans followed by telephone calls to non-respondents to complete a computer-assisted telephone interview (CATI) interview. The survey questionnaire (Attachment 7) is estimated to take approximately 35 minutes to complete. Attachment 9, Data Collection Flow, provides a flowchart of the data collection methodology. Sampled veterans will first be contacted and invited to participate in the web survey via a mailed a letter (Attachment 2) with study information that contains the web survey URL and a unique PIN to access the survey. A set of frequently asked questions (FAQs) (Attachment 6) will also be included with the invitation letter, as well as a fact sheet with information about the Veterans Crisis Line. Two weeks following the initial invitation mailing, non-respondents will receive the first of three weekly reminders via U.S. postal mail encouraging their participation in the survey (Attachments 3, 4, 5). About 3,880 completed web surveys are expected assuming a 20 percent response rate to the web survey.
Beginning the 6th week of the data collection field period, all non-respondents to the web survey will move on to the telephone phase of the study where veterans will be contacted using the CATI method. During the CATI phase of data collection, for efficiency purposes, the web survey will undergo minimal revision for telephone administration by trained interviewers. We expect the CATI interviews to be in the field approximately 10 weeks in order for all cases to get through the entire calling algorithm within our scheduler. Assuming a 33 percent response rate to the CATI survey, we expect to get about 5,121 completed CATI surveys, yielding approximately 9,000 total completed surveys.
The telephone instrument version of the survey will be identical to the web version, with some format adaptations appropriate for interviewer administration and the addition of soft warnings to ensure interviewers record answers for each survey question. The instrument will be integrated with the survey management system and the call scheduler.
Call Scheduler and Calling Algorithm: Westat’s proprietary survey delivery system (SurveyBuilder) integrates a customized survey software application and management system to accommodate the large volume of respondents and the simultaneous administration of multiple surveys.
When we enter the telephone phase, telephone numbers will be loaded into the CATI database and become available to interviewers through the call scheduler. All telephone calls are automatically dialed by Westat’s auto dialer, reducing interviewer dialing time as well as eliminating errors in manual dialing a number. The scheduler ensures that cases will be called at the appropriate times, using rules developed to minimize the number of calls to any given respondent and to reduce nonresponse. For example, the week is divided into day and time “slices” through which the system moves each case in a specified pattern of call attempts. If the first call attempt is in the evening and results in no answer, the scheduler can automatically set the next call attempt for another time of day and a different day of the week. A maximum of five calls will be placed to potential participants.
Technical Support: The study’s toll-free number and project email address will be monitored by study staff to provide a prompt response to questions about the study and questionnaire (web or CATI). Westat will respond to inquiries using trained and experienced staff who can directly address issues or concerns raised by sample members or respondents. The web survey, as well as study information materials, will prominently display the toll-free number and email contact information to the survey support center. Each page on the web survey will include an email hyperlink. Voicemail and email will be available 24 hours a day, 7 days a week but staffed Monday through Friday from 9 a.m. to 5 p.m. (ET). All messages left after normal business hours will be addressed the following workday. Westat will update FAQs as needed, train staff on the types of issues that may arise, and enter all inquiries and their resolutions into the Survey Management System.
Methods to Maximize Response Rates and Deal with Nonresponse
Please see the response to Question 9 in the previous section for details about the incentive strategy. Regarding non-response, OMB guidelines require that a nonresponse bias analysis (NRBA) be conducted when the response rate for a Federally-sponsored survey is less than 80 percent, which we expect will be the case for this survey. Because one of the reasons for developing and then using analysis weights is to reduce the nonresponse bias in resulting estimates, an NRBA includes many of the analysis procedures we use in developing weights and in verifying they were calculated correctly. A nonresponse adjusted sample weight will be calculated for each respondent regardless of whether or not the respondent screened positive for needing mental health services. These weights will permit Westat and IOM project staff to estimate means, percentages, and totals from the collected data that will be representative of the population of OEF/OIF/OND veterans. The same weights can be used to calculate sub-class means for the veterans that screen positive for needing mental health services. For this purpose it will be necessary for analysts to perform domain analyses in which the domain variable is the survey outcome for screening positive.
The goal of weighting is to make the weighted survey estimates approximately unbiased for the corresponding population parameters. The weights first reflect the selection probabilities of the sampled veterans (the base weights) and then adjustments to the base weights to compensate for nonresponse and to make the weighted distributions for some key variables conform to known or well-estimated distributions for those variables. Because the proposed sample design is for a two-phase sample, the weights will be the product of a first-phase weight and a second phase weight. The first-phase weight will be the approximately 4, the reciprocal of the sampling rate for the first-phase sample, which is approximately 25 percent. The second-phase base weights will be the reciprocals of the conditional probability that a veteran was selected for the second-phase sample given that the veteran had been selected for the first-phase sample selected by VA. The VA-provided information for the veterans in the first-phase sample (contained in a subset of the 27 frame variables) will be used to adjust the second-phase base weights for nonresponse. The final weights (the product of the first- and second-phase weights) will be raked to available population totals if population control totals are available and if VA provides the first-phase weights. Lastly, if there are outliers in the final weights that would cause large losses in precision due to weight variability, they will be trimmed to reduce the variability of the weights.
Three analyses of the survey’s nonresponse properties will be carried out using a data set that contains the computed weights and the frame data provided by VA for all veterans in the first-phase sample. The first analysis will compare the second-phase survey response rates for different levels of categorical frame variables. These variables include gender, military-service branch, military-service component, rank at separation (enlisted versus officer), deployment status, and whether or not the veteran used VA-provided mental health-care services. The NRBA will include this first analysis, which will also help determine the frame variables to be used in weighting to create post-stratification cells or to rake the second-phase base weights.
The second analysis will use multiple sets of weights to compare weighted distributions of the categorical frame variables used in the first analysis. Four sets of weighted estimates will be compared:
Using data for all veterans in the first-phase sample, the estimated frequencies computed with first-phase weights (With an equal probability first-phase sample, this can be an unweighted calculation.);
Using data for all veterans sampled for the second-phase sample, the estimated frequencies computed with second-phase base weights;
Using data for all veterans responding to the second-phase sample, the estimated frequencies computed with second-phase base weights; and
Using all veterans responding to the second-phase sample, computed with final weights (that is, the product of the first and second phase weights).
This second analysis included in the NRBA will also be able to identify additional frame variables that should be used in creating nonresponse response adjustment cells. This analysis permits one to estimate the nonresponse bias in subgroup means for each variable being analyzed. Hence, this analysis allows us to check if the weighting adjustments were effective because estimated nonresponse biases for the variables used to make weighting adjustments should be essentially zero.
The third analyses will be similar to the second analysis but will compare weighted means computed from the following continuous frame variables and will also compare estimated regression coefficients for a set of models in which the dependent variable in each model is one of the following variables:
For each veteran, the total number of VA outpatient health care encounters since separation;
For each veteran, the total number of VA inpatient health care stays since separation;
For each veteran, the total number of VA outpatient mental health service encounters since separation, defined as an encounter in which the diagnosis includes an International Classification of Diseases, Ninth Revision (ICD-9) code in the subset of mental health ICD-9 codes; and
For each veteran, the total number of VA inpatient mental health service stays since separation, defined as an encounter in which the diagnosis includes an ICD-9 code in the subset of mental health ICD-9 codes.
Test of Procedures or Methods to be Undertaken
Westat implemented its protocol for conducting cognitive interviews (approved by the NAS IRB) with nine individuals from the target group. The purpose of the testing was to assess whether veterans comprehended the instructions and the questions as intended in order to complete the survey, and to gauge the sensitivity of certain questions. Based on these interviews, we made minor revisions to the wording, response options, or skip patterns for some questions; however, the time it takes to complete the survey (35 minutes) did not change as a result of the modifications.
Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
Attachment 10, Data Analyses, provides a description of the statistical analyses for the proposed data collection. The data collection and analysis plans were developed by Westat, experts in survey research, with extensive input from the IOM study committee overseeing the data collection effort. The IOM study committee is comprised of 18 members with substantial expertise in the following areas: survey and data analysis, health services research, epidemiology, biostatistics, clinical medicine, psychiatry, psychology, sociology, and other mental health professions.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Mixon, Joni |
| File Modified | 0000-00-00 |
| File Created | 2021-01-25 |