Supporting Statement _3-7-16

Evaluation of the Department of Veteran Affairs Mental Health Services-- OEF/OIF/OND Veterans’ Access to Health Services Survey

OMB No. 2900-xxxx
Estimated Burden: 35 minutes

Expiration Date: xx/xx/xxxx

The Paperwork Reduction Act of 1995: This information is collected in accordance with section 3507 of the Paperwork Reduction Act of 1995. Accordingly, we may not conduct or sponsor and you are not required to respond to, a collection of information unless it displays a valid OMB number. We anticipate that the time expended by all individuals who complete this survey will average 35 minutes. This includes the time it will take to follow instructions, gather the necessary facts and respond to questions asked. The purpose of this web-based survey is to help VA to better understand why Veterans choose to use or not use VA mental health services available to them. The survey results will lead to improvements in the quality of service delivery by helping to improve Veterans’ access to VA mental health services. Participation in this survey is voluntary and failure to respond will have no impact on benefits to which you may be entitled.

Evaluation of the Department of Veterans Affairs Mental Health Services

Supporting Material for OMB Clearance

OMB No. 2900-XXXX

1. Justification

1. Circumstances Making the Collection of Information Necessary

This is a congressionally-mandated research study to evaluate mental health services provided by the Department of Veteran Affairs (VA). Congress directed the VA to conduct a survey of veterans with assistance from the Institute of Medicine (IOM) of the National Academies. Attachment 1 contains the authorizing legislation, the National Defense Authorization Act for Fiscal Year 2013, Section 726.

Following the large number of deployments and operations in Iraq and Afghanistan, the number of military members with mental health problems has been rising. All Veterans who need mental health services do not seek them it or receive them from the Department of Veterans Affairs (VA) health care system. This study is to assess barriers to receiving mental health care services among veterans.

2. Purpose and Use of the Information Collection

The purpose of survey is to help the VA to better understand unmet needs and barriers to receiving care among veterans of Operation Enduring Freedom, Operation Iraqi Freedom, and Operation New Dawn (OEF/OIF/OND). The VA can use the results to improve veteran access to high-quality mental health care services.

3. Use of Improved Information Technology and Burden Reduction

To carry out the survey task, IOM contracted with Westat, a highly-qualified research company, with years of experience conducting large-scale surveys using state-of-the-art information technology for web-based surveys. For this study, a web survey will be accessible across a wide array of computer configurations. The survey application will also be programmed to work more effectively (reduce potential respondent error) when accessed by a tablet or smartphone. The web survey will be tested on multiple operating systems including Windows XP, Windows 7, Windows 8, and Mac OS X. The survey will also be tested across multiple browsers including Internet Explorer 8, 10, and 11; Firefox 26 and 30; Google Chrome 36; Opera 10; and Safari 6. In addition to desktop PCs and browsers, the web survey will be tested on mobile devices, including the iPhone 4, 5, and 6, iPad, Nexus 7, and Windows Surface. Westat will secure a domain name (URL) for the study, which will be register as a “.org” site to ensure the project retains all rights to this name for the study’s duration.

All nonrespondents to the web survey will move on to the telephone phase of the study where veterans will be contacted using the computer-assisted telephone interviewing (CATI) method.

The Survey Management System (SMS) is Westat’s data management system for the study that includes automated systems for managing and tracking respondent information and survey completions. It also includes systems to support the processing of web survey and CATI data. The SMS will be designed to allow study staff to upload and manage the sample of veterans; log and manage any incoming calls/emails; receive postal non-deliverables (PNDs); assign case disposition codes to cases such as complete, refused, deceased, PNDs, and ineligible; monitor and track survey completion status by mode; and provide data extraction tools to support data analysis, reporting, and delivery.

4. Efforts to Identify Duplication and Use of Similar Information

A comprehensive literature review performed by IOM staff did not reveal other studies that have assessed the same research questions, and that have collected and analyzed data in the same manner proposed in this survey effort.

5. Impact on Small Businesses or Other Small Entities

We are not aware of any potential impact on small businesses or other small entities.

6. Consequences of Collecting the Information Less Frequently

This data collection effort is mandated by Congress. It is a one-time data collection effort that is scheduled to begin in early 2016 and end in summer 2016.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances relating to the general requirements cited in 5 CFR 1320.5.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

We are not aware of any comments at this time.

9. Explanation of Any Payment or Gift to Respondents

The focus of this survey, mental health services, is a subject that people generally do not want to engage in because of perceived stigma associated with mental health problems. Therefore, it is imperative we use available resources and evidence-based survey practices to encourage participation and achieve the highest response rates possible. Research unequivocally demonstrates that incentives improve response rates.

We will mail a $2 pre-incentive in the first postal mailing to all sampled veterans. Also, a post-incentive will be given to respondents who complete the survey. Our data collection strategy includes a progressive incentive scheme that will increase the incentive amount towards the end of the field period. Westat reviewed the literature on incentive schemes and concluded that implementing this strategy offers two possible advantages: (1) to increase the overall response rate and (2) to improve the response rate among the non-user population; a key objective of this research is to understand why veterans who are eligible for VA services elect not to use VA for mental health care. We expect that Veterans who are classified as users of services will be more willing to cooperate on the survey and will be disproportionately represented among the early responders. By increasing the incentive amount towards the end of the field period, we might have more success in getting completes from non-users. A related advantage is that this design may increase the opportunity to assess non-response bias.

It is important to significantly increase the incentive amount to motivate respondents who have not been cooperative. To this end, our data collection strategy includes starting with a $5 promised incentive through the start of the CATI phase of data collection, and increasing it to $20 if we are not on target to meet our desired number of completes. Our planned field period for data collection includes a 10 week CATI phase. Two weeks after the start of the CATI phase, we will assess the number of completed interviews obtained by then, and project at what point during the remaining 8 weeks of the CATI phase we will implement the progressive incentive scheme if we are not on track to reach our target number of completes. At that point, CATI interviewers will mention the $20 incentive amount when introducing the survey and when leaving an answering machine message for the first time. This progressive scheme is included as part of our planned effort to maximize response rates, and will also be used to assess non-response bias, assuming that the less cooperative respondents resemble those that never respond at all. We will be able to assess whether those who are only motivated by the larger incentive differ significantly from those that cooperate early.

We will implement a tracking system within our Survey Management System for managing all activities associated with the incentive task. These activities will include receipt and safeguarding of cash, inserting the incentives into postal mailings and tracking them, and receipt/processing of incentives returned with post non-deliverables.

10. Assurance of Confidentiality Provided to Respondents

The National Academy of Sciences’ Institutional Review Board approved the survey research protocol on December 18, 2015. On February 2, 2016, the National Institute of Mental Health issued a Confidentiality Certificate, protecting the identity of research subjects in the project. The Certificate reference number is CC-MH-16-158 and it expires on 12/31/2019 (see Attachment 12).

The proposed collection of information is voluntary and information will be kept confidential. Westat has extensive experience conducting data collection in a secure environment. This study will be conducted within Westat’s FISMA HIGH data security enclave, which has been approved by the VA.

The survey materials provide respondents with assurances of the voluntary nature of participation and the confidentiality of respondent information as follows.

The letter inviting veterans to participate in the survey (Attachment 2) states:

“THE SURVEY IS CONFIDENTIAL. The information you provide on the survey will be combined with administrative data provided by the VA about health services you may have used. Results of the survey and administrative data will be reported in a way that does not identify you or any individuals who participated in the survey.”

“Your participation is voluntary. Refusal to participate will involve no penalty or loss of benefits to which you are otherwise entitled, and you may discontinue participation at any time without penalty or loss of benefits to which you are otherwise entitled.”

The three reminders letters (Attachments 3, 4, 5) have statements that are similar to those in the invitation letter.

A set of Frequently Asked Questions (Attachment 6), which will be mailed with the invitation letter and the reminder letters states:

“Will my information be kept confidential? Yes. We will do everything we can to keep all data confidential including your survey responses and administrative data that Westat, our contractor, receives from the VA about health services you may have used. Only researchers at Westat and IOM-appointed experts who are approved to work on this study and who have signed an agreement to keep all data confidential will have access to individual survey and administrative data for analysis purposes. Westat will provide the VA with the survey responses, but will have deleted your name and any other information that could be used to identify you. The IOM will release a publicly available report in 2017. When reporting the results of this study, all information about you will be combined with information from other Veterans, and only group statistics will be reported. We will not disclose your responses or data to anyone who could use it to identify you or any other participants. Westat will destroy all data in its possession no later than one year after the study has been completed or, if the VA requests additional analysis, after that analysis has been completed.”

“To further help us protect your privacy, we have obtained a Certificate of Confidentiality from the United States Department of Health and Human Services (DHHS). With this Certificate, we cannot be forced (for example by court order or subpoena) to disclose information that may identify you in any federal, state, local, civil, criminal, legislative, administrative, or other proceedings. The researchers will use the Certificate to resist any demands for information that would identify you, except to prevent serious harm to you or others, and as explained below. You should understand that a Certificate of Confidentiality does not prevent you, or a member of your family, from voluntarily releasing information about yourself, your family, or your involvement in this study. If an insurer or employer learns about your participation, and obtains your consent to receive research information, then we may not use the Certificate of Confidentiality to withhold this information. This means that you and your family must also actively protect your own privacy. You should understand that we will in all cases, take the necessary action, including reporting to authorities, to prevent serious harm to yourself, children, or others. A Certificate of Confidentiality does not represent an endorsement of the research study by the Department of Health and Human Services or the National Institutes of Health.”

“Am I required to complete this survey? No. You do not have to complete the survey; participation is entirely voluntary and there are no penalties for refusing to answer any question or stopping at any time. However, this is an important study and we hope you will want to participate. Your responses are important. Understanding the experiences and perspectives of Veterans is important for making improvements in the mental health services provided by the VA.”

“Will my VA benefits be affected if I decide not to participate? No. Refusal to participate will involve no penalty or loss of benefits to which you are otherwise entitled, and you may discontinue participation at any time without penalty or loss of benefits to which you are otherwise entitled. However, by participating in this study and telling us about your experiences, you (and other Veterans who participate) may help to improve the mental health services provided by the VA.”

The first page of the questionnaire (Attachment 7) states:

“VOLUNTARY RESPONSE/CONFIDENTIALITY: Your participation is voluntary. Refusal to participate will involve no penalty or loss of benefits to which you are otherwise entitled, and you may discontinue participation at any time without penalty or loss of benefits. We will do everything we can to keep all data confidential including your survey responses and administrative data that Westat, our contractor, receives from the VA about health services you may have used. Only researchers at Westat and IOM-appointed experts who are approved to work on this study and who have signed an agreement to keep all data confidential will have access to individual survey and administrative data for analysis purposes. Westat will provide the VA with the survey responses, but will have deleted your name and any other information that could be used to identify you. The IOM will release a publicly available report in 2017. When reporting the results of this study, all information about you will be combined with information from other Veterans, and only group statistics will be reported. We will not disclose your responses or data to anyone who could use it to identify you or any other participants. Westat will destroy all data in its possession no later than one year after the study has been completed or, if the VA requests additional analysis, after that analysis has been completed.”

“To further help us protect your privacy, we have obtained a Certificate of Confidentiality from the United States Department of Health and Human Services (DHHS). With this Certificate, we cannot be forced (for example by court order or subpoena) to disclose information that may identify you in any federal, state, local, civil, criminal, legislative, administrative, or other proceedings. The researchers will use the Certificate to resist any demands for information that would identify you, except to prevent serious harm to you or others, and as explained below. You should understand that a Certificate of Confidentiality does not prevent you, or a member of your family, from voluntarily releasing information about yourself, your family, or your involvement in this study. If an insurer or employer learns about your participation, and obtains your consent to receive research information, then we may not use the Certificate of Confidentiality to withhold this information. This means that you and your family must also actively protect your own privacy. You should understand that we will in all cases, take the necessary action, including reporting to authorities, to prevent serious harm to yourself, children, or others. A Certificate of Confidentiality does not represent an endorsement of the research study by the Department of Health and Human Services or the National Institutes of Health.”

For the CATI portion of the survey:

The survey will include the use of a script (Attachment 11) that states exactly what the interviewer reads to the sample member during the consent process, thus ensuring that sample members receive the necessary information about the study purpose, the voluntary nature of their participation, and protection of their confidential information. The CATI consent statement uses the same wording that appears in the web-based questionnaire.

11. Justification for Sensitive Questions

The survey questionnaire includes a subset of questions asking the veteran about his/her mental health and well-being, as well questions about wartime experiences. These questions are necessary to determine the factors that underlie veterans’ mental health needs and their use of services. In the web survey, respondents are directed to click the “Mental Health Resource” button located at the bottom of each page if they feel a need for assistance. The button links to resources for follow-on referrals to counseling hotlines and services, such as the Veterans Crisis Line. Westat’s survey support staff and CATI data collectors will be trained to follow a distressed caller protocol in the event that a sample member or respondent becomes seriously distressed during a telephone interview or survey support center interaction. The protocol includes use of available resources for follow-on referrals to counseling hotlines and services for veterans.

12. Estimates of Annualized Burden Hours and Costs

The table provides the number of respondents, frequency of responses, and annual hour burden (5,192 hours) for the survey data collection.

Study Component	Form	No. of respondents	x No. of responses	x No. of minutes	÷ by 60 =	Annual Burden Hours
Survey	Survey	8,900	x 1 = 8,900	x 35 = 311,500		5,192
TOTAL		8,900				5,192

13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

There is no additional burden.

14. Annualized Cost to the Federal Government

The amount is $0.

15. Explanation for Program Changes or Adjustments

Not applicable; this is an initial submission.

16. Plans for Tabulation and Publication and Project Time Schedule

The project time schedule for this congressionally-mandated study is set by a contract between IOM and VA and must be completed on time. Survey data collection will begin as soon as OMB approval is received. Survey implementation will occur over a four-month period, and subsequent data analysis and report writing will occur over a six-month period. The draft final survey report is scheduled to be completed by summer 2016. The survey is one of several tasks under this evaluation. The final report for the overall study with survey results, as well as findings from site visits and a comprehensive literature review, will be released to the VA, Congress, and the public in early summer 2017.

17. Reason(s) Display of OMB Expiration Date is Inappropriate

It is appropriate for the proposed data collection materials to display the OMB expiration date.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

We are not requesting any exceptions to certification under the Paperwork Reduction Act (PRA). This package is our submission to comply with the PRA.

2. Statistical Methods (used for collection of information employing statistical methods)

1. Respondent Universe and Sampling Methods

A two-phase sample design will be employed for the survey of OEF/OIF/OND veterans. In the first phase of sampling, the VA will select a random sample of approximately 25 percent of all OEF/OIF/OND veterans, yielding a sample of approximately 150,000 veterans. Through a Data Use Agreement, the VA will provide Westat with a data file containing a non-personally identifiable information (PII) identifier for each veteran in the first-phase sample, along with other non-PII variables—such as, age, gender, military-service characteristics, and utilization of VA health care services—which Westat will use to stratify the first-phase sample and then select a stratified second-phase sample. All of Westat’s processing to select the second-phase sample will be conducted in Westat’s secure FISMA High Enclave (FHE) enclave with access given only to Westat project staff. The identifiers for the second-phase sample will be provided to VA, who will provide back to Westat the identities and contact information for the veterans in the second-phase sample. Westat will use the contact information to conduct data collection. The total targeted sample size will be 8,900 completed cases, which, assuming a response rate of 46 percent, will require a fielded sample size of 19,400.

Attachment 8, Sampling Overview, provides a simplified graphic overview of the sampling plan. The survey instrument includes validated questions that will identify veterans with current mental health needs. The target number of completed surveys is 4,000 such veterans—2,000 veterans who need mental health services and are using VA mental health services and 2,000 veterans who also need mental health services but have not elected to use the mental health services provided by VA.

2. Procedures for the Collection of Information

Tracing: VA will provide Westat with administrative data files that will include all the information needed for sampling and fielding the survey, including contact information of sample members. For all postal nondeliverables (PNDs) received, we will conduct individual-level tracing using various sources, including LexisNexis and the National Change of Address database maintained by the U.S. Postal Service.

Administration: The survey will use a multi-mode data collection approach that includes the administration of a web survey to sampled veterans followed by telephone calls to nonrespondents to complete a computer-assisted telephone interview (CATI) interview. The survey questionnaire (Attachment 7) is estimated to take approximately 35 minutes to complete. Attachment 9, Data Collection Flow, provides a flowchart of the data collection methodology. Sampled veterans will first be contacted and invited to participate in the web survey via a mailed a letter (Attachment 2) with study information that contains the web survey URL and a unique PIN to access the survey. A set of frequently asked questions (FAQs) (Attachment 6) will also be included with the invitation letter, as well as a fact sheet with information about the Veterans Crisis Line. Two weeks following the initial invitation mailing, nonrespondents will receive the first of three weekly reminders via U.S. postal mail encouraging their participation in the survey (Attachments 3, 4, 5). About 3,880 completed web surveys are expected assuming a 20 percent response rate to the web survey.

Beginning the 6th week of the data collection field period, all nonrespondents to the web survey will move on to the telephone phase of the study where veterans will be contacted using the CATI method. During the CATI phase of data collection, for efficiency purposes, the web survey will undergo minimal revision for telephone administration by trained interviewers. We expect the CATI interviews to be in the field approximately 10 weeks in order for all cases to get through the entire calling algorithm within our scheduler. Assuming a 33 percent response rate to the CATI survey, we expect to get about 5,121 completed CATI surveys, yielding approximately 9,000 total completed surveys.

The telephone instrument version of the survey will be identical to the web version, with some format adaptations appropriate for interviewer administration and the addition of soft warnings to ensure interviewers record answers for each survey question. The instrument will be integrated with the survey management system and the call scheduler.

Call Scheduler and Calling Algorithm: Westat’s proprietary survey delivery system (SurveyBuilder) integrates a customized survey software application and management system to accommodate the large volume of respondents and the simultaneous administration of multiple surveys.

When we enter the telephone phase, telephone numbers will be loaded into the CATI database and become available to interviewers through the call scheduler. All telephone calls are automatically dialed by Westat’s autodialer, reducing interviewer dialing time as well as eliminating errors in manual dialing a number. The scheduler ensures that cases will be called at the appropriate times, using rules developed to minimize the number of calls to any given respondent and to reduce nonresponse. For example, the week is divided into day and time “slices” through which the system moves each case in a specified pattern of call attempts. If the first call attempt is in the evening and results in no answer, the scheduler can automatically set the next call attempt for another time of day and a different day of the week. A maximum of five calls will be placed to potential participants.

Technical Support: The study’s toll-free number and project email address will be monitored by study staff to provide a prompt response to questions about the study and questionnaire (web or CATI). Westat will respond to inquiries using trained and experienced staff who can directly address issues or concerns raised by sample members or respondents. The web survey, as well as study information materials, will prominently display the toll-free number and email contact information to the survey support center. Each page on the web survey will include an email hyperlink. Voicemail and email will be available 24 hours a day, 7 days a week but staffed Monday through Friday from 9 a.m. to 5 p.m. (ET). All messages left after normal business hours will be addressed the following workday. Westat will update FAQs as needed, train staff on the types of issues that may arise, and enter all inquiries and their resolutions into the Survey Management System.

3. Methods to Maximize Response Rates and Deal with Nonresponse

Please see the response to Question 9 in the previous section for details about the incentive strategy. Regarding non-response, OMB guidelines require that a nonresponse bias analysis (NRBA) be conducted when the response rate for a Federally-sponsored survey is less than 80 percent, which we expect will be the case for this survey. Because one of the reasons for developing and then using analysis weights is to reduce the nonresponse bias in resulting estimates, an NRBA includes many of the analysis procedures we use in developing weights and in verifying they were calculated correctly. A nonresponse adjusted sample weight will be calculated for each respondent regardless of whether or not the respondent screened positive for needing mental health services. These weights will permit Westat and IOM project staff to estimate means, percentages, and totals from the collected data that will be representative of the population of OEF/OIF/OND veterans. The same weights can be used to calculate sub-class means for the veterans that screen positive for needing mental health services. For this purpose it will be necessary for analysts to perform domain analyses in which the domain variable is the survey outcome for screening positive.

The goal of weighting is to make the weighted survey estimates approximately unbiased for the corresponding population parameters. The weights first reflect the selection probabilities of the sampled veterans (the base weights) and then adjustments to the base weights to compensate for nonresponse and to make the weighted distributions for some key variables conform to known or well-estimated distributions for those variables. Because the proposed sample design is for a two-phase sample, the weights will be the product of a first-phase weight and a second phase weight. The first-phase weight will be the approximately 4, the reciprocal of the sampling rate for the first-phase sample, which is approximately 25 percent. The second-phase base weights will be the reciprocals of the conditional probability that a veteran was selected for the second-phase sample given that the veteran had been selected for the first-phase sample selected by VA. The VA-provided information for the veterans in the first-phase sample (contained in a subset of the 27 frame variables) will be used to adjust the second-phase base weights for nonresponse. The final weights (the product of the first- and second-phase weights) will be raked to available population totals if population control totals are available and if VA provides the first-phase weights. Lastly, if there are outliers in the final weights that would cause large losses in precision due to weight variability, they will be trimmed to reduce the variability of the weights.

Three analyses of the survey’s nonresponse properties will be carried out using a data set that contains the computed weights and the frame data provided by VA for all veterans in the first-phase sample. The first analysis will compare the second-phase survey response rates for different levels of categorical frame variables. These variables include gender, military-service branch, military-service component, rank at separation (enlisted versus officer), deployment status, and whether or not the veteran used VA-provided mental health-care services. The NRBA will include this first analysis, which will also help determine the frame variables to be used in weighting to create post-stratification cells or to rake the second-phase base weights.

The second analysis will use multiple sets of weights to compare weighted distributions of the categorical frame variables used in the first analysis. Four sets of weighted estimates will be compared:

• Using data for all veterans in the first-phase sample, the estimated frequencies computed with first-phase weights (With an equal probability first-phase sample, this can be an unweighted calculation.);

• Using data for all veterans sampled for the second-phase sample, the estimated frequencies computed with second-phase base weights;

• Using data for all veterans responding to the second-phase sample, the estimated frequencies computed with second-phase base weights; and

• Using all veterans responding to the second-phase sample, computed with final weights (that is, the product of the first and second phase weights).

This second analysis included in the NRBA will also be able to identify additional frame variables that should be used in creating nonresponse response adjustment cells. This analysis permits one to estimate the nonresponse bias in subgroup means for each variable being analyzed. Hence, this analysis allows us to check if the weighting adjustments were effective because estimated nonresponse biases for the variables used to make weighting adjustments should be essentially zero.

The third analyses will be similar to the second analysis but will compare weighted means computed from the following continuous frame variables and will also compare estimated regression coefficients for a set of models in which the dependent variable in each model is one of the following variables:

• For each veteran, the total number of VA outpatient health care encounters since separation;

• For each veteran, the total number of VA inpatient health care stays since separation;

• For each veteran, the total number of VA outpatient mental health service encounters since separation, defined as an encounter in which the diagnosis includes an International Classification of Diseases, Ninth Revision (ICD-9) code in the subset of mental health ICD-9 codes; and

• For each veteran, the total number of VA inpatient mental health service stays since separation, defined as an encounter in which the diagnosis includes an ICD-9 code in the subset of mental health ICD-9 codes.

4. Test of Procedures or Methods to be Undertaken

Westat implemented its protocol for conducting cognitive interviews (approved by the NAS IRB) with nine individuals from the target group. The purpose of the testing was to assess whether veterans comprehended the instructions and the questions as intended in order to complete the survey, and to gauge the sensitivity of certain questions. Based on these interviews, we made minor revisions to the wording, response options, or skip patterns for some questions; however, the time it takes to complete the survey (35 minutes) did not change as a result of the modifications.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Attachment 10, Data Analyses, provides a description of the statistical analyses for the proposed data collection. The data collection and analysis plans were developed by Westat, experts in survey research, with extensive input from the IOM study committee overseeing the data collection effort. The IOM study committee is comprised of 18 members with substantial expertise in the following areas: survey and data analysis, health services research, epidemiology, biostatistics, clinical medicine, psychiatry, psychology, sociology, and other mental health professions.