DATE: August 20, 2020
MEMORANDUM TO: Tom Garin
National Center for Veterans Analysis and Statistics
Department of Veterans Affairs
Luke Murren
Veterans’ Employment and Training Service
Department of Labor
Dori Allard
OEUS Division of Labor Force Statistics
Bureau of Labor Statistics
MEMORANDUM FROM: Erica Yu and Struther Van Horn
Office of Survey Methods Research
Bureau of Labor Statistics
SUBJECT: Recommended revisions to the Veterans Supplement to the CPS
______________________________________________________________________________
Introduction
The Office of Survey Methods Research (OSMR) was asked to review and cognitively test a set of proposed questions to be added to the Current Population Survey (CPS) 2021 Veterans Supplement (Supplement) This report summarizes the results of OSMR’s expert review and cognitive interview data collection from research participants.
Supplement stakeholders, the Department of Veterans Affairs (VA) and the Department of Labor’s Veterans’ Employment and Training Service (DOL VETS) proposed a set of questions that target new concepts that were not measured in previous versions of the Supplement. Full documentation of the proposed questions and the stakeholder motivations for collecting each piece of information are included in Appendix 1. The main goals can be summarized in the following points:
What are the perceived causes of veteran unemployment and underemployment?
What DoD-VA, state workforce system, or other organization benefits and services are veterans using to address unemployment and underemployment?
How do veterans perceive the transition to civilian employment and the services designed to help them through transition?
What motivates key transition decisions like type of job, benefits usage, and location?
How is military spouses’ employment affected, and how do spouses affect veterans’ employment decisions?
How does a veteran’s service-connected disability affect civilian employment?
What demographics such as rank, education, and military occupational specialty code help explain different post-service employment outcomes?
Given the complexity of these concepts, cognitive testing is important. Results from cognitive testing can reduce measurement error by informing revisions to question wording and question order as well as instructions to both respondents and interviewers. Additionally, review by OSMR staff experienced with questionnaire design can make cognitive testing studies more efficient by addressing basic question design issues before testing with research participants and designing probes to understand causes for response difficulties.
The goals of this study were to:
Explore whether the questions successfully measure the target concepts and, if not, develop new questions.
Determine whether the language used in the questionnaire works across different branches of the armed forces and time periods that veterans may have served.
Determine whether questions are appropriate to be asked of all veterans and, if not, identify appropriate sub-groups or develop screening questions.
Determine if introductory or transition language is needed to explain survey goals and reduce respondent concerns, such as concerns about confidentiality.
Determine what interviewer instructions are necessary to explain key concepts.
Test minor wording changes to the questions that had already been administered in previous Supplements.
Several issues emerged at the start of the study that warranted substantial initial revisions to the proposed questions. First, stakeholders raised concerns that the length of the proposed questions exceeded the allowed length of the Supplement of approximately 10 minutes. It would not be possible to ask all of the proposed questions within the Supplement time limit.
Second, upon our initial review of the proposed questions, OSMR raised concerns about the breadth and complexity of the proposed research goals. It would not be possible to accurately measure the proposed concepts within the Supplement time limit nor sufficiently cognitively test the concepts within the timeframe of the study.
Third, shortly before data collection was to begin, COVID-19 limitations on in-person data collection methods reduced OSMR’s ability to conduct in-person data collection. Telephone interviews were the only method available and we were unable to offer incentive payments. We anticipated that these data collection changes would limit the number and diversity of participants we could recruit as well as the speed of recruitment. Additionally, data collection was delayed due to the required re-submission of the information collection request submitted to the Office of Management and Budget to account for the COVID-19-imposed changes to data collection methods.
Given these issues, OSMR reduced the scope of the changes to the Supplement to concentrate our efforts on a subset of items that we expected to be able to cognitively test effectively. Attempting to address all of the proposed revisions and new questions would likely have resulted in insufficient evidence for recommendations for most newly proposed items.
This report summarizes OSMR’s review and findings from cognitive interviews with research participants1. We developed recommendations for a final questionnaire based on our understanding of stakeholder goals from initial development conversations and interim feedback from a stakeholder meeting in May 2020 and final review conversations in July 2020. A high-level summary of the scope of the final recommendations compared to the original proposed questionnaire can be found in Appendix 2. Stakeholders should review these findings and justifications to understand our recommendations. After these recommendations were shared and discussed with stakeholders, questionnaire requirements were sent to the Census Bureau for instrument development. Additional changes to the questionnaire may be made by Census and not reflected in this report.
Methods
Expert Review
Two researchers reviewed the proposed questions and identified which items were beyond the scope of this study and which items needed revision before testing with research participants. The researchers considered issues such as whether an item adequately measured the target concepts and the potential for generating feelings of sensitivity, which could result in non-response or poor data quality. In addition, the researchers conducted early stage scoping interviews with three research participants. During these interviews, the researchers solicited feedback on the target concepts to guide questionnaire and instrument development. Expert review findings and revisions are summarized as part of our recommendations within each section.
Cognitive Interviews
Design
As this was a cognitive testing study, we made iterative revisions to survey questions during data collection. For some issues, feedback from one participant was enough evidence to indicate a revision. For other issues, we waited until we saw consistent feedback from several participants before making a revision.
After the first round of data collection (25 interviews), we summarized the preliminary findings and shared them with stakeholders for feedback. This input was then used to revise the questions for the final round of interviews (38 interviews). After the final interviews, we submitted our recommendations for the questionnaire and shared them with stakeholders for further feedback. That input was then used to revise the questions a final time. Additional changes to the questionnaire may need to be made by Census and not reflected in this report.
Participants
A total of 63 participants were recruited for cognitive interviews from three different sources:
Agency Veteran Employment Program Managers, working with the Veterans Services organization within the Office of Personnel Management, sent an email to known veterans in the federal government asking for volunteers. The message described the research study goals and sponsorship, and asked veterans to e-mail BLS if they were interested in participating. This recruitment method was nationwide but reached only veterans employed by the federal government. A total of 26 participants was recruited using this method.
A “quick poll” was sent to members of the Veteran Insights Panel, the online non-probability panel of the Veterans Health Administration. The survey described the research study goals, sponsors, and asked volunteers to submit their name and contact information for BLS to contact them with further information. This recruitment method was nationwide but reached only veterans who had volunteered to participate in the ongoing VHA online panel. A total of 35 participants were recruited using this method.
Advertisements were posted on Craigslist in the “labor gigs’ or “domestic gigs’ sections in Dallas, TX, Los Angeles, CA, Orlando, FL, and Washington, DC. The advertisements described the research study topic and asked for volunteers to contact OSMR for information. Two participants were recruited using this method.
At the time of contact with the OSMR recruiter, participants were screened for the characteristics being targeted in the study, including separation date, service-connected disability, National Guard or Reserve status, retired from the armed forces, and whether they lived with a veteran and could answer on the veteran’s behalf as a proxy respondent. Recruitment priorities for these characteristics changed over the data collection period as sub-group targets were reached.
Below is a summary of the characteristics of the participants interviewed for this study:
Characteristic |
Count |
|
Characteristic |
Count |
Sex |
|
|
Branch of armed forces |
|
Male |
46 |
|
Air Force |
10 |
Female |
17 |
|
Army |
30 |
|
|
|
Coast Guard |
4 |
Race |
|
|
Marine Corps |
8 |
White |
44 |
|
Navy |
11 |
Black or African American |
17 |
|
|
|
Asian |
1 |
|
Period of service, selected intervals |
|
Refused |
1 |
|
Vietnam Era (Aug 1964 to April 1975) |
21 |
|
|
|
May 1975 to July 1990 |
6 |
Age, mean |
57.1 years |
|
August 1990 to August 2001 |
11 |
|
|
|
September 2001 or later |
28 |
Education, median |
Bachelor’s degree |
|
|
|
|
|
|
Rank |
|
Employment status |
|
|
Enlisted |
46 |
Employed |
37 |
|
Officer |
17 |
Unemployed |
5 |
|
|
|
Retired |
16 |
|
Separation date |
|
Disabled |
5 |
|
Within the last 10 years |
19 |
|
|
|
|
|
|
|
|
National Guard or Reserve |
|
|
|
|
Ever served |
29 |
|
|
|
|
|
|
|
|
Length of active duty service, median |
6-9 years |
|
|
|
|
|
|
|
|
Service-connected disability |
|
|
|
|
Has rating |
51 |
|
|
|
Currently processing |
4 |
The sample yielded good coverage of issues related to disabilities, with a high number of participants reporting a service-connected disability, with disability ratings ranging from 0% to 100%, and participants waiting to receive a rating. The sample also yielded good coverage of issues related to National Guard and Reserve service, with a high number of participants reporting having served in the National Guard or Reserve.
All branches of the armed forces were represented, with participants from both officer and enlisted ranks. Most time periods of service were also represented, though there was a higher than expected number of participants who had served during the Vietnam Era. These participants may have different experiences than veterans who served during different eras.
Relatedly, the sample skewed older, with most separation dates more than 10 years ago. Education levels were also high, with 31 participants reporting a Master’s or doctorate degree. Feedback from these participants may be different from what we would have learned from a younger or less-educated sample.
The sample did not yield good coverage of employment status, with little representation of unemployed veterans. Further, many of the veterans worked for the federal government. Feedback from these participants may be different than veterans who are not employed or choose not to work for the government.
The sample is also limited by the low number of proxy respondents. Further, the feedback we did collect from proxies is from participants who are veterans themselves and knowledgeable about military and transition issues and military terminology. Feedback from these participants may be different than other household proxy respondents who did not serve in the military.
Procedures
Cognitive interviews were conducted remotely over the telephone between May 4 and June 5, 2020. During each interview, the researcher described the purpose of the study, obtained informed consent, administered selected CPS Basic questions and the full Supplement, and then debriefed the research participant. Debriefings varied by individual and focused on the Supplement items that the individual participant seemed to have had difficulty with or that were relevant to targeted characteristics about the individual, such as a disability that affected a job or separation within the last 10 years.
Results
Findings from the expert review and interview data collection are presented by section of the Supplement: Time in Military, Disability, Transition to Civilian Employment, Training, State Workforce Agency, and Benefits. These recommended section groupings differ from the original proposed sections. These revisions were based on iterative changes made during testing to improve coherence and flow of the items.
Each section includes a discussion of:
Initial revisions based on expert review
Major issues uncovered during data collection
Summary of iterative revisions made during data collection
Summary of feedback from stakeholders
Recommended question wording or other revision(s)
Summary of outstanding limitations and considerations for future research
Supplement Universe
Overall, the stakeholders from DOL and VA agreed that the main focus of the Supplement is on the experience of transition to civilian employment. Given this focus, we proposed asking the majority of the Supplement of only that sub-group of veterans who transitioned after 2010. Selected items, such as the items collecting service in particular combat theaters, must be asked of all veterans. This revision is shown in Appendix 3 alongside other instructions regarding universe.
Time in Military Section
Expert Review
Prior to cognitive testing, we identified question ordering as our main concern. Items about the veteran’s time in military and other service details were spread throughout the Supplement despite being related more closely to each other than to other items. To ease the cognitive burden on respondents, we revised the question order in this section and brought in items that had originally been proposed for other sections.
During review of the Supplement questionnaire, we learned that veterans who are currently in the armed forces (per their response to PEAFNOW) are excluded from the Supplement. Given that these veterans should not be in the Supplement survey at all, we recommend using their answers to the existing item “Are you currently a member of the Reserve or National Guard?” (Time in Military, Item S2) to route respondents to exit the Supplement.
Below is a summary of the concerns identified and revisions made during expert review:
Potential sensitivity of starting with questions related to war and combat zones
Beginning the Supplement with items about war and combat zones may lead respondents to react to the content of those items and believe the survey will be focused on potentially sensitive items dealing with those subjects. To address this concern, we changed the question order of the supplement to start with a gentler and potentially easier question to answer.
Original Proposed Wording |
Revised |
Item 1: Were you on active duty in Vietnam, Laos, or Cambodia; in the waters in or around these countries; or did you fly missions over these areas at any time between February 28, 1961 and May 7, 1975? |
Item 1: Have you ever been a member of the Reserve or National Guard? |
Scattered questions about service
In the original proposed questionnaire, items related to the veteran’s service in the military appeared in multiple sections despite being related to each other. To ease respondent burden, we consolidated items in the “Time in Military” section (“What was your rank at separation?” and “What was your primary MOS?”), re-ordered National Guard and Reserve items to appear together, and re-ordered combat zone items to appear together. We deliberately began the combat zone series with the general item about any service, so as to reduce the potential for redundancy when answering yes to the specific geographic area items.
Original Proposed Wording |
Revised |
Items about service appear throughout “Time in Military”, “Transition”, and “Employment” |
Items about service appear in “Time in Military” only |
Low value of distinguishing between Reserve and National Guard
We could not identify any analytic purpose in asking veterans to distinguish between being in the Reserve or National Guard. Further, the cell sizes for each group are likely to be small, which may limit usage of the data.
Original Proposed Wording |
Revised |
Was that the Reserve or National Guard? |
Drop item |
Ambiguity as to which MOS to report
Given that service members can, and often do, have more than one specialty (simultaneously or over time), respondents may have difficulty knowing which occupational specialty to report. The item should instruct respondents how to choose which specialty to report. We modified the item to refer to “primary” MOS, expecting that this term allowed respondents the flexibility to choose the MOS they felt was most important.
Original Proposed Wording |
Revised |
What was your military occupational specialty? |
What was your primary military occupational specialty? |
Interview debriefings focused on whether veterans who had served in the Reserves or National Guard would have difficulty answering questions about their time in the military. Probes included how being in the Reserve/National Guard affected calculations for length of active service and year of last active service. We were also anticipated that “active duty” time calculations may not be consistent for these Reserve/National Guard participants and probed on this item as well. Other probes focused on how veterans define and report their “primary” military occupation specialty (MOS) and how veterans report their rank at separation.
Cognitive Interviews
All 63 participants were asked the items in the “Time in Military” section. At least 3 participants indicated that they would not respond to some items in this section, due to their information, such as MOS or combat/war zone questions, as still being classified. Overall, the feedback from the participants was positive for this section. The majority of participants were able to answer the questions related to their time in the military and felt that the questions were relatively easy to answer and not burdensome or sensitive in nature. The concerns that emerged from the feedback from participants are summarized below:
Participants did not understand the survey goals
During the initial rounds of cognitive testing, no formal introduction to the Supplement was provided to participants. Early feedback from the participants included questioning what the purpose of collecting the data was and how the data were going to be used. To address this concern, we modified the wording of the introduction from the 2019 Supplement. After this introduction was added, several participants noted that they didn’t “see" the point of the questions and that the questions weren’t written "for" veterans using plain language. Older veterans consistently noted that this introduction gave a misleading explanation about the kinds of questions that they would be asked, given that they were not in the universe for most employment and transition questions. Several participants also mentioned that the introduction sounded very much like “legal jargon”. To address this feedback, further edits were made to the introduction. We replaced the "formulate policies” verbiage with more direct plain language.
This feedback is limited, however, because they were collected within the context of a research interview, which itself had an introductory statement and informed consent procedure. Several participants confused these two introductions. Many participants felt that the explanation for informed consent did help them understand the goals of the survey: “the text is good and it is helpful. It’s upfront and helps frame it.”
Original Proposed wording |
Revised |
The information you give is important. The Department of Veterans Affairs (VA), the Bureau of Labor Statistics, and the Veterans Employment and Training Service sponsor the Veterans Supplement. They will analyze these data to measure trends in veteran employment and unemployment and to formulate policies and programs regarding employment and job training for veterans. |
I now have a few questions to ask you about your service in the Armed Forces. The Department of Veterans Affairs and the Veterans’ Employment and Training Service of the Department of Labor sponsor the following questions. The goal of these questions is to understand what services veterans need to help with the transition to civilian employment.
As a reminder, your responses will not impact your benefits and will not be seen by employers. Please do not disclose any classified information. |
Inconsistency with reporting rank
During early cognitive testing, several participants who had been officers mentioned that the questions in this section were tailored for enlisted service members. Also, participants sometimes gave vague answers that did not meet the desired level of detail. Based on our research on ranks, we understood “paygrade” to be a more universal and common standard across branches of the armed forces.
After a discussion of these findings, a further streamlining of this question was made to simplify data entry and circumvent the need for open-ended text entry: collapse ranks into a multiple choice format.
Original Proposed Wording |
Revised |
What was your rank at separation? |
What was your pay grade at
separation? 2- E-4 to E-6 3- E-7 to E-9 4- O-1 to O-10 5- W-1 to W-50 |
Terminology differences between military branches for job specialty
During early cognitive testing, 6 participants mentioned that the term "military occupational specialty" was Army- and enlisted-centric. Participants indicated that, while they understood what underlying concept we were targeting, it would be better to use enlisted- or officer-specific terminology for each branch of the military.
Feedback after these revisions was positive. Participants typically reported their alphanumeric code and their job title. However, several participants were unable to recall and report their alphanumeric code; it is likely that proxy respondents will have similar difficulties. We recommend that if this question is included in the questionnaire, the instrument must be able to accept longer verbatim responses in addition to alphanumeric codes.
After discussion of these findings, stakeholders modified the target concept of this question from military occupational specialty code to occupation, and provided examples. The revised question is based on the CPS Basic question that collects occupation, with modified examples. While we believe that generic occupations will be easier to collect and code than military occupation, we caution that the question will not measure the same concept as originally intended.
Original Proposed Wording |
Revised |
What was your military occupational specialty? |
What kind of work did (NAME/you) do, that is, what was (your/his/her) primary occupation? For example, geospatial engineer or combat medic specialist. *Enter verbatim response.
What were (your/his/her) usual activities or duties at this job? *Probe if necessary: For example - create geographic data and compile into maps using software, administer emergency medical treatment *Enter verbatim response. |
Missing collection of skills beyond primary job specialty
Participants were asked how they interpreted “primary” in relation to their job specialty. There was a range of interpretations: some said it was the occupational specialty that they held the longest, while others mentioned that it was the occupational specialty they had at time of their separation. For the purposes of the survey, that flexibility is desirable as it allows the respondent to choose the one that they felt was their main occupation.
During cognitive testing, several participants mentioned that veterans can have multiple occupational specialties at one time or over time and that a secondary occupational specialty is often the one that translates to civilian employment. As such, it is important to allow respondents to report more than just their primary occupational specialty.
For those who did have a secondary occupational specialty, the question tested well and feedback was positive from participants. For example, several participants mentioned being glad that we were capturing both primary and secondary occupational specialties. However, not all veterans had a secondary occupational specialty, and some may not recall it.
After discussion of these findings and the subsequent revision to target collection of occupation rather than military occupational specialty, we no longer recommend collecting a secondary occupational specialty.
Complexity of reasons for “retirement”
Cognitive testing revealed that the originally proposed question and response options were not clearly understood. Participants distinguished between medical discharge and separation and retirement and at least 1 participant indicated that he had to take a medical forced retirement. Given that the original motivation for adding this item was to understand the effect of retirement on employment decisions, we tested a modified “yes”/”no” version of the question about whether or not they have retired from the military. Participants were able to answer this question accurately and easily.
Original Proposed Wording |
Revised |
Did you retire from the Armed Forces for either medical or length of service reasons? |
Did you retire from the Armed Forces? |
Difficulty reporting total active duty service
Cognitive testing revealed that there were difficulties with how National Guard and Reserve members defined and calculated “active duty” service. Several participants mentioned that they did not include time that they served in the National Guard/Reserve in their total time served. Other participants referred to the “points” system of accounting for active duty time in the National Guard/Reserve or that they think of the total active duty time served based on retirement calculations (e.g., creditable service recorded on their DD-214). During data collection, we tested several versions of this item, including explicit instructions to participants to include or exclude certain service time and using different terms to refer to active duty service (e.g., “Include any active time due to call-ups from the Reserve or National Guard”). Every version caused response difficulties. For example, adding language about “reserve” time or “inactive” time caused more confusion, such as being unsure whether to include time spent in the Individual Ready Reserve. We ultimately do not recommend any revisions to this question wording. However, stakeholders should be aware of this issue when using these data.
Additionally, several participants reported their precise total time served, accurate to the number of days. For some participants, this was a difficult task. We considered instructing interviewers to read the list of response options aloud as scripted but reading the full list is tedious and time-consuming. Instead, we recommend training interviewers to instruct respondents, if necessary, about the level of precision needed.
Original Proposed Wording |
Revised |
In total, how long did you serve on active duty in the Armed Forces? |
How long did you serve on ACTIVE DUTY in the Armed Forces? *Do not
read response options aloud. |
Inconsistent wording for Vietnam and Iraq active duty items
In the original proposed questionnaire, the four combat theater items were revised to reduce respondent burden down to two items: one item for each geographic area. The proposed revision kept most of the original 2019 question wording but modified the reference period. Although the question wording of the two items differed, they are intended to collect the same information about two different theaters of operation. The language of the laws mandating the collection of these data does not specify particular wording must be used and we recommend using the simpler language for both items. This recommendation also removes a parenthetical clause, which leads to non-standardized question reading as it leaves the reading of those words up to the discretion of the interviewer.
Original Proposed Wording |
Revised |
Were you on active duty in Vietnam, Laos, or Cambodia; in the waters in or around these countries; or did you fly missions over these areas at any time between February 28, 1961 and May 7, 1975? |
Did you serve on active duty on the ground, in the air, or at sea in Vietnam, Laos, or Cambodia at any time between February 28, 1961 and May 7, 1975? |
Earlier it was reported that you served on active duty in the US Armed Forces. Did you serve in (on the ground, air, or sea) Iraq, Afghanistan, or any other conflict zone at any time since October 2001? |
Did you serve on active duty on the ground, in the air, or at sea in Iraq, Afghanistan, or any other conflict zone at any time since October 1, 2001? |
Although we recommend revising these items to reduce respondent burden and improve clarity, there are concerns that altering the question wording may change the way that respondents answer these questions. Ultimately, the stakeholders decided not to implement this revision and the original 2019 question wording will be used in the upcoming Supplement.
Overall, participants were able to answer the recommended series of questions. Compared to the 2019 Supplement, we believe that these recommendations will lower respondent burden by organizing related questions together and add value to the data set by adding new items of analytic interest.
During cognitive interviews, another issue emerged that we identified as substantive but not appropriate for addressing in the current Supplement. The “call-up” language used in the question, “Was your LAST period on active duty a result of a call-up from the Reserve or National Guard?” may be too specific. One participant noted that they were in Active Guard Reserves, for which their entire service time counted as active duty time; the “call-up” language was confusing and too specific to account for their situation. Stakeholders may consider matching the wording of this question directly to the congressional mandate, which says "veterans who were called to active duty while members of the National Guard or a Reserve Component". Additional research is needed before considering whether to include this revision in a future Supplement.
The final set of questions for this section is summarized below and appears in full, along with universe and skip instructions, in Appendix 3.
Question wording |
Response options |
I now have a few questions to ask you about [your/NAME’s] service in the Armed Forces. The Department of Veterans Affairs and the Veterans’ Employment and Training Service of the Department of Labor sponsor the following questions. The goal of these questions is to understand what services veterans need to help with the transition to civilian employment.
As a reminder, your responses will not impact [your/NAME's] benefits and will not be seen by employers. Please do not disclose any classified information. |
|
[Have/Has] [you/NAME] ever been a member of the Reserve or National Guard? |
1 - Yes |
[Are/Is] [you/NAME] currently a member of the Reserve or National Guard? |
1 - Yes |
Was any of [your/NAME's] active service the result of a call-up from the Reserve or National Guard? |
1 – Yes 2 - No |
Was [your/NAME's] LAST period on active duty a result of a call-up from the Reserve or National Guard? |
1 - Yes |
How long did [you/NAME] serve on
ACTIVE DUTY in the Armed Forces? |
1 - 6 months or less |
Did [you/NAME] EVER serve in a combat or war zone? Persons serving in a combat or war zone often receive combat zone tax exclusion, Imminent Danger Pay, or Hostile Fire Pay. |
1 - Yes |
[(Were you)/(Was he/she)] on active duty in Vietnam, Laos, or Cambodia; in the waters in or around these countries; or fly missions over these areas at any time between August 5, 1964 and May 7, 1975? |
1 - Yes |
[(Were you)/(Was he/she)] on active duty in Vietnam, Cambodia, or Laos; in the waters in or around these countries; or fly missions over these areas at any time between February 28, 1961 and August 4, 1964? |
1 - Yes |
Did you serve in Iraq, off the coast of Iraq, or did you fly missions over Iraq at anytime since March 2003? |
1 - Yes |
Did you serve in Afghanistan, or did you fly missions over Afghanistan, at anytime since October 2001? |
1 - Yes |
In what year [were/was] [you/NAME]
LAST released from active duty? |
Enter year |
From which branch of the Armed
Forces [were/was] [you/NAME] last released from active duty? |
1 - Air Force |
What was [your/NAME's] pay grade at separation? *Do not read list of response options aloud |
1- E-1 to E-3 2- E-4 to E-6 3- E-7 to E-9 4- O-1 to O-10 5- W-1 to W-5 |
What kind of work did (NAME/you) do, that is, what was (your/his/her) primary occupation? For example, geospatial engineer or combat medic specialist. *Enter verbatim response. |
Enter verbatim response |
What were (your/his/her) usual activities or duties at this job? *Probe if necessary: For example - create geographic data and compile into maps using software, administer emergency medical treatment *Enter verbatim response. |
Enter verbatim response |
Did [you/NAME] retire from the Armed Forces? |
1 - Yes |
Disability Section
Expert Review
When reviewing the proposed questions in the “Disability” section, we identified potential concerns regarding respondent perceptions of sensitivity, respondent comprehension of terms such as “individual unemployability”, and the ability of the items to accurately measure the target concepts. An item of special interest was the disability rating item, which showed relatively high rates of refusal in analyses of 2016-2018 Supplement data. Before cognitive testing began, we identified the concerns below and made revisions.
Ambiguity as to which disability rating to report
Respondents may have disability ratings from both the VA and the Department of Defense; the question should explicitly ask respondents to report one to ensure consistency.
Original Proposed Wording |
Revised |
What is your current service connected disability rating? |
What is your current service-connected VA disability rating? |
Redundancy of collecting work status
Other items collected in CPS Basic provide the target information about work status. It is not necessary to collect it again in the Supplement.
Original Proposed Wording |
Revised |
Do you receive Individual Unemployability from the VA, or has the VA found you to be unemployable?
If yes: Are you working anyway? |
Drop item |
Difficulty of measuring reasons for working despite being unemployable
We anticipated this question was potentially sensitive and would be difficult for respondents to answer. Pre-testing the appropriate response options for this item was out of scope for this study.
Original Proposed Wording |
Revised |
Do you receive Individual Unemployability from the VA, or has the VA found you to be unemployable?
If yes: Are you working anyway?
If yes: What reasons are you working? |
Drop item |
Given the concerns about sensitivity and knowledge of different terms, interview debriefings focused on probing how the participant decided which rating to report, whether the items clearly distinguished between payment for disability and retirement, and whether the items were perceived as sensitive. Debriefing also included general probes to prompt participants to describe any other aspects of employment and disability that were not captured by the questions.
Cognitive Interviews
A total of 51 participants reported having filed for or received a rating for a service-connected disability, of which 4 participants reported that they were waiting for a decision pending an initial claim or appeal and 1 participant reported that a claim was denied. Two additional participants reported their claims were denied. The concerns that emerged from feedback from participants are summarized below:
Sensitivity to disclosure of personal information
At the start of cognitive interviews, we administered this section in the same way as the 2019 Supplement, that is, without assurances to reduce sensitivity concerns. However, 2 out of the 3 initial participants with a service-connected disability receiving this version of the questionnaire expressed concern about how the collected data would be used. Given this feedback, we introduced an assurance at the start of the Supplement but more than half of participants who received this version still expressed sensitivity concerns. To address that concern, we added a Disability section introduction and only a few participants receiving that version of the questionnaire noted sensitivity concerns.
Feedback from participants also indicated that the disability rating item, which collects relatively fine-grain information about the participant, was the most sensitive (“Whose business is it, what my rating is? That’s something between me and the VA.”). Other items, which collect “yes”/”no” responses, were perceived as less personal.
Original Proposed Wording |
Revised |
None |
The following questions are about the impact of service-connected disabilities on jobs. Your responses will not impact your benefits and will not be seen by employers. |
Exclusion of veterans who have a service-connected disability but have not received a rating
Four participants were waiting for a rating decision and several others reported that their disability claim had been denied. The original wording of the item (“Has … determined that you have a service-connected disability…”) systematically excludes these circumstances and therefore does not collect responses from all respondents whose employment may be impacted by their service-connected disability. Although the recommended language is broader and screens in veterans whose disability has been determined to not be service-connected (claims denied), these veterans perceive their disability to be so and their experience may also be of analytic interest. We recommend including these veterans but distinguishing them from veterans who have a confirmed service-connected disability.
This revision requires the addition of a screening question to identify those veterans who have received a rating from the VA. During pre-testing, we found that this question worked for most participants; however, participants with 0% ratings may answer incorrectly and say they do not have a rating. We recommend training interviewers to understand that 0% ratings should be collected as valid ratings.
Original Proposed Wording |
Revised |
Has the Department of Veterans Affairs or Department of Defense determined that you have a service-connected disability; that is, a health condition or impairment caused or made worse by military service? |
Have you filed a claim for or received a rating from the Department of Veterans Affairs or the Department of Defense confirming that you have a service-connected disability; that is, a health condition or impairment caused or made worse by military service? |
None |
Did you receive a VA disability rating? *Probe if necessary: If respondent answers “no”, probe whether claim was denied or still pending. 1 - Yes |
Comprehension difficulty with the term “Individual Unemployability”
In this sample, two participants reported receiving Individual Unemployability. These participants were able to answer the question accurately.
However, the question appeared to cause confusion for participants who were not familiar with the Individual Unemployability benefit program. At least 5 participants interpreted the question incorrectly (e.g., not realizing that Individual Unemployability is a defined benefit). Most participants relied on the additional definitional information provided (“… has the VA found you to be unemployable”), which is vulnerable to misinterpretation. Further, the cell sizes for this group are likely to be small, which may limit usage of the data.
Original Proposed Wording |
Revised |
Do you receive Individual Unemployability from the VA, or has the VA found you to be unemployable? |
Drop item |
Missing measurement of disability impact on career jobs
At least 4 participants gave feedback that, although their service-connected disability did not prevent them from getting or holding “a job”, it did prevent them from getting or holding a job that they desired. Several participants said that their service-connected disability meant that they did not meet application or entry requirements (e.g., hearing, running) for a job in their ideal career path and so they ended up pursuing other jobs.
During the data collection period, we modified the wording to target “a job that meets your long-term career goals”. We considered using a simpler term such as “a job that you wanted” but felt that this wording was too broad and could be interpreted to include circumstances such as any job that paid enough money, which was not the intended concept for this question. Overall, participants understood this term as intended and were able to answer accurately. However, at least 2 participants had difficulty answering the question, apparently due to a shift in expectations whereby they felt that being “realistic” meant that they had to limit their career goals to jobs they were able to do given their disability (e.g., 100% disability rating). It is likely that no version of this question would work for these respondents.
Original Proposed Wording |
Revised |
Did your service connected disability ever prevent you from getting or holding a job in the past? |
Has your service-connected disability EVER prevented you from getting or holding a job that met your long-term career goals? |
Does this disability currently keep you from getting or holding a job? |
Does your service-connected disability CURRENTLY keep you from getting or holding a job that meets your long-term career goals? |
Missing understanding of future impacts of disability
When asked about the impact of service-connected disability on past and current employment, at least 4 participants volunteered that their disability had “not yet” had an impact. Participants described health conditions becoming more severe as they aged, which they anticipated would eventually cause them to retire earlier than desired, get a different a job than they wanted, or otherwise become unable to hold their jobs.
Original Proposed Wording |
Revised |
None |
Do you think that your service-connected disability will EVENTUALLY keep you from getting or holding a job that meets your long-term career goals? 1 – Yes 2 – No 3 – Don’t know |
Missing measurement of mitigating factors such as workplace accommodations
At least seven participants reported either that accommodations for their disability have been critical to enabling them to do their jobs or that the lack of accommodations is the primary reason why doing their job is difficult or they are unable to progress in their career. One participant mentioned as an example of a critical accommodation her employer’s scheduling flexibilities while dealing with the effects of post-traumatic stress disorder.
We recommend adding items to collect information about workplace accommodations. Although the Supplement would not collect detail about the disability or the accommodation, understanding whether veterans are in need of accommodations may inform programs and policies. The recommended item is a version of an item from the CPS Disability Supplement, modified to specify accommodations for service-connected disabilities specifically. However, in the Disability Supplement, the item is asked of all respondents, not only disabled respondents; the analytic value of this item may be limited.
Original Proposed Wording |
Revised |
None |
Have you ever requested any change in your workplace to help you do your job better and accommodate your service-connected disability? For example, changes in work policies, equipment, or schedules? |
None |
Were any of those requests granted? |
Unnecessary question about a potentially sensitive topic
At least 4 participants felt that asking about monthly payments was redundant after also asking about rating, given that payments were based on ratings. These participants, and others, also perceived these questions about payments as sensitive; several participants reported that this item caused them to believe that the survey was going to ask for more personal information, such as the amount of their monthly payment.
While the precise payment amount may vary by individual circumstances, having a non-zero rating should accurately indicate whether the veteran receives any payment. The rating item has suffered relatively high non-response in previous years, but we expect a lower non-response rate given the addition of the introduction and confidentiality assurances to the “Disability” section.
Original Proposed Wording |
Revised |
Do you currently receive a monthly payment for a service connected disability from either the VA or a branch of the military service? |
Drop item |
Questions perceived as irrelevant to their circumstances
Older participants who are retired consistently reported that the questions about work in this section felt irrelevant to their lives. Although they were eager to respond to the survey, they felt that their responses to these items were not useful. Given that their responses are not tied to any future work plans, their responses are not needed.
Original Proposed Wording |
Revised |
D4: Ask of all veterans who respond to D3 with any answer other than “no” |
D4: Ask of all veterans who respond to D3 with any answer other than “no” AND any labor force status other than “retired” |
Overall, participants were satisfied that the recommended series of questions captured important concepts about the impact of service-connected disability on employment. Compared to the 2019 Supplement, we believe that these recommendations will lower non-response rates and present a fuller picture of the impact of disability on employment, with an emphasis on meaningful employment and veterans’ long-term concerns.
During cognitive interviews, two issues emerged that we identified as substantive but not appropriate for addressing in the current Supplement. At least 4 participants brought up vocational rehab and reintegration programs as being important for veterans with disabilities to help with job training. And several participants mentioned sensitivity about employers and disclosure of their disability, noting that disclosure can prevent someone from serving in the Reserves or discourage employers who perceive veterans with disabilities as problematic. At least 1 participant mentioned that she thought that disclosing her disability ought to translate to preferences in hiring but it did not seem to do so in her experience. Additional research is needed before considering whether to include these topics in a future Supplement.
The final set of questions for this section is summarized below and appears in full, along with universe and skip instructions, in Appendix 3.
Question wording |
Response options |
The following questions are about the impact of service-connected disabilities on jobs. Your responses will not impact your benefits and will not be seen by employers. |
|
Have you filed a claim for or received a rating from the Department of Veterans Affairs or the Department of Defense confirming that you have a service-connected disability; that is, a health condition or impairment caused or made worse by military service? |
1 - Yes |
Did you receive
a VA disability rating? |
1 - Yes |
What is your
current service-connected VA disability rating? |
0 - 0 percent
|
Has your service-connected disability EVER prevented you from getting or holding a job that met your long-term career goals? |
1 - Yes |
Does your service-connected disability CURRENTLY keep you from getting or holding a job that meets your long-term career goals? |
1 - Yes |
Do you think that your service-connected disability will EVENTUALLY keep you from getting or holding a job that meets your long-term career goals? |
1 - Yes |
What is the
last year you worked at a job or business? |
Numeric Response |
Have you ever requested any change in your workplace to help you do your job better and accommodate your service-connected disability? For example, changes in work policies, equipment, or schedules? |
1 - Yes |
Were any of those requests granted? |
1 - Yes 2 - No |
Transition to Employment Section
Expert Review
When reviewing the proposed questions in the “Transition to Employment” section, we identified potential concerns regarding the ability to accurately measure the target concepts. Before cognitive testing began, we identified the following concerns and made revisions:
Difficulty of measuring reasons for location decisions, satisfaction with job, and impact of military service
We anticipated that several of the target concepts would not be possible to either measure in the limited space available in the Supplement or cognitively test in this study. The items on understanding location decisions, satisfaction with wages, line of work, and time to find a job, and the impact of military experience overall on civilian work potential were too complex for the study.
Original Proposed Wording |
Revised |
When you left active service, did you stay near your last duty station, return to your home of record, or transition to a different location? |
Drop item |
Why did you live and work where you did? |
Drop item |
Have you moved in the past three years? If so, why? |
Drop item |
Did you find this satisfactory, considering your wages earned, line of work, desired hours, and time to find a job? |
Drop item |
Missing screener questions to ensure respondents are only asked relevant questions
In the original proposed questionnaire, items about civilian employment assume that the veteran has been offered and accepted a civilian job. Elsewhere in the CPS, information about the veteran’s current employment status is collected, but status at the time of the interview may not be the same as status at the time of transition; a veteran could be unemployed at the time of interview but had previously had a civilian job. Past job information is not available from elsewhere in the CPS and must be collected in the Supplement if desired for analysis.
Original Proposed Wording |
Revised |
None |
When did you first apply for a civilian job? |
None |
Did you not want a job, return to a previous job, go back to school, or something else? |
None |
Have you received an offer for a civilian job? |
Cognitive interview debriefings focused on probing how participants transitioned to their first civilian job, including how they decided what jobs to apply for, whether they used the specialized training they acquired in the military, and how prepared for transition they felt at the time of their separation.
Cognitive Interviews
A total of 19 participants separated from the armed forces in the last 10 years and went through the “Transition to Employment” section. Although we were not able to interview as many veterans in this sub-group as we had planned, we obtained valuable feedback for revising this section of the Supplement. The main recommendation is to add items to understand veterans’ first civilian job and how it relates to their military training.
In early cognitive interviews, we included items about job search at the time of transition in this section but later relocated those items to a different section of the Supplement in order to focus on the civilian job itself in this section.
The feedback from participants is summarized below:
Missing background information about transition preparation
Participants’ experiences with preparation for transition to civilian employment varied – some participants attended a Transition Assistance Program (TAP) workshop while others did not. Although TAP attendance was recently made mandatory, respondents may have transitioned earlier than when that policy went into effect and may not have gone through TAP, or may not have attended TAP for other reasons. For the purposes of understanding transition experiences, it may be important to separate these groups in data analysis. We recommend using a version of the item used in the 2019 Supplement.
Original Proposed Wording |
Revised |
None |
While still on active duty, did you
attend any of the Transition Assistance Program workshops, known
as TAP or A-CAP? |
Missing whether veterans want to use their specialized training in civilian employment
When discussing how their military training applied to their civilian employment, several participants reported that they had not sought a job that used their military training. Participants reported that this disconnect was common, especially among veterans in “blue collar”, “low-paying” occupational specialties, or specialties unique to military operations (e.g. infantry). For the purposes of developing a descriptive crosswalk between occupational training and the jobs that veterans ultimately work in, it may be helpful to understand whether veterans want to use their occupational training in civilian employment. Even in the absence of a detailed crosswalk, these data may provide insight into which military occupational specialty codes may not lead directly to desirable civilian employment.
Original Proposed Wording |
Revised |
None |
When first applying for a civilian job, did you want a job that used the specialized training you acquired in your military occupation? |
Missing information about first civilian job
In order to develop a meaningful crosswalk between occupational training in the military and civilian employment, we recommend collecting the title of the first civilian job after separation. First civilian job title is a more direct link to the occupational training acquired in the military; more so than current job title, which is what is collected elsewhere in the CPS. These job titles may reveal that a veteran was not able to find meaningful employment immediately after separation.
After discussion of these findings, stakeholders requested the below examples to be read aloud as part of the scripted question.
Original Proposed Wording |
Revised |
None |
What was that first civilian job, that is, what was your occupation? For example, information security officer or management analyst. *Enter verbatim response. |
|
What were your usual activities or duties at this job? *Probe if necessary: For example - encrypt data and assess risk on computer systems; study work problems and recommend new systems, procedures, or organizational changes *Enter verbatim response. |
Missing information about satisfaction with first civilian job
After discussion of our questionnaire recommendations, stakeholders requested that a measure of satisfaction with first civilian job be added. This question wording was not tested with research participants.
Original Proposed Wording |
Revised |
Did you find this satisfactory, considering your wages earned, line of work, desired hours, and time to find a job? |
Was that the job that you wanted at the time of separation? |
Retired participants feel these questions are not relevant to them
At the onset of interviews, we limited the universe for this section to those who had recently separated, believing that responses from veterans who separated in the 1980s or earlier, for example, would not be as useful. Although this section of questions was limited only to participants who had separated recently, that universe still included veterans who are out of the labor force because they are retired or disabled. These participants expressed feeling that the questions about transition to civilian employment are not relevant to them and that their responses were not useful.
Original Proposed Wording |
Revised |
Ask of all veterans who separated after 2010 |
Ask of all veterans who separated after 2010 and are of working age |
Age of retirement is uncertain, even to older veterans
Participants had difficulty reporting an age of anticipated retirement. Even participants in their 60s were uncertain about their retirement plans. When participants did provide an answer, it was often vague (referring to a range of years) or qualified by a statement that they would then look into starting their own business or looking for part-time work. Feedback from CPS staff converges with this finding that it is difficult to collect anticipated retirement age.
Original Proposed Wording |
Revised |
How much longer do you intend/plan to work? |
Drop item |
Overall, participants were satisfied that the recommended series of questions captured important concepts about how they perceived their occupational training affecting civilian employment. Of the many new questions proposed for this section on transition, we believe that the recommended questions provide clear and useful data about the transition experience from the veteran perspective and provide data that can be used to understand the connection between specialized military training and civilian jobs.
The final set of questions for this section is summarized below and appears in full, along with universe and skip instructions, in Appendix 3.
Question wording |
Response options |
The following questions are about your transition to civilian employment. |
|
While still on active duty, did you
attend any of the Transition Assistance Program workshops, known
as TAP or A-CAP? |
1 - Yes |
When did you first apply for a
civilian job? |
1 - Before
separation |
Did you not want a job, return to a previous job, go back to school, or something else? |
1 - Not want a
job 4 - Something else |
When first applying for a civilian job, did you want a job that used the specialized training you acquired in your military occupation? |
1 - Yes |
Have you received an offer for a civilian job? |
1 - Yes |
When did you accept the offer for
your first civilian job? |
1 - Before
separation |
What was that first civilian job,
that is, what was your occupation? For
example, information security officer or management
analyst. |
Enter verbatim response |
What were your usual activities or duties at this job? *Probe if necessary: For example - encrypt data and assess risk on computer systems; study work problems and recommend new systems, procedures, or organizational changes *Enter verbatim response. |
Enter verbatim response
|
Was that the job that you wanted at the time of separation? |
1 - Yes 2 - No |
Training Section
Expert Review
When reviewing the proposed questions in the “Training” section, we identified potential concerns regarding the terminology used and the ability of the items to accurately measure the target concepts. Before cognitive testing began, we identified the concerns that follow and made revisions.
Ambiguity of the term “obtain”
In the original proposed questionnaire, a series of items asks whether the veteran sought to obtain training or education. It is unclear whether the term “obtain” in this context means to “find” or “complete”. As these are both important concepts to understand, we revised the series to first ask about finding training and then separately about completing training.
Original Proposed Wording |
Revised |
Since leaving active duty, have you tried to obtain formal job training or job-related schooling? |
Since separation, have you TRIED to obtain any formal job training or job-related schooling? |
Were you able to obtain the training or schooling you were seeking? |
Did you find training or schooling that met your needs? |
None |
Were you able to complete the training or schooling? |
Low value in measuring “interest” on a 5-point scale
In the original proposed questionnaire, one item asks respondents to rate interest in job training or education on a 5-point scale. We anticipated that “interest” would not be a valuable concept to measure, given that it does not require commitment. Instead, asking about how “useful” training or education is within a defined reference period may be more likely to measure actual training and education needs and behaviors. We also anticipated that respondents would be unlikely to accurately use the full range of a 5-point scale in the telephone mode (e.g., primacy or recency effects) and that, for the purposes of analyses, a 3-point scale would provide the information needed for the stakeholders. Although the revised question is double-barreled (finding new employment or improving current employment situation), we anticipated that the value of cueing both concepts was more valuable than the ability to distinguish between these reasons.
Original Proposed Wording |
Revised |
On a scale of 1-5, with 5 being the highest interest, how would you rate your interest / willingness in additional training or education for new or higher level employment? |
How useful would additional training, school, or an apprenticeship be for finding new employment or improving your current employment situation? 1 – Very useful 2 – Somewhat useful 3 – Not at all useful |
Multiple concepts measured within a single question
The initially proposed item on GI Bill benefits covered multiple concepts in a single question. We split the item into two, asking first whether they received GI Bill benefits, then who did they use the benefits or who they plan to use the benefits for.
Original Proposed Wording |
Revised |
Did you use any of your GI Bill benefits? If yes, and if eligible, did you use it for you or transfer to a son/daughter? |
Did you receive any GI Bill benefits? |
Who used the benefits or who plans to use them? |
Complexity of measuring educational attainment relative to military service
The initially proposed questionnaire included items on the veteran’s education. The veteran’s highest level of education is already collected elsewhere in the CPS and does not need to be asked again in the Supplement. As to when the education was attained relative to military service, we determined that the concept was too complex to be within scope for this study, given the other research goals. For example, collecting when relative to military service the veteran began taking courses and when the veteran received the degree requires multiple survey items.
Original Proposed Wording |
Revised |
What is the highest level of education attained? Was the degree attained before, during, or after military service? |
Collect highest level of education from elsewhere in CPS. Drop item on timing relative to military service. |
Difficulty measuring what partners the veteran received information from
In the original proposed questionnaire, one item asks respondents what, if any, partners the veteran received job search and related services or information from. This question, even within the context of job training, is so broad as to encompass a wide variety of valid responses. It was not within the scope of this study to establish the appropriate response categories for this item. Furthermore, the information would need to be accompanied by several other items in order to be useful, such as the name of the organization, what information or service was received, and whether that information or service was helpful.
Original Proposed Wording |
Revised |
What, if any, non-DOL or VA partners did you receive services/information from? |
Drop item |
Interview debriefings focused on probing the experience of veterans who had sought out job-related training or education. Debriefings also included general probes to prompt participants to describe any other aspects of training that were not captured by the questions.
Cognitive Interviews
At the beginning of data collection, we asked this Supplement section on training of all participants. However, feedback from older veterans who had retired or were otherwise out of the labor force was consistent: they felt these questions were not relevant to their circumstances. These participants did not have any interest in job-related training or education, currently or in the future. We revised this Supplement section to be asked only of veterans who are in the labor force, for whom job-related training and education are relevant. Nonetheless, we believe it is important to ask the Supplement questions of veterans of all eras as older veterans may still be in need of job-related training.
All respondents were asked questions about their GI Bill benefit use and who used the benefits because these items were initially included in a separate “Benefits” section that was not conditional on separation date or labor force status. The majority of respondents indicated that they had used GI bill benefits (48 participants indicated that they had used the benefits). At least 7 participants mentioned that that they had used their GI bill benefits for other dependents, such as their spouses or children.
The concerns that emerged from feedback from participants are summarized below:
Collecting aspirations rather than concrete plans
Most participants who were still in the workforce answered “yes”, that additional training or education would be useful. However, not all participants had concrete plans to seek out training or education, instead referring to long-term plans such as eventually completing a graduate degree. To better inform development of programs and services, we revised the item to ask about plans for training and education within a defined reference period. The exact reference period chosen was 4 years, which is parallel to the retrospective reference period used elsewhere in the Supplement. We recommend that any reference period within the 3-5 year range will likely be appropriate for the sponsors’ goals of understanding training and education decisions. Using a “yes”/”no” response format, the question elicited mostly “no” responses, with participants thinking more concretely (e.g., cannot afford training or education at this time and so no plans to do so). Ultimately, we recommend a short three-point scale to measure how likely the veteran is to seek out training or education.
Original Proposed Wording |
Revised |
On a scale of 1-5, with 5 being the highest interest, how would you rate your interest / willingness in additional training or education for new or higher level employment? |
In the next four years, how likely is it that you will seek job-related training or education? 1 – Very likely 2 – Somewhat likely 3 – Not at all likely |
Confusion of items related to first civilian job and items related to any civilian job
During early cognitive testing, we found that the reference contexts of different items were unclear: some items focused on the veteran’s first job while other items focused on any job. For example, the item collecting whether the veteran’s specialized training is used is meant to focus on the full civilian career. We recommend developing two separate Supplement sections (first civilian job vs. general career) with a clear division to create distance between these concepts.
Original Proposed Wording |
Revised |
None |
The following questions are about your transition to civilian employment. |
None |
The next questions are about training for civilian employment. |
The term “technical skills” was not understood as intended
Participants were confused about the term “technical skills,” often interpreting it to refer to skills related to heavy equipment or computer software and excluding intelligence skills. Alternative wording versions tried during testing elicited interpretations of “soft skills” such as discipline and organization, which also was not the intended goal for this question. After receiving this feedback, we tested a version of the question using the term “specialized training”, which was interpreted as referring to the skills acquired as part of their occupational specialty, as intended. For example, one participant answered that he did not use his specialized training as a civilian, explaining that he now taught business law, not military law, which had been his specialty as a service member.
Original Proposed Wording |
Revised |
Were technical skills that you acquired in your military occupation used in civilian employment? |
Have you used the specialized training that you acquired in your military occupation in ANY civilian job? |
Topic of training no longer needed
After discussing these findings, stakeholders identified the topic of training as no longer being a data requirement, due partly to the complexity of data collection.
Original Proposed Wording |
Revised |
What was the topic or subject of that education or training that you were not able to find? 1
- COMPUTER procedures, programming or software training
|
Drop item |
Scattered items about education and training
Initially, the items asking about GI Bill benefits were located at the end of the Supplement along with other items about benefits; however, the GI Bill items seemed more closely related to the training and education items in this section. When describing how their GI Bill had been used, participants frequently referred back to the education or training that they had described in this section.
Original Proposed Wording |
Revised |
An “Education and Benefits’ section |
“Education” items are narrowed in scope and included as part of “Training” |
Ambiguity between on-the-job training and apprenticeships
During cognitive testing, we asked participants who said they wanted additional training or education whether they wanted training, education, or an apprenticeship. In our sample, no participants selected an apprenticeship. However, during additional discussion with other researchers about these categories, we understand that there is no distinction between on-the-job training and apprenticeships that is universally understood. Therefore, we recommend combining these two categories into one. Although this approach does not collect information on apprenticeships in particular, it is highly likely that data collected about “apprenticeships” would have substantial measurement error.
After discussing these findings, stakeholders requested the addition of two categories: occupational certification and entrepreneurship training. These terms were not tested with research participants.
Original Proposed Wording |
Revised |
Based on the previous response, would you be most interested in more training, more education, or an apprenticeship? |
Are you planning to seek out any of the following? 1
– Occupational certification 4 – School-based education |
Ambiguity about how to answer if sought out or completed multiple trainings
Initially, items asking about whether the veteran obtained training or education referred simply to the outcomes of past training or education. This scope led to difficulty for respondents who had sought out or completed multiple trainings or education courses – they were not sure which training the questions referred to. Given the stakeholder interest in understanding obstacles to obtaining training or education, we revised the items to probe for whether the veteran had been unable to find or complete all training or education they had sought out and, if not, why. However, this revision still led to difficulty instructing participants which training to consider when answering follow-up questions about reasons for failing to find or obtain training. Ultimately, we recommend addressing this challenge by narrowing the scope to the most recent training or education only.
Original Proposed Wording |
Revised |
Were you able to obtain the training or schooling you were seeking? |
Were you ABLE to find the job-related education or training that you looked for most recently? |
Difficulty with GI Bill terminology
The majority of participants indicated that they had used the GI bill and knew what it was. However, several participants had difficulty with the GI bill terminology. Several participants distinguished between types of education benefits, such as those offered related to a disability (vocational rehabilitation). One participant was not sure if the question included “Education Savings”, which was something the participant had put money into prior to the GI bill. An additional participant mentioned not being familiar with what the GI bill is. We recommend an interviewer instruction that provides additional information about the GI Bill.
Original Proposed Wording |
Revised |
Did you receive any GI Bill benefits? |
Did you receive any GI Bill benefits? *The GI Bill is an education benefit that helps to cover the costs associated with getting an education or training. It includes: Post-9/11 GI Bill, Montgomery GI Bill, Dependents’ Education Assistance, and Fry Scholarship. |
Overall, participants were satisfied that the recommended series of questions captured important concepts about how they perceived their training needs. Many of the items in this section are substantively unchanged from the 2019 Supplement. Of the new questions proposed for this section on training, we believe that the recommended questions provide clear and useful data that can be used to develop services for veterans.
The final set of questions for this section is summarized below and appears in full, along with universe and skip instructions, in Appendix 3.
Question wording |
Response options |
The next questions are about training for civilian employment. |
|
Have you used the specialized training that you acquired in your military occupation in ANY civilian job? |
1 - Yes |
Since separation, have you TRIED to
find any job-related education or training? |
1 - Yes |
Were you ABLE to find the job-related education or training that you looked for most recently? |
1 - Yes 2 - No |
Were you able to complete that education or training? |
1 - Yes |
What was the main reason you weren’t
able to [find/complete] that education or training? |
1 - CONFLICT with job
responsibilities |
In the next four years, how likely is it that you will seek job-related education or training? |
1 – Very
likely |
Are you planning to seek out any of the following? 1
– Occupational certification 4 –
School-based education |
Numeric
response option |
Did you receive any GI Bill benefits? *The GI Bill is an education benefit that helps to cover the costs associated with getting an education or training. It includes: Post-9/11 GI Bill, Montgomery GI Bill, Dependents’ Education Assistance, and Fry Scholarship |
1 - Yes |
Who used the benefits or who plans to use them? 1 –
Yourself |
Numeric
response options |
State Workforce Agency Section
Expert Review
When reviewing the proposed questions in the “State Workforce Agency” section, we identified potential concerns regarding the value of some items; these concerns are summarized below along with the revisions made before cognitive testing began.
Scattered items about job search services
In the original proposed questionnaire, items about the veteran’s search for a job were located in both the “Transition” section and the “State Workforce Agency” section. Given that job search needs extend beyond just the search for the veteran’s first civilian job, we re-ordered the items so that job search items were located alongside the state workforce agency items.
Original Proposed Wording |
Revised |
Items in “Transition” and “Job Training – Workforce system” |
Items in “State Workforce Agency” |
Undefined context for assessing use of virtual job search services
The original proposed questions about virtual services were broadly about the any use of any job search services, without limiting the context to services offered by a state workforce agency or other named organization. We anticipated that the value of collecting such broad answers would not be useful and developed a series of items asking about organizations that the veteran used during their job search and then what modes of service the veteran had used from that organization and what mode of service the veteran preferred.
Original Proposed Wording |
Revised |
In your transition and job search, did you use virtual services, in-person services, both, or neither? |
Which services did you use? Please select all that apply. 1 - Veterans Administration 2 - State workforce or employment services agency also known as “job service”, “unemployment offices” or “one-stop service centers” 3 – Non-profit organization – please specify 4 - Other – please specify
If use State workforce services agency: Which types of State workforce or employment services did you use at least once? 1 - Web-based services 2 - Telephone services 3 - In-person services |
The assessment of the state workforce agency’s impact may be affected by speculation and external factors
In the original proposed questionnaire, an item evaluating the veteran’s experience using the state workforce agency relied on outcomes that could only be answered by speculation and factors outside of the control of either the veteran or the state workforce agency (e.g., no jobs in the veteran’s desired industry in the local area, general downturn in hiring, veteran did not receive feedback from employers about reasons for not being hired). Developing appropriate questions and response options to measure the impact of the state workforce agency was out of the scope of this cognitive testing study.
Original Proposed Wording |
Revised |
What was your experience after using the state workforce system? (Gain employment faster, increased wages, career advancement, more quickly change career paths, enrolled in school or training) |
Drop item |
Low information value of “yes”/”no” questions about “other” behaviors
In the original proposed questionnaire, two existing items from the 2019 Supplement collected what strategies veterans had used to find jobs and whether the strategies were useful. In our review, we found that the value of these items was low, given that they are collected at a “yes”/”no” level of detail without also collecting the detail of what strategies were used. However, given the complexity of these target concepts, these items were out of scope for this cognitive testing study.
Original Proposed Wording |
Revised |
Since leaving active duty have you used any other strategies for finding a job or obtaining job-related training, such as networking, checking the internet, responding to newspaper ads, contacting a private employment service, or seeking employment through a temporary staffing agency? |
Drop item |
Were any of these strategies useful in identifying employment opportunities, applying for jobs, getting a job, or obtaining the job-related training you were seeking? |
Drop item |
Potential difficulty with reliably field coding long lists of response categories
In the 2019 Supplement, the item collecting reasons why the veteran did not use the state workforce agency had 13 categories. A list of this length can be difficult to field code quickly and reliably, whereby interviewers may not consider all items before recording a response. In our review, we found some items that seemed unlikely to be useful for stakeholders and other items that could be revised to be made clearer. We also found some viewpoints that we heard in the interviews were not represented in this list (found a job himself/herself and did not use any job search services). We revised the categories to address the main concepts while reducing the length.
Original Proposed Wording |
Revised |
1 – Already have JOB lined up 2 – Was not looking for JOB or WORK; did not want to work 3 – Have FAMILY OBLIGATIONS 4 – DISABLED or UNABLE to work 5 – Did not think specialist could help (NO FAITH in system) 6 – Contacted VA or other government office for help 7 – PRIDE or STIGMA associated with going to “unemployment office” 8 – COULD NOT FIND employment service office 9 – Specialists do not have information on GOOD JOBS 10 – Was UNAWARE OF SERVICES provided by employment offices and their specialists 11 – Too OLD to re-train, go back to school, or switch occupations 12 – Prefer or had access to INTERNET or job WEBSITES 13 – Other specify |
1 - Already had
a JOB lined up 5 - Thought centers are NOT USEFUL (NO FAITH in system) 6
– Thought centers do not have GOOD JOBS 9 - NOT AWARE OF these centers 10 - Found a job HIMSELF/HERSELF 11 – Other specify |
Interview debriefings focused on probing for details about the participant’s experience with state workforce agencies and other organizations used during job searches. We also probed on whether the mode (in-person vs. online) was a significant factor in their past experience or interest in working with the state workforce agency in the future.
Cognitive Interviews
Participants who had recently separated from the armed forces were asked this Supplement section. However, part way through data collection, we revised the universe for the items concerning future needs for training to be asked of all participants, given that any veteran may want job-related training or education even if they had not recently separated.
Early on in cognitive interviews, we concluded that service mode, online vs. in-person, did not affect the veterans’ interest in services. However, given stakeholder feedback that understanding demand for different types of services to be offered online, we revised those items toward collecting information about specific services. Additionally, based on stakeholder feedback after early data collection, we revised the section to focus on American Job Centers and probed for familiarity with American Job Centers and needs for services.
The concerns based on feedback from participants are summarized below:
Retired participants feel these questions are not relevant to them
Although this section of questions was limited only to participants who had separated recently, that universe still included veterans who are currently out of the labor force because they are retired. These participants felt that the questions about job-related training and education are not relevant to them and that their responses were not useful.
Original Proposed Wording |
Revised |
Ask of all veterans |
Ask of veterans of working age |
Participants were not familiar with “American Job Centers”
Throughout testing, we used terms such as “state employment or career services agency” and “American Job Center” (AJC), per stakeholder interest in understanding the value of that branding. Participants were familiar with either the generic ‘state” employment office or their state-specific office name but no participants were familiar with the “American Job Center” branding. Given this widespread unfamiliarity with the AJC term, we do not recommend asking questions based only on AJC terminology. Other wording, like the term “unemployment offices” used in the 2019 Supplement, led participants to think of out-of-scope activities like filing for unemployment.
Original Proposed Wording |
Revised |
Since leaving active duty, have you visited a state workforce or employment service office or logged onto one of their internet websites to review reference materials on job openings, training opportunities, or higher-education options? |
Since separation, have you ever visited a website for or made face-to-face or telephone contact with a state employment or career services agency, sometimes known as American Job Centers, for help finding a job or job-related training? |
Missing understanding of how state workforce agencies can improve
Although the 2019 Supplement item asking whether the agency was helpful was cut from the original proposed questionnaire, one of the goals of the Supplement is to understand how veterans use state workforce agencies and what services the centers should offer. This 2019 Supplement item seems to be an effective way to collect that information.
After discussing this recommendation, stakeholders determined that the open-ended item collecting the reason that the center was not helpful could not be collected as recommended, partly due to costs. This item “What was the main reason the state employment or career services center or website was not helpful?” was dropped from the questionnaire.
Original Proposed Wording |
Revised |
None |
Were you able to get all of the help that you needed from the state employment or career services center or website? |
Ambiguity in negative responses to virtual services
Early feedback from participants indicated that the virtual services questions were too broad and did not capture an actionable understanding of what job search services veterans need. For subsequent interviews, we revised the items to identify specific services that could be offered at AJCs, which we understood to be the underlying target of the proposed question.
Participants who answered that they did not want specific online services sometimes wanted an in-person version of the service but other times did not want the service at all. The items were then revised to be able to collect which mode the veteran prefers, or that the veteran does not want the service at all.
Original Proposed Wording |
Revised |
None |
I am going to read a list of services that a state employment or career services center might offer to help with your next job search. For each service, please tell me if you would prefer to use the service in-person, online, or not at all. |
Overall, participants were satisfied that the recommended series of questions captured important concepts about how they perceived their use of and needs from state workforce agencies. We believe that the recommended questions provide clear and useful data about the how state workforce agencies can develop their services for veterans.
The final set of questions for this section is summarized below and appears in full, along with universe and skip instructions, in Appendix 3.
Question wording |
Response options |
The following questions are about your experience with state employment or career services agencies. |
|
Since separation, have you ever visited a website for or made face-to-face or telephone contact with a state employment or career services agency, sometimes known as American Job Centers, for help finding a job or job-related training? |
1 - Yes 2 - No |
What were the reasons you did not visit or contact a state employment or career services center or website? 1
- Already had a JOB lined up 5 - Thought centers are NOT USEFUL (NO FAITH in system) 6
– Thought centers do not have GOOD JOBS 9 - NOT AWARE OF these centers 10 - Found a job HIMSELF/HERSELF 11 –
Other specify |
Numeric response options |
Were you able to get all of the help that you needed from the state employment or career services center or website? |
1 - Yes |
I am going to read a list of services that a state employment or career services center might offer to help with your next job search. For each service, please tell me if you would prefer to use the service in-person, online, or not at all. |
|
Résumé help |
1 - In-person
|
Job application help |
1 - In-person
|
Finding job opportunities |
1 - In-person
|
Developing career goals |
1 - In-person
|
Translating military work experience for civilian employers |
1 - In-person
|
Benefits Section
Expert Review
After removing several Education items from the originally proposed “Benefits and Education” section during pre-testing, only a Benefits-related item ultimately remained and this Supplement section became a “Benefits” section. When reviewing the proposed question, we identified a potential concern regarding the ability of this study to adequately pre-test the item.
Complexity of collecting which military and government benefits received
In the initially proposed questionnaire, an item asked about whether the veteran received benefits from the military or the government and, if so, what benefits. We determined that it was out-of-scope for this study to develop appropriate and useful response categories for this item given that it would require a comprehensive list of all military and government benefits available. We were also concerned that such a question would be relatively burdensome for interviewers because they would need to understand each benefit program in order to accurately probe and provide clarifications.
Original Proposed Wording |
Revised |
Are you currently receiving any military or government benefits? |
Drop item |
Interview debriefings for this section focused on participants’ perceptions about benefits, to understand what their concerns were that may be relevant to the VA or DOL. As we added Supplement items to address these concerns about benefits, we probed on evaluating current outreach efforts of the VA and how veterans would prefer outreach to occur in the future.
Cognitive Interviews
Initially, due to concerns about accurately measuring veterans’ use of a wide range of military and government benefits, we dropped the Benefits item from the Supplement. Feedback from veterans led us to revise this revision to instead ask two questions about veteran’s confidence in their knowledge about benefits and how they would prefer to learn about benefits in the future.
The concerns that emerged from feedback from participants are summarized below:
Missing measurement of benefit awareness
Although we did not initially include an item about what benefits veterans received, we did probe participants about benefits and any issues they believed were important. Feedback from early interviews was that veterans did not feel as though they were aware of all of their potential benefits from the VA. Across multiple cognitive interviews, veterans shared that they perceived awareness of benefits to be more important for survey sponsors to understand than which benefits they do or not have. Given that awareness itself is not possible to measure without a burdensome review of a comprehensive list of benefits, we recommend focusing on confidence in awareness, which can be used to understand which sub-groups of veterans are not well-served by current outreach efforts.
Original Proposed Wording |
Revised |
None |
How confident are you that you are aware of all the benefits that you are eligible for from the VA? 1
- Not at all confident 3 - Very confident |
Missing feedback on preferred methods of outreach.
During cognitive testing, several participants mentioned that they felt they had to be proactive to learn about their benefits and often had to rely on “word of mouth”. In order to collect constructive information from veterans about what the VA could be doing, we added an item about the veteran’s preferred way of learning about benefits. Options offered included website, online accounts, personal letters, and e-mails. Feedback from participants led to the addition of social media and in-person visits to VA offices.
While the majority of participants who were asked this question felt that they were easily able to answer this question, it is important to note that one participant refused to answer this question as he indicated that he didn’t want to be contacted by the VA in the future.
After discussing these findings, stakeholders requested that “personal text message” be added as a response option. This option was not tested with research participants.
Original Proposed Wording |
Revised |
None |
Finally, I’d like to read you a
list of different ways the VA could reach out to you about the
benefits that you are eligible for. For each one, please let me
know if that’s a way that you would like to learn about
your benefits. 1
- In-person visit to my local VA office 7 – Personal text message |
Overall, participants felt that these questions captured veterans’ perceptions about both current VA outreach efforts and ways in which the VA could reach out to them in the future. Of the new questions proposed for this section on benefits, we believe that the recommended questions provide clear and useful data that can be used to develop future outreach strategies for veterans.
The final set of questions for this section is summarized below and appears in full, including universe and skip instructions, in Appendix 3.
Question wording |
Response options |
And now, the last questions are about your VA benefits. |
|
How confident are you that you are aware of all the benefits that you are eligible for from the VA? |
1 - Not at all
confident |
Finally, I'd like to read you a list
of different ways the VA could reach out to you about the
benefits that you are eligible for. For each one, please let me
know if that's a way that you would like to learn about your
benefits. *Read
each item aloud. |
Numeric
response options |
Discussion
Cognitive interviews with a range of veterans provided valuable insights to revise the Supplement. Veterans identified areas that were not covered by the Supplement but were perceived to be important to understanding and serving transitioning veterans. The cognitive interviews also provided feedback for revising question wording and response options to ensure that the questions were understood as intended and measured the targeted concepts. Although the sample of veterans that we were able to interview was not as diverse as we had originally planned, these recommendations should serve the majority of veterans.
1 The original study design included online data collection, which was ultimately terminated after pilot results indicated a low incidence rate of the target sub-groups in the recruitment platform population and poor data quality.
Page
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Yu, Erica - BLS |
| File Modified | 0000-00-00 |
| File Created | 2021-04-24 |