Supporting Statement Part A

OMB Control Number: 1290-0043

National Worker Survey

OMB Expiration Date: 10/31/25

This is a generic Information Collection to add to the umbrella Information Collection Request (ICR) for OMB Control Number 1290 – 0043.

National Worker Survey

Cognitive Interview Data Collections for DOL Research

1290 - 0043

Supporting Statement

Part A

March 2024

Submitted By:

Chief Evaluation Office
U.S. Department of Labor
200 Constitution Ave NW
Room S-4307
Washington, DC 20210

Project Officer:

Kacie Chang

A1. Necessity for the Data Collection

The Chief Evaluation Office (CEO) at the U.S. Department of Labor (DOL) seeks approval for conducting cognitive interviews (also known as cognitive tests) to make sure the questions in the Worker Survey are easy to understand and answer and understood consistently across participants. Cognitive interviews with a large sample (i.e., more than nine individuals) of demographically diverse respondents will help CEO ensure that the wording and concepts presented in the questionnaire are unambiguous, comprehensible, and understood by a variety of individuals from different backgrounds and subgroups of interest (e.g., of different industries/occupations and pay types such as hourly workers, salaried workers, and workers paid by the piece or job). This is an essential step in the development of the instrument.

This proposed information collection meets the following goals of DOL’s generic clearance for formative data collections (1290-0043):

• inform the development of DOL research;

• maintain a research agenda that is rigorous and relevant; and

• ensure that research products are as current as possible.

Study Background

The Fair Labor Standards Act (FLSA) establishes national standards on minimum wages, the payment rate of overtime work, employer recordkeeping, and limitations on youth employment. Although certain workers are exempt from these provisions, the standards set by the FLSA apply to more than 143 million workers and 9.8 million workplaces in the United States.¹

The Wage and Hour Division (WHD) in the U.S. Department of Labor (DOL) promotes compliance with the FLSA through a balance of strategies, including by investigating allegations of violations and by undertaking industry-based high-impact investigations that employ large numbers of low-wage workers who are members of vulnerable demographic groups.² There is no agreed upon definition of low-wage workers, and different definitions based on hourly wage cutoffs result in populations with different demographic, social, and economic characteristics.³ Sector-based investigations in high-priority industries have typically been based on the analyses of industries with a history of FLSA violations that employ workers who are less likely to file complaints.⁴

Ideally, agency-initiated investigations would complement complaint-driven investigations by uncovering the most severe of the unreported FLSA violations. However, the data and resources available to identify industry sectors that have a high likelihood of wage and hour violations are limited.

CEO, in collaboration with WHD, contracted with Westat to conduct a nationally representative worker survey to understand the prevalence and severity of violations of workers’ rights under the FLSA. The study will design a nationally representative survey of workers with oversamples of workers in low-wage industries where violations are most likely to occur.

Westat will conduct cognitive interviews to gain feedback to improve the questionnaire. Cognitive interviewing uses in-depth interviewing that assesses how people interpret, process, retrieve, and respond to words, phrases, questions, and response options.⁵ We will conduct three rounds of cognitive interviews. Each round will consist of 15 interviews. Round 2 will test revisions to the instrument based on Round 1. Round 3 will be optional if any problems with questions persist after Round 2. This methodology reflects current best practices.⁶ Cognitive interviews will be conducted virtually and respondents will be paid a $50 incentive for their participation in the form of an Amazon e-gift card.

Because of the complex nature of FLSA violations, it is important to test questions and concepts on workers with a wide variety of job characteristics, including hourly workers, salaried workers, and workers who are paid by the job or piece. Workers in low-wage industries may also speak languages other than English, and it will be important to have accurate translations of the instrument. In order to interview individuals with a variety of job characteristics, and test in different languages, it is necessary to interview more than nine participants. The need for cognitive testing with more than nine participants is further underscored by the fact that many of the questions in the survey are new or have not been widely used in national labor market surveys.

Legal or Administrative Requirements that Necessitate the Collection

There are no legal or administrative requirements that necessitate the collection. CEO is undertaking the collection at the discretion of the agency.

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

The FLSA protects covered “employees” as defined by the Act and the relevant court interpretations. Independent contractors who are not economically dependent on the contracting party under the “economic realities test” are not covered. In addition, the FLSA exempts certain executive, administrative, and professional employees, as well as outside salespeople and some computer-related employees who are paid on a salaried basis because they are not viewed as economically vulnerable. Currently, these employees must be paid a salary of at least $684 per week and meet specified tests as to the nature of their duties to be eligible for these exemptions. The goal of the worker survey is to produce reliable estimates for all covered, nonexempt workers and especially for workers in low-wage industries.

In order to produce these estimates, the questions and response options on the survey need to be clear, easy to answer, and understood consistently across respondents. We will conduct cognitive interviews to gain feedback on the questionnaire, including how participants interpret key questions and terms, how easy or difficult it is to provide answers, and whether response options are appropriate for participants.

The interviews will occur in three rounds with 15 respondents in each round. The first round will focus on the understanding of survey questions and response options using probing approaches. Participants will answer the survey questions, and then the interviewer will ask probing questions to gather insight on their interpretation of each question, understanding of certain terms, and how they arrived at their answers. The cognitive interviews will mostly focus on new questions not used in existing national surveys. This round will occur in English only.

The second round will include probing on any questions revised after the first round. During this round, we will also time participants to gain a better understanding of the length of the survey. This round will include English, Spanish, Chinese, Vietnamese, and Haitian versions of the instrument. Only the English version of the instrument is included with the current request. After the first round, DOL will submit a change request with the revised instrument in English and translated into additional languages. Submitting a change request for testing in additional languages will increase efficiencies by reducing the need to translate questions that may be revised as a result of the first round of cognitive interviews.

The third round, if necessary, will be used to further explore questions revised after the second round in each language.

Research Questions

Are the survey questions straightforward and easy to understand and answer?

Are the response options appropriate?

Are all questions and response options understood the same way by all participants?

Study Design

Cognitive interviewing is an evidence-based, qualitative method specifically designed to investigate whether a survey question—whether attitudinal, behavioral, or factual in nature—fulfills its intended purpose. The method relies on interviews with individuals who are purposively recruited. These individuals are presented with survey questions in much the same way as survey respondents will be administered the final draft of the questionnaire. The cognitive interviews seek to understand if respondents understand the questions as intended and are able to recall the information about which we ask.

For the respondent universe, DOL estimates that about 143 million workers are covered by the FLSA. Respondents for the cognitive interviews will be recruited via Westat’s intranet (WesInfo) (only family or friends of Westat employees will be eligible), Craigslist, social media, and requests to workers’ rights organizations for up to 45 individuals who work in low-wage industries for up to 90-minute cognitive interviews. Research has shown that Craigslist and social media are effective, cost-efficient methods for recruiting participants for studies.⁷ The WesInfo advertisement (Appendix A) and the social media advertisement (Appendix B) will direct individuals to complete a brief web screener to determine if they are eligible. We will enclose the social media advertisement in an email message (Appendix C) to workers’ rights organizations asking that they share it with their members.

Interested persons will complete a brief web screener (Appendix D). A Westat recruiter will review screener responses to determine if the individual is eligible. If eligible, a Westat recruiter will contact the individual by phone call to schedule the virtual interview. For those who are ineligible, or no longer interested when contacted by the Westat recruiter, no identifying information will be retained. Recruitment data will be maintained in separate, encrypted files, with password protection and access limited to authorized personnel.

We will use Zoom to conduct the interviews. To reduce the number of "no shows" for cognitive interviews, the recruiter will send an email reminder one day prior to the scheduled virtual interview (Appendix E).

Each interview will begin with the interviewer reading the informed consent. Participants will have an opportunity to ask any questions they may have at this time. Next, the interviewer will ask the participant’s permission to audio- and video-record the interview and explain why the recording is important. If the participant consents to recording, the recording will begin and the participant will again be asked to state, for the record, that he or she gives permission to be recorded.

The protocol (Appendix F) will focus on identifying question wordings that are either misunderstood or understood differently by different respondents; vague definitions or ambiguous instructions that may be interpreted differently; items that ask for information that the respondent does not have access to; and confusing response options or response formats. The cognitive interviews will at a minimum explore: adaptation of several items from the interviewer-administered Unregulated Work Survey to a self-administered web survey; recall of hours worked; appropriateness of recall periods for different types of violations; descriptions of industries and occupations; and instructions for recruiting eligible peers using e-coupons in respondent-driven sampling (RDS). The informed consent form is in Appendix G.

We will use a semi-structured probing approach that includes scripted probes tailored to the testing goals, while still allowing for open-ended probing on unanticipated problems that may arise. We will use a combination of concurrent probing techniques (immediately asking respondents about their answer to a question) and retrospective techniques (asking respondents to reflect back on how they interpreted the question at the end of each section of the instrument). Occasionally, we may employ a think-aloud approach, which asks respondents to verbalize their thought processes as they answer questions, with little probing by the interviewer. This approach can work well for certain questions, particularly those that require estimating an answer. However, not all respondents feel comfortable ‘“thinking ’aloud” and data can vary in quality between individuals, so this approach should be used selectively.⁸ All probes will be scripted to be neutral so as not to bias respondents or introduce reactivity in their responses.⁹

The interviewer will then ask the participant both the scripted probes and unscripted probes for any anomalies that may have been observed as the participant was completing the questionnaire (e.g., when trying to answer a question, the respondent may have paused for an extended period of time and look confused). At the end of the interview debriefing, the interviewer will thank each participant for their time and email a link with the $50 Amazon e-gift card.

For the first round, the interviewer will display one question on the screen at a time. Each question or section will be probed on individually as they proceed through the questionnaire.

For the second round, in order to gauge timing, the interviewer will display the whole survey, showing one full page at a time. Respondents will be asked to answer each question, one after the other, and the interviewer will scroll through the word document as the respondent proceeds. When there is skip logic, the interviewer will instruct the respondent on which questions to answer next. The second round will also include cognitive testing using scripted and unscripted probes in Spanish, Chinese, Vietnamese, and Haitian.

A third round, if needed, will test changes to questions made in the second round.

Following the cognitive interviews, interviewers will review their notes and prepare a summary report of the findings with suggested revisions to the instrument.

Universe of Data Collection Efforts

The universe includes workers who are nonexempt from the FLSA including those from selected industries of interest to DOL.

A3. Improved Information Technology to Reduce Burden

We will use electronic Zoom to conduct cognitive interviews. This will allow us to obtain participants from across the country and reduce the burden on the participants by eliminating the need to travel.

A4. Efforts to Identify Duplication

Cognitive interviews conducted under this Generic Clearance will not duplicate instrument development activities by any other studies. Many questions in the survey are new or have not been used in a self-administered survey previously. Because of this, the cognitive testing is necessary to ensure the accuracy of the survey. The information is not available elsewhere.

A5. Involvement of Small Organizations

The cognitive interviews will impose a minimal burden on this sector of the economy. For recruitment of workers for the cognitive interviews, we will ask several workers’ rights organizations to distribute our advertisement to their members.

A6. Consequences of Less Frequent Data Collection

This is a one-time data collection to inform survey design, conducted in three rounds.

A7. Special Circumstances

We will use the race and ethnicity questions with minimum categories only and examples as opposed to the version including multiple detailed checkboxes and write-in response areas. The screener is a web screener for qualitative research. The small, purposive sample is not intended to be representative or used for statistical comparisons. The purpose of asking the race/ethnicity question is only to gauge whether of mix of participants by broad race/ethnicity categories is achieved. The additional detail will not be used and we do not believe justifies the additional burden to respondents. The race and ethnicity question in the main survey will not be asked in the cognitive interviews.

A8. Federal Register Notice and Consultation

No public comments are requested for this information collection.

Consultation with Experts Outside of the Study

The project has a Technical Working Group (TWG) consisting of five experts. The TWG met to review and provide comments on the survey instrument on October 10, 2023. TWG members also provided suggestions for areas to be explored in cognitive interviews.

A9. Incentives for Respondents

We propose a $50 gift card incentive for respondents for this information collection.

A10. Privacy of Respondents

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law.

As specified in the contract, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Safety and Monitoring Plan that assesses all protections of respondents’ personally identifiable information. The Contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. All project staff are required to sign the Contractor’s own Pledge of Confidentiality.

As specified in the contract, the Contractor shall use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard.  The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor shall submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information that ensures secure storage and limits on access.  

A11. Sensitive Questions

There are no sensitive questions in this data collection.

A12. Estimation of Information Collection Burden

Total Burden Requested Under this Information Collection

The table below summaries the burden by instrument. The average hourly wage for respondents to the screener and interview guide is based on estimates from the Economic Policy Institute.¹⁰

Instrument		Total Number of Respondents	Annual Number of Respondents11	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage	Total Annual Cost
Respondent Screener		60	20	1	.0835	1.67	$13.60	$22.71
Interview Guide		45	15	1	1.5	22.5	$13.60	$306
Total Number of Respondents	105		35	Estimated Annual Burden Total		24.17		$328.71

Total Annual Cost

The total annual cost is $18,109. This includes the annual burden plus contractor labor hours, and incentive payments of $50 per completed interview.

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The total cost for the data collection activities under this current request will be $54,568.

A15. Change in Burden

This is for an individual information collection under the umbrella formative generic clearance for DOL research (1290-0043).

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

The data collection is scheduled to occur in March 2024.

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

1 U.S. DOL (2009 July). Fact Sheet #14: Coverage under the Fair Labor Standards Act (FLSA). Wage and Hour Division. https://www.dol.gov/agencies/whd/fact-sheets/14-flsa-coverage

2 Dolfin, S., Maxwell, N., Gould-Werth, A., Yanez, A., Deutsch, J., and Hendrix, L. (2020). Compliance Evaluation Strategies Literature and Database Review. Department of Labor. https://www.dol.gov/sites/dolgov/files/OASP/evaluation/pdf/WHD_ComplianceStrategies_LiteratureReview_Oct2020.pdf.

3 Fusaro, V. A., and Shaefer, H. L. (2016 October). How should we define “low-wage” work? An analysis using the Current Population Survey. Monthly Labor Review, U.S. Bureau of Labor Statistics. https://doi.org/10.21916/mlr.2016.44

4 Bernhardt, A., Milkman, R., Heckathorn, R., Theodore, N., Auer, M., DeFilippis, J., Gonzalez, A. L., Narro, V., Perelshteyn, J., Polson, D., and Spiller, M. (2009). Broken laws, unprotected workers: Violations of employment and labor laws in America’s cities. National Employment Law Project (NELP) Report.
https://s27147.pcdn.co/wp-content/uploads/2015/03/BrokenLawsReport2009.pdf

5 Willis, G. B. (2015). Analysis of the cognitive interview in questionnaire design. Oxford University Press.

6 ibid

7 Gioia C.J., Sobell L.C., Sobell M.B., and Agrawal S. (2016) Craigslist versus print newspaper advertising for recruiting research participants for alcohol studies: cost and participant characteristics. Addictive Behavior, 54:24–32; Whitaker C., Stevelink S., and Fear N. (2017). The Use of Facebook in Recruiting Participants for Health Research Purposes: A Systematic Review. Journal of Medical Internet Research. 19(8):e290. doi: 10.2196/jmir.7071. PMID: 28851679; PMCID: PMC5594255; Darko, E.M., Kleib, M., and Olson, J. (2022). Social Media Use for Research Participant Recruitment: Integrative Literature Review. Journal of Medical Internet Research, 24(8):e38015. doi: 10.2196/38015. PMID: 35925655; PMCID: PMC9389385.

8 Collins, Debbie, ed. Cognitive Interviewing Practice. London: SAGE Publications Ltd, 2015.

9 Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data. The MIT Press.

10 Gould, E. and K. DeCourcy. State of Working America Wages 2022: Low-wage workers have seen historically fast real wage growth in the pandemic business cycle. Economic Policy Institute. March 2023. In 2022, the 10th-percentile wage was $12.57. In states that saw increases in the minimum wage between 2019 and 2022, the average 10th-percentile wage was $13.60 in 2022.

11 Annualized over the three year period of the umbrella generic

8