Attachment I - Cognitive Testing Report

Attachment I
TO:

Karen Kosanovich
Office of Employment and Unemployment Statistics

FROM:

Robin Kaplan and Tywanquila Walker
Office of Survey Methods Research (OSMR)

DATE:

11/29/2024

SUBJECT: Report on Cognitive Testing for the 2025 Contingent Work Supplement
Introduction
The Bureau of Labor Statistics Office of Employment and Unemployment Statistics (OEUS) is
revising the Contingent Work and Alternate Work Arrangement Supplement (CWS), which will
be collected in 2025 as part of the Current Population Survey (CPS). Survey Methodologists in
the Office of Survey Methods Research (OSMR) conducted questionnaire pre-testing from May
to September of 2024 to help inform changes to the survey.
The CWS was last conducted in 2023. The current testing focus is on questions about task- and
app-based work that could be used to measure those who obtain work or pick tasks by using an
app or website to directly connect them with customers, sometimes referred to as digital platform
workers. There are several criteria that may distinguish work obtained using apps or websites,
such as whether payment for work is made through the app or website, whether the worker is
paid by the task or project, whether the worker creates a profile or receives a customer rating
through the app, and how long the typical task or project lasts.
To learn more about this work, CPS added several questions to the CWS. For this testing, we
aimed to understand whether respondents understand the questions and can answer the questions
as intended. The purpose of the cognitive testing was to ensure that (a) the questions show
construct validity and will serve to address the issues outlined above, (b) respondents to the
survey understand the terminology used in the questions, (c) respondents have the requisite
knowledge to answer the questions for both themselves and for other members of their
household, and (d) whether respondents have difficulty or sensitivity answering the supplement
questions.
Methods
OSMR conducted two rounds of cognitive testing, for a total of 25 participants (16 participants
in Round 1 and 9 participants in Round 2). Using an iterative design, questions were revised
between the first and second rounds of testing. See Appendix A and B for the CWS
questionnaires used in the first and second round of testing, respectively.
All participants indicated that they worked full or part time doing one of the following types of
work: task- or app-based work, freelancing, self-employed, or as an independent contractor, or
lived with someone else who did so (i.e., proxy response). Some participants or proxies who had
traditional wage and salary jobs were included in the sample to determine how the questions
1

functioned for non-platform workers. Participants were offered a $50 incentive. OSMR
interviewers administered a brief set of CPS labor force questions to confirm that they were
currently working, as well as a household roster of other adults aged 15 and above. Afterward,
participants answered the CWS questionnaire for themselves and up to two additional household
members. After completing the questionnaire, participants answered debriefing questions about
their experience answering the questions.
Participants
Round 1

A total of 16 (10 female, 6 male) participants were included in the first round of cognitive
testing. The average age of the participants was 38 years old, with a range of 25 to 63 years old.
Ten participants had a Bachelor’s degree or higher, 3 had some college, and 3 had a high school
diploma. In addition to the 16 self-reports by the participants, 14 proxy reports were also
provided. A total of 17 participants (12 self-response and 5 proxy-response) had multiple jobs.
The tables below display additional information about participants’ work status.
Table 1. Employment Status, Round 1
Employment Status
Family Business
Government
Non-profit Organization
Private Company
Self-employed
Don't Know
Refused
Total

Self

1
1
1
6
7
0
0
16

Main Job
Proxy
Total
1
2
5
6
1
2
4
10
2
9
1
1
0
0
14
30

Self

0
0
0
2
9
0
0
11

Second Job
Proxy
Total
0
0
0
0
1
1
1
3
3
12
0
0
0
0
5
16

Table 2. Worker Classification, Round 1
Classification
Traditional Wage/Salary Worker
Self-employed
Independent Contractor/Freelance Worker
App/Website Worker
Task-based Worker
Temp Worker
Don't Know
Refused
Total

Self
9
7
10
7
4
5
0
0
42
2

Main Job
Proxy Total
11
20
2
9
5
15
4
11
3
7
2
7
1
1
0
0
28
70

Second Job
Self Proxy Total
2
2
4
9
3
12
11
4
15
10
2
12
2
0
2
2
2
4
0
0
0
0
0
0
36
13
49

Note: Classifications were determined by how participants answered questions in the survey, and categories were
not mutually exclusive, that is, participants could be included in multiple classifications.

Round 2
A total of 9 (5 female, 4 male) participants were included in the second round of testing. The
average age of the participants was 35 years old, with a range of 25 to 49 years old. All nine
participants had a Bachelor’s degree or higher. In addition to the 9 self-reports by the
participants, 10 proxy reports were also provided. Participants were a mix of task- or app-based
workers, self-employed, freelancers, and independent contractors. More business owners and
self-employed participants were recruited for the second round to see how these types of
respondents would interpret and respond to the questions. A total of 8 participants (7 selfresponse and 1 proxy-response) had multiple jobs. The tables below display additional
information about participants’ work status.
Table 3. Employment Status, Round 2
Employment Status
Family Business
Government
Non-profit Organization
Private Company
Self-employed
Don't Know
Refused
Total

Self

0
0
0
5
4
0
0
9

Main Job
Proxy
Total
0
0
0
0
1
1
5
10
2
6
0
0
1
1
9
18

Self

2
0
0
2
3
0
0
7

Second Job
Proxy
Total
0
0
0
1
0
0
0
1

2
0
0
3
3
0
0
8

Table 4. Worker Classification, Round 2
Classification
Traditional Wage/Salary Worker
Self-employed
Independent Contractor/Freelance Worker
App/Website Worker
Task-based Worker
Temp Worker
Don't Know
Refused
Total

Self
5
4
5
3
1
0
0
0
18

Main Job
Proxy Total
6
11
2
6
3
8
2
5
3
4
1
1
0
0
1
1
18
36

Second Job
Self Proxy Total
4
1
5
3
0
3
6
1
7
4
1
5
1
0
1
3
1
4
0
0
0
0
0
0
21
4
25

Note: Classifications were determined by how participants answered questions in the survey, and categories were
not mutually exclusive, that is, participants could be included in multiple classifications.

3

Results
General feedback
Distinguishing task- and app-based work from a traditional job

During both rounds of cognitive interviews, most participants could answer the CWS questions
for themselves and as a proxy for other household members. However, participants had varying
interpretations of what they considered their task- and app-based work to be, which did not
always map onto a traditional job:
“Something I do in my free time.”
“A hobby.”
“A way to earn money on the side.”
“A one-time thing that I did, which is hard to remember.”
“Things I do to find short-term work.”
As noted, during Round 2, more business owners and self-employed participants were recruited
for the second round to see how these types of respondents would interpret and respond to the
questions. Those who did task-based work for their businesses tended to comment that the
questions were geared more toward people who do app-based work. They could answer the
questions but found them more challenging. Examples included participants who used websites
like Craigslist, social media sites like NextDoor or Facebook (to advertise or find opportunities),
or LinkedIn, Home Advisor, or WhatsApp (to schedule work and communicate with clients).
Participants who cobbled together many different types of task- and app-based work at times
struggled with some of the questions about how many jobs they have. This was because they did
not view the platforms as them having multiple clients or customers, but rather the
platform/app/website has the customers, as there is no direct contact between them and the
customers/clients.
Participants had different general conceptions of the term “gig work.” When asked what this
term meant to them, some answered that it is defined by the length of the task:
“Someone who works short term gigs to determine their own hours.”
“Something fast, something they need in the moment.”
“Someone who picks up small jobs and short-term jobs for quick compensation and
looking for side hustle jobs.”
“Gig work means an individual engages in a short-term job, usually means you found
something online.”
Others defined it in contrast to having a “traditional” job or a freelance job:
“Just somebody who picks up random side jobs here and there.”
“Someone who has a primary job who’s doing something on the side that’s unrelated to
their primary job, maybe a skill they have or a short-term opportunity. Those people are
younger who might want to supplement an income or see an opportunity to grab some
money, and don't consider it a work situation.”

4

“Work that’s not really structured, not considered real work, it’s a little gig or a one-time
task, a person may require a temporary service, it’s the opposite of secure ongoing work.”
“Gig workers go from gig-to-gig; freelancers go from employer-to-employer.”
S7
Round 1: Last week, were you working as an independent contractor, an independent
consultant, or a free-lance worker (on your main job)?
(Read during debriefing): Independent contractors, independent consultants, and free-lancers
typically do not receive W-2 tax forms, rather they often receive 1099s. These workers often
use schedule C when filing taxes and pay self-employment taxes.
1. Yes
2. No
Round 2: Last week, were you working as any of the following: an independent
contractor, an independent consultant, or a free-lance worker (on your main job)?*
(Read during debriefing): Independent contractors, independent consultants, and free-lancers
typically do not receive W-2 tax forms, rather they often receive 1099s. These workers often
use schedule C when filing taxes and pay self-employment taxes.
1. Yes
2. No
*Note that in Round 1, many participants did not realize this was a yes or no question. Instead, they answered
“freelance” or “independent contractor”. In Round 2, the question was modified so that the response options aligned
better with the question.

Most participants were familiar with the tax forms mentioned in the debriefing in S7, especially
if they used 1099s for their job. Most participants also said the descriptions of the tax forms
clarified the question and helped them understand the difference between an independent
contractor/freelancer versus a wage and salary worker. However, some participants were
unfamiliar with Schedule C. Additionally, a few participants weren’t familiar with these forms
because they have someone else do their taxes.
One participant changed her response from no to yes after reading the paragraph about tax forms.
She explained, “I used to receive 1099s, prior to the pandemic. But after, no, I haven’t gotten one
of those in a while. After seeing the examples, I would say no, because I haven’t received a
1099.” This participant also explained that she does not earn enough to receive any of these tax
forms: “I don’t make enough to file for taxes. Until you make $10,000 or more, you don’t need
to file taxes. The question assumes people make enough to file taxes, which isn’t always the
case.” She felt the question was rigid and wasn’t worded in a way that applied well to her: “I
think it should open up the question more, to other situations where you just don’t make enough
to even file taxes. Not just assume there’s a 1099, and that you’re going to file taxes. My
5

situation is different.” Additionally, one participant was confused by this question because she
provides a W-9 for her work: “I got a W9 for an event we did. Made me think which one I’m
supposed to get for gig work doing events.”
Some participants (unprompted) answered the question based on the types of taxes they file
(even prior to seeing the paragraph about tax forms), indicating they used their knowledge of
their own tax forms to help answer the question. Several participants did not understand the
nuances between independent contractor, consultant, or freelance workers. They either just chose
a label they identified with most, or adopted the label the company uses (e.g., independent
contractors for Uber drivers). Most participants indicated that this was not a sensitive question
for them, and the inclusion of tax forms in the question would not cause them discomfort or
change their response.
In Round 2, changing the wording to add “any of the following…” did not seem to help
participants realize that it was a Yes/No question. Most participants did not provide a yes or no
response and instead specified whether they were an independent contractor, consultant, or
freelance worker.
P1
Round 1: (Option A) Does your work involve you getting short-term tasks or projects to do,
often for different customers or employers?
For example, lining up work as a handyman, substitute teacher, freelance computer worker, in-home
health care aide, or delivery driver.
o Yes
o No
Debriefing probe (Option B): Some people’s work involves arranging for and completing a series of
short-term tasks or projects, often for different customers or employers. Does your work involve you
getting short-term tasks or projects to do?
For example, lining up work as a handyman, substitute teaching, doing freelance computer work, providing
in-home health care, or using an app to find people to drive somewhere or deliver items.

o Yes
o No

Round 2: Does your work involve you getting short-term tasks, jobs, or projects to do, often for
different customers or employers?
For example, lining up work as a handyman, freelance graphic designer, in-home health care aide,
substitute teacher, or rideshare or food-delivery driver.
o Yes
o No

Note: In Round 1 of testing, we administered Option A during the survey questionnaire, and then asked participants
to compare Option A to an alternative version of the question (Option B).

6

During Round 1 question administration (Option A), several participants mentioned that if this
question were read aloud without the examples, the response would be “yes” for just about
anyone who works. For example, as one participant remarked, “Even making a Whopper for a
customer, or a lawyer completing a case for a client, can be a short-term task.” Another
participant said that his household member who works as a waitress does “short-term tasks for
customers.” When discussing someone who worked as a teacher in their household, another
participant mentioned that the parents at a school could be considered the teacher’s “customers.”
Relatedly, one participant said yes to this question for another household member who worked as
an intern for a private company. However, the household member handles IT support tickets for
one company and explained that “the customers submitting the tickets are different, but his
employer is the same.”
Several participants were confused by the question due to the terms used, such as “task or
project.” For example, a Lyft driver answered yes because the terms “tasks or projects” did not
resonate with him. After hearing the examples, which included delivery driver, he wanted to
answer yes “because the people I’m driving are the customers.” One participant reporting on
another household member (a waitress) indicated that he would answer yes to this question: “Um
yeah, I guess if I’m hearing that right she has to do different things (short term) for people that
come in to the restaurant” and “She does get paid a low $2 minimum, but everything she makes
is based on tips (for each task/project).”
Another participant worked doing several jobs, including mystery shopping, surveys, and
annotation work. The term ‘customer’ did not resonate for her: “I guess because to me that’s if
you have an actual position then you’d have customers, I just do a job for a company. They’re not
my customers or clients. The companies are not considered a customer/client. I’m an
independent contractor of the company.” After looking closely at the examples during debriefing,
most of these participants understood the scope of the question better and indicated that they
would change their original answer.
In addition, a couple of participants didn’t understand the concept of “short-term task” well
because the task could differ in duration or longevity:
“I think it’s the short-term, I don’t know what that means. If you say handyman, it is a
one-time task, but in-home health aide could be the same person over and over. Delivery
driver could be UberEats or someone who delivers for Amazon going to the same
neighborhood. It’s too vague.”
“My answer is no, but the way you’re looking at it may be yes. A handyman does 50
kinds of things. A substitute teacher does the same thing over and over.”
“For a substitute teacher, it would be a repetitive type of thing. This doesn’t apply to my
work since it’s not necessarily a series, I may do something once, then never hear about it
again. There is no continuation of the same task, it’s one and done and nothing following
after.”

7

During debriefing, we asked for feedback on the examples provided. Some felt the example
“freelance computer worker” was dated, but they understood it. Some participants mentioned
that “delivery driver” was very general and could include someone who works for UPS (as a
wage and salary worker) or someone who works at DoorDash.They noted that the example from
Option B, “or using an app to find people to drive somewhere or deliver items,” might help
clarify this distinction. Several participants mentioned they were surprised to see “substitute
teacher” in the examples, but then it made sense in the context of the question.
During debriefing, we asked participants which option they preferred (A or B). As seen in the
table below, 12 out of 16 participants selected Option A. The reasons for this preference included
finding Option A more direct and to the point and that Option B was too wordy and detailed.

Round 1 – Question P1
Comparing Option A to Option B, which do you
prefer?
14
12

12

10
8
6

4

4
2
0

Option A

Option B

To help respondents understand the question better, in Round 2 the examples were read aloud for
the first household member and then as needed for additional household members. Additionally,
the word “job” was added to the question stem as some participants felt this term resonated more
than “tasks or projects.” During testing, no participants seemed confused or concerned about the
addition of the word “jobs” in the question stem and indicated they understood its meaning as
intended.
Reading the examples aloud seemed to help most participants arrive at a correct answer. The
sentiment from several participants was similar to Round 1, that without reading the examples
most people could say yes to the question. For example, one participant stated: “I paused when I
answered it for my main job because there are a lot of short-term tasks that I’m doing sometimes
for different internal people. The example was helpful, I knew the answer was no.” One
participant, a business owner who also indicated that he worked for a private company, found the
8

phrase “for different customers or employers” confusing. He expressed that “customers are
different from employers; customers pay you and you work for employers.” He later said that the
examples helped him better understand the context.
P2a
Round 1: Do you use an app or website to get these short-term tasks or projects?
o Yes
o No
o Sometimes

Round 2: Do you use an app or website to get these short-term tasks, jobs*, or projects?
o Yes
o No
o Sometimes

*Note: The term “job” was added to the question stem in Round 2 to improve clarity of the question.

Similar to previous questions, some participants answered yes to the question who shouldn’t
have, for example, if their work uses a website or app for a supervisor to assign work to
employees, such as Workday or an IT ticket system. This brings respondents through a path of
questions that don’t always apply to them. Similarly, one participant answered yes to the
question because he uses apps/websites to do his work; he did not seem to understand the
question was about getting the work. This participant went through the series of questions about
apps and websites, and most items did not apply, creating some confusion. Another participant
mentioned she occasionally gets direct texts or emails to do work, rather than through the
app/website: “I guess it’s what I do. I go into an app and find the task. Both jobs work the same.
Every now and again I might have someone text me for work, but that job info is still on the
app/website.”
During Round 2 of testing, most participants seemed to understand this question and did not have
issues answering. One participant answered no to the question because she thought the question
was only asking about apps. After seeing the question in the debriefing, she changed her answer
to yes, because she realized that it also included websites. Another participant in Round 2
mentioned that this question was geared toward people who use pet service apps, like Rover, but
she gets customer through word of mouth or social media, like NextDoor. While she technically
used a website, the wording of “to get these tasks” did not resonate with her: “I think of those as
dedicated for that purpose, you’re using Rover for that, you set up a profile to find work.
Everything I use to advertise is a social media website and myriad other purposes. I didn’t
mention Craigslist, but I use that as well. But none of them are dedicated to the specific tasks.”

9

P4
On your job, are you ONLY paid by the task or project?
o
o
o

Yes
No
Paid hourly

No major issues were observed with this question. Participants seemed to understand the question
and provided an accurate response. In Round 1, some participants specified in the debriefing they
are paid by the hour upon seeing the response category. One participant was confused by this
question because “it depends on the amount I work, or the amount I charge. Sometimes I charge
different amounts for different customers or how long I’ve known them.”

10

P5
Round 1:
Option A: Does the app or website you use to get most of your work manage customers’
payments? If you use more than one app or website, think about the one you use to get most of your
work.
o Yes
o No
o Sometimes
Option B:
Do customers pay for your work through the SAME app or website you use to get that work?
If you use more than one app or website, think about the one you use to get most of your work.
o Yes
o No
o Sometimes
Round 2:
Option A:
Do you get paid by (APP NAME/the app or website you use to get most of your work) or directly
by the customer? If you use more than one app or website, think about the one you use to get most of
your work
o The app or website
o Directly by customer
o Both
Option B:
Do customers pay for your work through (APP NAME/the app or website you use to get most of
your work)? If you use more than one app or website, think about the one you use to get most of your
work
o Yes
o No
o Sometimes
*In both rounds of testing, participants were presented with two versions of P5. Option A was asked during question
administration, and Option B was shown during debriefing, where both options were compared.

In Round 1, several participants found the phrase “manage customers’ payments” to be vague
and nonspecific. For example, they weren’t sure if the question was asking whether the app
collects money directly from customers, how the workers get paid by customers, or if it referred
to managing customer’s tax documents, or something else. Some workers mentioned they were
removed from the payment process or did not interact directly with any customers (e.g., user
testing surveys), and they did not know how to answer the question, as one participant noted,
“Um, I don’t know. I don’t handle that side.”
11

One participant interpreted the question as asking about how he gets paid (as a DoorDash driver),
instead of how the customers make payments. He explained the answer wouldn’t differ either
way, since DoorDash handles both. Similarly, a participant describing how they get work doing
online surveys had trouble coming up with an answer, “Um, gee, I’m not sure because it’s a
crowdfunding platform, it’s simply a mediator between the researcher and participant. I’m not
sure if the price that they charge is influenced by anything the platform is doing, or if it’s just
their price. When I participate, they tell me how much I’m getting paid, and that’s coming from
the researcher. I don’t know if they take some of the pay. I guess my answer is no.”
During debriefing in Round 1, most participants (10 out of 14) seemed to prefer the alternative
version of the question (Option B), as seen in the figure below:

Round 1 – Question P5
Comparing Option A to Option B, which is clearer or
easier to understand?
12

10

10
8
6
4

4

2
0

Option A

Option B

Participants stated that they preferred Option B because they found it clearer and more direct.
Asking about “the same” app or website seemed simpler than asking how the app or website
handles payments. Some participants felt Option A was confusing and may have referred to tax
documents. The participants who preferred Option A noted that Option B was hard to understand
and felt that the “same app to get work and pay” could be referring to two different things.
In Round 2 of testing, a revised version of Option A was asked during question administration to
try to clarify the question, while the same Option B question wording from Round 1 was asked in
debriefing. One participant, who worked for UberEats and DoorDash, commented that the
question was confusing “because Uber and Doordash pay a part of it, but the customer also tips.
If a customer has a coupon; the payout is through their services but the customer pays for the
product or service.”

12

In addition, this question was confusing for several participants who were task-based workers
that primarily used sites like Craigslist, WhatsApp (to communicate with clients), or Facebook
(to advertise or find surveys to do online). They were able to answer this question (and others in
the series), although they mentioned the questions did not seem to apply because payments are
not made through these websites or platforms. They noted that the questions seemed tailored to
those who use apps to get work only.
When asked in debriefing whether they preferred Option A or B, participants had mixed
opinions. As seen in the figure below, out of 8 participants, five preferred Option A and three
preferred Option B. Note that one participant said she would not select a preference because
“neither applied to her.” This participant was a task-based worker who did dog-walking through
social media ads and word of mouth.

Round 2 – Question P5
Comparing Option A to Option B, which is clearer or
easier to understand?
6
5

5

4

3

3
2
1
0

Option A

Option B

Preference for Option A was usually because the question laid out the two categories: the app or
website versus the customer. Participants liked that the question laid out these two options
clearly and they could select between them. Additionally, as one participant pointed out, “Option
A is broader and could apply to a wide range of workers, while Option B only applies to app
workers.”
Preference for Option B was often because participants felt it was more streamlined, as it was
asking a simple yes or no question as to whether customers pay through the app. One participant
commented that for delivery drivers, Option B would be easier to answer. She felt that Option A
may be confusing for people who work for an app but also use the app as a customer: “It’s
complicated because you’re technically being paid by the customer, but it comes through the
app. The payroll comes through the platform. Maybe rewording [Option A] to ‘who pays’ would
help.” Another participant felt Option B was more straightforward because Option A might
depend on how a person interprets the term “directly by the customer,” because a service like
13

Uber has the customer pay through Uber and some people may “think the payment is filtered by
the service.” Option B is “just asking how you get compensated.”
Some participants felt the questions were asking different things. For example, one participant
stated, “They seem slightly different to me, but seem like they’re geared toward people who
work for apps like Uber.” This participant also felt there was a logic flow problem: for people
who indicate they only use the website for advertising, then they could be skipped out of these
questions. Another participant stated that Option A is “asking whether you are paid by the app or
the customer, whereas B is only asking if the customer pays through the app.” Other participants
felt both options were asking the same thing, but Option A allowed them to specify whether it’s
the app or customer.
P9.5NEW
Round 2:
When you get tasks, jobs, or projects through (APP_NAME/this app or website), does it:
Select the one that gets you most of your work.
(1) Automatically assign you work
(2) List work that you can select or bid on
(3) Allow potential customers to select you from a list of workers
(4) Only give you a way to advertise
(5) Or does it do something else? (Specify: ___________________)
In Round 2 of testing, a new question (P9.5NEW) was added to assess how the participants’ app
or website was used to get work. During question administration, most participants asked for this
question be repeated one or more times. The interviewer had to read the question slowly, reading
the numbers before each option to cue participants that multiple categories would be read.
Participants almost always answered by giving a number instead of the category label, since they
couldn’t recall what the category said. This process of reading and repeating the question and
categories was often time-consuming. The question was easier to understand when participants
could view the showcards in debriefing.
Comprehension of the question categories was poor. For example, participants didn’t understand
category 1 well. They explained that most of these apps do not automatically assign you work,
rather the worker has the flexibility to accept or decline work. Additionally, the terms “select or
bid on” and “list work” in category 2 were also confusing for participants. Participants explained
that the apps do not always list work, but they may send a notification when another task is
available. The term “selecting” generally made sense to participants, but for those unfamiliar
with apps that involve bidding (e.g., Fiverr), this term seemed out of place.
Participants who used websites to search for tasks or projects tended to select category 5,
specifying that they use websites like Craigslist to search for job opportunities or answer ads.
One participant, an UberEats and DoorDash driver, explained that the app will let you see 2-3
jobs and then she can select one of them. Regarding option 2, which she selected, she explained
that she did not agree with the portion that states “list work” but did agree that she can select
14

work. She also noted that option 1 would not apply because work is not automatically assigned.
When asked about whether alternative language such as “Automatically suggest or assign work,”
the participant agreed this would have made more sense. She also indicated that she could select
multiple categories (e.g., 1, 2, or 5) depending on the situation or context. Similarly, another
participant, answering as a proxy for an Uber driver, noted that the jobs aren’t “listed” in the app,
but they appear as quick blurbs and the drivers are given a short period of time to accept or reject
them, but “there is no bidding.” He felt that “a portion of category 2” was correct – the app
presents work that you can select. During debriefing, when participants could see the question
categories, one participant changed his answer from 5 to 2. The word “bid” did not resonate with
him during question administration. As he expressed, “after thinking more about it, I do bid on
projects for secret shopping tasks - you have to put a price on what you want to be paid for the
task.”
P12
Some people get their next assignment from their boss or supervisor. Other people only work when they
line up a task, *job, or project on their own, perhaps using an app or website. At your MAIN job, once a
task, *job, or project is completed, how do you get your next one?
(1) Boss or supervisor assigns it
(2) Line up own work/responsible for finding next task
(3) Use app or website
(4) Advertising
(5) Customers come through word of mouth, networking, or referrals
(6) Repeat customers
(7) Something else (Specify)
*Note the only difference between Round 1 and Round 2 wording was the addition of the word “job” in the question
stem (noted by an asterisk and blue font).

In both rounds of testing, participants understood this question well. However, response selection
was sometimes challenging. For example, choosing between options 2 versus 3, 5 versus 6, and 2
versus 5 was sometimes difficult for participants, and they felt multiple categories applied or it
was a mix of two categories. For other participants, their responses did not always line-up with
the provided categories well. In these situations, many responses were coded as “7- something
else,” such as looking at Craigslist ads or using social media. These responses could align with
multiple categories, including options 2, 3, or 4. One participant, an Uber driver, answered that
he “moves from place to place, visiting locations he thinks people will need rides.” After
repeating the question, he explained that Uber “sends requests to him for new rides.” This
participant also answered for a household member using UpWork for writing jobs explained,
“you can bid on a job, or a customer can approach you to do a job.”
Another participant, an UberEats driver, felt this was question too long: “You lost me midway
through the first or second sentence, I think it could be shortened. Personally, I think the average
person would get lost too. For gig work, the first sentence doesn’t usually apply.” She also felt
the category about advertising did not make sense. One participant noted that options 2-6 could
apply to her, but she selected 2 since it felt like an umbrella to her: “I would choose number 2
because even if I have an app or website or use word of mouth, I’m still lining it up on my own.
15

It feels like an umbrella to me, but 2 through 6 all apply to me. Numbers 3 and 4 also apply
because I’m using an app or websites.”
One participant thought the lead-in sentence was misleading for those who are self-employed
since they do not have a boss or supervisor. The same participant also recommended asking
about “jobs” or “job assignments” instead of “tasks or projects” because “I get hung up on task
or project. I would use ‘job’ as a plumber gets a job assignment. ‘Project’ might be something
longer-term or more involved.”
Recommendations
Based on two rounds of cognitive testing, participants had a good general understanding of most
questions on the supplement, and classified their category of work correctly for both themselves
and other household members. Specific recommendations by question are detailed below:
•
•
•
•
•

•

•

•

S7: Interviewers can use the read if necessary text to clarify the meaning of the question
only if respondents ask for clarification related to tax forms, or use their tax documents as
a basis for answering the question and need clarification.
Retain the phrase “short-term tasks, jobs, or projects” for items P1, P2a, P4, and other
items where applicable, as respondents understood this phrase and it improved
comprehension.
P1: Read the examples for the first household member; include in interviewer training
that even if respondents interrupt, make sure to read the examples to clarify the meaning
of the question.
P1/P2a/P2b: Specify “main job/second job” for these questions.
P5: Consider whether asking if people get paid THROUGH the app would meet
measurement goals, as this would provide a more clean-cut response (are they paid
through the app, or does the customer pay them directly some other way, not using the
app?). Consider whether the “Both” option should be read aloud and offered as a
response category to reduce confusion about how payments are handled.
P9.5NEW: Consider dropping this question or reducing the number of
categories/simplifying the categories (e.g., Assign work; browse or apply for work; or
advertise your work). Consider making the question “Select all” as the categories may not
be mutually exclusive.
P12: Consider whether this question could be dropped for time. This question can
sometimes be difficult to field code as respondents' answers do not map well to the
categories; additional interviewer training and probing instructions will be necessary to
arrive at a code 1 through 6 and avoid many responses being placed in Something else,
specify. The question may also benefit from an optional probe instructing respondents to
select the MAIN way they get their next task. Consider whether the preamble is needed,
or could also be read if necessary text to shorten the question, or changing the question to
a yes/no format, asking if a boss or supervisor assigns their work.
P13/P14: These questions could be dropped for time if needed; most responses seemed to
indicate participants preferred not having a boss or supervisor assign their work and
enjoyed the flexibility of their work.

16

Appendix

Appendix A.docx

Appendix B.docx

17