Cognitive interviewing Report

Attachment 6

Findings and Recommendations from
Cognitive Testing for the Quarterly
Summary of State and Local Government
Tax Revenue (QTAX)(Excerpt)
Prepared for:
Tyler Kerr, Economy-wide Statistics Division (EWD)
Kristina Frates, EWD

Prepared by:
Kristin Stettler, Data Collection Methodology and Research Branch (DCMRB)
Krysten Mesner, DCMRB

Economic Directorate
U.S. Census Bureau

September 30, 2020

Contents
Research Objectives ...................................................................................................................................... 3
Research Methodology ................................................................................................................................. 4
Participants ................................................................................................................................................... 5
Findings and Recommendations ................................................................................................................... 6
Item Specific Findings and Recommendations ......................................................................................... 6
Items 4h, i, and j, Medical Marijuana sales, Recreational marijuana sales, and Total marijuana sales (h
plus j) ......................................................................................................................................................... 1
Finding #1: Items 4h, i, and j are clear ...................................................................................................... 6
Finding #2: Creating codes and separate accounts for marijuana sales would require effort ................. 6
Finding #3: A couple participants noted that they had some confusion between general sales tax and a
specific gross receipts tax on marijuana ................................................................................................... 6
Item 5h, Marijuana licenses ...................................................................................................................... 6
Finding #1: Item 5h was clear to participants ........................................................................................... 6
Finding #2: Many states do not have a separate code for Marijuana licenses ........................................ 6
Appendix A: Draft Questions ........................................................................................................................ 7
Appendix B: New Tax Code Descriptions for F-72 Classification Manual ..................................................... 9
Appendix C: Interviewer Protocol ............................................................................................................... 10

Research Objectives
The Quarterly Summary of State and Local Government Tax Revenue (QTAX, F-72) provides quarterly
estimates of state and local government tax revenue at a national level, as well as detailed tax revenue
data for individual states. The redesign of the QTAX proposes to add several new data items to include
new or different forms of sales and license taxes. The State Finance and Tax Statistics Branch proposed
these new items due to increasing requests for their publication from data users. The proposed new
data items have been adopted across many states, and continue to increase as a percentage of state
total tax collections. The U.S. Congress, federal agencies, state and local governments, education and
research organizations, investment and banking institutions, and the general public employ these
results. Some major uses include the following:




Development of government component of the gross domestic product estimates
Development of the national income accounts
Tax policy research

The draft questions were discussed with current QTAX respondents using cognitive interviewing. The
objectives of these interviews were to learn about the following:






Do respondents understand what information they are being asked to provide with these new
questions?
Do respondents understand the terminology used in the questions?
Are respondents able to answer the questions that are being asked?
Is the requested information available in respondent records?
Are the instructions helpful to respondents?

The questionnaire draft can be found in Appendix A. The new tax code description for F-72 classification
manual can be found in Appendix B. The interviewer protocol can be found in Appendix C.

Research Methodology
From January through May of 2020, staff from the U.S. Census Bureau’s Data Collection and
Methodology Research Branch (DCRMB) conducted 14 cognitive interviews over one round of testing,
regarding proposed new questions. Participants in the interviews were recruited from a list of recent
respondents to the Quarterly Summary of State and Local Government Tax Revenue. Generally, those
recent respondents were the primary participants in the interviews, but on occasion these respondents
brought one or more colleagues to the interviews, because they were integral to the typical survey data
collection process. On occasion, these people who accompanied the original participant turned into the
primary participant in these interviews. Interviews were conducted over the phone.
Cognitive interviews were the testing methodology used for this project. Cognitive interviews are used
to, ‘(a) understand the thought processes used to answer survey items, and (b) to use this knowledge to
find better ways of constructing, formulating, and asking survey questions’ (Forsyth and Lessler, 1991).1
Cognitive interviews traditionally focus on the four steps of Tourangeau’s (1984) cognitive response
model: comprehension, retrieval, judgment, and communication/reporting.2 Comprehension refers to
the respondent’s interpretation and understanding of the question’s language, structure, and grammar.
In order to answer the question, a respondent must understand what information is being requested on
the survey. Retrieval is the step where relevant information is obtained, either from records or from
memory. The next step, judgment, describes the respondent’s evaluation of the completeness or
relevance of the data obtained. It is here that estimates are made based on partial or incomplete data.
The last step, communication or reporting, deals with mapping the response to the answer space
provided and possibly altering the answer.
While Tourangeau’s model is suitable for household and social surveys, the establishment survey setting
presents additional factors that must be considered. First, instead of or in addition to a reliance on
memory, establishment surveys rely heavily on records and the information contained within them.
Second, organizations tend to have distributed knowledge. Some people are experts in one type of
information, while others keep information about something else. Third, competing priorities, both for
the organization and the individual(s) completing the questionnaire, mean that the survey sometimes
does not receive the amount of attention that researchers and data collectors would like. Finally,
organizations regularly authorize only a few individuals to release data. If the data provider is not
authorized to release the data, an additional step must be added to the response process. Tourangeau’s
model was expanded by Sudman et al (2000) to account for these factors. 3

1

Forsyth, B.H. and Lessler, J.T. (1991). ‘Cognitive Laboratory Methods: A Taxonomy.’ In Measurement Errors in Surveys, P.P. Biemer, R.M.
Groves, L.E. Lyberg, N.A. Mathiowitz, S. Sudman (eds). New York: John Wiley & Sons, Inc.
2

Tourangeau, R. (1984). ‘Cognitive Sciences and Survey Methods.’ In Cognitive Aspects of Survey Methodology, T.B. Jabine, M.L. Straf, J.M.
Tanur, and R. Tourangeau (eds). Washington, DC: National Academy Press.
3 Sudman, S., Willimack, D.K., Nichols, E., and Mesenbourg, T.L. (2000). ‘Exploratory Research at the U.S. Census Bureau on the Survey Response Process in Large Companies.’ Paper prepared for presentation at the Second
International Conference on Establishment Surveys, Buffalo, NY.

Participants
Participants were current QTAX respondents. Interviews were conducted with a variety of sizes and
types of state governments.
Table 1. State governments participating in this study
State governments
Total: 14

The job titles of participants varied, including Financial Reporting Supervisor, Accounting Officer Senior,
Accountant, Chief Economist, Assistant Director of General Accounting, State Financial Consultant,
Management Auditor, Director of the Bureau of Research, Manager, Federal Reporting Team Lead,
Supervisor of Financial Reporting and Management, Financial Reporting Manager, Accountant,
Economist, Administrator, Financial Reporting Supervisor, and Finance Administrator.

Findings and Recommendations
Item Specific Findings and Recommendations
Items 4h, i, and j, Medical Marijuana sales, Recreational marijuana sales, and Total
marijuana sales (h plus j)
Finding #1: Items 4h, i, and j are clear
The concept of 4h, i, and j are clear to participants. The definition of these items matched the
participants’ personal definitions of these items.

Finding #2: Creating codes and separate accounts for marijuana sales would require
effort
For the states that have legalized recreational and/or medical marijuana, many are not sure where this
data would come from or if marijuana is differentiated by recreational or medical. It would be a process
for many states to create separate accounts and parse out marijuana by type of use.

Finding #3: A couple participants noted that they had some confusion between general
sales tax and a specific gross receipts tax on marijuana
A couple participants noted that they had some confusion about the distinction between general sales
tax and a specific gross receipts tax on marijuana. They had to work through whether general sales taxes
on marijuana should be reported here, even though it was already reported in Item 2a. This seemed like a
more general issue as well.

Item 5h, Marijuana licenses
Finding #1: Item 5h was clear to participants
Most participants understood the concept of Marijuana licenses, though there was some confusion re:
whether permits or fees should be included.

Finding #2: Many states do not have a separate code for Marijuana licenses
For states where marijuana use is legalized, many states reported that they do not have a separate code
for marijuana licenses. Several participants noted they do have access and could easily break out.
Several others said they would need to reach out to others to see if this is obtainable.

Appendix A: Draft Questions
OMB No.: 0607-0112 Approval
Expires: 08/31/2019

F-72 draft (8-14-19)
State
name

ID

Period covered
by this
report------------------------->

Calendar
Qtr.

Tot
al

Februar
y

TAX COLLECTIONS

1st2019

Fiscal Qtr.
Omit
cents

(Net of refunds, except when otherwise indicated)
Please report either in terms of 3-month totals or
in terms of individual month amounts, whichever is

more convenient.
1. Property taxes
a. Directly State-collected

T01

b. Received from local government collection
agencies

T02

2. General sales and gross receipts taxes
(and related use taxes)
a. State imposed taxes (plus any retained
portion of local supplements)

T09

b. Collections of locally imposed supplements
to State general sales taxes (net of
State-retained amount)
c. Online sales and gross receipts taxes
Portion of 2a collected from internet-based
retailers and "remote sellers"
3. Motor fuel sales taxes
a. Gross collections
b. Refunds during period
c. Net collections (a minus b)

T13

4. Selective sales and gross receipts taxes,
other than motor fuel
a. Alcoholic beverages

T10

b. Public utilities

T15

c. Insurance

T12

d. Tobacco products

T16

e. Pari-mutuels

T14

f. Amusements

T11

g. Sports betting

T18

h. Medical marijuana sales
i. Recreational marijuana sales
j. Total marijuana sales (h plus i)
k. Other selective sales and gross
receipts

T17
T19

January

Marc
h

5. License taxes
a. Alcoholic beverages

T20

b. Public utilities

T27

c. Motor vehicle

T24

d. Motor vehicle operator

T25

e. Corporations in general

T22

f. Hunting and fishing licenses

T23

g. Amusements

T21

h. Marijuana licenses

T26

i. Occupation and business licenses not
elsewhere classified

T28

j. Other license taxes

T29

6. Individual income taxes
a. Gross collections
b. Refunds during period
c. Net collections (a minus b)

T40

7. Corporation net income taxes
T41
8. Death and gift taxes
T50
9. Severance taxes
T53
10. Documentary and stock transfer taxes
T51
11. Other miscellaneous taxes
T99
13.
Remarks

CENSUS USE ONLY
T77
12

Name

DATA
SUPPLI
ED

Title

Local
Total

Email

BY
Telephone

Area
Code

Numb
er

Extension

Appendix B: New Tax Code Descriptions for F-72 Classification Manual
New Tax Code Descriptions for
F-72 Classification Manual
Code T09 General Sales and Gross Receipts Taxes
Definition: Taxes applicable with only specified exceptions (e.g., food and prescribed
medicines) to sales of all types of goods and services or to all gross receipts, whether at a
single rate or at classified rates; and sales use taxes.
Includes: This includes sales or gross receipts taxes on the purchase or lease of motor
vehicles, if there is no specific and separate tax law covering this activity.
Excludes: Taxes imposed distinctively on sales of or gross receipts from selected
commodities, services, or businesses (report at appropriate Selective Sales and Gross
Receipts Taxes, codes T10 - T19). If a sales tax on vehicles is authorized by a law
distinctly separate from a general sales tax law, use Other Selective Sales and Gross
Receipts, code T19. Also excludes sales tax from internet-retailers (online) and remote
sellers (use Online Sales and Gross Receipts Tax, Code T06).
Code T17 Marijuana Sales Tax
Definition: Taxes on marijuana and marijuana products, including both medical and
recreational purposes. Medical marijuana is defined as taxes on marijuana and marijuana
products sold for use as a medicinal tool as prescribed by a doctor/medical professional.
Recreational marijuana is defined as taxes on marijuana and marijuana products sold for
use other than medical purposes.
Excludes: Criminal or civil fines for possession of marijuana in excess of
allowable amount (use Fines and Forfeits, code U30). For states that do not have
a specific marijuana and marijuana products tax rate, and charge at the general
sales tax rate (use General Sales and Gross Receipts, code T09).

Code T26 Marijuana License Tax
Definition: Licenses for growing, importing, wholesaling, manufacturing, retailing, and
use of marijuana products.
Excludes: Taxes based on volume or value of transactions (report at Marijuana
Sales Taxes, code T17) or on assessed value of property (use Property Taxes,
code T01).

Appendix C: Interviewer Protocol
Draft Cognitive Interview Protocol, 8/14/19
(Kristin Stettler and Krysten Mesner)

Research Questions:


Do respondents understand what information they are being asked to provide with these new
questions?



Do respondents understand the terminology used in the questions?



Are respondents able to answer the questions that are being asked?



Is the requested information available in respondent’s records?



Are the instructions helpful to respondents?

Expected Length of Interview: 30-60 minutes
Materials Needed:


Draft questions



New definitions from Classification Manual



Consent forms



Digital recorder

General probes that may be used throughout the interview:


Reflect back on respondent’s answer: “you said…”



In your own words, what is this question asking?



How did you arrive at this number/answer this question?



What records (if any) did you look at? What line or lines were of interest?



Specifically, what did you include in this number? What did you exclude?



Would you consult other people to obtain this answer?

INTRODUCTION



Explain purpose of meeting: get feedback on potential redesigned questions for the Quarterly
Summary of State and Local Government Tax Revenue (F-72).



There may be times throughout this interview that this seems like a test of you, I assure you that
it is not. The purpose of these interviews is to ensure that we are communicating the data
requests clearly with you, so in that way, these interviews are actually a test of us. On that note,
we want your honest feedback throughout, and if it’s not clear to you how to answer, please do
your best. There will be ample opportunities for you to let us know how to make improvements
so that you could do a better job answering.

ABOUT THE RESPONDENT(S)
First, I’d like to learn a bit about each of you, and your roles at your organization.


What is your job title? What is your background, in terms of education and job experience?



What is your role in completing government surveys?



How much experience do you have completing the QTAX survey?



Do you work with anyone else in your organization to get the data for the QTAX survey?
o

How are those other people involved?

o

How many people are involved?

OVERALL QUESTIONS
Before we get started with our draft questions, I want to ask you to about your financial records.


What sorts of financial records are available?



What type of financial records systems do you have?



[If in person] Are you able to show us your system(s)?

COGNITIVE INTERVIEW QUESTIONS
Thank you for that background information! Now I’d like to have you review the current version of the
QTAX survey, just to refresh your memory.


What was your experience completing this survey?



How long would you estimate it takes to complete, each quarter, including gathering data, etc.?
What takes the longest?



Have you looked through the Classification Manual? Just once, or do you refer to it regularly?



Do you have any comments or suggestions for us regarding the current form?

Next, I am going to show you the revised version that we have drafted, along with some new definitions
for the Classification Manual. Glancing through the form, what is your first impression?
Next let’s look at Items 4h, i, and j, Medical marijuana sales, Recreational marijuana sales and Total
marijuana sales (h plus i).


In your own words, what are we asking for here?



How would you answer these questions?



Where would the data come from in your records?



Do you have this information in your records?



What difficulties might you have retrieving this data?



What types of things would you be including here?



What does “medical marijuana” and “recreational marijuana” mean to you?



Do you have any other terms for this?



Is there any additional information you would like to see provided here?



Would you be likely to look at the definition in the Classification Manual?



Looking at the definition in the Classification Manual, what is your reaction?



Would you like any additional information or clarifications?



Could anything be deleted?



If you were not able to provide medical and recreational marijuana separately, would you be
able to provide the total, in j?



Where was this being recorded in previous forms?



Do you have any other thoughts or suggestions on this item?

OK, let’s look at Item 5h, Marijuana licenses.


In your own words, what are we asking for here?



How would you answer this question?



Where would the data come from in your records?



Do you have this information in your records?



What difficulties might you have retrieving this data?



What does “Marijuana licenses” mean to you?



Do you have any other terms for this?



What types of things would you be including here?



Are individual licenses included here, or somewhere else?



Is there any additional information you would like to see provided here?



Would you be likely to look at the definition in the Classification Manual?



Looking at the definition in the Classification Manual, what is your reaction?



Would you like any additional information or clarifications?



Could anything be deleted?



Where was this being recorded in previous forms?



Do you have any other thoughts or suggestions on this item?

Classification Manual
Now I would like to review the current Classification Manual.


In general, what do you think of the definitions?



Are there additional definitions/instructions you think would be helpful?



Are there any definitions/instructions you think are NOT necessary?



Do you have any suggestions for improvement?

WRAP-UP

Wrap-up
We are almost done, and I just have a few wrap-up questions.


Overall, what did you think about the new questions we reviewed?



Do you have any suggestions for improvement?



Do you think these questions would be easy to answer, difficult to answer, or somewhere in
between? Why?



Could you provide an estimate of how long you think it would take you to answer just the
questions we reviewed today? What is driving that estimate?



Compared to the current version of the QTAX form, what do you think of this revised version?
Would this version be easier or harder for you to complete?



Is there anything else we haven’t touched on today, that you think is relevant?



Do you have any questions for us?

Thank you very much for your time today!