Cognitive Interviewing for Q Tax, the Quarterly Survey of Local Government Taxes (F-70), Round 2
Prepared for:
Carma Ray Hogue, GOVS
Brian Zamperini, GOVS
Cheryl Lee, GOVS
Prepared by:
Kristin Stettler, RIRS
Elizabeth Sinclair, RIRS
Office of Economic Planning & Innovation
Economic Programs Directorate
U.S. Census Bureau
October 11, 2012 (revised)
In Round 2, we again found that respondents were easily able to report property taxes, even though the item had not been asked on the form previously. The local income taxes were relevant to some of our respondents and they seemed to understand the questions and be able to report the data. All of the data items were generally available in the respondents’ own records.
Again in Round 2, some respondents seemed to want to report all their tax revenues, not just the specified taxes asked about on the form. We are recommending that several methods be used to make it clearer to respondents that we don’t want all their tax revenues.
Respondents again had mixed reactions to the shorter form, with several feeling like the current survey was easier for them, since it was more representative of their government and since they already have their data broken out into those categories and could pull the numbers directly from their books.
Additional issues observed during testing are documented later in this report.
We recommend considering alternative designs, as well as conducting additional cognitive testing, both in person and over the phone. We also recommend considering conducting a pilot test with a small test sample of cases before the survey goes into the field.
Currently data on quarterly taxes by governments are collected on two separate forms. The Quarterly Survey of Property Tax Collections has one data item (property taxes). The Quarterly Survey of Non-Property Taxes asks for eleven data items, including sales taxes, license and permit fees, and income taxes. The Governments division proposed combining the two surveys into one shorter survey. The move to a single questionnaire and the collection of fewer taxes will reduce respondent burden and costs of collection. If it is possible to combine these collections, a single sample would significantly decrease the sample size. A smaller sample size would allow analysts more time to focus on the sample units, thus leading to better quality data that can be released earlier.
Cognitive testing was proposed to gauge respondents’ reactions to combining the two surveys into one form. The new survey form is going to those who have previously completed the Non-Property Tax form, so we were looking for their reaction to the addition of the Property Tax question and the reduced length. Additionally, we tried to learn about how respondents interpreted the questions, what records they used to answer the questions, how burdensome the form was and who the best respondent is.
In Round 1, we tested a draft form with 9 respondents in the Kansas City metropolitan area. Based on findings from that round, we conducted 17 cognitive interviews in Round 2.
In Round 2, RIRS conducted 17 cognitive interviews from September 10-13, 2012. Eight were conducted in the Pittsburgh area and nine were conducted in the Northern Kentucky/Cincinnati metropolitan area. (Several additional non-response follow-up (NRFU) cases were also completed.) Interviews were in-person and typically lasted about one hour. Interviews included the respondent, the interviewer from Census (RIRS) and an observer from Census (GOVS). All of the interviews were audio-recorded with permission. Cases were selected purposively, primarily with regard to size of the government and likelihood of having multiple items.
In general, the respondents were pleased to be asked to participate and were happy their feedback was being considered.
Table 1: Characteristics of Governments Visited
Type of Government |
|
|||
County |
City/Town |
|
|
|
5 |
12 |
|
|
|
Generally, respondents thought this page was “fine,” that it “looks like another government form”.
Some respondents thought the form title, specifically the word “Taxes,” is too vague. Respondents believe that based on the title it is not clear which taxes the survey will be asking about.
Several respondents could not find the online reporting instructions and thought it might be because the website address did not begin with www or http://.
Recommendation: Consider indenting the REPORT ONLINE section (could make the arrow shorter), maybe adding a computer icon in front, and adding space above and below if possible. Also, add “www” or “http:// in front of the website address.
Generally, respondents thought the question was clear, though it was pointed out that since the question is so broad, it sets the expectation that we want respondents to report “all taxes”. The phrase “all taxes” leads the respondent to think that all of the government/agency’s revenue amounts from all taxes will be reported. However, below the question, there are only four specific taxes or items being reported. The respondents are wondering “Where do I report the rest of my revenue?”
When asked what “imposed” meant, respondents provided reasonable definitions, usually relating to levies their government has authority to collect, for example, “taxes approved by our board within the city”. Several did note that “imposed” has negative connotations. Some alternative suggestions were “levied” or “enacted” or “collected” Most respondents thought the instruction to “Report zero. . . “ was adequate, though some had either not noticed it or forgotten it by the time they were entering their data.
Recommendation: Consider underlining “imposed”. Consider rewording the question to be narrower in scope, perhaps “What was the amount of tax collections from the selected taxes below, imposed by your government for the quarter ending June 30, 2012?” Consider revising the title of the survey to reflect that the form only asks for a few select taxes, perhaps “Quarterly Survey of Selected Local Government Taxes”. Consider adding an introduction above the General Instructions that explains why we are only looking for a small number of taxes, and only those imposed by the government. Perhaps: “This survey collects tax data only for a small number of specific taxes and only those imposed by your government. All taxes imposed by your government or tax revenues received by your government may not be reported.” Consider adding an example below item 2 to clarify this issue as well.
Respondents seemed to understand that these includes/excludes applied to all of the questions. The includes made sense to respondents. The second exclude, “Taxes not measured by value” was not understood by most respondents. They couldn’t even guess what it might mean. One respondent said, “even a per capita tax is still a value.” The third exclude, “payments in lieu of taxes” was interpreted slightly differently by different respondents, though it seemed like most would have excluded these anyway. One respondent did say he would include these, since they were in the real estate line of his books and it would be difficult to back them out. The fourth exclude, “Special assessments based on area or footage” was not always clear to respondents. Some did not know what we meant by it. Several respondents noted that the last two excludes seemed to only apply to property taxes.
Recommendation: Delete second exclude bullet, “Taxes not measured by value.” Consider moving the third and fourth bullet back under property taxes. Consider moving exclude “shares of taxes imposed by another government” here. Consider adding some space above and below the includes/excludes to set them off a bit.
Respondents did not seem to have any difficulty with this item, either understanding it or reporting it. Most knew exactly where to pull the figures from their books, and estimated it would take 5-10 minutes or less, on average. One respondent mentioned that she liked the examples in the include statement as she would not have originally thought to include automobile property tax. The former excludes (service charges, interest earnings, fines) were not being included by any respondents. The former includes (levies for debt service, etc) were all being included where appropriate.
Recommendation: No recommendation needed.
In Pennsylvania and Ohio, municipalities cannot impose general sales or gross receipts taxes. However, several respondents from Pennsylvania still would have reported something here. Several were tempted to report the Regional Access District (RAD) tax, which is an Allegheny County sales tax of 1%, collected by the state along with its own 6% sales tax and then distributed to the county and its municipalities. Even when respondents saw the exclude about “shares of taxes imposed by another government” they still wanted to report it because they felt it was a big tax revenue for them and it was a “sales and use tax” as specified under the last include. Also, one respondent included the amounts of general sales tax that their golf course collected FOR the state and another respondent was tempted to include a similar amount, but did not. Allegheny County, which perhaps should have reported the RAD, did not want to, because they felt it was not imposed by them, but was mandated by the state legislature in Harrisburg.
In Kentucky, governments either have a Gross Receipts tax or Corporation net income tax, but they cannot have both.
In general, respondents thought the include and exclude statements were clear. When asked about whether they would include the items mentioned in the second exclude (hotel motel, car rental, amusement) they generally said they would not. Also, they did not think they would include motor fuels, public utilities, alcohol or tobacco taxes, although when asked about them specifically, some were tempted to add them in. Several respondents who previously completed the F-73 were wondering where to put the additional items. In Pennsylvania, this was a tricky item, as it should not have been relevant for the cities/towns, yet several of them found something to report here.
Recommendation: Consider moving the include statement, “Only taxes imposed by your government” and the exclude statement, “Shares of taxes imposed by another government” into the includes/excludes above 2A.
Most of the Pennsylvania municipalities would include earned income tax (EIT) here. Pennsylvania now mandates that EIT be collected by a private collection agency, which complicates the issue. There is now a lag until local governments get their revenues. One respondent would not have included EIT here, because it was not collected by their government any more. Several respondents were unsure whether they should include their local services tax here ($1/week on workers).
Respondents from Kentucky and Ohio seemed to understand immediately if this item was imposed by their government. For the governments that these items pertain to, respondents generally understood what the items are asking for. The governments mentioned that they have an “occupational tax” or an “occupation license tax”. A few respondents mentioned the need to calculate and divide out the portion that is given to the county or state, but that that would be a relatively easy task. The City of Cincinnati said that they would have to contact the tax department to have the individual/corporate taxes split. Currently they report one number and make a comment in the Remarks section that the number reported is a combination of local individual and corporate net income taxes.
Generally, the include statement was interpreted correctly, though one or two respondents were not sure whether “net” income was accurate. The exclude statement was confusing for most respondents. Most respondents found this very easy to report, estimating it would take 5 minutes or less, on average.
Recommendation: Consider clarifying the second include statement under Item 2 to include private agencies, perhaps adding “or authorized agency”. Consider breaking the include into two statements. Consider replacing the exclude statement with simpler wording , such as “Shares of income taxes imposed by another government”. (This recommendation may be irrelevant if the exclude is moved above 2a, as suggested in Finding #3.) Consider whether an exclude is needed to rule out “commuter” or “head” taxes or other flat taxes.
In general, this was not applicable to Pennsylvania respondents, who would usually check the NA box. One respondent noted that he would not have included income taxes on unincorporated businesses here because he would not have read the include statement, only the header (though I believe it wasn’t applicable in any event, since it was not “taxed separately from individual income”).
This item did apply to a couple governments in Kentucky. For the governments that this item pertains to, respondents generally understood what the item is asking for. The governments mentioned that they have a “corporate net profit tax.” The counties in Kentucky mentioned that this item would be on the same report as the Local Individual Income Tax and would be relatively easy to report.
Recommendation: No recommendation necessary.
Generally, the data requested was easily available to the respondents themselves. However it was brought to our attention that in Ohio, there is often a third party that processes and tracks income taxes for certain governments. A respondent did mention that in order to answer the Individual Income Tax (2c1) he would need to get the data from a report from the third party.
Recommendation: No recommendation necessary.
Respondents generally said it could take from 30 minutes up to several hours to complete the form the first time, but that it would usually take 10-60 minutes after that. Quite a few respondents had to add three months data together to get the quarterly figure. Some preferred to take a year to date number and then back out the other months. Also, many respondents had to add multiple data items and/or funds together. Some would program a report so that after the first time, they could just run that report. Others would just note what they did so they could do it again by hand the next time.
It should be noted that mostly in Ohio, there was pushback on filling out the form, even if it only takes 30 minutes. It was conveyed that with decreasing budgets and increasing workforce layoffs, that even a 30 minute voluntary survey would be put to the side, and possibly not be completed at all.
Recommendation: No recommendation necessary.
For the second round of testing, respondents had similar reactions to the first round. Most respondents couldn’t imagine being willing to revise their data. Most respondents were not aware they could edit their data and most said this is not something they would utilize, “once the books are closed and numbers are finalized, they don’t change”. One or two people noticed the fourth bullet under General Instructions. Most of the respondents looked for this information in the Remarks section.
When respondents saw that they should call Census to revise their data, they noted that they may be put on hold and transferred often before reaching the correct person. One respondent said, “I’m not going to call, email maybe”. The form does not clearly state that the phone number directly dials to a census analyst.
Recommendation: Add wording to the end of the fourth instruction in the General Instructions to make it clear that the number will directly put them in touch with an analyst, perhaps “to speak with an analyst”.
Most respondents did not read these in detail. No respondents had any negative reactions.
Recommendation: No recommendation needed.
Suggestions for addressing the survey included: Finance Director, Tax Collector, Tax Department, Chief Financial Officer, Treasurer, Controller, City Manager, City Clerk, and City Administrator. Several respondents noted that it is not a good idea to address it any of the elected officials, such as Chairman or Council President.
Recommendation: No recommendation necessary.
Most respondents preferred being able to use the internet, saying for example, that it’s “easier to report online, then it is done”. One respondent did not like the idea of the user ID and password being printed on the form. It was mentioned that someone else, if not multiple people, would open up the envelope and see the survey before it reached his desk. He did not think that would be very secure. He did not like the idea of someone else being able to access his log-in and report for his city. One respondent complained about not being able to print her completed survey. Two respondents admitted they would only mail back their forms.
Recommendation: Emphasize the internet as the preferred data collection method for respondents. Resolve the printing issue.
Again, quite a few respondents wanted to know how the data is used and why it is important. One respondent suggested emphasizing that completing the form was their “civic duty”. The idea of shortening the survey was well received and respondents mentioned that it would be a good idea that for the first data collection with the shortened survey to mention that in the cover letter.
Recommendation: Emphasize importance of data.
Most respondents were amenable to receiving notifications/reminders via email, though they noted that errors might occur if the usual respondent leaves the government. Several respondents were worried that an email might get lost, and would like the next contact to be by a different method.
Recommendation: Consider collecting a “backup” email address.
Most respondents preferred the short form, saying things like “sometimes less is more”. Respondents thought it was easier to respond to, especially in Kentucky where a lot of the non-property items do not apply. However, several respondents felt that the information provided was not a true representation of their governments. In one municipality, occupation and business licenses are one of the major sources of revenue and if the non-property form was filled out it would provide a better picture of his government. Additionally a few other governments mentioned that the non-property form breaks out the data in a way they already have.
Recommendation: No recommendation necessary.
Respondents felt that quarterly response was a reasonable burden, but wanted to know how the data was used and why such frequent response was necessary.
Recommendation: No recommendation necessary.
Consider alternative designs.
Draft a revised form.
Draft a revised cover letter.
Conduct additional cognitive testing -locally, if possible, prior to the QuIP trip scheduled for November and perhaps over the telephone with some respondents from different parts of the country.
Consider conducting a pilot test with a small test sample of cases (maybe 500? Or a split sample?) before the survey goes into the field.
The Response Improvement Research Staff (RIRS) in the Office of Economic Planning and Innovation (OEPI) assists economic survey program areas and other governmental agencies with research associated with the behavioral aspects of survey response and data collection. The mission of RIRS is to improve data quality in surveys while reducing survey nonresponse and respondent burden. This mission is achieved by:
Conducting expert reviews, cognitive pretesting, site visits and usability testing, along with post-collection evaluation methods, to assess the effectiveness and efficiency of the data collection instruments and associated materials.
Assisting program areas with the development and use of nonresponse reduction methods and contact strategies.
Conducting empirical research to help better understand behavioral aspects of survey response, with the aim of identifying areas for further improvement as well as evaluating the effectiveness of qualitative research.
For more information on how RIRS can assist your economic survey program area or agency, please contact Diane K. Willimack at Diane.K.Willimack@census.gov.
F-73 Cognitive Interview Protocol
Final: September 7, 2012
(This protocol is a guide – the questions presented here will not necessarily be asked exactly as worded in the protocol or in this order. It is important to note that not all questions will be asked in every interview.)
Research Questions:
Can the two surveys be combined?
Can the tax imposers provide responses for the new survey?
1. Introduce observers and their background
2. Purpose of visit:
Let me start by telling you a little about what we will be doing today. Currently there are two quarterly tax surveys for local governments collected by the U.S. Census Bureau: Quarterly Survey of Property Tax Collections and Quarterly Survey of Non-Property Taxes.
The Census Bureau is considering combining the two surveys into a four question survey form. We would like your input and feedback on the possibility of combining the two surveys.
This is not a test of you but the form, and getting information from you about interpretations, impressions, and opinions of the form. There are no right or wrong answers. All feedback is useful.
I will ask you general and specific questions about your government and the form itself. Some questions may seem odd or redundant because we are not accountants, and because we’re meeting with various local governments, that have different organizational structures and ways of keeping records.
Do you have any questions?
3. Permission to audio-tape discussion? Have R sign consent form.
Before we get started: I'd like to audio tape this interview, so I don't have to rely on my memory later. This session is confidential. Only persons connected with this project will have access to your tape. If that is all right with you, please sign this consent form. It also tells you about the confidentiality of this session.
Background Information
1. Respondent Background
Tell me a little about this agency:
What sorts of activities is it involved in?
What is your role in this agency?
Have you filled out government forms in the past? If so, which ones?
Generally speaking, how do government forms get completed at this agency?
Does more than one person fill out a form? If so, what is that process? Do you communicate with other people to complete the form?
When does your fiscal year end? About how long does it take for quarterly figures to be ready?
Now I am going to hand you a copy of the questionnaire. We will go through it page by page. Please write on it, and complete it as if we were not here. I will ask you questions when you have finished the page.
Cognitive Interview
General Probes
In your own words, what was that question asking?
What are you thinking about?
Can you tell me more about that?
You answered, “…” because…?
I want to make sure I understand, can you explain that again?
Can you tell me how you arrived at that answer? What did you include? What did you exclude? What records did you use?
Reflect back on R’s answer and ask R to correct (“I want to make sure I have it right. I think you said, “…?”)
1. Cover page
What is your initial impression upon seeing the form?
In your opinion, what is the most important information on this page?
What is the least important?
Is there any (other) information you would like to see here? What would that be?
What information do you need prior to filling out the form?
Did you notice the UserID and password?
2. Instructions
-Probe on descriptions offered of records, procedures, interpretations, etc.
-Observe whether R appears to read the question and instructions
Did you read any of the information on this page?
(If yes) What did you read?
3. Item 2 (Question Stem and Includes/Excludes)
What do you notice when you look at Question 2?
What does the word “imposed” mean to you?
What is your reaction to the Include and Exclude statements?
(If a government that collects taxes FOR others, ask how quickly they distribute that information?)
4. Item 2A (Property taxes)
How would you go about answering this question? What records would you need to consult in order to answer this question?
Is this information readily available?
Is this information available here or would you need to go somewhere else to get this information? Where?
How long would it take to obtain the information?
Would you include taxes collected by others?
What is your reaction to the Include statement?
How easy or difficult would you say this question is?
What does “all taxes on property, real or personal” mean to you?
Would you include receipts from service charges? Interest earnings? Fines?
Would you include levies for debt service? Levies for contributions to pension funds? Levies for other funds or purposes?
5. Item 2B (General sales and gross receipts taxes)
How would you go about answering this question? What records would you need to consult in order to answer this question?
Is this information readily available?
Is this information available here or would you need to go somewhere else to get this information? Where?
What is your reaction to the Include and Exclude statements?
What does the second exclude statement mean to you?
Would you include motor fuels sales taxes? Public utilities sales taxes? Alcoholic beverage sales taxes? Tobacco products sales taxes? Any other sales or use taxes?
How easy or difficult would you say this question is?
6. Item 2C1 (Local individual income taxes)
How would you go about answering this question? What records would you need to consult in order to answer this question?
Is this information readily available?
Is this information available here or would you need to go somewhere else to get this information? Where?
What is your reaction to the Include and Exclude statements?
How easy or difficult would you say this question is?
7. Item 2C2 (Local corporation net income taxes)
How would you go about answering this question? What records would you need to consult in order to answer this question?
Is this information readily available?
Is this information available here or would you need to go somewhere else to get this information? Where?
What is your reaction to the Include statement?
Would you include commuter or head taxes?
How easy or difficult would you say this question is?
8. Page 2 Layout
What is your overall impression of the layout of this page?
What is your reaction to the example at the top of the page?
What do you think the shaded box around the include/exclude statements below Item 2 means?
What do you think about the location of the N/A box?
Wrap-up
Would the information requested on this form generally be available in this department, or would information have to come from some other department/staff member?
How would you go about getting that information?
Would you pass the form over to that department, or just get information from them over the phone/email/in-person?
When would the data be available from others?
How much time would it take you to complete this report (actual time, not elapsed time)?
Is collecting on a quarterly basis reasonable? If no, how often should this information be collected?
If you wanted to revise data for previous quarters, what would you do? How could we tell you that you can revise up to 7 previous quarters?
Overall, how easy or difficult was it to complete this form? Which were the easiest questions to answer? Which were the hardest to answer?
What are you thoughts on completing this form on the internet? Were you aware that was an option?
Who would be the best person to address this form to? What other titles might be applicable at other comparable governments?
(If respondent/government had not returned survey previously) Why have you not returned this survey in the past?
What is your reaction to this shorter version of the form? (show them the previous version if necessary)
How would you prefer to be contacted about this survey initially? If we emailed you, would you find that acceptable?
How would you prefer to be reminded if this survey form was overdue? If we emailed you, would you find that acceptable?
Is there anything else we might do to encourage response to this survey?
Any other comments?
Review cover letter (if available).
Thank you for your assistance today.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | pick0002 |
| File Modified | 0000-00-00 |
| File Created | 2021-01-29 |