QSS - Attachment 6: Cognitive Testing Initial

Results from Cognitive Testing of the

Quarterly Services Survey (QSS)

Prepared by Rebecca Morrison (ESMPD)

15 September 2003

1. Due Date

Many respondents will have trouble submitting a report by the 18^th of the month following the close of the calendar quarter, primarily because their records are not yet ready. While some companies close out their quarters within 10 days, many more companies require 30-45 days to close their books. While estimates are more important than book figures to the Census Bureau, the primary respondents tend to be accountants, who prefer holding onto the form until a firm number is available.

Training respondents to provide early estimates, rather than late book figures, will take time and effort – analysts and clerks will likely need to make a significant number of phone calls during early iterations of the survey so that information is obtained in a timely manner.

2. Survey Coverage

During the two rounds of testing, we varied the survey coverage descriptions. In some cases, we used a 3-digit NAICS statement, while in other cases, we used a 4- or 5-digit statement. Generally speaking, respondents were confused by descriptions that were too vague, and were annoyed when a more specific description did not apply to their business. Finding a suitable “middle-ground” option may prove difficult.

It is preferable to err on the specific side, and be prepared to a) change the company’s NAICS code accordingly and b) send appropriate forms in the future. If a company will continue to receive an “incorrect” form, the respondent should be informed, preferably by phone call, and encouraged to continue responding.

3. Revenue

Quarterly revenue numbers were generally easy for companies to obtain. For some companies that operated on a 5/4/4 accounting schedule, or a fiscal year with non-calendar quarters, respondents generally made some effort to provide calendar quarter revenue numbers. Sometimes, the figure(s) provided did not exactly match the dates requested – they added a couple of days or cut off a couple of days, or they provided different months. However, companies generally seemed willing to provide the revenue information for a comparable period of time, then specifying that time period in a subsequent question.

For some companies, revenue remains relatively stable from quarter to quarter. Few companies we spoke with during testing indicated that there was any seasonality to their business cycle. Variations tended to be the result of the general state of the economy.

4. Source of Revenue

Companies tended to know their customer base, and could easily categorize their customers into the three categories provided (government; business firms and not-for-profit organizations; household consumers and individual users). For some companies, providing an actual percentage – or even a decent estimate – would be difficult since this information usually isn’t tracked in company records. These companies would tend to provide a top-of-the-head estimate, which has the potential to be different from reality.

The breakout of sources of revenue does not appear to be likely to change from quarter to quarter; virtually all respondents said this remains stable.

When respondents were presented with hypothetical scenarios containing fictitious information about companies, they generally responded properly. However, one scenario had a large amount of variation. In this scenario, respondents were asked about revenue from three quasi-governmental entities: the U.S. Postal Service, regional airport authorities, and transit systems. While respondents put the Postal Service in the governmental category, there was a great deal of confusion about whether regional airport authorities and transit systems should be considered governmental entities or business entities. Though this does not cause much concern for most companies, if they are likely to have quasi-governmental customers, there is likely to be variation across respondents in the way they are reported.

5. Voluntary vs. Mandatory Survey

Since the questionnaire is coming from the Census Bureau, it is largely thought to be a mandatory survey. Though some respondents noted that it is in fact voluntary, they said they would be likely to complete and return it because of its brief nature and overall simplicity.

6. Electronic Reporting

Most respondents liked the idea of reporting electronically, saying it would make things easier. In addition, some respondents commented that reporting electronically would mean they would be able to provide a better estimate than one that would be mailed in, since they could hold off on reporting by a few days.