Cog Testing Round 2

 

TABLE OF CONTENTS

 

OVERVIEW

ROUND ONE RECAP

• □Participants were introduced to the organization of their establishments by sector. Initial reactions to grouping this way was well received 

• □Burden was anticipated to be about the same in terms of pulling data, but respondents felt time savings would be in lack of redundancy with separate surveys &/or less need for coordination of due dates 

• □Respondents expressed preference to answering via spreadsheets and being able to utilize pdfs 

• □Some feedback as to not being aware of delegation feature in e-corr 

• □Participants generally expected certain data (e.g. addresses) to be pre-filled 

ROUND TWO RESEARCH GOALS

• □Gather further feedback regarding the content of the modules and how they relate. 

• • • ▫Should payroll be separated? 

    • ▫How does a need for delegation play into the organization of the modules? 

• □Further explore respondents’ preferences for acquiring data at the establishment versus industry level 

• • • • ▫How does the level of industry affect this? 

• □What are respondents' thoughts about organizing their establishments by NAICS? 

<number>

 

OVERVIEW

• □Over twenty interviews conducted from 1/20/2022 to 2/17/2022 

• □Hour long interviews conducted over MS Teams 

• □Participants recruited via Qualtrics.  

• □Protocol Structure: 

• • ▫Respondents were shown several “mockups” of certain key screens 

• • • ▫They were told these screens were not going to reflect the final design of the instrument but were a tool to investigate the principle of the content/functionality.  

  • ▫Respondents reviewed a consolidated list of the question topics within each module and were asked to discuss any potential areas of difficulty and general thoughts about the groupings. 

  • ▫Finally, respondents were asked to reflect on the concept of AIES as a whole, their thoughts about the overall design, and their thoughts on its affect on their burden.  

<number>

 

DEMOGRAPHICS: TITLES

Respondents were selected from companies that had responded to at least two annual Census surveys in the past. The companies were of medium to large size. Several operated in more than one sector, but not all. Others often operated in more than one NAICS, within one sector.

TITLES:

• □Accounting Director 

• □Associate Vice President of Finance and Risk 

• □Chief Financial Officer   

• □Cooperate Controller 

• □Director of Finance  

• □Manager of Financial Reporting 

• □President 

• □Senior Accountant 

• □Treasury Manager  

<number>

 

DEMOGRAPHICS: NAICS

221100	Oil and Gas Extraction
561900	Other Support Services
722000	Food Services and Drinking Places.
541300	Architectural, Engineering, and Related Services
312100	Beverage Manufacturing
445000	Food and Beverage Stores
311000	Food Manufacturing
622100	General Medical and Surgical Hospitals

524100	Insurance Carriers
441000	Motor Vehicle and Parts Dealers
623000	Nursing and Residential Care Facilities
813100	Religious Organizations
624000	Social Assistance
236000	Specialty Trade Contractors
238000	Specialty Trade Contractors
325900	Structural clay products, nec

<number>

<number>

<number>

 

 

<number>

 

OVERVIEW

F I N D I N G S

• □The overview made sense to respondents 

• □It was very common for respondents to report that they would want the ability to see the full list of questions prior to answering the survey, (list of topics; pdf) 

• • ▫Many respondents gather all the relevant data, then enter it all in as a final step. 

• • • ▫“I would probably pull the instructions to see everything that's being collected.”  

T E R M I N O L O G Y : “Company Level Data”

• □“Company level” was understandable and generally evoked the correct assumptions, but there was a strong preference for the word “Consolidated Company Data”.  

• • ▫Some thought “level” was a little vague. Others said “Company Level” referred more to organizational structural and less about monetary questions  

  • ▫“Consolidated” seems to more appropriately cover both structure and financial information 

<number>

 

<number>

 

REPORTING METHOD

F I N D I N G S

• □Essentially all respondents indicated wanting to use the spreadsheet option 

Interactive Spreadsheet:

• □Those who were asked about an interactive spreadsheet mentioned that they would still want a downloadable template.  

• • • □Many R’s like to print off the template. 

• • • • • □Some fear losing data by going back and forth in the instrument or leaving their responses to sit too long/ being erased 

    • □Difficult to discern how this would be beneficial in terms of time savings, as many R’s want a template to fill out and/or delegate.  

    • □A few people might use, but majority wanted to use a template and delegate/share it.  

Recommendation: Provide both options (downloadable spreadsheet + online version).

<number>

 

<number>

 

MODULE ONE

F I N D I N G S

• □In general, the higher the level of the request, the easier the data will be to pull.  

• • • ▫The information requested in Module One has a greater likelihood to be the easiest to pull.  

    • ▫Payroll and Cap ex were mentioned repeatedly as data that would require reaching out.  

    • ▫There are cases wherein the data are easier to first pull at the establishment level 

• • • • ▫One person mentioned cap ex in particular would require gathering at the establishment level before rolling up. There are likely other topics like this as well, (e.g., robotics was also mentioned) 

Other considerations:

• □Some data will be difficult to pull simply because of the volume (e.g., capital lease agreements) or may just take time to gather (e.g., new lease agreements) 

• □R’s may need to reach out to multiple people/departments, regardless of how easy or difficult the data are to pull 

<number>

 

<number>

 

Company Summary

F I N D I N G S

• □This higher level grouping seemed to make sense to respondents 

• • • ▫Some commented that seeing this breakout would be helpful.  

• □Many respondents assumed they would only be listed in one sector 

• • • ▫(Note that R’s do not generally differentiate between our terminology “sector” versus “industry”. They’re likely assuming that their one primary business line coincides with our definition of a sector.)  

• □There was some question how warehouses would be treated. Another mentioned e-commerce shipping centers 

• □Recommendation: May be helpful to offer an explanation of the groupings/sectors.  

<number>

 

<number>

 

Primary Business Activity

F I N D I N G S

• □Many found this to be a good way to organize the data. “Straightforward” “Prefiltered is easier” 

• • ▫Some R’s did not immediately notice the NAICS description (need to make more prominent) 

  • ▫Several R’s mentioned the description of NAICS was helpful 

• • • ▫“This is how they do it for insurance purposes too. Makes a lot of sense. Easier to understand to be honest” 

• □There was variability in respondent’s familiarity with NAICS 

• □Respondents often don’t understand how/why they’re classified a certain way 

U P D A T I N G  N A I C S

• □The functionality seemed clear for how to update their industry.  

• • • ▫Some mentioned they likely would use this feature.  

<number>

 

Primary Business Activity

ESTABLISHMENTS WITH MULTIPLE PBA’S:

• □Some R’s mentioned they did have locations wherein more than one PBA might be relevant. This pre-grouping might make reporting easier, as opposed to breaking out.  

• • • ▫“Do have more than one code [line of business]. Picking one, that makes it easier…Won’t paint a picture of what's going on in that facility.” 

• • • • • ▫Be clear in instructions that respondents are not  being asked to exclude data 

• □Some R’s assumed this would be where they would update location data (address, Op. status) 

• • ▫Recommendations:  

• • • ▫There should be an area for them to enter in comments regarding any confusion/frustration with their classification.  

    • ▫PBA should be addressed next to any questions regarding COS (address; operational status) May need to be on its own page to minimize any confusion and allow respondents to focus on this task alone.  

<number>

 

Primary Business Activity

RECORD KEEPING AND NAICS

• □Industry questions divided at the 6-digit NAICS, likely more difficult to parse than how R’s are used to reporting.  

• • • ▫For example, ACES is rolled up to four digits not six 

• • • • • ▫This has implications for R’s ability to provide estimates 

    • ▫Respondents will sometimes combine data that we would like them to split (they have one bucket but we want two) 

      • • ▫Respondents often give us what’s easier for them.  

        • ▫“So talking about the buckets earlier, might not be exactly how we have our buckets [organized]. We might combine them where you want them separate. It gets time consuming and very manual to realign into these buckets. The more general the bucket you ask for potentially more easy to provide, not needing to get down to a granular level. One broad general number then I don't need to split it out.“ 

Recommendation

• □Give a heads up for AIES fundamental changes. Especially if there will be a change in industry reporting (i.e., from 4 digit to more complex) offer guidance and let respondents know ahead of time.  

• • • ▫“I go in and use last years guidelines. I try to follow the same process from earlier years.” 

• □Note clearly: “For the purposes of this survey, all business lines will be grouped into one industry code per establishment” 

<number>

KS: R’s won’t be able to fudge as easily either.  Can’t arbitrarily group their data. There are struggles with this. (At that point you’ve almost circled back to estab level)

<number>

<number>

 

 

MODULE TWO

What is easier for respondents, obtaining data at the establishment level, or industry level?

☁ ☁ ☁

• □This is a nuanced question because generally a respondent will have to pull establishment data in order to roll up to industry. Industry is not its own entity. 

• • • ▫For this reason, sometimes pulling at the establishment level is the likely first step.  

• □Rolling up to the industry level is simple for some- but it also depends what we mean by industry. What level industry? Requiring data at the 6+ level NAICS might essentially be equivalent to an establishment level for many companies. 

  • • ▫Flexibility would be ideal 

    • ▫Recommendation: Only ask respondents to confirm their establishment data at the NAICS level that is the broadest necessary level 

☁ General rule of thumb: The more detailed the data request is by establishment, the more difficult it is to collect.☁

<number>

 

MODULE TWO

OBTAINING DATA AT THE ESTABLISHMENTS LEVEL

• □Respondents' ability to pull data at the establishment level is dependent on the individual company’s records  

• • ▫“Boss wanted an [item] report. How many were sold at the store, on that date.” 

  • ▫“Absolutely can roll up to engineering level. Company model is easiest opposed to by location. By industry is easier than by location.” 

  • ▫Establishment is definitely easier- first then break out, because you'll always have the data based on that plants [activities]...but we do look by product line. Readily available. I have legal entity drilled in my end, so start with establishment to look for data. 

• □Sometimes the ability to pull data at the establishment level is dependent on the state the establishment is in. Data may be captured differently and require different ways of looking up the information.  

• • • ▫Examples: Those in the utilities industry; Insurance companies/clinics often do not parse things by location in their records; Contract based services often do not organize records by establishment. 

• • • • ▫“Financials not broken out by location, but by service lines or business lines” 

    • ▫Recommendation: Alert respondents of fundamental collection changes prior to mailout 

<number>

 

MODULE TWO

OBTAINING DATA AT THE ESTABLISHMENTS LEVEL

• □Some R’s will need to reach out to several people/departments. (payroll was frequently mentioned) 

• • • • ▫“It requires multiple people regardless of module.” 

• □Some R’s will need to reach out to each establishment if the question is detailed: 

  • • • ▫“But if the info is asked by each operating unit/call center it will take more time. Consolidated means less time to fill out the report” 

FREQUENTLY MENTIONED TOPICS THAT MAY BE DIFFICULT  

Some are industry specific

• □E-Commerce: “E-commerce is a little tricky- don't have a good way to break that out. What is e-commerce?” 

• □Recommendation: Group examples of e-commerce by industry  

<number>

 

<number>

 

MODULE THREE

ROLL UP ABILITIES

• □Completely dependent on their record structure. For some will be simple, some companies have their location data already broken out, so it’s just another step to roll it up. Some mentioned that rolling up to company level is easiest. 

• □One R mentioned this would require manual work because their system cannot run reports on all their locations at once. Others do not separate their branch data (the utilities person) not easy at all at the establishment level.  

• • ▫“With it currently we can combine 10 or 15 stores and then another 10 or 15 then repeated then group them all together. manual process by individual store.” 

  • ▫“By location just because how our system works.”  

• • • ▫He was saying they could roll up by industry but because of the way their software is structured, it's geographical. So location based is actually easier. Rolling up is just an extra step. 

<number>

 

MODULE THREE

POTENTIAL AREAS OF DIFFICULTY

• □Inventories: Some concern (potentially Manufacturing specific-maybe not all?) 

• □Cap Ex: Record organization varies greatly. Brough up several times.  

• □Cap Ex for some industries may be particularly difficult to pull, either by establishment or industry because the equipment moves. 

• • ▫E.g., equipment currently in one location may have different uses for different activities- difficult to know how to break that out. 

• • • • ▫ Medical industry break out their records by specialty- difficult to do so if the equipment is utilized by multiple locations.  

      • ▫Cap ex may only be associated with locations where property taxes have been paid, even though the equipment might move from place to place. It was impossible to calculate cap ex if they didn’t own the location.  

      • ▫Recommendation: consider offering guidance for this scenario. 

• □Recommendation: If respondents have already provided all relevant establishment figures for a given NAICS, prefill that data so the respondent is not doing redundant work. 

<number>

 

DELEGATION

VARIOUS USES

• □Nearly all respondents thought the ability to delegate would be beneficial. 

• □Some R’s will have to reach out per topic (i.e., cap ex, payroll, R&D etc.) because each topic is associated with different departments. 

• • • ▫“Everyone has their piece of the pie.”  

• □Some R’s wanted the ability to see all similar topics grouped together. This was relevant for delegation wherein you may have to reach out to numerous other departments and it would be good to have all relevant questions ready to send them.  

  • • ▫“I think it's just how we collect and report versus your buckets takes time to sort. I don't want to send to [someone] in payroll fill out page 2, 16, 42, and 26.” 

<number>

 

DELEGATION OF PAYROLL

DELEGATING PAYROLL:

• □Some respondents were enthused by the idea of breaking this into it’s own section, some were more neutral to the idea. 

• • • ▫“Then for payroll we have to go to HR once, put it in and be done- versus mixing payroll and financials going in 2 different directions.” 

    • ▫“Not everyone has access to the payroll, so if we wanted to get all the info about cap ex then everyone has that, but only higher level have payroll.” 

• □Payroll can be a sensitive topic, smaller companies may be at a higher risk of exposing salaries; more of a benefit to have it separated. 

  • • ▫“I wouldn't want like the R&D people to see payroll. Don't want that mixed in with operational.” 

    • ▫“not too many people share [payroll access]. portion that one out. fixed assets aren't going to know payroll. Great one to have maybe sent as a different part. Don’t want people knowing what you make” 

• □Payroll is frequently housed by a third party (e.g., ADP), or housed internally in its own department. 

<number>

 

MODULE GROUPINGS

THOUGHTS ON GROUPINGS

• □Some seemed to assume that the topics would be grouped together– but separate modules have overlapping topics (payroll, cap ex) which may not have been immediately clear.  

• • • ▫“I think the more consolidated in general, the easier to pull the numbers…takes less digging.” 

    • ▫“For me it's all in one spot (similar records) so really going category by category- finding the info. Going to same spot doing the same thing. I don't have to log into all these different places.” 

    • ▫“To me it's easier to break it out into those categories than mixing them all together. Summary module level. Just payroll, just expense. Capital expense and payroll expense… those are independent. Everything payroll related- ALL in one section regardless of what asking for. Anything for capital specific to expenses. What takes precedent.” 

• □Recommendation: To mitigate this slightly, clearly lay out the topics within the modules before the respondent answers questions.  

<number>

 

BURDEN

THOUGHTS ON BURDEN

• □Generally, the idea of something consolidated seemed good but the number of surveys they receive, and their company’s structure plays a big role.  

• □Respondents who only respond to 1 or 2 surveys may perceive AIES as more work/more difficult 

• • • ▫In general, the benefits of AIES are clearer for complex companies 

    • ▫Recommendation: In pre-canvas or other initial contacts regarding of changes before mailout, consider ways to “market” the survey in a way that highlights how these changes may benefit companies large and small.  

• □For companies with more compartmentalized departments, consolidating the surveys may create more difficulty. 

  • • ▫Sometimes one individual may be tasked with filling out specific surveys, so this change may seem overwhelming.  

<number>

 

BURDEN

THOUGHTS ON BURDEN

• □Some mentioned concerns about saving progress as you go. 

• □Several mentioned concerns about the time it would take to complete 

• • • ▫“What I'm wondering is how long it would take me to get through this. I don't know about timesaving; it's pulling the data and confirming it.” 

    • ▫If it was one survey for everything NO thank you. Would be hard to delegate…I want them separately otherwise have to block off a whole week for this one survey.”  

• □As with R1 findings, compiling data is generally most time-consuming  

• □Readjusting/reorganizing the data can be time-consuming as well. May be particularly difficult with regard to breaking out data by detailed NAICS. 

  • • ▫“Large part will be how do you structure this. How different are your buckets to ours?”  

• □May not really save time but be easier to keep track of one versus multiple, Just ONE due date. 

  • • ▫“I hope it would be better, because for me I would be sending to people. All at once and not needing to track and not filling out the front and back stuff over and over.” 

<number>

 

SUBMITTING DATA & TIMING

SUBMITTING DATA

• □Majority expressed preference for submitting all the modules at the end.  

• • • ▫This has implications for linking the modules within Centurion, versus separating them in respondent portal 

TIMING (MAILOUT) CONCERNS

• □Many R’s mentioned concerns about timing of submission and assumed they would need to extend the submission date. 

  • • ▫“I think I would rather have one (survey), but not during property taxes which are up until March and April. I already have enough.” 

• □One mentioned calendar vs fiscal year 

• • • ▫“We follow a fiscal year not a calendar year. So, I have to pull it twice. Then I have to clean up the irrelevant dates.”  

• □One respondent mentioned how helpful providing an alternate contact would be for cases when the primary contact is temporarily unavailable or switching roles 

<number>

 

Key Takeaways

<number>

 

Key Takeaways

<number>

 

T H A N K  Y O U
Q U E S T I O N S?

 

File Type	application/vnd.openxmlformats-officedocument.presentationml.presentation
File Modified	0000-00-00
File Created	0000-00-00

---

© 2024 OMB.report | Privacy Policy

---