2016 ASE Cognitive Testing Report

Attachment D - 2016 ASE Cognitive Testing Report

MEMORANDUM
To:

Patrice Norman, Economy-Wide Statistics Division
Beth Goldberg, Economy-Wide Statistics Division
Naomi Blackman, Economy-Wide Statistics Division

From:

Michael Brennan, Data Collection Methodology & Research Branch, Economic Statistical
Methods Division
Krysten Mesner, Data Collection Methodology & Research Branch, Economic Statistical
Methods Division

Date:

December 28, 2016

Re:

Second Round of Cognitive Testing of the 2016 Annual Survey of Entrepreneurs

This memorandum summarizes the second round of research conducted by the Data Collection
Methodology & Research Branch on the Business Screener, Family Ownership, Banking Relationship and
Finance, Government Regulations, and Business Advice and Mentoring questions being considered for
the 2016 ASE. In December, we conducted cognitive interviews with 18 companies in the San Diego,
California and Washington, DC metropolitan areas. The distribution of companies by size (number of
employees) and NAICS sector is summarized in the tables below. Data for number of employees were
included in the recruiting files provided by EWD based on prior responses either to the 2012 SBO or the
2014 ASE, and may not reflect the most accurate data for the companies we interviewed.
Number of employees
1 to 4 employees
5 to 9 employees
10 to 19 employees
20 to 49 employees
50 to 99 employees
Total

Number of companies
7
3
5
2
1
18

Sectors
21: Mining, Quarrying and Oil and Gas Extraction
31-33: Manufacturing
42: Wholesale Trade
44-45: Retail Trade
53: Real Estate and Rental and Leasing
54: Professional, Scientific, and Technical Services
62: Healthcare and Social Assistance
72: Accommodation and Food Services
Total

Number of companies
1
1
4
5
1
1
2
3
18

We summarize the findings of the research and provide recommendations below. The interview
protocol and test questionnaire used for the interviews are included as appendices to this
memorandum.

Attachment D - 2016 ASE Cognitive Testing Report

Business Screener Questions
Finding #1 - Type of Ownership
Consistent with our findings in the first round of cognitive testing, some respondents expect to see
response options that match the legal structure of their business. For example, several respondents
noted they would look for “LLC”, “Corporation”, “S-Corp”, or “Private equity”. While most identified a
best fit option from the choices provided, the remaining indicated they would use the “Other” option to
write in their legal ownership structure.
Recommendation: We recommend dropping this question and using the questions found on the
2015 ASE.
Finding #2 – Number of Owners
A few respondents commented that the “Do not combine two or more owners to create one owner”
instruction was unclear and uninformative.
While the majority of respondents were able to answer the question, we observed that the “Business is
owned by a combination of the following: individuals, parent companies, estates, trusts, or entities”
response option was confusing to those who read it. One respondent’s interpretation of the category
was “like a joint venture”, while two other respondents felt a business typically wouldn’t have a
combination of these types of owners. Unlike the round one testing, none of the respondents we tested
this question with had an ownership structure where this response option would be the appropriate
answer.
Recommendations:
•
•

Remove or add clarity to the instruction bullet: “Do not combine two or more owners to
create one owner”.
We recommend removing “Business is owned by a combination of the following:
individuals, parent companies, estates, trusts, or entities” response option.

Finding #3 – Business – 10% or More Ownership
Respondents did not have difficulty understanding this question. However, it did seem to influence
some respondents’ interpretation of Question 4 – Percent Ownership. These respondents assumed only
owners with greater than 10% ownership should be reported in Question 4.
Recommendation: No recommendation needed.
Findings #4 – Percent Ownership
As stated in Finding #3, some respondents would not report owners with less than 10% ownership. In
one instance, one of the owners of a small company would not include his wife (3% ownership) in

Attachment D - 2016 ASE Cognitive Testing Report

Question 4, but included her in his interpretation of Question 10 - Family Ownership since combined
they accounted for over 50% ownership of the company.
Consistent with our first round of testing, most businesses did not have any issues providing the owners’
names for this survey. Examples of responses included “No, I think that’s public knowledge”, “Someone
could just google or look that up on Facebook”, and “I don’t think most companies would have an issue
with this”. However, two respondents felt very strongly that this information was none of the Census
Bureau’s business as it is already reported to the IRS.
When filling out the form on paper, we observed some users instinctively wrote the name of the owners
in the first column and the percentage owned in the second column.
Recommendations:
•
•

If subject matter experts desire to have any percentage of ownership reported rather than
percentages greater than 10% it should be noted in the instructions.
Reversing the Name and Ownership Percentage column so respondents fill in the owners’
names first may make the table more user friendly and easier to read.

Owner Specific Questions
Finding #5 – Owner 1 – Education
We observed respondents missing the phrase “prior to establishing, purchasing, or acquiring the
business” in both Question 5, where it appears at the end of the question, and Question 6, where it
appears at the end of the question. However, most respondents indicated it would not change the way
they would report. A few respondents did note that this difference would add uncertainty to their
answer.
Several respondents were unfamiliar with the educational background of their owners. In this scenario
they would either assume the answer or need to reach out to someone who knew the information.
One respondent, who was from Europe, would use “university” to refer to undergraduate level
coursework. A “College”, in their case, is a school a person attends to acquire a trade, technical, or
vocational education.
Recommendations:
•
•

Change the title of the question to “Owner 1 - Education Prior to Business” in hopes that
respondents associate the question with the owners’ pre-business education.
Consider adding a “Don’t know” option.

Finding #6 – Owner 1 – Field of Degree
As stated in Findings #5, respondents did not notice the phrase “prior to establishing, purchasing, or
acquiring the business” when it was located at the beginning or end of the question. Also, respondents

Attachment D - 2016 ASE Cognitive Testing Report

who did not have information regarding the owners’ education in Question 5 did not have an answer for
Question 6.
In a few cases, respondents who did not answer bachelor’s degree or higher in Question 5 missed the
response option “No Bachelor’s, Master’s, Doctoral, or Professional degree” and selected a field related
to their trade or college coursework.
Some respondents had issues with finding or selecting response options. Some examples include:
•
•

•

One respondent who scanned the list for Finance did not notice it was combined in “Business
and Finance”.
Another respondent, who’s business owner was a software engineer, was not sure whether to
check “Engineering and Related Technologies” or “Information Technology or Computer
Science”. She did not notice the “select all that apply” instruction and would only have chosen
one.
Multiple respondents were unsure what was meant by “Personal Services.”

Recommendations:
•
•
•
•

Change the title of the question to “Owner 1 – Field of Degree Prior to Business” in hopes
that respondents associate the question with the owners’ pre-business field of degree.
Consider a “Don’t know” option for respondents who are unfamiliar with the owner’s
degree.
Change “Personal Services” to “Personal Care Services” if this more accurately describes the
response.
We recommend the sponsors look at the classification of educational programs used by
other sources such as the National Survey of College Graduates and the Integrated
Postsecondary Education System.

Spouse and Family Ownership and Operation Questions
Finding #7 – Ownership or Operation
Generally, the majority of respondents were able to answer this question correctly. We observed that
respondents with multiple owners were confused to whom the question was referring. Some of these
respondents felt the question implied there was a relationship between the owners they listed, while
others interpreted the question as asking if the spouses or unmarried partners of the owners listed were
involved in the business. These respondents requested more detail be included in the question.
Recommendations:
•

We recommend this question be dropped and further testing be conducted to best
identify how to integrate Question 8, 9, and 10 into the screener section. Confusion
may be reduced if respondents answer this question before listing the owners of the
business.

Attachment D - 2016 ASE Cognitive Testing Report

•

Alternatively, it may be possible to incorporate these questions into the “Percent
Ownership” table.

Finding #8 – Jointly Owned
The majority of respondents we tested would have skipped Question 8 and Question 9 because they
answered “no” to Question 7. Respondents who read these questions had many interpretations of the
phrase “jointly owned”. Some respondents felt the term referred to an equal ownership between
spouses, while others described “jointly owned” as both parties having some degree of ownership
whether it was equal or not. Still others thought “jointly” referenced a legal entitlement or stake in the
business. Three respondents suggested using the phrase “domestic partners” rather than unmarried
partners.
Recommendations:
•
•

Add a definition/instruction for “jointly owned” if there if is a specific survey definition
that should be followed.
We recommend further testing of the integration of Question 8, 9, and 10 to the
“screener” portion of the questionnaire.

Finding #9 – Equal Operation
Respondents did not have any issue interpreting this question. However, when probed, they were
unsure why Owner 1 and Owner 2 would be the only response options if more than two owners were
listed. Respondents would prefer Owner 1 and Owner 2 be replaced with the names provided in
Question 4, but would have assumed that is what the response options were referring.
Recommendations:
•
•

If possible, autofill Owner names for the response options.
As stated in Finding #8 this question should be moved to the beginning of the
questionnaire as part of the “screener” section.

Finding # 10 – Family Ownership
The majority of respondents did not have any issues answering this question. One respondent originally
would not have considered him and his wife’s ownership as “family ownership” since he thought this
relationship would be covered by Question 8. Another respondent, whose family’s estate owned a large
portion of the business, was unsure if estate ownership would qualify in the context of this question.
Several respondents indicated the phrase “close relatives” was vague and were unsure who else should
be included. Examples of family members who respondents considered but wouldn’t necessarily include
were aunts, uncles, cousins, and in-laws.
No issues were found regarding including unmarried partners in the family definition.

Attachment D - 2016 ASE Cognitive Testing Report

Recommendations:
•
•

Clarify “close relatives” if there are limits to who should be included in “family
ownership”.
As stated in Finding #8 this question should be moved to the beginning of the
questionnaire as part of the “screener” section.

Banking Relationship and Finance Questions
Finding # 11 – Business Banking Relationships
Most respondents did not have difficulty understanding this question. However, the majority of
respondents do not have information on all banking relationships the owners may possess. Using the
information available to them, all of the respondents able to answer this question would select “Some
of the banking relationships were the same”. None of the respondents we interviewed would answer
“All of the banking relationships were the same”.
Some respondents, who lack knowledge of an owner’s personal details, may be able to answer for one
personal banking relationship based on an owner’s payroll information. Most respondents would need
to ask the owners.
Respondents with multiple owners typically guessed that “some were the same”, but would not be
confident assuming they knew “all” banking relationships that may exist.
A few respondents were unsure how the response option “The owners did not have any business
banking relationships” was possible. To these respondents any transactions conducted in the name of
the business would be “business banking relationships”.
Recommendations:
•
•

Add a “Don’t’ know” response option.
A “yes/no” question may be more appropriate for this situation since respondents typically
do not know all the owners’ personal banking relationships. A possible replacement
question could be:
“In 2016, were any of this business’s banking relationships with the same financial
institutions as any of the owners’ personal banking relationships? Banking relationships
include business checking or savings accounts, credit cards, loans, etc.





Yes
No
The owners had no business banking relationships
Don’t know

Attachment D - 2016 ASE Cognitive Testing Report

Finding #12 - Business Banking Duration
Several respondents had difficulty understanding this question, and a majority had to reread the
question aloud or to themselves multiple times. Multiple respondents identified a relationship between
Question 11 and Question 12, but were unsure which banking relationships should be included in the
context of Question 12. One respondent asked, “Is the question asking how long the banking
relationships that are the same were in place, or how long any personal banking relationship were in
place?”
Once respondents understood what the question was asking, they were unable to provide an answer
because the information was unknown to them. Most respondents indicated they would have to ask an
owner, and some doubted the owners would recall the answer if the business had been established a
long time ago. Other respondents were unsure how to answer for current owners who were not part of
the business when it was created. One respondent stated it would be possible for multiple business
owners to require multiple boxes be selected.
Finally, some respondents commented that the response options “seem odd” or “too short”. One
respondent noted “you’ll most likely get a lot of ‘more than 12 month responses’… seems like it should
be broken out by years.”
Recommendations:
•

•

We recommend this question be removed from the survey. This question is too complex for
companies that have been in business many years, have multiple owners, or have changed
ownership from the inception of the business. This information is not easily accessible to
respondents and may not provide a quality data product.
If the question is added to the survey, we recommend a “Don’t know” response option.

Finding #13 – Outstanding Loans
Overall, most respondents were able to answer this question without difficulty because the business did
not have any outstanding loans.
One respondent would have initially responded “No” but then stated “oh, you have ‘Business did not
obtain an outstanding loan’…I would pick that, I just thought I was answering no to needing a loan, not
no to providing collateral. Seems like two questions.”
Another respondent felt the word “outstanding” conflicted with the word “obtained”. They said, “Are
you asking only about loans ‘obtained’ in 2016 or any ‘outstanding’ loans we had through 2016?”
Some respondents could not provide an example of collateral, while others mentioned their house,
business, or cars.
Recommendations:

Attachment D - 2016 ASE Cognitive Testing Report

•
•

We recommend the question be reworded to: “In 2016, was this business required to
provide collateral or loan guarantee for any outstanding loan?”
If the number of questions added to the survey was not an issue, this question could be
divided into questions. The first question would address if the business had any outstanding
loans in 2016. The second would ask if any collateral was required for those who answered
“yes”.

Finding #14 – Avoidance of Additional Financing
Multiple respondents felt like this question was “asking two different things”. One respondent
suggested removing “and the owner(s) chose not to apply” from the question stem since it is one of the
response options.
When probed about their interpretation of additional financing most respondents said they felt
additional financing was “referring to loans”.
Recommendations:
•

Drop “and the owner(s) chose not to apply” from the question stem and lowercase
“DID” in the first response option. The question should be as follows:
At any time during 2016, did this business need additional financing?
Business needed additional financing and owner(s) chose not to apply
Business needed additional financing and owner (s) did apply
Business did not need additional financing

•

Similar to Finding #13, question number permitting, this question could be made
simpler by first asking if the business required additional financing, and then using a
follow up question to ask whether the owner(s) chose to apply.

Finding #15 – Purchases on Account
Similar to our first round findings, a few respondents indicated they were unfamiliar with the phrase
“purchases on account”.
None of the respondents we met with had heard of or used the term “trade credit”. Respondents used
“invoice payment terms” to understand the context of the question.
Some respondents, who worked for businesses who provide a service, felt these concepts did not apply
to their business since they do not regularly work with “suppliers”. In their case the only time they use
payment terms were for allowing clients to pay in installments.
Recommendations:

Attachment D - 2016 ASE Cognitive Testing Report

•
•

Remove the term “trade credits” from the question.
The question wording could be changed to:
o “In 2016, did this business make any purchases on account or by using invoice payment
terms? Invoice payment terms allow businesses to purchase goods or services from
suppliers now and pay at a later date.”
o Examples such as Net30, Net monthly, and Net 60 could be used if they capture the
context of the information the sponsor would like to collect.

Finding #16 – Negative Impact on Profitability
This question had no major issues. Regarding the addition of “Regulations”, most respondents indicated
they associated the response option with the time spent meeting requirements for regulations.
Recommendation: Change “each of the following” to “any of the following” in the question
wording.

Regulation Questions
Finding #17 - Types of Regulation
A few respondents did not notice the N/A option until half way through the question. These
respondents usually mentioned they would change some response options from neutral to N/A. One
respondent stated “if you had N/A in the first column, I’d probably just check that for most of these.”
When discussing regulations, several respondents mentioned regulations related to employee benefits
such as 401K programs. In many cases respondents would mention employee compensation, but noted
that they did not consider this the same as the “worker’s compensation” response option which they
felt represented the costs associated with workplace accidents. Some would write “benefits” in the
“Other” category while others would just answer for the options provided.
Other response options with issues included “employment records”, “border and trade/trade”, and
“financial regulations”. In all of these cases some respondents could not associate these options with
tangible regulations and were unsure to what they referred. In distinguishing between “border and
trade” and “trade” some respondents referred to “border and trade” as “international” and “tariffs”
while “trade” was associated with “commerce” and “domestic”.
Recommendations:
•
•
•
•
•
•
•

Change “which impact” to “what impact”.
Change “following government regulations” to “following types of government regulations”.
Change “Neutral” to “No Impact”.
Write out “Non-Applicable” rather than use “N/A”.
Add a “Regulations” title over the types of regulations response options.
Remove “regulations” from “Financial regulations”.
Remove “Border and Trade”.

Attachment D - 2016 ASE Cognitive Testing Report

•

Consider adding a category for employee benefit programs and clarify vague categories such
as “Employment Records”.

Finding #18 – Regulations and Starting or Acquiring the Business
Many respondents were not at the business when it was started or acquired, and in many cases would
need to reach out to the owners.
In some instances, respondents would answer the question based on their understanding of the
industry or market in the present and assume it was the same when the business was started.
A few respondents found it difficult to think about “all regulations” in the context of this question.
One respondent noted, “well, they couldn’t have been that bad since we’re in business now”. This
quote captures the possibility of a bias in the data collected from this question which is only asked to
businesses who were successfully started or acquired.
Recommendation: None needed.
Finding #19 – Regulations and Growth of the Business
All of the respondents we met with understood this question.
One respondent noted that in the case of regulations both positively and negatively impacting growth
they would need to somehow weight the impact of each to pick only one response option.
Respondents suggested including “increasing sales” or “hiring new employees” as examples of business
expansion.
Recommendation: None needed.

Business Advice and Mentoring Questions
Finding #20 - Reasons for Seeking Business Advice
Despite the inclusion of the phrase “paid or unpaid advice” many respondents only thought about paid
services or outsourced business functions such as legal advice, accounting, payroll, and banking.
One respondent stated “It’s hard to pick a primary reason and everything that applies… I feel like I
should be limiting myself but I’m not sure to how many”.
Some respondents felt ‘Business finances’, ‘Investment and access to capital’, and ‘Taxes and
accounting’ overlapped. One respondent said “I think you need to move ‘Taxes and Accounting’ to
before ‘Business Finances’…everyone will think about checking it for their accountant”.
Recommendations:

Attachment D - 2016 ASE Cognitive Testing Report

•

•
•
•
•

Remove “primary reason” and change the question wording to “During 2016, did this
business seek any paid or unpaid business advice or mentoring for the following
reasons? Select all that apply.
Move taxes and accounting to above business finances in the list.
Add an “Other” option.
Consider adding a “Legal” response option.
Having separate questions for unpaid advice or mentoring and paid services or
narrowing the questions focus to unpaid or paid services may help capture more
meaningful data.

Finding #21 – Providers of Business Advice
As seen in Round 1, some respondents selected “family”, “friends”, and “employees” without
mentioning any unpaid advice or mentoring in Question 20.
Several respondents felt that “Professional consultants” was not a proper response option to include
services or advice provided by lawyers, accountants, bankers, and IT. Some respondents would write
these title in “Other”, while others may just use “Professional consultants” but felt these providers
warranted their own category. As one respondent explained, “you go to a consultant for a specific need
or a unique situation for your business not something common like doing your taxes.”
A few respondents were unsure they would know all of the advice or mentoring sought by the business.
An Office Manager noted “This business? I know who I reach out to and I know who my boss may
consult...but I won’t know who our owners’ are talking to or who the CFO is talking to.”
Respondents who worked for consulting companies or other service providers felt “Suppliers” did not
apply to them.
Most respondents were familiar with the SBA but not as likely to know the SBDC, WBC, or MBDA.
Recommendation:
•
•

Consider changing professional consultants to “Legal and Professional Advisors”.
Alternatively, add examples of the groups the sponsor would like to be included in
“Professional Consultants”.

Finding #22 – Outcome of Advice or Mentoring
All the respondents we met with would be able to answer this question. A few respondents indicated
they may not know the outcome of the advice as it could be ongoing but could answer based on their
expectations. One respondent was unsure how to answer if they received positive and negative advice,
but said they could still likely answer the question.
Recommendation: None needed.

Attachment D - 2016 ASE Cognitive Testing Report

Finding # 23 – Exit Strategy
Similar to our round one findings, some respondents were unsure how to answer when multiple owners
need to be considered. One respondent assumed the question implied all of the owners would be
exiting at the same time and would have marked “Sell or merge business with another firm”, but when
probed on the matter she indicated she would expect a transfer between owners if only one owner was
leaving.
In most cases respondents would mark the scenario they felt was most likely to occur, not a scenario
that has been currently strategized by the owners.
Recommendation: Change the final response option to “Business does not currently have an
exit strategy for any owner”.

Attachment D - 2016 ASE Cognitive Testing Report

Appendix A - 2016 ASE Test Questionnaire

2016 Annual Survey of Entrepreneurs
Please answer the following questions for the business named in the mailing address. Answer even if the
business has been sold, reorganized, or ceased operation.
You may use estimates if this survey requests information that is not available in your business records.

. Type of Ownership
Which best describes the ownership of this business? Select all that apply.









Single owner, self-employed, or independent contractor
Business owned by multiple people
Another business or organization (e.g., parent business)
Estate or trust
Non-profit organization
Members from a cooperative or club
Government or tribal entity
Other (specify) - _____________________________

2. Number of Owners
In 2016, how many people owned this business?
•
•

Do not combine two or more owners to create one owner.
Count spouses and partners as separate owners.

1 person
2 people
3 people
4 people
5-10 people
11 or more people
Business is owned by a parent company, estate, trust, or entity
Business is owned by a combination of the following: individuals, parent companies,
estates, trusts, or entities
 Don’t know









Attachment D - 2016 ASE Cognitive Testing Report

3. Business – 10% or More Ownership
In 2016, did at least one person own 10% or more of this business? (Do not count parent
companies, estates, trust or other entities).
 Yes
 No – Select “No” ONLY if no person owned 10% or more of this business

4. Percent Ownership
For the person(s) owning the largest percentage(s) in the business in 2016, please list the
percentage owned by each person and his or her name.
•
•
•

Do not report percentages owned by parent companies, estates, trusts, or other entities
If more than 4 people owned this business equally, select any 4 people
Round percentages to whole numbers. For example, report 1/3 ownership as 33%
Percentage Owned
(Estimates are acceptable)

Owner
1:
Owner
2:
Owner
3:
Owner
4:

Name

Attachment D - 2016 ASE Cognitive Testing Report

5. Owner 1 –Education
What was the highest degree or level of school Owner 1 completed prior to establishing,
purchasing, or acquiring this business? Select one box only.
 Less than high school graduate
 High school graduate – Diploma or GED
 Technical, trade, or vocational school
 Some college, but no degree
 Associate Degree
 Bachelor’s Degree
 Master’s, Doctorate, or Professional Degree

6. Owner 1 – Field of Degree
Prior to establishing, purchasing, or acquiring this business, what was the field of the highest
degree completed for Owner 1? Select all that apply.
 No Bachelor’s, Master’s, Doctorate, or Professional Degree
 Agriculture, Environmental and Related
 Architecture and Building
 Business or Finance
 Education
 Engineering and Related Technologies
 Food, Hospitality, or Personal Services
 Health, Medicine, or Pharmacy
 Humanities or Arts
 Information Technology or Computer Science
 Law or Legal Studies
 Mathematics, Economics, or Statistics
 Natural and Physical Sciences
 Social Sciences
 Other (Specify) ___________________

7. Ownership or Operation
In 2016, was this business owned or operated by spouses or unmarried partners?
 Yes
 No – Skip to Question 10

8. Joint Ownership
In 2016, was this business jointly owned by spouses or unmarried partners?
 Yes
 No

Attachment D - 2016 ASE Cognitive Testing Report

9. Equal Operation
In 2016, was this business equally operated by spouses or unmarried partners?
 Yes
 No, primarily operated by Owner 1
 No, primarily operated by Owner 2

10. Family Ownership
In 2016, did two or more members of one family own more than 50% of this business? (Family
refers to spouses, unmarried partners, parents/guardians, children, siblings or close relatives)
 Yes
 No

11. Business Banking Relationships
In 2016, were this business’s banking relationships with the same financial institutions as any of
the owners’ personal banking relationships? Banking relationships include business checking or
savings accounts, credit cards, loans, etc. Select one box only.





All of the banking relationships were the same
Some of the banking relationships were the same
None of the banking relationships were the same – Skip to Outstanding Loans
The owners had no business banking relationships – Skip to Outstanding Loans

12. Banking Relationship Duration
How long were the owners' personal banking relationships in place before financial transactions
were first conducted by this business? Select one box only.





0 - 1 month
2 - 5 months
6 - 12 months
More than 12 months

13. Outstanding Loans
In 2016, was this business required to provide collateral or loan guarantee for any outstanding
loan the business obtained? Select one box only.





Business did not obtain an outstanding loan
Yes
No
Don’t Know

Attachment D - 2016 ASE Cognitive Testing Report

14. Avoidance of Additional Financing
At any time during 2016, did this business need additional financing and the owner(s) chose not
to apply?
 Business needed additional financing and owner(s) chose not to apply
 Business needed additional financing and owner (s) DID apply
 Business did not need additional financing

15. Purchases on Account
In 2016, did this business make any purchases on account or using trade credits? Trade credits
are invoice payment terms a business establishes with their suppliers allowing them to
purchase goods or services now and at a later date.
 Yes
 No

16. Negative Impact on Profitability
For 2016, did each of the following negatively impact the profitability of this business? Select
one in each row.
Yes

Access to financial capital
Cost of financial capital
Finding qualified labor
Taxes
Government regulations (for example, federal, state and/or local)
Slow business or lost sales
Customers or clients not making payments or paying late
The unpredictability of business conditions
Changes or updates in technology
Other (Specify)

□
□
□
□
□
□
□
□
□
□

No

□
□
□
□
□
□
□
□
□
□

Attachment D - 2016 ASE Cognitive Testing Report

17. Types of Regulations
For 2016, which impact did each of the following government regulations have on this
business’s profitability? Select one in each row.
Very
Somewhat
Negative Negative
Employee hiring
Workers’ compensation
Occupational health and safety
Health insurance
Employment records
Building and renovation permits
Business registration
Business and professional licensing
Health permits and inspections
Border and trade
Immigration and citizenship
Environmental
Trade
Financial regulations
Other (Specify)________________

□
□
□
□
□
□
□
□
□
□
□
□
□
□
□

□
□
□
□
□
□
□
□
□
□
□
□
□
□
□

Neutral

□
□
□
□
□
□
□
□
□
□
□
□
□
□
□

Somewhat Very
Positive
Positive

□
□
□
□
□
□
□
□
□
□
□
□
□
□
□

□
□
□
□
□
□
□
□
□
□
□
□
□
□
□

18. Regulations and Starting or Acquiring the Business
What impact did regulations have on the ability to initially start or acquire this business?
 Positive impact
 Negative impact
 No impact
 Don’t know

N/A

□
□
□
□
□
□
□
□
□
□
□
□
□
□
□

Attachment D - 2016 ASE Cognitive Testing Report

19. Regulations and Growth of the Business
During 2016, what impact did regulations have on expanding this business’s operations, such
as by increasing production, adding locations, or attaining new customers?






Positive impact
Negative impact
No impact
Business did not plan to expand operations
Don’t know

20. Reasons for Seeking Business Advice
During 2016, what was this business’s primary reason for seeking paid or unpaid business
advice or mentoring from others? Select all that apply..
 Business finances
 Employee relations (for example, hiring, workforce retention, employee
performance/growth, employee separation)
 Management and day-to-day operations
 Product development and innovation
 Investment and access to capital
 Succession planning and exit strategy
 Increasing sales
 Reducing costs
 Taxes and accounting
 Regulatory compliance
 Technology/ Information Technology
 Key performance indicators and business targets
 Copyrights, trademarks, and patents
 Did not seek advice/mentoring – Skip to Exit Strategy

21. Providers of Business Advice
During 2016, from whom did this business seek the advice or mentoring selected in the
‘Reasons for Seeking Business Advice’ question? Select all that apply.
 Family (Family refers to spouses or unmarried partners, parents/guardians, children,
siblings, or close relatives.)
 Friends
 Professional colleagues
 Employees
 Professional consultants
 Customers
 Suppliers
 Government-supported technical assistance program (for example, Small Business
Administration (SBA) Small Business Development Center, Women's Business Center, or
Minority Business Development Agency (MBDA) Business Center)
 Other (Specify)

Attachment D - 2016 ASE Cognitive Testing Report

22. Outcome of Advice or Mentoring
During 2016, did the advice or mentoring selected in the ‘Reasons for Seeking Business Advice’
question lead to positive business outcomes or changes in business operation that are
anticipated to be positive?
 Yes
 No

23. Exit Strategy
Which of the following best describes this business’s current exit strategy for any of the
owners? An exit strategy is a plan the business owners create to describe how they intend to
exit the business and capture their investment. Select all that apply.
 Walk away from the business
 Liquidate or sell off assets and repay the business’s liabilities
 Sell the business to employees or managers (for example, offer an Employee Stock
Ownership
Program (ESOP),
management buy-out, or employee buy-out)
 Sell or merge the business with another firm
 Sell the business to another individual that is not an owner of the same business
 Sell or transfer ownership to another owner of the same business
 Sell or transfer ownership of the business to a family member(s) that is not an owner
of the same business
 Prepare an Initial Public Offering (IPO)
 Other (Specify)
Business does not currently have an exit strategy

Attachment D - 2016 ASE Cognitive Testing Report

Appendix B – 2016 ASE Round 2 Protocol
2016 Annual Survey of Entrepreneurs – Cognitive Interview Protocol
Research Questions to Address:
• Are respondents able to answer the questions as intended?
• Are there any items that cause the respondents problems? What can be done to fix
those problems?
• Are there any order/context effects?
Materials:
• Copies of questionnaire testing draft
• Recorder, batteries
• Consent form
Procedure:
• Go through the Introduction and Before the Questionnaire questions.
• Then provide R with the draft questionnaire and ask R to fill it out. Once R completes
each question, ask the general probes and any item-specific follow-up probes.
• Note how R navigates the questionnaire and if they appear to be having any difficulty
with the questions. If R appears to have any problems (utterances like “Hmm”, confused
looks, etc.), probe on this.
• Go through probes about Wrap-up Questions
Introduction:
• Explain purpose of meeting: to understand the process of answering questions, and to
evaluate how new questions work or don’t work.
• Some questions will seem odd and/or obvious, but I don’t want to assume I know what
you are thinking.
• Explain that we are not testing the respondent – we only want to evaluate the
questionnaire
• Structure of meeting: Understand your business and your role in it, then review the
questionnaire and ask you questions as you complete it.
• Permission to record discussion? Solely for note-taking purposes. Have R sign consent
form.
Before the Questionnaire:
• What types of goods or services does this business provide?
• What is your role in the company? What kind of responsibilities do you have?
• Have you completed other surveys from the Census Bureau or other agencies? If so,
which ones?

Attachment D - 2016 ASE Cognitive Testing Report

•

What was your role in the process for responding to this questionnaire? (Gather data?
Enter data? Consult with data providers? Etc.)

Questionnaire test
Now let’s take a look at the draft questions. I would like you to go through and answer the
questions as if you were filling out an actual form. If you don’t have answers to any particular
questions, you can just make a best guess. I will ask you some questions as we go through
them.
These questions are intended for [name of business as listed in recruiting file].
General Probes:
• What does this question mean to you?
• How did you go about answering this question?
• Would you be able to answer this question yourself? (If no) Who would be the
appropriate person to answer it?
• (If R appears to have problem/confusion) What are you thinking about?
• Reflect R’s specific answer: You said ___. Can you tell me more about that?
• If R notices skip instructions: The electronic system would skip automatically, but we will
go over every question to make sure they are all clear.
Q1 – Type of Ownership
• How would you respond to this question for this business?
• Are these response options clear or unclear?
• Can you think of any types of ownership that may be missing?
Q2 – Number of Owners
• Note respondents selection.
• Who do you consider to be an owner?
Q3 – Business – 10% or More Ownership
• How did you decide which option to select?
• Would you need to consult any records to answer this question?
Q4 – Percent Ownership
• For each percentage: How did you arrive at this figure?
• If more than one owner: How did you select the order of Owners you provided?
• How do you feel about providing the Owners’ names?
Q5 – Owner 1 - Education

Attachment D - 2016 ASE Cognitive Testing Report

•
•

How would you answer this question?
Did you notice the words “Prior to”?
o If no: Would this change the way you answer the question?

Q6 – Owner 1 – Field of Degree
• Are these response categories clear?
• Are there any important categories that are missing in your opinion?
• Note how the respondent would respond if they answered < bachelor’s degree in Q5
Q7 – Ownership or Operation
• Is it clear or unclear who to include when answering this question?
Q8 – Joint Ownership
• What does the phrase “jointly owned” mean to you?
Q9 – Equal Operation
• What does the phrase “equally operated” mean to you?
• How would you know if one person was the primary operator?
• Would you expect “Owner 1” and “Owner 2” to contain the names of the Owners you
provided in Question 4?
Q10 - Family Ownership
• Who did you include when you answered this question?
• Does the definition for family provided match your personal definition of family?
Q11 – Business Banking Relationships
• What types of banking relationships did you include in your answer?
• In your own words, how would you describe each response option?
Q12 – Banking Relationship Duration
• In you own words, what is this question asking you?
• Would you have this information?
• What financial transaction did you consider?
Q13 – Outstanding Loans
• Is this information you would have access to in your records?
Q14 – Avoidance of Additional Financing
• What types of additional financing comes to mind?
• What does the phrase “did not apply” in the context of this question?
• Are the response options clear or unclear?

Attachment D - 2016 ASE Cognitive Testing Report

Q15 – Purchases on Account
• Is the phrase “purchases on account” clear or unclear?
• I the definition of “trade credits” clear or unclear?
• Does your business use the term “trade credits”?
Q16 – Negative Impact on Profitability
• What types of taxes did you consider?
• Can you think of any other negative impacts to profitability your business has
experienced? How do they fit into these categories?
Q17 – Types of Regulation
• How did you determine which column to place your answer for each row?
• Are any rows confusing to you?
• Can you give an example of a financial regulation?
Q18 –Regulations and Starting or Acquiring the Business
• What types of regulations did you consider when answering this question?
• Would you have this information?
Q19 – Regulations and Growth of the Business
• What does this question mean to you?
• What types of regulations came to mind when answering this question?
Q20 – Reasons for Seeking Business Advice
• How would you determine your business’s primary reason(s) for seeking advice?
• In your own words, what does succession planning and exit strategy mean to you?
• In your opinion, what is the different between “business finances” and “taxes and
accounting”?
• Can you think of any other categories that may be missing from this list?
• How would you answer this question if your business sought advice for something not
listed in these options?
Q21 – Providers of Business Advice
• How would you answer this question?
• Does your business have a formal advice or mentor seeking process?
• Is there anyone else your business seeks advice from?
Q22 – Outcome of Advice or Mentoring
• How would you answer this question?
Q23 – Exit Strategy
• Did you have any issues with the response options listed?

Attachment D - 2016 ASE Cognitive Testing Report

•
•
•

How would you answer if your business had an exit strategy that was not listed?
Can you think of any other types of exit strategies not included in this list?
Would your business be comfortable or uncomfortable sharing this information with the
Census Bureau?

Wrap-up Questions
• About how long did it take you to complete the survey before?
• How long do you think it would take you to complete these questions?
• Any other feedback you would like to give us on these questions or your experiences
with the survey?
Thanks for your time!