Various Economic Area Pretesting Activities

ASM-MOPS Cognitive Interview Protocol

Draft: April 8, 2010

Before the Questionnaire:

● Tell me a little about your plant:

○ What sorts of activities is it involved in?

● What is your role at this plant? Have you filled out government forms in the past? Which ones?

● Generally speaking, how do government forms get completed?

Now I’m going to hand you a copy of the questionnaire. We’ll go through it page by page. Please write on it, and complete it as if we weren’t here. I’ll ask you questions when you’ve finished the page/form.

● What is your initial impression upon seeing the form?

● What do you notice?

● What information do you need prior to beginning the survey? Of the

information on this page, what is the most important?

Item 1

● In your own words, what is this question asking you?

 Do you have access to this number? If not, how would you go about answering this question?

Item 2

● In your own words, what is the difference between each of the options?

 What does the term ‘establishment’ mean to you?

Item 3

● In your own words, what is this question asking you?

• Did you work in any other locations within the company? If so, did you include this time in your answer?

Item 4

● How would you go about answering this question?

 Do you have access to this information? If not, how would you get these figures?

 How accessible are the figures from 2007?

Item 5

● In your own words, what is this question asking you?

● How would you go about answering this question? Did you consult

with anyone? How would you calculate the percentage? Would you have to estimate?

● What does ‘production workers’ mean to you?

● (Probe their response…If R not around in 2007, ask how they would answer this question).

Item 6

● In your own words, what is this question asking you?

● How would you go about answering this question? Did you consult

with anyone? How would you calculate the percentage? Would you have to estimate?

● What does ‘non-production workers’ mean to you?

● (Probe their response…If R not around in 2007, ask how they would answer this question).

Item 7

● How would you go about completing this item?

● Was the example helpful to you? Are there other examples that should be included to help you or others answer this question?

 What does ‘process problem’ mean to you?

● (Probe their response…If R not around in 2007, ask how they would answer this question. Do they have documented official rules to follow for handling such situations?)

Were you aware of the procedures used in 2007?

Are there other procedures that you use that aren’t listed?

What are the types of performance indicators that you collect at this plant?

Is the term “production performance indicators” a relevant term?

Item 8

● How would you go about completing this item?

● Was the example helpful to you?

● What does the phrase ‘key production performance indicators’ mean to you? What determines which indicators are key and which ones are not?

● Describe the monitoring process.

● (Probe their response…if R not around in 2007, ask how they would answer this question.)

 Are you involved in generating or monitoring these performance indicators?

Item 9

● In your own words, what is this item asking you for?

● How would you go about completing this item?

● (Probe their response…if R not around in 2007, ask how they would answer this question.)

Item 10

● How would you go about completing this item?

● (Probe their response…if R not around in 2007, ask how they would answer this question. How are the indicators ‘shown’ to the plant manager?)

 (if not talking to the plant manager) How would you be able to find out this information?

Item 11

● In your own words, what is this item asking you for?

What do you consider to be non-managerial staff?

Are different non-managerial staff shown this information at different frequencies? If so, how did you choose the answer that you gave?

● (Probe their response… if R not around in 2007, ask how they would answer this question. How are the indicators ‘shown’ to non-managerial staff?)

Item 12

● In your own words, what is this item asking you for?

● What are production display boards? If have them, where are they usually located?

What are on your display boards?

● Did you find the examples helpful?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

Item 13

Are you involved with submitting indicators to your corporate headquarters? If not, how would you go about answering this question?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

Item 14

Do you have different teams on the production line?

If yes, do you track performance by production line team? Why?

● (Probe their response…if R not around in 2007, ask how would they answer this question

Item 15

● In your own words, what is this item asking you for? What is the difference between the different response items offered?

● How would you go about completing this item?

Do you have production targets aimed at the main product produced at this location?

How does your company define short term? Long term? Do the timeframes used for long term and short term in this question seem reasonable?

Are you the correct respondent for this item, or would someone else in the plant know more about this?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

Item 16

● In your own words, what is this item asking you for?

How comfortable do you feel answering this item?

What is the difference between the 2^nd and 3^rd options?

Did you have any problems answering this same item for 2007?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

Item 17

● In your own words, what is this item asking you for?

Are there rules or policies about who within the plant/establishment can know about production targets? If so, what are they?

Are you aware of who within the establishment knows this information?

Did you have any issues completing this for 2007?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

Item 18

● How would you go about completing this item?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

Do you have access to the information about who within the company was awarded when production targets were met?

Item 19

● In your own words, what is this item asking you for?

● How would you go about completing this item? What does each response option mean to you?

Are you involved in promotion activities for the employees at this location? Are you involved for all employees or just some?

Besides tenure, what other factors are used to promote employees at your establishment?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

Item 20

● In your own words, what is this item asking you for?

● How would you go about completing this item? Would you have to work with any others within the company in order to gather this information?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

● Are there documented official policies related to this topic?

What does the term “underperforming employee” meant to you?

What is your company’s policy for underperforming employees?

Does your company have a set level of performance that you expect your employees to meet? Do these levels differ based on function, team, or other factors?

Are you involved or informed of issues related to “underperforming employees”?

The second option mentions that underperforming employees are kept in their position for at least a year. What do you think of the timeframe referenced in this option? Does this seem reasonable? Too long, too short?

The third option mentions ‘rapidly helped and re-trained”. What time frame would you consider ‘rapid’ to refer to?

Item 21

● In your own words, what does “part of a firm with other establishments” mean to you?

● How would you go about completing this item? (IF ‘No’) Did you notice the skip pattern?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

Item 22

● How would you go about completing this item?

Are you involved in hiring decisions at this location?

Are there other options that would apply to your establishment? What are they?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

Item 23

● In your own words, what is this item asking you for

• Are you involved in decision related to employee compensation at this location?

• Would you need to work with anyone else in order to answer this item?

• Would your answer change if the percentage were more than 10%? Less than 10%

• Are there other options that would apply to your establishment? What are they?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

Item 24

● In your own words, what is this item asking you for?

• What does the term “new product introductions” mean to you?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

● Can you think of any examples that could apply to ‘Other’?

Item 25

● How would you go about completing this item?

• What does the term “product-pricing decisions” mean to you?

• Would you have to work with any others in the company to answer this item?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

● Can you think of any examples that could apply to ‘Other’?

Item 26

● In your own words, what is this item asking you for?

● How would you go about completing this item?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

● Can you think of any examples that could apply to ‘Other’?

Item 27

● How would you go about completing this item?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

● Describe the authorization process.

• Are requests for capital expenditures for equipment handled differently than requests for other types of capital expenditures?

• Do the rules for authorization differ based on the type of equipment that is being purchased? Or are the rules the same regardless of the type of equipment?

Item 28

● In your own words, what is this item asking you for?

Do you have access to this information, or do you need to get this from someone else?

Would you have any problems getting this information for 2007?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

Item 29

● In your own words, what is this item asking you for?

● How would you go about counting the layers of management?

● What does ‘layers of management’ mean to you?

What is the most senior management level at this establishment?

● Is the example helpful to you? How so?

Describe all of the different layers at this establishment.

● (Probe their response…if R not around in 2007, ask how would they answer this question)

Item 30

● In your own words, what is this item asking you for?

● What does ‘sets the pace of work’ mean to you?

What does the term ‘factory manager’ mean to you?

● (Probe their response…if R not around in 2007, ask how would they answer this question)

● Can you think of examples that could apply to ‘Other’?

Item 31

● How would you go about completing this item?

Do you have access to information about this item? Would you have to talk to others at the establishment, such as factory workers?

● (Probe their response…if R not around in 2007, ask how would they

answer this question)

● Can you think of examples that could apply to ‘Other’?

Wrap-up

● How much time would it take you to complete this report (actual time,

not elapsed time)?

● Overall, how easy or difficult was it to complete this form? Which were

the easiest questions to answer? Which were the hardest to answer?

● For any of the questions you cannot answer, how would you indicate

that? Would you mark a zero, leave it blank, put a dash, write in N/A, or use some combination of these?

Likely R:

● How does this establishment usually handle incoming mail that is addressed only to the establishment’s name? Who gets it?

● Does mail usually arrive on your desk opened or unopened?

● Are you the person who would probably answer these questions, or

would someone else answer them?

Electronic reporting:

● Let’s say you had the opportunity to complete this survey online. Do

you think you might do that?

● What factors would affect your decision to report online, instead of on

paper?

● What is the best way for us to tell you that you can complete this survey

online? (flyer in the mailing package, note on form, some other way?)

Any other comments?

THANK YOU VERY MUCH FOR YOUR HELP!!

12