Forms Redesign Team Final Report

The Bureau of Labor Statistics (BLS) is revising the data collection form for the Producer Price Index (PPI). The current form has not been revised in almost 30 years, so as the PPI program upgrades the mail and fax system, management would like to revise the form as well. The goals of the revisions are (1) to address data collection problems with the current form and (2) to adjust the forms to meet the requirements of the new system.

PPI management chartered the Repricing Form Redesign Team to design an improved form, along with a corresponding cover sheet with instructions. The team began work in November 2011 and completed work in December 2012.

The team accomplished the work by completing several tasks:

Reviewing the work done by previous teams, including

A 1997-1998 effort to redesign the PPI form
The recent Repricing Flexibility Needs Team
The Industrial Price Program (IPP) Form Team, which completed a redesign of IPP’s form in 2004

Developing a prototype form based on earlier work
Getting feedback on the form from relevant BLS staff
With the assistance of the Office of Survey Methods Research (OSMR), conducting cognitive evaluations of the form
Revising the form based on the feedback and the test results, in an iterative manner
Presenting a final recommendation

The team was instructed to explore ways to address the variety of needs across PPI sections. This could be accomplished by having different forms or by allowing customization of features on an individual form (e.g., by allowing Industry Analysts (IAs) to choose from terms like “margin” or “rate” in addition to “price”).

Ultimately, we decided to focus on one main form designed to accommodate all items and services as much as possible. We made this decision because (1) According to the Enhanced Repricing Request Methods (ERRM) team, PPI would only be able to support one form for a significant period of time, so this one form would need to work in all cases, and (2) there would need to be a backend system in place for IAs to select which form or tailoring to use.

Although we focused on just one form, we are submitting a second form as well. This form is basically the same as the primary form, but does not display the Adjustments to Price or Terms of Transaction. These two boxes contain information that is not relevant to some items and services in the index. Having the information on the form could be confusing to respondents. Plus, removing the boxes would provide more space for the description, possibly preventing some forms from going to an additional page. In addition, it was thought that for this change, there might be an easy way in the existing system(s) to identify which form to use.

The team members included:

Tiffany Hamilton (BIP-PPI, Leader)
Jean Fox (OSMR)
Barbara Giles (OFO-NO)
Jeff Guccini (BIP-PPI)
Alex Moore (BIP-PPI)
Ryan Ogden (BIMA-PPI)
Richard Regotti (OFO-RO)
Phil Sturm (BIP-IPP)

This report describes the process we followed to design and test the forms. The first section will summarize the steps we took, and subsequent sections will describe the steps in more detail.

Executive Summary

Initial Input

The team began the project by reviewing information already available. This included:

The prior Form Redesign Team’s prototype. This team developed a revised form in 1997-8. Although this form was never implemented, the work identified areas for the current team to consider.
The PPI Repricing Flexibility Needs Report. This was completed by a recent team to identify areas on the form where flexibility (e.g., different wording options) would be helpful.
IPP’s Form Team’s Final Report. This team revised the IPP form, completing their work in 2004.

The team also interviewed nine current respondents to get their input on the current form.

The team decided that, given the amount of work that went into the IPP form and the similarity between PPI and IPP, we would base the new PPI form on IPP’s form.

Initial Designs

The team took the IPP design and began to modify it to address PPI needs. We considered both methodological needs (e.g., PPI allows respondents to update four prior months of data) as well as system needs (e.g., locations of the four corner “anchors” and the barcode). We created these prototypes in Visio, which provided flexibility in designing the form and allowed us to make changes easily. The team also developed early designs for the cover sheet based on IPP’s cover sheet.

In the initial designs, we were still considering multiple versions of the form to accommodate the diverse needs of the program. As the team progressed, we realized that we should focus on just one form, since the PPI system would only accommodate one form, at least for some time. Therefore, we needed to be sure the form met the minimum needs for all items and services.

Internal Reviews

Once we had prototypes of each form, we presented the prototypes to the IAs in a series of meetings. During these meetings, we presented the proposed changes and sought feedback from the IAs. We revised the form based on the feedback we received. During this time we were also meeting with REPEST and presenting to them the various form designs including changes.

Internal Cognitive Testing

Next, the team conducted two rounds of cognitive testing, using internal BLS staff as participants. We recruited staff members from the Consumer Price Index (CPI) and IPP programs, so they would be somewhat familiar with PPI concepts. This was true to some extent, but there were still some terms the participants were not familiar with. However, these tests still helped us to identify parts of the form that needed to be changed.

For the testing, we developed a series of 14 scenarios. Each scenario had a form for the participant to complete and a description of the information needed to complete the form. The scenarios covered a range of situations in all sections. We presented the scenarios in random order. Half of the forms were in black-and-white, and the other half in color, because we were not sure whether PPI would be able to print the form in color. We recorded data such as the time to complete the scenarios, participant ratings and comments, and task success rates. After each round of testing, we made some modifications to the form and shared our prototypes with REPEST.

External Cognitive Testing

After the internal testing, the team conducted cognitive testing of the form with actual PPI respondents. We recruited local respondents for field visits and remote respondents for remote testing, in order to get input from geographically diverse participants. In the end, we tested the form with five local participants and one remote participant. For the local participants, we visited them and brought the scenarios and forms. For the remote participant, we sent the scenarios and forms, asked him to complete the forms and fax them back to us, then we scheduled a phone interview. It was significantly easier to recruit local participants, possibly because the burden to participate seemed higher for remote participants.

At this point, we became aware that the PPI system may be able to accommodate a second form, so we also created a version of the form without the Adjustments to Price or the Terms of Transaction sections. We attempted to find PPI respondents who could participate in a cognitive test of this revised form. However, this form would benefit a smaller group of PPI respondents, and we were not able to find any participants to test this version. Because this version is so similar to the version we did test, with the only change being that irrelevant information would be removed, the team felt that it would be okay not to test the revised version.

After this testing, the team prepared our final versions of the form. These versions will serve as the models for ERRM to follow as they build the forms. There may need to be slight differences in the final form, depending on what the system can accommodate.

Final Prototype

This section contains a description of each section of the form and how it is different from the current form. There is a copy of the final forms we submitted to REPEST and ERRM in Appendix A. These include the final basic form and the cover sheet, along with the alternative layout without adjustments to price and terms of transaction (and corresponding cover sheet). The initial form we started with (the current production form) is in Appendix B.

Overall

There are a couple of improvements that impacted the whole form. First, we eliminated the dense instructions on the left of the form. Instead, we incorporated the most important instructions in the form itself, where the information was needed. We incorporated the more complex instructions into the cover sheet, where there was more room to provide a full explanation.

Second, the font is larger, making it easier to read. And third, the information provided by the respondent will be printed in red Courier text, to easily distinguish it from the information provided by the program, which will be in black Helvetica text.

Header

The header contains the logo, survey title, the code from the Office of Management and Budget (OMB), the PPI URL, and PPI codes.

The item code is in the header, as well as at the bottom of the page. This is because IAs need to know the code when working with the item (e.g., entering data or discussing the form with the respondent). There have been occasions where one of the codes is written over, spilled on, or ripped off, so it is no longer available.

The bottom of the page displays only the item code, with the label “PPI Item Code” fully spelled out. With this design, when IAs are talking with the respondent, they can direct the respondent to “look for the PPI Item Code at the bottom of the page.” This makes it easy for the respondent to find the code.

Section 1: Instructions

The instruction section contains two parts: (1) standard instructions to be printed on all forms and (2) room for remarks for the IAs to give special instructions or leave comments for the respondent. The standard instructions encourage faxing the form back rather than mailing it by listing the fax number in bold before the address. These instructions also contain the contact information for the IA.

The remaining area in this section will accommodate the Remarks field. However, the configuration is slightly different from that on the current form. It is wider, but has fewer lines available. This may require some IAs to modify their remarks for a few items or services, but we did not think it would affect too many forms (although we were not able to get an exact count).

Also, on the new form, the remarks are at the top of the page, whereas they are in the middle of the page on the current form. This new layout allows IAs to include instructions for any part of the form, but any of the current instructions referring to changes “above” the instructions will have to be changed to “below.”

Section 2: Description, Adjustments to Price, and Terms of Transaction

This section retained the same basic question as on the current form. We switched the order of the yes/no responses so we could provide additional instructions when respondents have a change. We did not experience any problems with this in testing with current respondents.

Description

The space allocated to the description is wider than the current form. However, PPI systems are set up to work with the description that is 80 characters wide. Therefore, we recommend centering the description so there is not a big white space on either side of the page. Vertically, there are 16 lines available in this area, just as there are on the current form.

Adjustments to Price

This section is basically the same as the current form. The one difference is that we are changing the third header from “Already applied to reported price” to “Reflected in the price below.” In testing, we found that participants had trouble with the current wording because they had not yet reported a price. The newer wording is more of an instruction to the respondent.

An additional difference with the Adjustments to Price and the Terms of Transaction sections is the layout. Instead of being stacked vertically, they are presented horizontally across the page. This saves vertical space, but may result in more of the adjustments and terms wrapping to multiple lines.

Terms of Transaction

This section is basically the same as the current form.

Section 3: Previous Prices

In this section, we added a question about whether the previous prices have changed. We did this because participants often overlooked this section during testing. We felt that adding the question would encourage respondents to review the previous prices provided.

We also changed the wording used when a price was missing from “not yet received” to “Please provide.” We felt this new wording was a more active instruction on what to do.

In this section, the data entry fields are designed to allow for character recognition. This will make it faster and easier for IAs to review the previous prices because IAs will only have to check the numbers, rather than enter them, then check them.

Section 4: Current Price

We moved the “type of price” field to the current price section to make it clear to the respondents what to report. On the current form, this is listed before the previous reported prices.

We also included the unit of price (from the Adjustments to Price section) here as well, with the label “per.” We received feedback from IAs that this information was important to include in both places, using labels that were appropriate for each location. We did not experience any problems with this in user testing.

Section 5: Remarks

We separated out remarks to have one section for remarks from the IA and remarks from the respondent. Remarks from the IA will be in the instructions area at the top (see section 1 above). Remarks from the respondent will be in section 5. We put this field at the bottom of the form so respondents will have completed the whole form before entering comments. This should be a better workflow for respondents, because they won’t have to enter the price, then go back up to enter a comment to explain price changes.

Section 6: Address Block

We added instructions to encourage respondents to review the contact information.

Cover Sheet

We followed the format of the IPP cover sheet, changing it to accommodate the PPI form.

Details of the Interviews with Respondents about the Current Form

This section describes the interviews we conducted with PPI reporters to get their input on the current form.

Methodology

We put together a short interview, consisting of nine questions, to get some input from actual reporters on possible problems with the current form. The exact questions are shown in Appendix C.

Results

All of the participants’ responses are shown in Appendix D. In summary, we found that:

Some participants reported that the current form was easy to complete.
Others reported that it wasn’t so clear.
About half of the participants had looked at the instructions, but half of those only looked at them the first time they completed the form. Therefore, for the most part, people are not reading the instructions.
Two of the participants reported that the price history was helpful in identifying what to enter in the current price field.

Details of the Initial Reviews by IAs

Methodology

We conducted two focus groups in which we presented the proposed new form design. We presented three options for a basic form, with an additional option for services. These forms are available in the folder \\Dppssrv4\formredesign\Final Report Files. We explained the changes between the current and new form. We also gave participants the opportunity to provide feedback about the new form.

Participants

The participants included IA’s, team leaders and section chiefs from the Durables, Nondurables and Services Sections of the Producer Price Index.

Results

Several IAs suggested printing the reporter code and item code on the form more than once.
Some IA’s suggested revising the numbering system so that the previous reported price section would be numbered.
There were several questions regarding field space and if the new form could accommodate the current information.
Most participants preferred “per” instead of “unit of measure” next to the current price.
Most of the Services section wanted the continuation sheet to be entirely columnar spec.
Some of the Services section wanted additional templates; one with adjustment to price and terms of transactions sections removed, one that would use margin instead of price, and one that used a range of dates.
Some participants worried that the production cost question may be confusing.

Changes to the form

We made several changes as a result of the initial focus groups. We removed the page number from the form and replaced it with the item code. The item code is now located at both the top and bottom of the form. We added a number to the previous reported price section along with an instruction to update the form. “Unit of measure” was replaced by “per” next to the current price.

Details of the Internal Cognitive Testing

After incorporating feedback from the IAs, we conducted two rounds of cognitive testing using BLS staff as participants. Both rounds followed the same methodology.

Scenarios

We created 14 scenarios for the internal test. Each scenario consisted of a text description of the situation for the participants to price and a form with the description of the item or service. This description of the situation provided the information the participants needed to complete the form, such as the price, changes to the item, and address changes. The form had pre-printed data from the previous reporting period, as a respondent would normally receive it. See Appendix E for a sample of the forms and the cover sheet used in the first round, and Appendix F for the materials used in the second round.

The scenarios covered a range of items/services and situations. The scenarios we used for the first test are shown in Appendix G, and copies of each form and the scenarios for both tests are in \\Dppssrv4\formredesign\Final Report Files. These scenarios were basically the same across the two tests, with minor changes to correct small wording problems.

Methodology

Each test session started with the interviewer making introductions, gaining informed consent (see Appendix H), giving a short explanation of the procedures for the cognitive test, and describing the new form. Next, the interviewer provided the participant with the first item to price, along with the scenario, which provided the information needed to complete the form.

We asked the participants to complete one scenario; then got feedback on that scenario. This was intended to simulate how someone would complete the form on their own, allowing us to record the time needed. Once the participant completed all of these scenarios, the interviewer asked the participant to provide a few ratings; then asked for general feedback on the form (see Appendix I for the debriefing questions).

Participants

In the first round of testing, we had one pilot participant (from OSMR) and five test participants. However, since there were no changes made to the protocol after the pilot test, we included the pilot test results in our findings. In the second round of testing, we had six test participants.

The participants were recruited from the Consumer Price Index (CPI) and International Price Program (IPP) programs, because we expected they would have some working knowledge of the concepts of PPI. This was true to some extent, but there were still some terms the participants were not familiar with. However, these tests still helped us to identify parts of the form that needed to be changed.

Data Analysis

To evaluate the form, we considered several types of data from the test:

The time needed to complete the forms
The results of the post-test debriefing ratings and open-ended questions
How often participants made the changes we wanted them to make
How often participants made changes they should not have
The comments participants make while completing the form

In analyzing the data, we looked at each form to evaluate whether the participants had entered the data correctly. For each part of the form, we determined whether there should have been a change or not. If there should have been a change, we recorded whether the participant had made the correct change or not. If there should not have been a change, we recorded whether the participants wrote something anyway or not.

In cases where participants made an unneeded change, most of their entries were benign and could be ignored, but others were wrong and could have been a problem. However, in the context of this test, it was sometimes difficult to categorize the response, so instead, we created one category of “no updates needed, but something entered” to apply to all scenarios. (Note: This was not a problem in the external test, because the respondents knew what data to enter on their forms).

The team recommended improvements to the draft form based on the results of this test.

Results

Task Times

The graph and table below show the task times for both internal usability tests. Although it appears that the average task time was faster in the second test than in the first test for all scenarios, none of the differences were statistically significant. However, with only six participants in each test, the probability of finding a statistically significant difference is small. One thing to note about the results is that most of the scenarios were completed in less than three minutes on average.

Task	Test 1	Test 2
A	0:04:46	0:04:24
B	0:02:02	0:01:29
C	0:02:34	0:02:18
D	0:04:02	0:01:53
E	0:02:34	0:01:51
F	0:01:25	0:01:15
G	0:04:19	0:02:09
H	0:04:27	0:02:42
I	0:02:57	0:02:53
J	0:02:27	0:01:59
K	0:04:08	0:03:57
L	0:03:20	0:03:04
M	0:02:26	0:02:05
N	0:02:55	0:02:16

Ratings

Here are the results of the ratings (1 = Strongly Agree, 5 = Strongly Disagree). N=6 except where noted.

	Test 1	Test 2
The form was easy to complete.	2.4	1.7
I could easily find the information I needed on the form.	2.7	1.9
The use of color was helpful.	2.7	2.0
The instructions were helpful.	2.5 (n=3)	2.7 (n=5)

Overall, the ratings are slightly better than neutral (a lower number is better).

Summary of Findings from Round 1

Details of the participants’ performance (i.e., whether they completed the form correctly) are shown in Appendix J. The main findings are listed below:

Most of the participants never read the instructions at the top of the form.
All of the participants made more calculations than they needed to. For example, in the scenarios with columnar specs, the participants did all the math, when they only needed to report the rate changes.
Participants were not sure whether the instructions in item 1 included making changes to the terms of transaction and adjustments to price or just to the description.
Participants were confused by the adjustments to price column header “already applied to price” because they read that information before they entered the price.
Most of the respondents used information from the previous price to determine what value to enter for the current price.
Item 2, which asks for the current price, only asks whether the price has changed. This was confusing when the last few months have missing data. Participants could not tell whether to report changes since the last reported price or since the previous month.
Item 3, which asks for a change in production costs, was confusing to the participants. They generally entered just the change in reported price.
Participants were overall positive about the use of color.
Overall, the form was completed well enough that IAs would not need to make a phone call in 59% of the cases.

Changes Based on Round 1

Instead of placing the instructions at the top of the form above the item description box, we numbered instructions and put them in their own box. The goal of this was to make the instructions seem part of the form (and workflow) and not part of the header, in an effort to encourage people to review the instructions.
We changed the wording of the instructions in item 1 (item 2 in the revised form) to include adjustments to price and terms of transaction.
We switched the order of the adjustments to price and terms of transaction, so that the adjustments to price appeared on the left side of the page. We felt that the adjustments to price were more likely to change, so we wanted to make that section more salient to improve the chances of capturing those changes.
We changed the adjustments to price column header to read “reflected in the price below?” to help provide instructions on what to do (or not do) with the adjustments.
We added a question about whether the previously reported price had changed to encourage people to review that section.
In the previous price section, we added combs for each month’s entry, so the responses could be read automatically here at BLS by PPI systems. This is not based on test results, but based on input from REPSTEER.
We reworded all of the questions (about the description, previous price, and current price) so that a “no” response meant no change, and a “yes” response meant there was change. We provided further instructions to make the updates for each yes response.
We changed the question for the current price item to explicitly identify the period covered by the form (e.g., the wording was changed to “Has the price changed between <date 1> and <date 2>?”).
We removed item #3, which asks for a change in production costs. Although IPP staff finds this information helpful, PPI staff needs more complicated information (the change in production cost plus the markup, not just the change in production cost). The question was not getting the information the PPI IAs needed, so they would have to make a phone call anyway. Therefore, we decided to remove the question.
Like the instructions section, we incorporated the Remarks section into the form better to encourage respondents to enter a remark (it became a numbered item with specific instructions).
We removed the checkbox for changes to contact information, since it did not seem to add anything to the data review process.
We made the box with contact information at the bottom of the form smaller.
We did not change the cover sheet much, except to accommodate the changes to the form.

Summary of Findings from Round 2

Details of the participants’ performance (i.e., whether they completed the form correctly) for the second round of internal cognitive tests are shown in Appendix K. The main findings are listed below:

For two-page items/services, the instructions at the top of the page were confusing, since they were the same as on page 1 (e.g., they referred to completing sections 2-4, which do not appear on a continuation page). The instructions on the second page should be simplified to refer the respondent to the instructions on page 1.
All participants used the previous price to help guide what they should enter in the price fields.
The participants liked the color, stating that it helped identify what they should focus on.
A couple of participants expressed concern about using red, since it is usually reserved for emergencies or, as one participant put it, “scolding.” However, we felt that the color red helped to draw the respondents’ attention. Further, IPP has been using red on their form without a problem.
Although we left as much space as we could, several participants noted that the “remarks” area was small.
Overall, the form was completed well enough that IAs would not need to make a phone call in 69% of the cases.

Changes Based on Round 2

We made a clear indication on page one that there is a second page, and a clear indication on page 2 that it is the second page. For example, the instructions in item 1 on page 2 ask respondents to refer to the instruction on page 1.
We numbered the instruction and previously reported price sections.
We also right justified the dates in the previously reported price sections.
We added bold lines between the numbered sections.
We increased the size and added combs to the correction area under the previous reported price section. This will allow for character recognition.

Details of the External Cognitive Testing

Research Design

The team conducted cognitive interviews with actual PPI respondents, to get their input on the form.

Participants

We had planned to conduct about half of the tests in the field and the other half remotely. For the field test, we recruited participants from areas requiring only local travel (e.g., approximately from Richmond to Philadelphia). For the remote test, we recruited participants from other parts of the country and conducted the interview by phone. This allowed the team to observe some of the participants completing the forms in person, while also allowing us to include respondents from a wider geographical area.

We had hoped to interview 15 respondents on the basic form, with additional interviews for the alternative version of the form. In the end, we were only able to recruit six participants. We conducted five field tests and one remote test, all with the basic form. We had more trouble than expected in recruiting participants for the remote condition. We believe that it was perceived to be more burdensome, even though it was about the same amount of work (they had to complete the forms, fax them back, then schedule a meeting with us).

Also, we were not able to find participants to evaluate the alternative layout (without the Adjustments to Price and Terms of Transaction). This layout would benefit only a subset of PPI reporters, and we were not able to find a participant in that group. Although it would have been good to test this alternative, we did not believe testing was critical because the changes were minor (the only change in this alternative was that it eliminated two sections which are not relevant to some reporters).

Participants did not receive any payment for participating in this cognitive test.

Procedures

As with the internal tests, the scenarios provided a text description of the current situation we wanted them to price. However, unlike the internal tests, where participants each completed the same 14 scenarios, we developed individual scenarios for each participant in the external test, based on the items or services they are currently pricing for PPI. See a sample scenario, form, and cover sheet in Appendix L

We modified the participants’ current items/services so we could evaluate how they complete the forms for a variety of situations. These scenarios covered a range of situations, so we could test various parts of the form. However, some situations were irrelevant in certain industries, so we were not able to test all parts of the form with all participants. Overall, participants completed 1 to 5 scenarios. The test sessions lasted approximately 30 to 60 minutes.

Details of the protocol we followed (a basic script, which we adapted depending on the situation) are in Appendix M.

In-Person Interviews

After making introductions, gaining informed consent (see Appendix N), and giving a short explanation of the procedures for the cognitive test, the interviewer provided the participant with the first item to price, along with a scenario.

We asked the in-person participants to complete one scenario; then we got their feedback on that scenario. After completing their scenarios, the interviewer debriefed the participants (see the debriefing protocol in Appendix O). During the debriefing, the interviewer asked participants about their opinions of the new form.

Remote Interviews

The primary difference for the remote participant was that he completed the consent form and scenarios ahead of time, and faxed them back to us before the interview. This way, we could look at the forms during the interview. For the remote participant, we could not get feedback immediately after he completed each form, so we started with overall impressions. We conducted the remote interview by phone.

Data Analysis

The analyses for the external cognitive test were similar to the analyses for the internal tests. We counted how often participants make the changes we wanted them to make and how often they did not complete the form as we wanted. We also considered the comments participants made while completing the form, as well as the time needed to complete the forms, and the results of the post-test debriefing.

Results

Overall, participants found the form to be fairly easy to use (average rating 1.9, on a scale of 1-5, where 1 = “very easy” 5 = “very difficult”).

In the end, the six participants attempted a total of 22 scenarios. In most cases, participants had no trouble, completing the scenarios in less than 3 minutes. The average time to complete a scenario was approximately 2 minutes. One participant was unable to complete a scenario with a discontinued item—she just did not know what to do. We skipped the scenario, but noted that it would have required a phone call from the analyst.

A summary of the participants’ performance (i.e., how well they completed each form) is shown in Appendix P.

Participants reported a few problems as they completed the scenarios:

One participant had trouble finding the contact information at the bottom of the form
Two participants were not sure what to do about price changes when the price change occurred between dates in the previous price. For example, they weren’t sure what to do when there were previous prices for, say, December 14 and November 14, and the price change occurred on Dec 1.
One participant had trouble determining what to show as the price, given the discounts (the discount was 55%, and he wasn’t sure whether to put in 55% of the price or 45%)
One participant thought the flow was illogical because the address information was at the bottom of the page, but changes needed to be recorded in the section above it.
One participant did not know what to do with a discontinued item.
One participant felt that the terms of transaction were difficult to complete because he did not understand their purpose.
One participant did not know how to deal with the discounts.
In the scenario where reporting was being transferred to another employee, no price was given. One participant did not know what to do about the price.
One participant didn't know whether to provide previous prices for the new model or just the current price.
Overall, PPI IAs would not need to make a phone call in 82% of the cases.

During the debriefing at the end of the test session, we asked each participant what they thought about the form. Most people thought that the form is an improvement, noting that:

It was more user-friendly and straightforward.
The instructions were better.
The form was cleaner.

However, there were a few suggestions:

Provide more room in the remarks section.
Be clear about the date range for previous prices. A couple of participants were confused about what to do when the price changed between the dates displayed.

We also asked participants about the cover sheet. They noted that:

It is straightforward.
It was a little busy, but after reading it, it wasn’t too bad.
It is easier to read than the current instruction sheet.
It might be helpful to have the IA name and contact information on the cover sheet.

Finally, most respondents reported that completing the actual forms would not take much more time than they experienced in the study. They have easy access to their price database, and it is typically already open on their computer.

Changes after the External Cognitive Test

The team felt that the form did well enough in the external test, so we did not recommend any additional changes.

Additional Findings that Do Not Impact the Design of the Form

As the participants went through the different stages of testing, we heard some additional suggestions that might help respondents complete their forms. These suggestions did not relate to the form itself, but more to the way the content is provided. For example, some were suggestions about the layout of the description.

The description could be presented in a stacked vertical format, showing the label on the left, and the value on the right. This could make it easier for respondents to scan the values to identify what had changed. See an example below:

Model: 12AB

Color: Red

No. of features: 12

Another possibility is to combine this format with the “prose” format currently used, putting the most important features, or the ones most likely to change in the vertical format, with the others as prose.
It would help to have the most important features first, to make them easier to find.
For items requiring data entry in the description, the spec could include instructions adjacent to the data entry areas. For example, in construction, next to the data entry line for the fee percentage, there could be additional text instructing the respondent to put that value in the current price field.
Anything that respondents do not need should not be printed on the form. For example, some forms with columnar specs have columns for the rates, weights, and totals. On these forms, respondents only need to enter the rates, but in our tests, they did all the math to come up with total values, then adding them up for the final price. They do not need to do this, so it adds unnecessarily to their burden. If the weights and total columns were not displayed, respondents would not feel obligated to complete the math. (This recommendation would require some additional instructions in the description to tell respondents what to do about the price field).
PPI might consider collecting information on whether an item/service was sold during the current month. In the external test, a couple of the current respondents said that they were pricing items that could still be sold, but which they hadn’t produced or sold in years. The form does not collect this kind of information, so the Industry Analyst will never know when an item or service becomes obsolete. The IPP form does collect this information—it asks whether the price was estimated or based on an actual transaction. When the form repeatedly comes back with an estimated price, the IA can look into substituting another item.
One participant noted that it wasn’t clear what the deadline of “return in 5 days” meant. She wasn’t sure if it meant 5 days after she got it or 5 days after it was mailed. The wording is intended to mean “as soon as possible,” but it is a little confusing and not straightforward. Perhaps there is another way to address the deadline.

Next Steps

As the process of deploying the new form moves forward, we recommend PPI consider two additional activities to be sure that the transition goes smoothly and that the form is working as expected. We are basing these recommendations on the work that IPP did as they deployed their new form.

First, it is important to carefully introduce the new form to all the stakeholders. This may include materials such as advance letters and instruction booklets for respondents, as well as training for Field Economists. Second, it is useful to know whether the new form is improving data collection and to identify where there are still problems. Therefore, it may be helpful to conduct a study of the results of data collection with both the current form and the new form after it is introduced. This evaluation would look at issues such as whether reporters selected an option for each of the Yes/No boxes, and whether the form would require a phone call. We believe these recommendations will help ensure a smooth transition to the new form for all stakeholders.

Conclusion

The PPI Form Team has worked diligently over the last year to design a new PPI repricing form. We incorporated input from many sources as we developed the form to ensure that it meets the needs of Industry Analysts as well as the respondents. We also worked with ERRM and other PPI staff so the form would work with the new PPI processing system. We expect that the form will improve the data collection process for PPI.

Final Forms and Cover Sheet

Basic Form

Basic Form, Page 2

Coversheet, Basic Form

Alternative Form with No Adjustments to Price or Terms of Transaction

Coversheet, Alternative Form

Current PPI Form

Protocol for the Initial Interview with PPI Respondents

IA: Date: Time:

Introduction

I would like to ask you a few questions about the form itself. We are revising the form and would like to get feedback and suggestions from people who complete the forms. Participation is voluntary, and we would use your responses only for improving the form. Would you have time to answer a few brief questions about the form now?

To the IA: Make sure they have a copy of the form with them.

Interview Questions

1. What are your overall impressions of the form?

2. How do you generally proceed through the form? For example, which section do you start with?

3. [If not reported in Q2] Have you ever used the instructions to help you complete the form?

 Yes

 No – skip to 5

 Don’t know – skip to 5

4. (If yes to Q3 or reported earlier) Were they helpful?

 Yes – if yes, How were they helpful?

 No – if no, Why not?

 Don’t know – skip to 5

5. Are there any terms, questions, or sections on the form that you find confusing?

 Yes – if yes, which ones?

 No

6. Have you ever had any other trouble completing the form?

 Yes – if yes, what?

 No

7. Is there anything missing from the form that would help you complete it?

 Yes – if yes, what would you add?

 No

8. Is there anything on the form that is particularly helpful?

 Yes – if yes, what?

 No

9. Do you have any other comments or suggestions to help us improve the form?

Results of Initial Respondent Interviews

	Interviewer	Overall Impressions	Process
1	Tiffany	Pretty simple. Straightforward. Likes the 4 months of history	Top of the form with the item description.
2	Tiffany	Easy	Price, then item description.
3	Tiffany	Doesn’t understand the reason for the form	Price, then item description.
4	Alex	Forms are simplistic, they are pretty good.	I just go through it. I start at the top and work my way down. I start with the item description.
5	Alex	Easy to fill out. Don’t think the item is a good item to reprice as the bid-ask price does not change often.	Doesn’t start with a section. Reporter simply looks at the bid-ask prices.
6	Alex	The reporter was new to repricing and was not familiar with anything. Reporter stated that form was very straightforward.	Reporter emails specific traders to receive information about spreads for each bond. The reporter then updates the spread on the item description.
7	Jeff	Redundant information, repetitive information. Respondent felt like there were some instances where information appeared several times, worded differently, but essentially telling/asking you the same thing	Top to bottom
8	Jeff	After he went through initial contact with the FE it was pretty simple. Says it may have been more difficult had he not had a verbal explanation, but after he did, it was easy.	Looks right at the OH&P question that we are most concerned about and makes any necessary changes. Does not look much at the other parts of the form.
9	Jeff	Not all that clear. Questions/information possibly in the wrong order. Set up/layout could be much better.	Looks right at the OH&P question that we are most concerned about and makes any necessary changes. Does not look much at the other parts of the form.

	Interviewer	Used Instructions?	Instructions Helpful?	Anything Confusing?	Any Other Trouble?
1	Tiffany	N	na	N	N
2	Tiffany	N	na	N	N
3	Tiffany	Y	Y	N	N
4	Alex	N ( Reporter did not know that there were even directions with the form)	na	N	N
5	Alex	N	na	N	Y-Reporter stated that it took a while to calculate the correct values on the form. Treasury quotes are stated in 32nd of dollars and the reporter had to translate into dollars.
6	Alex	N	na	N	Y-Reporter stated that it took a while to calculate the correct values on the form. Treasury quotes are stated in 32nd of dollars and the reporter had to translate into dollars.
7	Jeff	Y	Y-Nothing in particular, he just likes to remind himself each time to refresh his memory.	Y - It is sometimes confusing determining where one section ends and another begins.	N
8	Jeff	Y - Only the first time.	Y-He looked at them the first time he filled out the forms just to make sure he knew what he was doing, but hasn’t really looked at them since.	N	N
9	Jeff	Y	Y -Clear and precise. Respondent states that he probably only looked at them the first time he filled out the form, and then maybe once again when he made a change, but he doesn’t look at them every time.	N	N

	Interviewer	Anything Missing?	Anything Helpful?	Other Comments
1	Tiffany	N	Item description and price history	No
2	Tiffany	Y - Add a due date.	N	Add a due date.
3	Tiffany	N	Y-Analyst name, point of contact. (Note: This is already located on the form)	no
4	Alex	N	N	No
5	Alex	N	N	Reporter wanted BLS to collect benchmark treasuries. (10 year, 5 year, 2 year). The reporter preferred to report on the 10-year treasury and he currently reports the spread on the 2 year and 5 year treasury.
6	Alex	N	N	Reporter faxes forms and wanted an automated confirmation after faxing the forms.
7	Jeff	N	N	Shorter, more concise.
8	Jeff	N	N	No. Form is pretty straightforward.
9	Jeff	N	Price history information	No

Sample Form and Cover Sheet from the First Internal Cognitive Test

Sample Form and Cover Sheet from the Second Internal Cognitive Test

Scenarios, First Round of Internal Usability Testing

For each internal usability test session, we provided participants with the following scenarios to help them complete the forms we also provided. Copies of the forms themselves are available in our shared directory.

Scenario A

Your company produces grain combines. Sales of model number A123 have been low in the past year and your company discontinued producing this combine on March 1, 2012. The replacement grain combine, model B 1234, has an additional two rows and the cab has a premium windshield wiper and washer system. The price for model B1234 is $427,000. The mark up of both combines is the same. Model A123 cost you $150,000 to make, while model B1234 costs you $200,000.

Scenario B

Your company produces a variety of lighting products, including flashlights. Over the last month your company has modified the flashlight’s housing, so it is now 4 ½ inches in diameter, and 7 inches long. The new model number is AB12, and the price of the new item is $53.

Scenario C

Your company produces aluminum railings. Your company has moved production from Nebraska to Maine. The production manger of the Maine plant, Jane Smith, will now be responsible for completing the forms. You don’t know the current price, but you notice that the model number has a missing character and should read ABC123.

Here is the contact information you have for Jane:

4321 Here Avenue

Bridgeport, Maine 54321

janesmith@abccasting.com

Scenario D

Your company produces ductile iron castings. Recently, the cost of iron has been on the rise and your company has decided to increase the metal surcharge from 0.2002 a pound to 0.2172 a pound and has not been applied this to the distributors’ price. You expect this to be a temporary change, and you will review it in a few months.

Scenario E

Your company produces household ranges. The current range has been discontinued. The heating elements of the new range have been updated to have two 12 inch burners, one 6 inch burner, and one 6/9 inch burner. The new model number is XYZ123DE4, and the price is $574. There are no discounts offered for this transaction.

Scenario F

Your company manufactures carbon fiber. On March 1^st, the manufacturer’s price of this item went from $13 a pound to $25 a pound due to an increased copper precursor costs.

Scenario G

Your company manufactures construction equipment. In January, your company instituted a 5% standard discount on all wheel loaders. The manufacturer’s price of this wheel loader is $170,875. Your organization has a policy to only report price information with all discounts and surcharges already applied when submitting price information to outside parties,

Scenario H

You are an estimator employed by a concrete company that submits prices to the PPI. You reprice a slab on grade foundation. Last month you reported an overhead and profit markup (OH&P) of 12%. Since that time, however, you realize that the market has improved. Additionally, some overhead items, such as fuel and insurance costs have increased. Because of these changes you have decided to raise your OH&P markup to 15%.

Scenario I

You are the president of a General Contracting company that submits prices to the PPI. Over the last month, material costs increased by approximately 3%. This has caused the bids of your subcontractors to rise and therefore, your bid to the developer to increase as well. However, because of the difficult economy, you have been unable to increase your overhead and profit markup. It is still at 6%, the same as last month.

Scenario J

You are a roofing contractor that submits data to the PPI for its maintenance and repair indexes. The cost of a square of shingles is now $78.00. The job you reprice uses 10 squares of shingles. Therefore the total cost has increased to $780.00.

Scenario K

You are a managing director at a management consulting firm that specializes in human resources consulting.

You provide data to the Bureau of Labor Statistics regarding the standard rates that you charge most clients. Using the data below, please update the BLS form.

Professional Level	Name	Current Rate
Sr. Consultant	Kanter	$950
Mid-level Consultant	Erin	$680
Analytical Researcher	Ryan	$275
Technical Researcher	Rich	$815

Secretarial	Temp Help	$125

Claudio left the firm last month. He was of similar ability and education to Ryan, the new hire.

Scenario L

You are a large broker-dealer who specializes in making a marker in investment grade corporate bonds. You report to BLS the bid-ask spread for several corporate bonds. Several times throughout the year, you replace the bond that you report with a new bond from a similar company.

Using the information below, please fill out the form for BLS:

New Bond:

Marathon Oil Corp

Coupon Rate: 7.5%

Maturity: 2/15/2019

Credit rating: Ba

Ask: 118.19

Bid: 117.41

Scenario M

You are a large broker-dealer who specializes in making a marker in treasury bonds. You report to BLS the bid-ask prices and bid-ask spread to the BLS each month. Using the US Treasury bond data below, please fill out the BLS form:

Time to Maturity	Ask	Bid
1 month	100.01	100
3 months	101.01	101
1 year	101.50	101.48
5 year	102.55	102.53
10 year	102.75	102.72
30 year	103.00	102.97

Spreads have widened greatly as the market expects the United States will receive another credit rating downgrade.

Scenario N

You operate a retail lumber company where construction companies can purchase products needed to build various structures. You are providing the service of a retail store. You send BLS the mark-up you charge for a select group of items in the store. You reprice shingles for BLS. Using the information below, please fill out the form:

Product: Oak Pro 40 Driftwood

Average retail price: $85.00

Vendor Price $70.00

Gross Margin $15.00

Your store stopped carrying Oak Pro 30 Driftwood due to a change in customer preferences and because of the higher margin that can be obtained on Oak Pro 40 Driftwood.

Consent Form for Internal Tests

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you will be observed, and we will record the audio from the session. The audio will be used only to review the session by the team members.

We estimate it will take you about one hour to participate in this research (ranging from 30 minutes to 90 minutes).

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: [02/29/12]

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

Internal Usability Test Debriefing Questions

We asked the participants to provide their ratings on paper. We asked the additional questions orally.

Ratings of PPI Form

Please complete the following ratings.

1. The form was easy to complete.

Strongly

Agree

Neutral

Strongly

Disagree

2. I could easily find the information I needed on the form.

Strongly

Agree

Neutral

Strongly

Disagree

3. The use of color was helpful.

Strongly

Agree

Neutral

Strongly

Disagree

4. The instructions were helpful.

Strongly

Agree

Neutral

Strongly

Disagree

N/A

Did Not Use

Additional Questions

5. What did you dislike about the form?

6. Were there any parts of the form or instructions that were difficult to understand?

7. What did you like about the form?

8. Did you ever read the instructions at the top of the form? If so, what did you think of them? Were they helpful?

9. Did you review the cover sheet? If not, could you take a look now? What do you think?

10. Do you have any recommendations to improve the form or the instructions?

11. Do you have any other comments or suggestions?

Results from the First Internal Cognitive Test

	Total	% done correctly
Item 1
Selected correct option for Item 1 question	55	67%
Selected incorrect option for Item 1 question	4
Left Item 1 blank	23

Correctly updated the description	51	78%
Incorrectly updated the description	14
No updated needed, something entered	1
No update needed, nothing entered	16

Correctly updated the adjustments	6	50%
Incorrectly updated the adjustments	6
No updated needed, something entered	0
No update needed, nothing entered	70

Correctly updated the terms of transaction	0	n/a
Incorrectly updated the terms of transaction	0
No updated needed, something entered	0
No update needed, nothing entered	82

Correctly updated the previous price	6	26%
Incorrectly updated the previous price	17
No updated needed, something entered	0
No update needed, nothing entered	59

Item 2
Selected correct option for Item 2 question	64	78%
Selected incorrect option for Item 2 question	8
Left Item 2 blank	10

Correctly updated the current price	47	81%
Incorrectly updated the current price	11
No updated needed, something entered	3
No update needed, nothing entered	21

Item 3
Entered correct response for Item 3 question	8	67%
Entered incorrect response for Item 3 question	4
No updated needed, something entered	15
No update needed, nothing entered	55

Correctly updated the address change	5	83%
Incorrectly updated the address change	1
No updated needed, something entered	1
No update needed, nothing entered	75

Entered comments "correctly" (included all the comments expected).	21	36%
Entered some useful comments (but not all needed)	18
Entered comments, but nothing useful	4
Did not enter any needed comments	16
No comments needed, something entered	10
No comments needed, nothing entered	14

Overall
No phone call needed	48	59%
Phone call still needed	34

Results from the Second Internal Cognitive Test

	Total	% done correctly
Item 2
Selected correct option for Item 2 question	68	81%
Selected incorrect option for Item 2question	8
Left Item 2 blank	8

Correctly updated the description	56	93%
Incorrectly updated the description	4
No updated needed, something entered	2
No update needed, nothing entered	22

Correctly updated the adjustments	8	67%
Incorrectly updated the adjustments	4
No updated needed, something entered	0
No update needed, nothing entered	72

Correctly updated the terms of transaction	0	n/a
Incorrectly updated the terms of transaction	0
No updated needed, something entered	0
No update needed, nothing entered	84

Item 3
Selected correct option for Item 3 question	36	43%
Selected incorrect option for Item 3 question	10
Left Item 3 blank	38

Correctly updated the previous price	6	25%
Incorrectly updated the previous price	18
No updated needed, something entered	0
No update needed, nothing entered	60

Item 4
Selected correct option for Item 4 question	67	80%
Selected incorrect option for Item 4 question	8
Left Item 4 blank	9

Correctly updated the current price	59	82%
Incorrectly updated the current price	13
No updated needed, something entered	1
No update needed, nothing entered	11

Item 5
Correctly updated the address change	6	100%
Incorrectly updated the address change	0
No updated needed, something entered	0
No update needed, nothing entered	78

Entered comments "correctly" (included all the comments expected).	39	54%
Entered some useful comments (but not all needed)	16
Entered comments, but nothing useful	4
Did not enter any needed comments	13
No comments needed, something entered	4
No comments needed, nothing entered	8

Overall
No phone call needed	58	69%
Phone call still needed	26

Sample Scenario, Form, and Cover Sheet from the External Cognitive Test

External Test Cognitive Interview Protocol

In-person Interviews

Thank you for participating today.
I am …[This is my colleague ( ) who will be taking notes for us today]
Explain cognitive testing
- The goal of our session is for us to evaluate a new form we are developing for PPI. We want to see how you complete the form. We are looking for what works well and what is more difficult.
- During this session, we will give you several simulated forms to complete, along with scenarios that provide the information you will need to complete the forms. The items/services in the forms will be similar to what you already price for us. However, the scenarios may or may not match actual situations in your business.
- Please use the information in the scenarios to complete the forms. If a scenario is significantly different from what you would normally experience, please let us know.
- We are testing the form, not you, so any problems you have are problems with the form that we need to fix. We would like to hear all your comments and suggestions so we can improve the form as much as possible.
- The information you provide today will only be used for the purposes of improving the form. It will not be used for any other purpose, and it is not part of the regular PPI data collection.
Administer consent form.
Have the participant complete the scenarios.
Conduct the debriefing (see Attachment D).
Thank the participant.

Remote Participants

Instructions during initial call

Thank you for agreeing to participate.
Explain cognitive testing
- The goal of our session is for us to evaluate a new form we are developing for PPI. We want to see how you complete the form. We are looking for what works well and what is more difficult.
- We will send you several simulated forms to complete, along with scenarios that provide the information to complete the forms. The items/services in the forms will be similar to what you already price for us. However, the scenarios may or may not match actual situations in your business.
- Please use the information in the scenarios to complete the forms. If a scenario is significantly different from what you would normally experience, please let us know.
- We will also send a consent form, which states that participation is voluntary, and that you may stop at any time.
- As you are entering the data, please note how long it takes to complete each form.
- We would like you to fax back the completed PPI forms along with the consent form.
- We will then arrange a call with you, so you we can get feedback on your experience with the new form.
Set up time to call back

Introduction during the test session

Thank you for participating today.
I am… [My colleague(s) <names> will also be listening today]
Explain cognitive testing
- We have a series of questions to ask you about the forms and your experience.
- We are testing the form, not you, so any problems you have are problems with the form that we need to fix.
- As a reminder, your participation is completely voluntary and you can decline to answer any question at any time.
- The information you provide today will only be used for the purposes of improving the form. It will not be used for any other purpose, and it is not part of the regular PPI data collection.
- We would like to hear all your comments and suggestions so we can improve the form as much as possible.
Conduct the debriefing (see Attachment D).
Thank the participant.

External Test Consent Form

CONSENT FORM

We estimate it will take you an average of 60 minutes to participate in this research (ranging from 45 minutes to 1 hour 15 minutes).

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires February 28, 2015.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: 02-28-2015

PRIVACY ACT STATEMENT

External Test Debriefing Items

Complete questions 1 – 4 for each item or service.

For the in-person participants, we asked these four questions after they completed each form, then started with question 5 (Q5) after they finished all the forms. For the remote participants, we started with Q5, then inserted the item-specific questions after Q13.

Q What steps did you take to complete the form? (Probe: What information did you report on the form?

Q Did you have any difficulty completing the form? (Probe: Was anything particularly confusing or challenging?)

Yes
No (go to Q5)
Don’t Know/Not Sure/No Opinion (go to Q5)

Q What was difficult?

Q (this question was only used as needed. The test moderator reviewed the form to determine if there were any other questions to ask, such as how the participant decided on a specific entry).

Q I’d like to begin by asking about your initial reaction to the form. (Probes: When you first saw the form, what did you think about it?)

Q How easy or difficult was it to complete the form? Very Easy, Easy, Neither Easy nor Difficult, Difficult, Very Difficult.

Very Easy
Easy
Neither Easy nor Difficult
Difficult
Very Difficult
No Opinion/Don’t Know/Not Sure

Q Did you look at the cover sheet at all? (check one)

Yes
No (go to Q15)
Don’t Know / Not Sure / No Opinion (go to Q15)

Q How helpful was the cover sheet? Not At All Helpful, Slightly Helpful, Somewhat Helpful, or Very Helpful.

Not At All Helpful
Slightly Helpful
Somewhat Helpful
Very Helpful

Q When did you look at the coversheet? (check all that apply)

Before starting
When I wasn’t sure what to do
Other, please specify
Don’t Know / Not Sure / No Opinion

Q What was your general reaction to the coversheet? (Probes: When you first saw the coversheet, what did you think about it?)

Q Was there anything about the cover sheet that was particularly confusing or difficult to understand?

Q Was there anything about the cover sheet that was particularly helpful?

Q Do you have any other comments on the cover sheet?

For remote participants, Q1-Q4 will be here, repeated for each form

Instructions for remote participants: “Next I’d like to look at the form itself. Let’s start/continue with <brief item description & item code>.”

Q [Additional question for remote participants] Approximately how long did it take you to complete this form?

Q If you were completing these forms with real information, how much additional time would it take for you to locate the information you need for a single form?

Q Overall, what didn’t you like about the form?

Q Overall, what did you like about the form?

Q (this question will only be used as needed. The test moderator will review the discussion so far to determine if there are any other questions to ask).

Q Did you print out the forms in color? (If yes) what did you think of the color?

Q Do you have any other comments that would help us improve the form?

Results of External Test

Across the 6 participants, there were 22 scenarios. One participant did not know how to complete one of the scenarios at all (involving a discontinued item), so it is not included in the analyses here, except for the “Overall” category, since it would have required a phone call.

	Total	% done correctly
Item 2
Selected correct option for Item 2 question	17	81%
Selected incorrect option for Item 2 question	1
Left Item 2 blank	3

Correctly updated the description	7	78%
Incorrectly updated the description	2
No update needed	12

Correctly updated the adjustments	1	25%
Incorrectly updated the adjustments	3
No update needed	17

Correctly updated the terms of transaction	0	N/A
Incorrectly updated the terms of transaction	0
No update needed	21

Item 3
Selected correct option for Item 3 question	14	67%
Selected incorrect option for Item 3 question	2
Left Item 3 blank	5

Correctly updated the previous price	4	44%
Incorrectly updated the previous price	5
No update needed	12


Item 4
Selected correct option for Item 4 question	16	76%
Selected incorrect option for Item 4 question	1
Left Item 4 blank	4

Correctly updated the current price	3	38%
Incorrectly updated the current price	5
No update needed	13

Item 5
Correctly updated the address change	8	100%
Incorrectly updated the address change	0
No update needed	13

Entered comments "correctly" (included all the comments expected).	1	20%
Entered some useful comments (but not all desired)	4
Entered comments, but nothing useful	0
Did not enter any needed comments	0
No comments needed	16

Overall
No phone call needed	18	82%
Phone call still needed	4

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	fox_j
File Modified	0000-00-00
File Created	2021-01-29

Forms Redesign Team Final Report

PPI Final Report.docx

Producer Price Index Survey