Attachment D - CE Mobile Diary Phase II Testing Report

Internal CE Document

(May be shared with CE/POC Team members at the Census Bureau)

Summary Report of the Phases II & III Usability Tests of the

CE Mobile-Optimized Web Diary

Brandon Kopp, Jean Fox, and Erica Yu

Office of Survey Methods Research

Nhien To

Division of Consumer Expenditure Surveys

February 2014

Executive Summary

In June 2013, ten usability testing sessions were conducted on the CE Mobile Diary using a high-fidelity wireframe instrument. Recommendations based on those tests were integrated into the production of a functioning, Internet-accessible version of the instrument. An additional 29 usability tests were conducted to assess the functionality of the new instrument. Due to project time constraints, these 29 tests were broken up into two phases. Twelve usability testing sessions were conducted in December 2013 and recommendations based on these tests were submitted to project staff and developers on December 20^th (referred to as Phase II below). Several of these recommendations were enacted in the instrument and an additional round of 17 usability tests were conducted in February 2014 (referred to as Phase III below).

The main findings of Usability Testing Phases II and III are:

1. A significant number of participants, particularly those who self-identified as having a low or moderate amount of experience using a smartphone, had difficulty simply navigating to the site and logging in.

2. Participants had extensive difficulty with changing their password. None of the nine participants who attempted the task were able to complete it without triggering an edit, and only three successfully completed it. The most common problem was participants’ failure to read the password requirements.

3. Participants did not have much difficulty with the basic functions of the diary, once logged in, but had significant problems with entering data in a way that meets CE data requirements. In particular, participants were often confused about whether to itemize their purchases. More often than not, they attempted to enter total values for a shopping trip; several even combined multiple receipts into a single entry. This occurred despite a full “placement” explanation of data requirements by the experimenter.

4. Of the issues noted during testing, the most worrisome are the problems participants had with simply logging in to the instrument. This process caused many participants difficulty and others were unable to log in without help from the test moderator. This becomes a larger potential problem when a Diary respondent will be expected to log in each time he/she uses the Diary. The burdensome nature of logging in will make it less likely that respondents will use the Diary as intended (i.e., fill it out at or near the time of purchase) or at all. Usernames and passwords should be simplified as much as possible while maintaining a reasonable level of security. As web/mobile data collection becomes more prevalent, Census, the Department of Commerce, NIST, and BLS should reevaluate the credentialing process to ensure password requirements are not negatively impacting response rates.

5. Before assigning a respondent to the CE Mobile Diary, survey administrators should screen respondents to ensure they not only have a smartphone but feel comfortable with navigating the phone’s web browser and entering data. Targeting more proficient smartphone users could help limit issues with Internet and Mobile Diary instrument navigation. Screening questions should target those who use their phone’s web browser frequently (not just applications). These questions should be developed in consultation with questionnaire design and Mobile Diary project staff. Ideally, these questions would be cognitively tested prior to implementation.

6. It is the opinion of the experimenters that the Mobile Diary should not be used as a standalone mode of response for the diary survey. The Mobile and Web Diaries should be linked and respondents should be able to access both during the diary period. This interoperability should be instituted as soon as possible, ideally for the CE Gemini Proof-of-Concept test.

While the above findings may seem pessimistic about the success of the Mobile Diary, it should be noted that testing was conducted with a small, convenience sample of participants. The primary use of usability testing is to identify potential issues, not to determine their prevalence in the CE Diary production sample. The rates of successful task completion are provided in this report only to provide background for the recommendations listed below. They should not be interpreted as applying to all potential CE Mobile Diary respondents.

Recommendations based on these issues are listed below. It is noted where changes that were recommended in Phase II have already been made. It is also noted where discussions with Census security and development stakeholders have shown certain changes are not allowed or feasible. The recommendations are focused strictly on change requests. If you would like further rationale for why these changes were recommended, please read the report or contact Brandon Kopp.

Recommendations

Overall

1. Two participants had an issue with what appeared like an accessibility feature on their phone. The font and the buttons appeared larger than normal. This caused numerous data entry and navigation issues because the text in the fields would only be half visible. See below for details.

These participants also had trouble because a set of magnification buttons would appear over the Save button. They had to wait, not touching the screen, until the buttons disappeared.

If there is any way this can be handled programmatically, that would be helpful. For example, if the size of the fields could expand to compensate for the increased size of the text. Accomodating accessibility features (at least eventually) would be necessary for 508 compliance.

Navigating To The Site

2. The root web address (respond.census.gov) should have a directory of available surveys. It is standard practice on a website that anything following the “/” can be reached through browsing and clicking rather than typing in a long URL.

3. Use a URL shortener to create a short and memorable URL (e.g., go.usa.gov/CEdiary).

4. If possible, email a Mobile Diary link to respondents.

5. Many participants typed the web address into a search bar. Work to ensure that the top search result in Google and Bing refers specifically to the CE Mobile Diary Diary.

Login Screen

After the “change password” task, logging in was the most difficult. This is particularly troublesome because any impediment gives respondents a ready excuse not to enter data. This task needs to be made easier. Below are several options:

6. NOT ALLOWED Have a checkbox that allows the user to unmask their password and see what they typed in. There were a lot of issues with miskeying on the small smartphone keyboard. This could help with that.

7. NOT ALLOWED Simplify the password. Allow participants to enter a 4-digit PIN. This is a common level of security for smartphones and applications. The telephone keypad could be called up, which has larger numbers. This would limit issues with miskeying.

8. If only numbers are going to be used for the username, then the number keypad should come up by default when selecting that field.

9. Assign respondents easier to remember usernames. If numbers are used, do not list more than two of the same digits in a row. Also, if only numbers are used, the number keypad should appear.

10. Limit the use of characters that look alike (e.g., l and 1, O and 0, etc.) in usernames and passwords. Preferably, these characters would be excluded entirely.

Initial Setup Screen (start date selection)

No change. Participants had no trouble with this task.

Change Password Screen

This was, by far, the most difficult task for participants. None of the participants who were presented this task made it through without errors, several did not make it through at all. For those that did eventually complete this task, it took a long time (10 or more minutes).

11. CHANGES MADE The first and most apparent problem is that there is no way to navigate away from the screen. Participants may get frustrated with the task or may have ended up on that screen by accident and there is no way to leave it except for the browser’s back button. If the participant has spent several minutes trying different passwords tapping the change password button several times, then pressing the back button will no longer get them back to the home screen.

    • Add a home icon to the top bar of the screen.

    • Add a Cancel button to the bottom of the screen, next to the Change Password button.

12. CHANGE MADE Make the “Change Password” label at the top of the page more prominent so people understand that that is what this page is.

    • Bold the font. Make it 2pt size larger than the other text.

13. CHANGE MADE The bolded statement that is currently at the top of the page does not contain useful information. People know what passwords are for. This can/should be removed.

14. CHANGES MADE The password requirements are not straightforward. At least one participant misread them, as did most of the BLS staff. In particular, “Passwords must contain a minimum of the following: 8 characters in length” was read to mean that the password must be exactly 8 characters long. We recommend rewriting the requirements to make them more clear.

Current Wording	Recommended Wording
Passwords must contain a minimum of the following: 8 characters in length 1 uppercase character 1 lowercase character 1 number 1 special character from the following: ! # $ * & ? ~	Passwords must contain all of the following: At least 8 characters At least 1 uppercase letter At least 1 lowercase letter At least 1 number At least 1 special character from the following: ! # $ * & ? ~

15. Further, these are onerous restrictions; more stringent than the original password respondents are given that only contains numbers and upper and lower case letters. As was recommended earlier, these should be simplified.

16. Participants largely ignored the password requirements and went straight to the Current Password field. Selecting this field moved the requirements off of the screen. It wasn’t until participants received the error message that they realized there are requirements. They need to be made more prominent.

    • As was suggested earlier, getting rid of the bolded statement at the top of the screen will help cut down the distraction from the important information on the page.

    • The password requirements could be bolded

    • Alternatively, the order of the screen could be rearranged and presented as a series of steps. For example, (1) enter current password, (2) enter new password that meets the following requirements…, (3) enter security information

Below is an illustration of what the screen may look like, given the recommendations above:

17. NOT ALLOWED Participants had considerable difficulty with getting their new password and new password confirmation to match. There should at least be an option to “unmask” these as well.

18. CHANGE MADE The security questions do not follow best practices. They should ask users about facts (e.g., what is your mother’s maiden name) not opinions/preferences (e.g., what is your favorite color). Opinions change over time and can be ill-defined. A participant may have two favorite colors, for example.

19. One participant had a technical issue with the change password screen. She made numerous mistakes in entering information into the boxes, triggering various alerts (password does not meet requirements, passwords do not match, incorrect current password, email addresses do not match). Once all of the errors were corrected, the “Change Password” screen reloaded showing no errors and with the security information questions removed. The password had changed but there was no indication that it had changed and the participant was not returned to the home screen. The participant used an iPhone and the Google Chrome browser.

20. Several users appeared confused as to what to do next after submitting a new password because it was not clear upon the reload of the screen whether anything had happened. The errors and notifications should appear at the top of the page (in addition to where they appear now) to alert users of the problem (e.g., “Password does not meet requirements”).

21. When testing this feature, testers should make several mistakes to ensure it is still functional. Users will make mistakes on this page.

22. The current icon for the Change Password function, with two interlocking gears, is difficult to see. It should be replaced with a single gear icon. That icon should be larger and fill the majority of the button as in the example below.

Recommended

Data Entry Screens

23. The optimal keyboard should appear for each field selected. That is, for cost fields, the alphabet keyboard still appears (at least on an iPhone). The numeric keyboard should appear and users should only be able to enter numbers, decimals, and dollar signs (possibly also commas, but this is not necessary). All other characers should be removed. The current reformatting done in the cost field (changing all values to have a dollar sign and two decimal places) is great and should be maintained.

24. CHANGE MADE Need to ensure that there is “padding” at the bottom of the screens. On a couple participants’ phones the button was at the very bottom of the screen. This could put the button right next to browser or phone buttons and could result in problems with navigation. One participant who used Safari on an iPhone did accidently hit the browser button when trying to save an expense.

25. CHANGE MADE In the “Alcohol Included?” box on the “Food and Drink Away From Home” data entry screen, (select all that apply) should be added to the question text.

26. This was mentioned in Phase II, but the most important fix that should be made is for the correct virtual keyboard to appear when a field is entered. The number keyboard should appear when a number field is selected, the email keyboard should appear when an email field is selected (in the Change Password screen), and so on. None of the participants mentioned this, but it would go a long way toward streamlining the process of data entry.

27. The “Go” or “Enter” button on the virtual keyboard should not initiate the save process. Ideally, it would be disabled and do nothing, but otherwise it could tab to the next field. Failure to disable the save function of the “Enter” key will result in half-finished diary entries and perhaps double-entries.

Edit Screens

28. CHANGE MADE Similar to the data entry screens, buttons should be given padding so that they are offset from the bottom of the screen.

Info Screen

29. Add a home icon to the top bar of the screen.

30. CHANGE MADE Move the “Return to Survey” button above the blue box containing the OMB Number.

Changes to Protocol/Pamphlet/Placement (Not Instrument)

31. Recommend that swiping and voice-to-text be suggested to participants to use this if their phone includes it?

32. Recommend during placement that respondents create a bookmark and/or add a link to the instrument to their home screen?

33. Several participants have assumed that they would have access to this information for personal use later. If we are not planning to allow this, we need to explain this to the respondent so they understand.

34. Several improvements should be made to the pamphlet to make data requirements more obvious.

    • Use visual formatting to denote that the data requirements for “Food and Drink Away From Home” are different from the other categories.

    • Add 2 examples:

      • Have an example of a “Food and Drink Away From Home” receipt being added to the Diary (arrows denoting where information from the receipt would be entered into the Diary).

      • Have an example for everything else, perhaps the Val-U-Mart receipt, showing that we need item level information and prices without tax.

35. The subpages on the INFO screen should be revised for better readability on a mobile device and for more helpful content.

36. Training materials that make use of the phones’ Internet connectivity should be considered. For example, web videos could be created that cover topics that would ordinarily be covered during placement. This would allow for training of household members who were not present at the placement interview, and for additional help to cut down on calls to the help desk and FRs. Based on the usability testing, the following topics should be covered:

1. Navigating to the site and logging in

2. CE Diary data needs

3. Entering and editing expenses

37. As part of diary placement, the FRs could offer to assist the respondent in navigating to the site, logging in, and doing the initial setup. Getting respondents to this point will make it much more likely that they use the Diary.

Future Changes

Not all changes are expected to be made prior to the Individual Diaries test. If the Mobile Diary is deemed worthy of future use in the CE, then the following should be considered for testing and implementation in subsequent generations of the instrument.

38. In the Phase I report, it was recommended that a tabbed format be used for the Home page. This would sync the format of the Mobile Diary with the web and paper diaries. Aligning the mobile and web versions will be especially helpful when/if they are linked together so that a respondent can use them interchangeably.
    
    It would also allow for distinct data entry pages for each expenditure category. That is, if one taps on “Add a New Food Away Expense,” the full data-entry page would open on the subsequent screen with category specific prompts for entering a general description of the whole meal and the cost of the meal including tax and tip. This format could also remind respondents that we have different data requirements for the different categories.
    
    It should be noted that these suggestions are based on judgments made by the experimenters involved in the usability tests. Either of these designs should be tested prior to implementation.

Tabbed Browsing

Select Category First w/Instructions

39. Ideally, the keypad that pops up when the cost field is selected should consist of only numbers and a decimal point. After some investigation, we discovered that this is not possible (at least on iOS devices) outside of actual phone applications. The number/symbol keypad mentioned above (once implemented) should not be considered the end point. As operating systems evolve, developers should reevaluate this.

40. A means of adjusting the size of the text within the instrument should be added. This is often done on standard government webpages. Perhaps this could be added to a “Setup” screen that allows the participant to personalize instrument for their needs. The example below is from the BLS.gov website.

1. Overview

The Consumer Expenditure Survey (CE) Program currently uses a paper diary to collect household expenditures. As part of ongoing improvements to the survey, the Bureau of Labor Statistics (BLS) and the Census Bureau (Census) have begun field testing a web-based diary instrument. The web diary may help some respondents with data entry and it has the potential to lead to more complete responses through periodic data checks by field representatives during the collection period. It does not solve one recurring data collection problem however; collecting accurate data on those purchases that do not yield a receipt and/or are forgotten before one returns home to enter items into the Diary. To help solve this issue, BLS and Census are designing a version of the web-diary specifically for use on a smartphone. The usability testing described below will provide feedback on an initial version of the mobile Internet optimized CE Diary survey.

2. Methods

2.1 Participants

Twenty-nine participants (14 female, 15 male) attended individual testing sessions that lasted about one hour. Participants were compensated $40 for their time. The twelve participants in Phase II were recruited by the Census Bureau and the seventeen in Phase III were recruited by BLS. One participant was removed from Phase III because his phone did not have a QWERTY keyboard (physical or virtual) which would have made it extremely difficult to complete the tasks as intended. While this participant did have a smartphone, he would likely have been screened out of the study if we had used the screening questions from the Individual Diaries Feasibility Test (IDFT) which ask if he accesses the Internet using his phone. This resulted in a total of 28 sessions

Participants in both phases were screened based on their prior experience with smartphones. Specifically, only those who reported owning a smartphone and having some experience with it were eligible to participate in this study. Experimenters made a special effort in Phase III to bring in participants who had a smartphone but had little experience using it. Nine participants reported having “A little” experience with smartphones, eleven participants reported having “A moderate amount” of experience, and nine reported having “A lot” of experience. Experimenters also made an attempt to bring in participants who had a range of smartphone operating systems (OS). This allowed us to see (1) whether performance on tasks differed across types of phone users and (2) whether there are significant differences in the way the instrument is displayed on different phone operating systems and browsers. See Table 1 for a breakdown of the study sample by OS type and experience levels.

Table 1: The sample sizes for each cell in the sample design.

		How much experience do you have with using a smartphone?
		A Little	A Moderate Amount	A Lot
Which operating system (OS) is on your device?	iOS/Apple/iPhone	3	2	4
	Android/Google/Galaxy	5	8	5
	Blackberry/RIM Windows Palm Other Don’t Know/Not Sure	1	0	0

OS Specific Technical Problems

1. For one participant, the category-specific questions (e.g., meal, vendor type, and alcohol included for Food and Drink Away From Home) did not appear when she selected the category from the drop-down menu. The category-specific questions did display when she tapped on the edit buttons on the home screen. (Older-model Blackberry phone, not sure which version of the operating system).

2. For another participant, when attempting to select the "Delete Item" button in the deletion popup box, the instrument would open the category selection dropdown. That dropdown was underneath the button though a thin line appeared through the popup box as if the box was overlaid on top of the popup. The participant could not select "Cancel" either. He used the back button to return to the instrument. (HTC Desire Phone, Android OS, using default browser)

3. For that same participant, whenever a text field was selected (description, password, cost field, etc.) the text "User ID" appeared. It would disappear when he started to enter text. (HTC Desire Phone, Android OS, using default browser)

2.2 Procedure

Participants came individually to the usability lab in the Office of Survey Methods Research at BLS. After the experimenter explained the purpose of the study and obtained informed consent, he explained the basic functions of the Diary instrument as well as the data needs for the Diary using a pamphlet designed for use in the upcoming IDFT (see Appendix B). This was meant to simulate Mobile Diary placement.

Participants in this study were asked to bring their own smartphone to the interview. That way, experimenters could ensure the participant was comfortable with the function of the operating system, the web browser, and the various keyboards. The participant was asked to place their smartphone on a sled that had a web camera suspended above it which gave the experimenter and observers a view of the participants’ screen and hands. Morae software recorded video of the participants’ smartphone screen and the audio from the participant and experimenter interaction.

The experimenter remained in the room with the participant and walked him or her through the tasks and debriefing questions (described below). Several observers monitored each session from an adjacent room. The observers’ task was to watch the webcam view of the participants’ screen, listen to the conversation between the experimenter and participant, and take notes on any difficulty the participant had completing each task and any feedback (positive or negative) they expressed during the testing session. Observers used a specially designed form to record their feedback.

During each testing session, the experimenter would read the task instructions and the participant would then attempt to complete the task. There were 13 tasks, as described in the next section. After the 13 tasks were complete, the experimenter asked a series of follow-up questions about the participant’s experience with the Diary.

2.3 Tasks

The 13 tasks used in this study covered the basic tasks CE Diary respondents would perform to complete the Diary survey using a mobile device. That is, they would need to navigate and log in to the Diary, perform the initial setup, enter a variety of purchases, and edit previous purchase entries. There was also a task for changing one’s password. Changing a password is not necessary for completion of the Diary, though it could make it more likely that a respondent will remember their password and thus use the Diary more often.

The tasks were divided into three blocks, shown below. The “Getting Started” block, by necessity, always came first. Blocks A and B were counterbalanced so that half of the participants received Block A then B and the other half received Block B then A.

Participants were read scenarios or given receipts for data entry for 11 of the 13 tasks. For two of the tasks, 4 and 9, participants entered expenses of their own. These non-directed tasks make the data entry more true to what a respondent’s experience would be with the Mobile Diary.

Due to technical issues with the change password function and time constraints, only 9 of the participants were asked to set a personalized password.

Getting Started

	Task Name	Text Read to Respondents
1.	Login	Let’s get started. First, I’d like you to use this username and password to login to the diary…
2.	Start Date	Next, you will see a screen asking you to select your start date. Please select October 21st as your start date. Below the start date, you will see that we ask for your e-mail address. You can skip that box. Please select the “Continue” button.
3.	Set Personalized Password	Next you will be given the option to set a personalized password. You will be logging into the diary several times, so you will want to use something you can remember. Please do not use a password that you use somewhere else, like your email. It’s important that you don’t forget the password since we can’t quickly reset it, so do whatever you would normally do to keep track of a password.

Block A

	Task Name	Text Read to Respondents
4.	Own Food	Think back to the last food purchase you made. Please add that item to the diary as if the purchase was made on [DATE].
5.	Enter Book for Friend, Enter Jeans for Self	On [DATE], you go shopping and buy a book for a friend and a pair of jeans for yourself. Here are the receipts. Please enter these expenses into the diary.
6.	Enter Car Insurance Bill	Later on [DATE], you pay your car insurance bill online. This is the billing statement. Please enter this expense into the diary.
7.	Delete Pants	The next day, on [DATE], you decide to return the pants that you had bought. Please go back and delete that item.
8.	Change Book Details	You also decide that, rather than give the book to your friend, you are going to keep it for yourself. Please update that item to reflect that the book was purchased for you.

Block B

	Task Name	Text Read to Respondents
9.	Enter Own Non-Food Purchase	Think back to the last purchase you made, other than food. Please add that item to the diary as if the purchase was made on [DATE].
10.	Enter Dinner, Enter Movie	On [DATE], you treat a friend to dinner and a movie and you pay for both. You decide to enter the purchases into the diary as you’re waiting for the movie to begin. Here is your ticket stub and the receipt from dinner.
11.	Enter Drinks	On the way home from the movie, you and your friend stop to get a couple drinks. Here is the receipt. Please enter this expense into the diary.
12.	Edit Price of Drinks Purchase	The next day you realize that you had left a $5 bill as a tip for the drinks you purchased, but forgot to enter that as part of the expense. Please change the entry to reflect the full price paid for the drinks.
13.	Enter Long Receipt	On [DATE], you go to the Val-U-Mart superstore to buy a few things for your house – enter your expenses from this receipt.

2.4 Test Metrics

2.4.1 Task Success. The session observers determined whether each task was completed successfully or not. Successful completion means that the participant completed the task as intended, without help from the experimenter. Not successful means a participant did not complete the task, completed the task in a way that would not lead to a valid response, or required help from the test moderator.

When only one observer was present, his or her rating of task success was used. When multiple observers were present, they did not always agree on their rating of task success. In this situation, the majority opinion was used. If there were an even number of observers and tie in ratings (e.g., one rates “Successful”, another rates “Not Successful”), an additional observer watched the video of that session and broke the tie.

2.4.2 Debriefing Questions. After the tasks had been completed, the experimenter asked participants several closed and open-ended questions regarding their experience with the mobile diary and their suggestions for improvements.

2.4.3 Other Quantitative Metrics. Ordinarily, the amount of time it takes to complete teach task and participants’ ratings of task difficulty would be reported in a usability test report. Several difficulties arose during the administration of the test that made it difficult to obtain meaningful measures of these constructs. For example, several participants had significant difficulties completing the tasks and were unable to complete them all. Also, the experimenter would occasionally modify the scope of a task (e.g., not requiring a respondent to start a task from the login screen) in order to get the participant through as many tasks as possible. Given these difficulties, only task success and participants’ answers to debriefing questions will be reported here.

3. Results

3.1 Task Success.

Overall, participants had moderate difficulty completing the tasks. The percentage of participants who successfully completed each task is shown in Table 2. On average, each participant successfully completed 66% of the tasks that they worked on (not all participants completed all tasks). This ranged from as low as 29% correct to one participant who successfully completed all of the tasks he worked on.

The percentage of successfully completed tasks did not increase linearly across the experience categories. Those who reported having “A Little” experience with smartphones successfully completed, on average, 65% of their tasks, those with “A Moderate Amount” of experience completed 59% of their tasks successfully, and those with “A Lot” of experience successfully completed 74% of the tasks. Many of those with little experience with smartphones proved to be more conscientious about the data requirements of the Diary. They were less certain about their skills and would refer to the pamphlet more often.

Table 2: Number of participants who completed a task (N) and the percentage who completed it successfully.

	Task Name	N	Percent Successful
1.	Navigate & Login	28	46%
2.	Start Date	28	96%
3.	Set Personalized Password	9	33%
4.	Own Food	25	80%
5.	Book for Friend, Jeans for Self	24	63%
6.	Enter Car Insurance Bill	22	95%
7.	Delete Pants	21	90%
8.	Change Book Details	18	100%
9.	Own Non-Food Purchase	28	93%
10.	Dinner, Movie	26	35%
11.	Drinks	25	48%
12.	Edit Price	24	58%
13.	Long Receipt	22	27%

3.1.1 Navigating to the Site and Logging In. Participants had difficulty right from the beginning of the test. Only 46% were able to navigate to the site and log-in without some major problem. Although the majority of those who had difficulties with this task reported having little or moderate experience, two of the participants who reported a lot of experience also did not complete this task successfully. Most of these participants had difficulty logging in; mistyping their password two or more times. Several participants did not realize at first that the password is case sensitive, others hit the wrong keys by accident, and others had difficulty distinguishing between similar looking letters/numbers (e.g., l or 1, O or 0, etc.). When participants mistyped their password, they often believed there was something wrong with the credentialing system rather than believe they had typed in something incorrectly.

The protocol of the usability tests originally called for logging into the instrument multiple times. Almost every task was meant to start from the login screen. Since the instrument automatically logs users out after approximately 15 minutes, this would more closely match real-world scenarios. This additional login requirement for each task was quickly abandon when it was apparent that it would take up too much of the testing session, even for experienced users.

A smaller, though still significant group of participants had difficulty navigating to the website. The most common error here was that participants typed the URL into a search box rather than their browser’s navigation bar, this was especially common on Android phones which have a readily accessible Google search bar. The results of a Google search are shown in the screenshot below. The first result links to the correct URL, but the page title “Logout – respond” made some participants apprehensive about clicking on it. Several clicked on the second link because of the large text for CEDM, Census, and Respond. Another navigation error was people who would type in the URL but stop once they’d typed in “respond.census.gov” thinking that once they landed on that site, they’d be able to find a link to where they’re going.

Navigation and login problems should be particularly concerning to CE Mobile Diary stakeholders. First, these problems exist outside of the instrument so they are harder to address. Second, as was mentioned earlier, participants were quick to think that something was wrong with the instrument. These problems could result in a large number of calls to FRs or the help number. Even worse, initial difficulty could give respondents a reason to break-off from the Diary entirely. Finally, we are expecting participants to login to the Mobile Diary instrument perhaps dozens of times over the Diary period. While they will likely get better at this task over time, any difficulty they have will be magnified by the fact that they have to complete this burdensome/difficult task again and again.

3.1.2 Changing Password. Another difficult task for participants was changing their password. Performance on this task went so poorly, in fact, that testing was suspended after only a few cases in Phase II because it took up too much of the testing session and there was no way to navigate away from the page. The navigation issue was fixed in Phase III (as well as several other changes to the Change Password screen) but participants continued to have difficulty with it.

Of the 9 participants who attempted this task across both Phases, none of them made it through without triggering an edit message. The most common problem was that participants did not read the requirements for a valid password. None of the participants added a symbol to their initial password. The edit message caused them to focus on the requirements and choose a valid password. When the edit message appeared, several participants acknowledged that they had not read the requirements.

Many other edits were triggered throughout the testing (e.g., email addresses do not match, incorrect current password, failure to fill out required field) often more than one per participant. In the end, 3 participants (33% of those who attempted it) were able to successfully complete this task, but only after encountering at least one edit message. This task will likely confuse all users; of the three who successfully completed the task, one reported a lot of experience with using smartphones, one reported a moderate amount, and one reported a little experience.

In addition to difficulties with the task, several participants had difficulty just finding the Change Password screen. This was apparent especially in those who mentioned having difficulty seeing the small type on the screen. The small form factor of the Mobile Diary necessitates the use of an icon rather than text. The gear icon is likely still the best option, though it should be made larger.

3.1.3 Initial Setup Screen. The task of selecting a start date was very easy for participants. Only one participant was unable to complete this task without help from the experimenter. That participant, who reported little smartphone experience, was unfamiliar with how the dropdown menu worked.

3.1.4 Data Entry Tasks. As with Phase I of testing on the Mobile Diary, participants had considerable difficulty with data entry. In Phase I, participants’ task success was primarily rated on their ability to press the correct key sequence, put the item in the correct category, and complete all of the required fields. They were not strictly rated on how closely their answer met the needs of the CE Diary survey, because the data needs had not been explained to them. In Phases II and III, participants did receive a placement briefing from the experimenter explaining the key concepts of the diary (e.g., what types of items needed to be entered separately, what types of descriptions were needed, etc.) so their responses were evaluated more closely.

For the most part, participants understood the basic functions of the Diary. They knew to press “Add an expense” to start the process, how to use the dropdowns and data entry fields, and how to save an entry. Participants mostly struggled with meeting the complex data needs of the CE Diary. For testing purposes, participants were given some difficult situations. When participants were given a straightforward data entry task such as the car insurance entry, they performed quite well, with 95% completing the task successfully. Participants mainly struggled with multiple item receipts, with Food Away from Home purchases, and with gifts. The specific issues they had are listed in the Data Entry and Editing Problems box below.

3.1.5 Data Editing Tasks. Participants did not have much of an issue with editing responses. The deletion and item detail change tasks had 90% and 100% success rates, respectively. Participants did have trouble with editing the cost of alcoholic drinks, but this was mainly due to difficulty understanding whether the cost should be changed in the total cost field, the alcohol included cost field, or both.

Several participants did have trouble accessing the edit screen, but managed to successfully complete the task. Several times, the experimenter read the deletion task while the participant was on a blank data entry screen. Rather than navigate to the home screen and tap edit, the respondents used the browser’s back button. This effect was mitigated when the experimenter would ask the participant to navigate to the home screen or login screen prior to reading the task; starting a data editing task at one of these screens more closely approximates what a respondent would encounter in a real-life situation.

Data Entry and Editing Problems

1. As with Phase I, participants continued to enter items cumulatively. The majority of the 73% who did not successfully complete the long receipt task attempted to enter a total amount; this despite being told specifically not to do so. Some participants attempted to combine multiple receipts (the book and pants, the dinner and movie) into a single data entry.

2. Several participants had the opposite problem; they wanted to itemize everything, including the dinner receipt. Again, the reporting of the total receipt was emphasized in placement.

3. Several participants disregarded the category-specific follow-up questions. One field that was missed several times was the “Type of Vendor” question under “Food and Drink Away From Home.”

4. Two participants initially thought the Home Screen showed entries grouped by date. This is not an unreasonable assumption for a diary. When asked to enter a second item for the same day, the participants tapped the edit button next to the first item and looked for a way to enter the data there.

5. Several participants pressed the “Go” or “Enter” button on the virtual keyboard when they did not mean to. They pressed it because they could not get the keyboard to disappear and needed to scroll to the next field. They either thought it would work like a tab key or that it would hide the keyboard. Instead the button initiates the save process. When the entry is saved, participants thought there was a technical error and restarted the entry. This will result in a high number of half entered lines of data.

6. Several participants were unsure how to properly enter expenses for the purchase of alcohol at a bar. Participants had difficulty deciding whether to put the expense in the total cost field, the alcohol included cost field, or both. When a tip was to be added, they were also unsure which field it should be added to.

7. Several participants were unsure how to use the “Purchased for someone outside your household” checkbox. On the task that asked participants to enter movie tickets that they had purchased for themselves and a friend, some participants expressed confusion when they got to the checkbox. They were unsure whether the two movie tickets should be entered separately (with the box checked for one of them) or whether they could be entered together and the box could be checked. Most participants chose the latter option.

3.4 General Reactions

Participants were positive in their overall ratings (see Table 3). Thirteen participants rated the Mobile Diary as Extremely Easy or Very Easy to use. Only two rated the Diary as Somewhat Difficult to use. Confidence in completing the Diary fell roughly in the middle of the scale with an average rating of 2.56 out of 4. Almost all of the participants said that using the Diary would require at least some training.

Interestingly, only nine of the 28 participants who answered the follow-up questions said that they would complete entries right away; a key selling point of the Mobile Diary. The remainder of the answers specified some time later that they would likely complete the Diary; most likely later in the day they made the purchase. Twenty-four of the 28 participants said that they would keep a receipt if one was offered to them and use that when recording their expenses. Only five said that they would rely on memory and none of them said that as their sole answer.

Table 3: Frequency and average overall ratings of ease of use, confidence, and need for training. Note that for Question 3 regarding training, a lower score is preferable, while for the other two a higher score is preferred.

Question	Response Options	Frequency	Average Score
Was the mobile diary easy or difficult to use?	Extremely Difficult (-3)	0	1.33
	Very Difficult (-2)	0
	Somewhat Difficult (-1)	2
	Neither Easy Nor Difficult (0)	3
	Somewhat Easy (1)	9
	Very Easy (2)	10
	Extremely Easy (3)	3
How confident did you feel in filling out the entries in the diary?	Not At All Confident (0)	0	2.56
	A Little Confident (1)	4
	Somewhat Confident (2)	9
	Very Confident (3)	9
	Extremely Confident (4)	5
How much training do you think the average person would need to get started using the diary?	None (0)	2	1.30
	A Little (1)	17
	A Moderate Amount (2)	6
	A Lot (3)	2
When would you record your expenses? (check all that apply)	Right away	9	N/A
	Sometime the same day	10
	At the end of the day	10
	Sometime throughout the week	3
	At the end of the week	2
	Other	0
How would you record your expenses? (check all that apply)	Keep the receipts	24	N/A
	When I don’t have receipt	15
	When I only have 1 expense	2
	Refer to paper notes	6
	Refer to budget application	1
	Enter expenses from memory	5

3.5 Participant Recommendations

Throughout the survey and through direct, open-ended questions during the debriefing, participants offered the following recommendations for improving the Mobile Diary.

Participant Recommendations

1. Several participants indicated that they would like to see additional categories added to the category selection box. Some examples given were “entertainment” and “bills.” It should be noted that these participants thought of the CE Diary as a budgeting application and not a data collection instrument. This view was commonly held across participants.

2. A few participants commented during the completion of their tasks that they would like the ability to enter information into the Diary as it might appear on a receipt. That is, they would like to start with a single entry and then break that into the constituent items using sub-forms.

3. Several participants had difficulty reading the text in the instrument. They did not attempt to zoom the text or adjust their settings to accommodate the small font size. These participants recommended a means of adjusting the size of the text within the instrument. This is often done on standard government webpages. Perhaps this could be added to a “setup” screen that allows the participant to personalize instrument for their needs.

4. Several participant suggested fixes for the Diary that are not technically feasible in a web-based instrument. For example, several recommended the ability to photograph receipts and have data extracted from the receipts. While these aren’t possible in the current design of the instrument, if an app-based instrument is developed, these types of feature should be explored.

5. One participant commented on the overall look of the Diary saying that it looked “governmental” and the color of the background especially was flat and bland.

4. Conclusion

Despite the advanced stage of the instrument, participants in this usability test had significant difficulties entering data in a way that would be usable to the Consumer Expenditure program. Many of the issues participants are having with understanding CE data needs, in particular the itemized entry of purchases, are likely common across the web and paper diaries as well. The difference between the CE Mobile Diary and these other versions of the diary survey is that there is limited screen space to fit instructions and examples (similar to what is in the initial pages of the paper diary). This is an area that warrants further thought as the Mobile Diary progresses through testing. There will likely be need for additional written instructions within the instrument as well as a more extensive booklet of instructions and/or training videos. These issues may also be mitigated through placement procedures used by field representatives who have encountered these issues with the paper diary.

It is the opinion of the experimenters that a standalone Mobile Diary instrument will not work for CE data collection. If the Mobile Diary is to be successful, it should be linked and used with the Web Diary. Data quality and burden may be negatively impacted by mode, even for respondents who volunteer to use and are comfortable with the Mobile Diary. In our sample of participants, several individuals had little mobile experience and made many data entry errors but nonetheless expressed a preference for mobile over web or paper. The Mobile Diary could serve as a supplement to data entry or even the main source of data entry for household members with fewer expenses, but should not be considered the right or only solution for a whole household.

The mobile device is still a relatively new concept to people and the idea of using their phone or tablet for data entry and for something that doesn’t specifically benefit them is still foreign. A large number of participants (close to half) spontaneously mentioned that they thought the diary would be useful for them and that they would be interested in using it for budgeting purposes. That is, they didn’t understand that BLS needs their data for our purposes and that their access to the site would be cut off immediately after their diary period. Instead, they were focused on entering data in a way that they might find personally useful and their expectation was that they could use this to track their own expenses over time. These attitudes may shift over time, but in the short term may make it difficult to convince respondents to enter “wheat bread” instead of “groceries” because they don’t care about how much they spend on bread, let alone different types of bread.

On a slightly more optimistic note, it is likely that some of the issues outlined in this report can be addressed simply by devising the right set of screening questions to ensure only those with the prerequisite skills end up using the Mobile Diary. The performance of the screening questions in the IDFT will help understand who ends up being directed toward the Mobile Diary instrument.

4.1 Limitations

The main limitation of this study was the narrow range of participant demographics and phone types that were sampled. With the proliferation of smartphones, there are likely many different “types” of users. We sampled across perceived experience with phones, though the participants (especially those with limited experience) were mostly older, over 45. These are often our target survey respondents, but for an individual diary, a wider range of participants would have been preferable. It should also be noted that this usability test used a small, convenience sample of participants. The results outlined above are meant to highlight potential problems with the Mobile Diary, not to determine the potential prevalence of those problems.

We also tested only a narrow range of phones, operating systems, and browsers. While we tested the most popular operating systems (Android and iOS) there were enough technical issues with those and the few other devices we saw (e.g., a Tracfone, a Blackberry) that there are likely a significant number of issues we haven’t accounted for with this set of recommendations.

We would have liked to compare alternate versions of the Mobile Diary to ensure that our recommendations were based in evidence. Timing and resources did not allow for the development of multiple versions of the instrument, however.

Appendix A: Mobile Diary Screenshots

Login	Initial Setup	Expense Summary
Common Entry	Food Away From Home	Food At Home

Clothing	Other Expense	Change Password

Appendix B: Placement Protocol and Pamphlet