Various Demographic Area Pretesting Activities

General Introduction for TIGERweb usability testing

Thank you for your time today. My name is (Test Administrator). I work here in the U.S. Census Bureau Usability Lab, and I will be working with you today. In this lab, we evaluate how easy or difficult our sites are to use. We bring in people like you who are potential users of our sites to try them out while there is still time to make changes to them. What works well, we keep. When potential users such as you have difficulty with something, we have an opportunity to fix it.

Today, we are working with a new version of a website, called TIGERweb. We are having you look at it now, in this early stage to get your feedback and comments so that as it is developed further we will have user input into what makes it better. We have put together a series of questions or tasks that we would like you to try to answer by using this website. I am going to give you (a number of) tasks to work on. Today, you need to read out each task and then try to accomplish the task using the website.

I want to tell you that you can’t make a mistake or do anything wrong here. Difficulties you may run into reflect the design of the website, not your skills or abilities. If you have a problem using parts of it, do not blame yourself. This site is intended for people like you. Where it works well, that’s great. Where it does not work well that is also great, because you will be able to help us identify things that can be corrected.

While you work through each task, you need to tell me what you are thinking as you try to find the answer. Your comments and thoughts are important. They will help the developers make changes to improve the site. So, please feel free to give me your impressions and your thoughts, both good and bad of what you see and what you experience on the site. I did not create the site, so do not feel like you have to hold back on your thoughts to be polite. Again, we are not evaluating you or your skills, but rather you are helping us see how well the site works. To do that, you need to share both your positive and negative reactions to the website.

So, while you are working, I’d like you to think aloud. In other words, I’d like you to tell me what you are thinking, describe the steps that you are taking, what you are expecting to see, why you are doing what you are doing, what you are going to do, and why. Tell me why you clicked on a link or where you expect the link to take you. Tell me if you are looking for something and what it is and whether you can find it or not. I will be here to help if you get stuck.

Only those of us connected with the project will review the results are they are used solely for research purposes. Your name will not be associated with the results, with the video recording or any of the other data collected during the session.

Okay now we’ll practice thinking aloud. (Test Administrator opens Craigslist/WTOP home page and asks practice question, “You have a few minutes around lunch time, you decide to check out what is going on in the news. You go on the WTOP website. Tell me from here what you would do.”)

Okay that was fine. Do you have any questions about the “think aloud” process we’ve just practiced and that I’ve asked you to use?

Now I am going to calibrate your eyes for the eye-tracking.

Do Calibration

Now that we have your eyes calibrated, we are ready to begin. I am going to go around to the other room to do a sound check. While I am doing that, please take a moment to complete this questionnaire. [Hand P questionnaire on Computer experience and demographics]

I’m going to leave but we will still be able to communicate through a series of microphones and speakers. Do you have any questions before we begin?

Questionnaire on Computer-and-Internet Experience and Demographics

1. During the last month, about how many hours did you use the Internet during a typical week? __________

2. Six months ago, about how many hours did you use the Internet during a typical week? __________

3. Five years ago, about how many hours did you use the Internet during a typical week? __________

4. How much experience have you had with computers to use the Internet? Please check one option.

G A great deal

G A lot

G A moderate amount

G A little

G None

5. How much experience have you had with computers to do things other than use the Internet?

Please check one option.

G A great deal

G A lot

G A moderate amount

G A little

G None

6. During the last year, how many times did you complete a survey on the Internet? _____________

7. During the last month, where were you when you used the Internet? Please select all places.

G Home

G Work

G School

G Library

G Another place (specify: _________________________________)

8. Do you have a computer with Internet access at home?

G Yes

G No (Skip to Question 9)

8a. What type of internet connection do you typically use at home? Please check all that apply.

G Cable Service

G Dial-up Service

G DSL Service

G Satellite Dish Service

G Fiber-Optic Service

G Mobile Broadband Plan for a Computer or Cell Phone

8b. Browsers are software on a computer used to surf the Internet. Last month, which Internet browser did you typically use at home? Please check all that apply.

G Firefox

G Google Chrome

G Internet Explorer

G Safari

G Other: __________________

8c. During the last month, what was the operating system on the computer you typically used at home?

Please check all that apply.

G Linux

G MAC OS

G Windows

G Other (specify: _____________)

9. During the last week, what are all the things you did on a computer? (Check all that apply.)

G Budget Expenses

G Conducted statistical analyses

G Looked at and/or edited photos

G Played video games

G Typed documents

G Used the Internet

G Other (specify ____________________________________)

G Other (specify ____________________________________)

G Other (specify ____________________________________)

G Other (specify ____________________________________)

10. How difficult is it for you to learn to use Web sites that you have not visited before?

G Extremely difficult

G Very difficult

G Moderately difficult

G Slightly difficult

G Not difficult at all

11. Computer windows can be minimized, resized, and scrolled through. How difficult is it for you to manipulate a computer window?

G Extremely difficult

G Very difficult

G Moderately difficult

G Slightly difficult

G Not difficult at all

12. How difficult is it for you to use the Internet?

G Extremely difficult

G Very difficult

G Moderately difficult

G Slightly difficult

G Not difficult at all

13. Not including email, how uncomfortable are you with providing personal information on Internet forms and surveys?

G Extremely uncomfortable

G Very uncomfortable

G Moderately uncomfortable

G Slightly uncomfortable

G Not uncomfortable at all

14. During last month, how many times did you do complex analyses of data (e.g., using SAS, SPSS, Excel) using a computer? _________

15. How familiar are you with the TIGERweb site, (e.g., location, tools, terms, data)?

G Extremely Familiar

G Very Familiar

G Moderately Familiar

G Slightly Familiar

G Not Familiar At All

Demographic Information

1. In which of these age categories do you belong?

❑ 30 or younger

❑ 31-45

❑ 46-60

❑ 61 or older

2. What is the highest grade or year of school you have completed?

❑Less than high school

❑Completed high school

❑Some college, no degree (indicate area of study: ____________________________________)

❑Associate’s degree (AA/AS) (indicate area of study: __________________________________)

❑Bachelor’s Degree (BA/BS) (indicate area of study: __________________________________)

❑Post Bachelor’s/Professional degree (indicate degree and area of study:_________________)

3. Are you male or female?

❑Male

❑Female

4. Are you of Hispanic, Latino, or Spanish origin?

❑Yes

❑No

5. What is your race? Choose one or more races.

❑White

❑Black or African American

❑American Indian or Alaska Native

❑Asian

❑Native Hawaiian or Other Pacific Islander

Tasks for users: Novice (1-4) Expert (1-7)

Question 1: Imagine you are moving to Virginia. You want to see a map of Virginia with the outline of all the counties and their names. Using TIGERweb, what would you do to find this information?

Question 2: Now imagine you are moving to Virginia and you have school-aged children. You heard Fairfax County has the best schools. You want to see the school boundaries for Fairfax county using TIGERweb. Once you find the information, trace the county boundary and the school boundary with your mouse.

Question 3: Using TigerWeb find the street that you live on. Display the street and county school district for your street.

Question 4: Using TIGERweb, what State Legislative District do you live in?

Question 5: Using TIGERweb, display the Washington DC, VA, MD urban area boundaries and the county boundaries that they are associated with. Also, obtain the urban area codes.

Question 6: An interviewer needs a map of roads in Suitland, Maryland that overlays places with census tracts. Using TIGERweb, create a map for this interviewer.

Question 7: Using TIGERweb, obtain number of housing units and number of people for the state of Delaware.

Post-task Questionnaire after each task for TIGERweb Usability Study.

Using TIGERweb is a frustrating experience.

Strongly Disagree						Strongly Agree
1	2	3	4	5	6	7

TIGERweb is easy to use.

Strongly Disagree						Strongly Agree
1	2	3	4	5	6	7

I have to spend too much time correcting things with TIGERweb.

Strongly Disagree						Strongly Agree
1	2	3	4	5	6	7

Overall Satisfaction Questionnaire for TIGERweb Usability Study.

Please circle the numbers that most appropriately reflect your impressions about using this Web-based instrument.

1. Overall reaction to the Web site:	terrible wonderful
		1	2	3	4	5	6	7	8	9	not applicable
2. Screen layouts:	confusing clear
		1	2	3	4	5	6	7	8	9	not applicable
3. Use of terminology throughout the Web site:	inconsistent consistent
		1	2	3	4	5	6	7	8	9	not applicable
4. Information displayed on the screens:	inadequate adequate
		1	2	3	4	5	6	7	8	9	not applicable
5. Arrangement of information on the screen:	illogical logical
		1	2	3	4	5	6	7	8	9	not applicable
6. Tasks can be performed in a straight-forward manner:	never always
		1	2	3	4	5	6	7	8	9	not applicable
7. Organization of information on the site:	confusing clear
		1	2	3	4	5	6	7	8	9	not applicable
8. Navigation:	impossible easy
		1	2	3	4	5	6	7	8	9	not applicable
9. Overall experience of finding information:	difficult easy
		1	2	3	4	5	6	7	8	9	not applicable
10. Census Bureau-specific terminology:	too frequent appropriate
		1	2	3	4	5	6	7	8	9	not applicable

11. Overall reaction to the Web site:

Terrible Wonderful

1 2 3 4 5 6 7

Frustrating Satisfying

1 2 3 4 5 6 7

Difficult Easy

1 2 3 4 5 6 7

13. Additional Comments (use the back of this paper if necessary):

Debriefing Questionnaire

1. Can you walk me through your thinking on why you marked (a particular item) especially low/high? (Do this for several low/high ratings).

2. What do you think of the basic screen layout?

a. Overall?

b. Colors?

c. The map?

d. Zooming and navigation tools?

e. Windows that opened up

f. How you selected an area

g. Other?

3. What did you like best about the page? Or What is something that you feel should stay the same?

4. What did you like least about the page? Or What is something that you feel should be changed?

5. How easy or difficult do you feel it was to make the choice on where you would go to find the information you were after?

6. Is there anything you’d like to mention that we haven’t talked about?