Sample Study Plan

IOE 2015 BCase01

State abbreviation Study Plan

Draft: July 9, 2018

Last revision: May 30, 2018

Version 4.0

Principal Investigator: Elizabeth M. Nichols

IOE 2015 BCase 01 Project Team

U.S. Census Bureau

Purpose: To investigate whether the states spelled out in a list of response choices allow interviewers to make a selection more quickly and accurately than when states are abbreviated.

1. Primary Research Questions

1. What display of state response options results in a faster interview in an interviewer-administered survey:

   1. state abbreviations in a dropdown

   2. the full spelling of states in a dropdown

   3. the combination of abbreviation and states

2. What display of state response options results in a more accurate interview in an interviewer-administered survey:

   1. state abbreviations in a dropdown

   2. the full spelling of states in a dropdown

   3. the combination of abbreviation and states

3. Is there a difference in user satisfaction between the three designs?

2. Outcome Measures (dependent variables)

1. Completion time¹: Time is measured for each screen; from the time the screen loads until the time the next button is selected.

2. Completion time to make a response choice: Time is measured for each screen; from the time the response option field becomes in focus until the time the next button is selected.

3. Accuracy of entry: Whether final data entry is correct or not based on the script entered. (1=Yes/ 0=No)

4. Task satisfaction score: (1-5)

5. Knowledge of state abbreviations: (0-51) This variable is a simple count of the number of states selected that the subject claims to know the abbreviation. If the subject selects “Select all” then this variable should equal 51.

Captured manually through audio recording.

1. Comments: Any comments about the response choices during the task.

3. Experimental Factors (independent variables)

1. Display of states (3 conditions)

   1. Condition 1: Abbreviation (all caps)

   2. Condition 2: Full state name (initial caps only)

   3. Condition 3: Abbreviation then state, like this MD - Maryland²

4. Experimental Design

This will be a between-subjects design with three conditions as described in (3 above). Subjects will be required to act as interviewers and conduct a face-to-face survey with the survey on an iPhone 6. The first task is for subjects to check off the states they know the abbreviation for. We will have to prescreen subjects and they must know at least 40³of the state abbreviations. If we allowed subjects without knowledge of the state abbreviations to take part in the sample, then any differences in accuracy or timing might be due to the lack of knowledge rather than because of the design. If the subject knows 40 of the 51 state abbreviations (DC is included), then they will proceed to the next step. At maximum 30 subjects will proceed to the next step. In that step, they will be assigned to one of 3 conditions, thus there are 10 subjects per condition. Then, they will receive the main survey. The subject will read the question, “What state shall I select?” aloud. The researcher will provide the full name of the state orally to the subject and the subject will enter the state and proceed to the next screen, where the same question is asked. This question and answer task will be repeated 51 times. The list of states and the District of Columbia will be orally stated in a randomized order. There will be 10 different randomizations of the states and the District of Columbia. There will not be a difference in how the states are presented orally to the subjects based on condition, that is, the states will always read aloud as the full name of the state because that is how typically respondents answer an address question. There will be a satisfaction question programmed into the survey after the subject has entered the data for the 51 states. The selected display of the response choice for the state task will use the format with a child window as shown in Figure 1.

5. Experimental Paradigms (tasks)

1. The subject identifies the states whose abbreviation they claim to know.

2. The subject records the answers to 51 questions on a smartphone.

3. The subject answers a satisfaction question.

6. Experimental Procedure

   1. TA does the following: loads the survey app, enters a pre-participant ID, hands survey with the states listed to the subject

   2. The first task is for the subject to check the states for which he or she knows the abbreviations.

   3. Then the subject hands the phone back to the TA. The TA checks to see whether the subject claims to know at least 40 abbreviations.

   4. If the subject has self-identified that they know at least 40 questions, the TA enters the condition and hands the phone back to the subject.

   5. The TA instructions the subject to ask the question as worded.

   6. Subjects then conduct the survey with the researcher reading one of 10 scripts to respond to the questions. A Macbook will be used to record the screen.

   7. Subjects will answer a satisfaction question.

7. Conditions

4. Condition 1: Abbreviation (all caps)

5. Condition 2: Full state name (initial caps only)

6. Condition 3: Abbreviation then state, like this MD - Maryland

8. Stimulus Design

This is the order of screens:

1. Screen for Pre-P #

2. Screen contains a long scrolling list of all 50 states and the District of Columbia in alphabetic order. The states and the District of Columbia, spelled out, which a checkbox to the left of each of the states. A “Select all” box at the top. The instruction reads, “Please select the states you know the state abbreviation for.” With a next button at the bottom of the screen.

3. “Thank you. Please hand to phone back to the test administrator.” With a continue button at the bottom.

4. Screen lists how many states were checked. There are two buttons on the screen (Conduct experiment/ Do not conduct experiment)

5. Conduct experiment  Figure 1: Screen with Pre-P# listed, P ID# entry field, Condition dropdown (1-3), Script dropdown (1-10) (with Continue button) (goes to (f))

Do not Conduct experiment  Figure 6 (End button and goes back to (a))

6. Figure 2: “On the next 51 screens you read the bolded question, the researcher will provide an answer, and you enter that answer into the response field. ” (Continue button).

7. Then, the questionnaire contains 51 screens where the only thing on the screen is “What state shall I select?” and a dropdown using the 50 states and the District of Columbia using the design in Figure 3. The dropdown uses the condition selected: Condition 1: Abbreviation (all caps); Condition 2: Full state name (initial caps only); Condition 3: Abbreviation then state, like this MD - Maryland

A next button is on the screen.

8. Figure 4: You have completed the survey. We now have a question about your experience with this survey. (Continue button).

9. Figure 5: How easy or difficult was it for you to accurately record the answers in the survey?

 1 = Very Easy

 2

 3

 4

 5 = Very Difficult

Submit button

10. Figure 6: Thank you for your participation. Please return the phone to your test administrator.

9. Data Set

Data set 1 – one row for every screen for every participant

1. Time date stamp

2. Pre-P_ID

3. P_ID: The participant’s identification number

4. Experiment: All cells will say experiment #

5. Condition (1-3)

6. Script # (1-10)

7. Question number (1-51)

8. Completion time: Time is measured for each screen; from the time the screen loads until the time the “next” button is selected.

9. Time within scroll window: Time is measured for each screen from the time the dropdown is activated until it is deactivated (if that occurs more than one time, then the times should be summed).

10. Final response: Final data entry record.

11. Correct or not (1=yes, correct; 0=no, not correct) (if this is too time consuming to program, I can)

Data set 2 – one row for every participant

1. Time date stamp

2. Pre-P_ID

3. P_ID: The participant’s identification number (will be missing for people who did not proceed into experiment)

4. Experiment: All cells will say experiment #

5. Condition (1-3) (will be missing for people who did not proceed into experiment)

6. Script # (1-10) (will be missing for people who did not proceed into experiment)

7. Knowledge of state abbreviations (so there should be 51 variables and use the state abbreviations as the variable name): (1=Yes checked it; 0=No did not check it)

8. Number of state abbreviations self-reported as knowing

9. Satisfaction score: (1-5) (will be missing for people who did not proceed into experiment)

10. Limitations

There are some limitations with the design but each of these limitations would, in theory, affect the conditions equally given large sample sizes.

1. Subjects might select more states than they know the state abbreviations for which is a confounding factor.

2. We are assuming that subject hearing is satisfactory, otherwise hearing the answer might be a confounding factor affecting both the time-on-task and accuracy measures.

3. Finally, the task is repetitive and the subject could grow tired affecting both the time-on-task and accuracy measures.

11. Data Analysis

First, we will use a Chi-Square statistic to model whether the accuracy of selection differed by condition. This will be a 3 x 2 Chi-Square (3 conditions by 2 accuracy). As this Chi-square does not take into account the knowledge of the state abbreviation, we will also model accuracy of selection with the independent variable of condition, and controlling for state and self-reported knowledge of state abbreviation and with a random effect for the subject. We will model completion time for questions with the independent variable of condition, and controlling for state and self-reported knowledge of the state abbreviation and with a random effect for the participant. Finally, we will tabulate the overall satisfaction scores to determine the variation between respondents and tabulate the knowledge data.

12. Participants

30 participants will need to get into the experiment (10 per condition), but there might be more who do the knowledge check and do not know at least 40 state abbreviations.

This will produce a total of 1,530 question-level observations (30 participants × 51 questions) or 510 observations per condition.

These participants will be from the general public. They should be age 18 and older and own a smartphone.

IOE Experiment Study Protocol

We would now like you to conduct a short survey on a smartphone. The first part of the survey is to see how many state abbreviations you know.

We will be recording the smartphone on this laptop while you are working on the survey and recording the audio. We will not be recording your face. Nor will we be associating your name with the recording.

Do you have any questions before we begin?

Ok, great. Now let me turn on the camera.

Step 1: [Start QuickTime Mobile Screen Recording] Command spacebar; search for quicktime; double click; file:/new movie recording; click on the arrow for the button and select iPhone, then select red button to record;

Step 2: Pick up phone and unlock. Exp 52 is one app. Tap the app. Enter pre-participant ID into the Pre-Participant ID field

Here is the smartphone. The first task is for you to select all the states that you know the abbreviations for. If you know all state abbreviations, you can select the checkbox at the top of the screen, and it will select all of them. If you don’t know them all, select the states that you know. Once you finish this task, you will see a screen that tells you to hand the phone back to me.

Are you ready? Yes. You may begin.

[P completes the task. Take hand-written notes of any relevant comments by P while taking the survey. If P asks a question, tell them: imagine that they are doing the task at home and they cannot look up any abbreviations.]

Step 3: [Take the smartphone back when finished] and see how many abbreviations they know. If they know more than 40, then proceed.

Step 4: If proceeding: Bring up one of the conditions on the smartphone determined from the assignment sheet and enter the next participant number.

The next part of the experiment is for you to ask questions, I will answer them, and you will record the answer into the phone. You should ask the questions exactly the way they are presented on the phone. You read aloud what is in bold. I will provide the response and then you enter it. The answers to the questions are the states in the U.S.

You will ask 51 questions altogether. You will be unable to ask for help. If you cannot find the state, that is okay. It is part of the experiment. In a moment, I will hand you the phone. The survey should take about 10 minutes to complete. Some of questions in this survey you might have already asked in another experiment.  After the questions, then you will be asked to record how satisfied you were with the design of the survey. You answer that for yourself – you do not ask me that question.

Step 4: Hand device to P.

Are you ready? Yes. You may begin.

[P completes the task. Take hand-written notes of any relevant comments by P while taking the survey. If P asks a question, tell them: imagine that they are doing the task at home and they cannot look up any abbreviations.]

Great, thanks. That will be all for this experiment.

Step 6: X in upper left, then save with the P number to the desktop in the folder.

Appendix B

Figure 1: Example of screening page but need 3 conditions, pre-ID, and script dropdown with 10 choices (1-10)

Figure 2: Change text to “On the next 51 screens you read the bolded question, the researcher will provide an answer, and you enter that answer into the response field.”

Example screen

Figure 3: This is how I want the dropdown to come up as a picklist

.

Figure 4: This is after the 51 screens

Figure 5: Change question to: How easy or difficult was it for you to accurately record the answers in the survey?

Figure 6: The end of the task

1 We do not know the effect size to be expected for either time variable as there is no prior literature on time for making a selection of state.

2 This is the typical order of the items.

3 We will determine how many abbreviations Census Bureau staff know. We might change the number 40 to a different number after our investigation.