Pilot - Draft Interview Protocol

Appendix C: Draft protocol for follow-up interviews with 15-20 pilot test respondents

Introduction

Thank you for agreeing to speak with me today. We would like to get some additional information about your response to the survey and your general thoughts on the study. I’d like to start by discussing the time and effort needed to complete the survey.

1. You indicated that you needed ____ minutes to complete the survey. Can you tell me more about that?

Probe as needed:

• What types of records did you need to consult?

• Did you need to work with anyone else to complete the survey?

• What questions were difficult to complete?

• What questions were easy to complete?

Instructions and definitions page

Let’s look at the second page of the questionnaire now.

2. Please look at the information under the heading “Survey Overview and Instructions.” What is your reaction to this information?

Probe as needed:

• What about the instruction in bold that begins “Please report based on…”? What is your interpretation of that instruction?

• Were you able to report based on tests used for students enrolled in [a program designed to transfer to a four-year institution]/[a liberal arts and sciences program]? Please explain.

Please look at the information under the “Definitions” heading.

3. What is your reaction to the definition of developmental or remedial courses?

Probe as needed:

• How does the definition apply to your institution?

• What do you think about the instruction for writing courses?

4. What is your reaction to the definition of entry-level college courses?

Probe as needed:

• What kinds of courses do you consider to be “entry-level?” Do you have examples?

5. What about the definition of entering students? What is your reaction to this definition?

Probe as needed:

• What types of students do you think fall under this definition?

• Are there types of students you think would be excluded?

Mathematics survey questions

Now I would like to look at the survey questions, starting with the mathematics section on page 3.

NOTE: Use the following sets of questions as appropriate:

• If math section was completed, start with question 7.

• If math section was not completed and reading section was completed start with question 6, then skip to reading section.

• If neither section was completed, ask questions 6, 7, 9, 10, and 13, then skip to reading section.

6. I noticed that you did not complete the mathematics questions. Can you tell me more about that?

Probe as needed:

• Were the mathematics questions not applicable to your institution? Please explain.

• Were you unable to obtain data to answer these questions? Please explain.

[SKIP TO READING SECTION IF READING SECTION WAS COMPLETED]

7. Let’s start by looking at question 1. Can you tell me in your own words what this question is asking?

Probe as needed:

• Please look at the first bulleted instruction under question 1. What is your interpretation of this instruction?

• How about the second bullet? What is your interpretation of this bullet?

  • What is your reaction to the portion of the instruction that says “even if your institution does not offer developmental or remedial mathematics courses?”

8. I noticed that you answered [yes]/[no] to question 1. Can you tell me how you arrived at your answer for this question?

Let’s look at question 2 now. I’d like to get your feedback on the two bulleted instructions.

9. Can you tell me in your own words what the first bullet is saying?

Probe as needed:

• Does this situation apply to your institution?

10. Let’s look at the second bullet now. What do you think this bullet is referring to?

Probe as needed:

• Does this situation apply to your institution?

[SKIP TO QUESTION 12 IF R ANSWERED “NO” TO SURVEY Q1]

11. Now let’s look at the list of tests in question 2. I noticed you reported scores for [NAME TESTS]. Can you tell me more about how you use these tests?

Probe as needed:

• Does your response to question 2 adequately represent your institution’s policy for evaluating entering students’ need for developmental or remedial mathematics courses? Please explain.

• Are the scores you reported representative of the level at or above which entering students are prepared for entry-level courses? Please explain.

• I noticed you wrote in [TEST NAME] as an “other” test. Can you tell me more about this test?

• I noticed you made a comment in the comment box for question 2 [SUMMARIZE COMMENT]. Can you tell me more about that?

12. Now let’s look at question 3. You answered [yes]/[no] to this question. Can you tell me more about how you arrived at this answer?

13. Let’s look at question 4. I’d like to get your thoughts on the list of criteria shown in this question.

Probe if needed:

• Are there any other criteria you think we should add to the list?

[SKIP TO READING SECTION IF R ANSWERED “NO” TO SURVEY Q3]

14. I see that you selected [NAME Q4 SELECTIONS] for question 4. Can you tell me more about how your institution uses these to evaluate entering students’ need for developmental or remedial mathematics courses?

Probe if needed:

• Do you use these in combination with test scores to make placement decisions? Please explain.

• I noticed you wrote in [SPECIFY WRITE-IN] in line f. Can you tell me more about how you use this criterion?

• I noticed you made a comment in the comment box below question 4 [SUMMARIZE COMMENT]. Can you tell me more about that?

Reading survey questions

NOTE: Use the following sets of questions as appropriate:

• If reading section was not completed, ask question 15, then skip to conclusion.

• If R completed both reading and mathematics sections, ask questions 16 and 17, then skip to conclusion section.

• If only reading section was completed, begin with question 18.

15. I noticed that you did not complete the reading questions. Can you tell me more about that?

Probe as needed:

• Were the reading questions not applicable to your institution? Please explain.

• Were you unable to obtain data to answer these questions? Please explain.

[SKIP TO CONCLUSION SECTION]

NOTE: If R completed both reading and mathematics sections, ask questions 16 and 17, then skip to conclusion section.

The questions in the reading section are very similar to those in the math section, with the exception of the tests listed. I would like to focus only questions 5 and 6 in the reading section, then we’ll wrap up.

16. I noticed that you answered [yes]/[no] to question 5. Can you tell me how you arrived at your answer for this question?

[SKIP TO CONCLUSION SECTION IF R ANSWERED “NO” TO SURVEY Q5]

17. Let’s look at question 6. I noticed you reported scores for [NAME TESTS]. Can you tell me more about how you use these tests?

Probe as needed:

• Does your response to question 6 adequately represent your institution’s policy for evaluating entering students’ need for developmental or remedial reading courses? Please explain.

• Are the scores you reported representative of the level at or above which entering students are prepared for entry-level courses? Please explain.

• I noticed you wrote in [TEST NAME] as an “other” test. Can you tell me more about this test?

• I noticed you made a comment in the comment box for question 2 [SUMMARIZE COMMENT]. Can you tell me more about that?

[SKIP TO CONCLUSION SECTION]

NOTE: Ask questions 18 through 25 if only the reading section was completed.

18. Let’s start by looking at question 5. Can you tell me in your own words what this question is asking?

Probe as needed:

• Please look at the first bulleted instruction under question 5. What is your interpretation of this instruction?

• How about the second bullet? What is your interpretation of this bullet?

  • What is your reaction to the portion of the instruction that says “even if your institution does not offer developmental or remedial reading courses?”

19. I noticed that you answered [yes]/[no] to question 5. Can you tell me how you arrived at your answer for this question?

Let’s look at question 6 now. I’d like to get your feedback on the two bulleted instructions.

20. Can you tell me in your own words what the first bullet is saying?

Probe as needed:

• Does this situation apply to your institution?

21. Let’s look at the second bullet now. What do you think this bullet is referring to?

Probe as needed:

• Does this situation apply to your institution?

[SKIP TO QUESTION 23 IF R ANSWERED “NO” TO SURVEY Q5]

22. Now let’s look at the list of tests in question 6. I noticed you reported scores for [NAME TESTS]. Can you tell me more about how you use these tests?

Probe as needed:

• Does your response to question 6 adequately represent your institution’s policy for evaluating entering students’ need for developmental or remedial reading courses? Please explain.

• Are the scores you reported representative of the level at or above which entering students are prepared for entry-level courses? Please explain.

• I noticed you wrote in [TEST NAME] as an “other” test. Can you tell me more about this test?

• I noticed you made a comment in the comment box for question 2 [SUMMARIZE COMMENT]. Can you tell me more about that?

23. Now let’s look at question 7. You answered [yes]/[no] to this question. Can you tell me more about how you arrived at this answer?

24. Let’s look at question 8. I’d like to get your thoughts on the list of criteria shown in this question.

Probe if needed:

• Are there any other criteria you think we should add to the list?

[SKIP TO CONCLUSION SECTION IF R ANSWERED “NO” TO SURVEY Q7]

25. I see that you selected [NAME Q8 SELECTIONS] for question 8. Can you tell me more about how your institution uses these to evaluate entering students’ need for developmental or remedial reading courses?

Probe if needed:

• Do you use these in combination with test scores to make placement decisions? Please explain.

• I noticed you wrote in [SPECIFY WRITE-IN] in line f. Can you tell me more about how you use this criterion?

• I noticed you made a comment in the comment box below question 8 [SUMMARIZE COMMENT]. Can you tell me more about that?

Conclusion

Thank you for participating in this discussion today. I have just a few additional questions before we finish.

26. We plan to administer this survey to a larger group of institutions later this year. What, if anything, do you think should be changed about the survey prior to the full study?

27. In general, do you think this survey is something institutions like yours would be interested and willing to participate in?

Probe as needed:

• Is there anything we could do to make institutions like yours more willing to participate in the survey?

28. Is there anything about your participation in the survey we didn’t discuss that you would like to add?

That concludes our questions. Thank you once again for participating.