ASMB SS #5 - Cognitive Testing Protocol

American Survey of Mortgage Borrowers
Cognitive Interview Protocol
Notes for the Interviewer
•
•
•
•

Interviews will be conducted via Zoom.gov. We will not be using webcams.
Record the Zoom session. Ensure respondent names or identifying information are not
captured on the recording.
Respondents will be sent a packet of materials (see content list below) via FedEx with
the request not to open the envelopes included inside.
Monitor the chat for questions from observers.

Packet Contents
•
•

Consent Form
Envelope #1 contains the cover letter and full questionnaire.
Do not have the respondent complete the full questionnaire. They will complete the shortened
testing version of the questionnaire (in the form of a fillable PDF) on Zoom.

•

Envelope #2 contains the shortened testing version of the questionnaire.
Only have the respondent open Envelope #2 in case technical difficulties prevent use of the
fillable PDF in Zoom.

Introduction
Hello, my name is ______________ and I work for Westat, a survey research company in
Rockville, Maryland. Thank you for taking the time to participate in this research study.
Westat is working with the Federal Housing Finance Agency and Consumer Financial Protection
Bureau to develop questions for a national survey of mortgage borrowers. Before surveys are
conducted, it’s important to try out questions with the help of people such as yourself.
It is important that the questions make sense, are easy to answer, and that everyone
understands the questions the same way. If you agree to take part in this study, we will ask you
to complete a questionnaire. After you finish, I will ask you some questions about the answers
you gave, including what you think the questions were asking or how you came up with your
answers. There are no right or wrong answers. Our purpose is not to compile information about
you. Instead, your interview along with those of others will show us how to improve these
questions.
We need the materials we sent you for our interview. Do you have the FedEx package we sent?





IF NO AND THE MATERIALS ARE CLOSE BY, ASK RESPONDENT TO RETRIEVE THEM.
IF MATERIALS AREN’T CLOSE BY, PROCEED WITH INTERVIEW USING ELECTRONIC
VERSIONS OF MATERIALS (CONSENT, LETTER, FULL QUESTIONNAIRE).
MODIFY INSTRUCTIONS TO RESPONDENT AS NEEDED.

1

Informed Consent
Before we get started, there are a few things I should mention. This is a research project, and
your participation is voluntary. If you prefer not to answer any questions just say so and we’ll go
on to the next one. It’s also okay if you change your mind after starting and would rather not
participate.
All your answers, everything you say, will be kept confidential. We will not use your name or
other identifying information in any reports, though we may include quotes you provide in our
reports. The interview will take about an hour and you will receive $100. We will also need to
record interview session on Zoom.gov. This helps me, so I can listen to what you are saying and
won’t have to take a lot of detailed notes while you are talking. It will also help when we write up
a summary of this interview. Zoom.gov will also record what happens on the computer screen
that we are sharing. We are not using webcams, so it will just record you clicking on answers to
questions. Only project staff will have access to the recordings and other project materials.
These materials will be destroyed once we have completed the project.
Finally, some of the researchers developing the questions are observing our interview today, to
learn if there are things that might need to be changed to improve the survey questionnaire.
There’s a sheet of paper in the FedEx envelope that has “Consent to Answer Questions to be
used in a Survey of Mortgage Borrowers.” It contains all of the things I just told you about your
rights in this interview. Please read it over and let me know when you finish. If you have any
questions, please ask.
ANSWER ANY QUESTIONS FROM THE RESPONDENT.
AFTER RESPONDENT FINISHES,

Do you agree to participate in this interview and to have the

interview recorded?
IF RESPONDENT ANSWERS NO, TERMINATE INTERVIEW.

The date and time is _______. Now that we are recording, let me
ask again, do you agree to participate in this interview? Is it okay with you if we record this
interview?

START ZOOM RECORDING.

2

Reactions to Survey Packet
Research Questions


Is there anything confusing or unclear in the cover letter and survey cover?



Do respondents understand the purpose of the survey?



Do respondents have any concerns about completing the survey?

Before we get to the survey itself, I’d like you to open Envelope #1. It contains a letter and
questionnaire. For the actual survey, this is what would be mailed to people. Unlike you, they
would not know ahead of time that they were getting it.
In addition to the letter and questionnaire, the envelope would contain a $10 bill as a token of
appreciation.
If you received this in the mail at home, what would you do?
IF NEEDED:

read it, toss it, respond to the survey right away, set it aside to respond to later, etc.

What reactions, if any, do you have to what’s inside the envelope?

Cover Letter
Okay, now I’d like you to read the cover letter. Let me know when you’re finished.
What do you think of the letter? What, if anything, grabs your attention?
What, if anything, is confusing or unclear?
If you received this in the mail, would you choose to complete the paper survey or to
complete it online? Why?

Survey Cover
Now I’d like you take a few moments to read the text on the cover of the stapled survey. Just
the first page—no need to flip it open. Let me know when you’re finished.
What do you think of the cover?
What, if anything, is confusing or unclear?

3

Think-Aloud Completion of Questionnaire
Your package contains the full questionnaire that we plan to mail to people in the sample for the
main study. Today, however, we’d like you to complete a shorter version that contains only
the questions we’d like to learn more about during this interview.
Now, I’d like you to complete the shorter version of the questionnaire on the computer. It’s here
on the screen, and I’m going to give you control so that you can answer the questions. You can
just move the mouse and click on the answer you want. If you decide to change your answer,
you can do that as well. You can scroll up and down to move around the survey.
You may notice that the questions are only on the left side of the page. That’s just to make it
easier to scroll through the survey during our interview today. When we mail the actual survey to
people, it will have two columns of questions side-by-side.
As I mentioned before, we’re interested in what you’re thinking while you’re filling out the survey.
So, I’m going to ask you to think out loud. It would be helpful if you could tell me about any
reactions—good or bad—that you have to the survey and the questions in it. You don’t have to
read the questions themselves out loud—just tell me what you’re thinking as you go through the
survey. I may sometimes interrupt you with a question.
WHILE THE RESPONDENT COMPLETES THE QUESTIONNAIRE, MAKE NOTES ON:
•

Any comments spoken aloud;

•

Any difficulty following skip items;

•

Any hesitation answering;

•

Any questions left blank, and

•

When respondent changes an answer.

IF RESPONDENT ASKS A QUESTION:

For now, please answer it the way you would if I wasn’t here

listening. I’m making a note of your question, and after you finish the survey, we’ll talk about it.

4

General Retrospective Probing
Thank you for completing the survey. Now I’m going to ask you some questions.

Overall Impressions
Research Questions
 What items are difficult for respondents for respondents to understand? What makes them difficult?
Signs of difficulty include: re-reading the question stem, voicing that a question is difficult to answer
 What items are difficult for respondents to answer? What makes them difficult?
Signs of difficulty include: taking a long time to answer, difficulty mapping answer to response options

First, just tell me your overall impressions of the survey.
IF NEEDED, PROBE ON SURVEY ORGANIZATION, INSTRUCTIONS.

Items and Skip Instructions
Was there anything in the survey that was unclear or confusing? What was unclear?
Were there any questions that seemed repetitive to you? Which ones?
How easy or difficult was it to follow the skip instructions?
IF DIFFICULT,

What was difficult and why?

Targeted Retrospective Probing
INTERVIEWER NOTE: [BRACKETED NUMBERS IN PURPLE] ARE ORIGINAL ITEM NUMBERS;
ONLY INCLUDED TO FACILITATE LATER ANALYSIS.

Now I’d like to talk about some of the questions in the survey. We may go out of order, but
let’s start back at the beginning.
TAKE CONTROL BACK ON ZOOM AND SCROLL THROUGH QUESTIONNAIRE AS PROBING.
FIRST, ASK SCRIPTED PROBES, WORKING THROUGH THE ENTIRE QUESTIONNAIRE.
(IF THE ITEMS WITH SCRIPTED PROBES ALSO HAD ISSUES THAT AROSE DURING THE
THINKALOUD, YOU MAY CHOOSE TO GO AHEAD AND PROBE ON THOSE. USE YOUR
BEST JUDGMENT IN TERMS OF TIME MANAGEMENT.)
NEXT, ASK ABOUT ANY OUTSTANDING ISSUES FROM YOUR OBSERVATIONS AND THE
RESPONDENT’S THINK-ALOUD COMMENTS.
.

5

Research Questions

Questionnaire with Embedded Probes

Questions 1 and 2 are new. Explore any issues
that emerge.
Question 3 has modified Q text and items. Probe for
issues with the Q or items.
Explore understanding of reference periods in Q1,
Q2, and Q3

INTRO PARAGRAPH: What comes to mind when
you read this paragraph? Is there any part of
this text that’s unclear to you?
Question 1 [Q1]
What does the phrase “Since the start of the
pandemic?” mean to you?
How easy or difficult was it to answer Q1?
What made it easy or difficult?
IF DIFFICULT: Which part of the question
was difficult?
IF R DID NOT ANSWER ALL 3 OF THE ITEMS:
I noticed you didn’t answer [ITEM] – can you tell
me why you skipped that one?

Question 2 [Q2]
What about Q2 – how easy or difficult was it to
answer that question?
What made it easy or difficult?
IF DIFFICULT: Which part of the question
was difficult?
IF R DID NOT ANSWER ALL 6 OF THE ITEMS:
I noticed you didn’t answer [ITEM] – can you tell
me why you skipped that one?

What about the phrase “During the pandemic?”
Can you tell me what that means to you?
Is there something else that affected your
household financially during the pandemic that
isn’t listed here?
Question 3 [Q3]
IF R DID NOT ANSWER ALL 6 OF THE ITEMS:
I noticed you didn’t answer [ITEM] – can you tell
me why you skipped that one?

Is there anything else that was necessary for
you to do during the pandemic that isn’t
covered here?

6

Research Questions
Q4 has modified Q text, reference period (explored
at Q2) and set of items. Probe for issues with the
Q/items or with choosing a response.
Q5 has modified reference period (explore).
Q6/Q7 have modified reference period (explored at
Q1). Revisit as needed.

Question 4 [Q4]
IF R DID NOT ANSWER ALL 8 OF THE ITEMS:
I noticed you didn’t answer [ITEM] – can you tell
me why you skipped that one?

Is there something that you did to address your
financial situation that isn’t listed here?
EXPLORE ANY REACTION R MAY HAVE TO Q’S 1-4 NOT
BEING ABOUT THEIR MORTGAGE EVEN THOUGH INTRO
SAID SURVEY IS ABOUT MORTGAGES?

Question 5 [Q5]
IF R ANSWERED “NO” TO Q5:
I noticed you answered “No” for question 5.
Can you tell me why you chose that response?

What does the phrase “At the start of the
pandemic?” mean to you?

7

Research Questions
Q8 has a modified reference period (probed at
Q2).Other issues to probe noted below.
Q10 has modified response categories. Probe for
issues with choosing a response.
Q11 has revised Q text and response categories.
Probe for issues with the Q or choosing a response.
Q12 is new. Explore any issues that emerge.

Question 8 [Q18]
Q8: Tell me about your answer to this question.
IF NO: Do you have an agreement with your
lender about missing payments or paying a lower
amount?
PROBE AS NEEDED TO DETERMINE REASON
FOR CHOICE OF “NO.”
Were you familiar with the term “forbearance” before
this study?

Question 10 [Q20]
Did you have any difficulty answering Q10?
IF DIFFICULT: What was difficult about making
that choice?

Question 11 [Q21]
Did you have any difficulty answering Q11?
IF DIFFICULT: What was difficult about making
that choice?
IF ANSWERED UNSURE/DON’T KNOW: Tell me more
about how you chose this answer.
IF R INDICATES THEY KIND OF KNEW BUT
WERE UNSURE: If the last option only said
“Don’t know,” would your answer have changed?
How would it have changed and why?

Question 12 [Q22]
How easy or difficult was it to answer Q12?
What made it easy or difficult?
IF DIFFICULT: Which part of the question was
difficult?
IF R DID NOT ANSWER ALL 4 ITEMS:
I noticed you didn’t answer [ITEM] – can you tell
me why you skipped that one?
IF OTHER (SPECIFY):
Did you consider choosing any of the other
answers listed?
IF YES, Why did you decide on “Other”?

8

Research Questions
Questions 13 and 14 are new. Explore any issues
that emerge.
Q15 has revised Q text and items. Probe for issues
with the Q or items.

Question 13 [Q23]
How easy or difficult was it to answer Q13?
What made it easy or difficult?
IF DIFFICULT: Which part of the question
was difficult?
IF R DID NOT ANSWER ALL 7 OF THE ITEMS:

I noticed you didn’t answer [ITEM] – can you
tell me why you skipped that one?

Question 14 [Q24]
IF R PROVIDED TEXT RESPONSE: Tell me about
your response to Q14.

Question 15 [Q25]
IF R DID NOT ANSWER ALL 11 OF THE ITEMS:

I noticed you didn’t answer [ITEM] – can you
tell me why you skipped that one?
Is there a reason that you did not get a
forbearance that isn’t listed here?

9

Research Questions
Q16 and Q17 are the same as in 2020.
Q18 and Q19 are new questions. Probe for any
issues.
Q20 has revised Q text. Probe for issues. Does the
R understand we’re asking about what happened
both during the pandemic and now?

Question 16 [Q26]
Q16 IF OTHER, Did you consider choosing any of
the other answers listed?
IF YES, Why

did you decide on “Other”?

Question 18 [Q44]
How easy or difficult was it to tell which
question you should answer next?
IF DIFFICULT: What

made it difficult?

Did you have any problems figuring out what
questions to skip and what questions to
answer over the next few pages?
DID R TRY TO CHOOSE MORE THAN ONE RESPONSE?
HOW DOES THAT AFFECT CHOOSING THE NEXT Q TO
ANSWER?
Question 19 [Q45]
What about question 19 – how easy or difficult
was it to answer that question?
What made it easy or difficult?
IF DIFFICULT: Which part of the question
was difficult?
Question 20 [Q46]
When did (or will) you obtain the new loan?
IF PAST/DURING PANDEMIC: Was it clear to
you that this question is also asking about
current refinancing plans?
IF PRESENT: Was it clear to you that this
question is also asking loans modified
during the pandemic?
IF R DID NOT ANSWER ALL 4 OF THE ITEMS:

I noticed you didn’t answer [ITEM] – can you
tell me why you skipped that one?

10

Research Questions
Q21 is the same as in 2020.
Q22 and Q24 have revised Q text. Probe for issues
with these Qs.
Q23 has revised Q text and response categories.
Probe for issues with the Q or choosing a response.
Q25 is new. Explore any issues that emerge.
Q26 has revised items. Probe for issues.

Question 21 [Q47]
IF R DID NOT ANSWER ALL 8 OF THE ITEMS:

I noticed you didn’t answer [ITEM] – can you
tell me why you skipped that one?

Question 23 [Q49]
Is it clear to you what we mean by “the
property” in this question? What does that
mean to you?

Question 24 [Q50]
IF R DID NOT ANSWER ALL 4 OF THE ITEMS:

I noticed you didn’t answer [ITEM] – can you
tell me why you skipped that one?
Is there a reason for not selling the property that
isn’t listed here?

Question 25 [Q51]
Tell me about answering question 25 – how easy
or difficult this question to answer?
What made it easy or difficult?
IF DIFFICULT: Which part of the question
was difficult?
Question 26 [Q52]
How easy or difficult was it to answer Q26?
What made it easy or difficult?
IF DIFFICULT: Which part of the question
was difficult?
IF R DID NOT ANSWER ALL 5 OF THE ITEMS:

I noticed you didn’t answer [ITEM] – can you
tell me why you skipped that one?

11

Research Questions
Q27 has revised response categories. Probe for
issues with choosing a response.
Q28, Q30 and Q31 are the same as in 2020.
Q29 has revised Q text and items. Probe for issues
with the Q or the items.

Question 27 [Q53]
Did you have any difficulty answering Q27?
IF DIFFICULT: What was difficult about
making that choice?
Q27 IF OTHER: Did you consider choosing any of
the other answers listed?
IF YES, Why

did you decide on “Other”?

Question 29 [Q56]
IF R DID NOT ANSWER ALL 8 OF THE ITEMS:

I noticed you didn’t answer [ITEM] – can you
tell me why you skipped that one?
Is there a reason for not having the property
that isn’t listed here?

12

Research Questions
Q32, Q33 and Q34 are the same as in 2020.
Q35 has revised Q text. Probe for issues with the Q.

Question 32 [Q59]
IF R DID NOT ANSWER ALL 7 OF THE ITEMS:

I noticed you didn’t answer [ITEM] – can you
tell me why you skipped that one?
Q32 IF OTHER (SPECIFY): Tell me about your
response of (OTHER TEXT) here.
Q32 IF “Nothing, will not buy again”: Tell me about your
choice of this response.

Question 35 [Q67]
How easy or difficult was it for you to answer
Question 35?
IF DIFFICULT: What

made it difficult

IF R DOES NOT SELECT AN ANSWER FOR “YOU” AND (IF
APPLICABLE) FOR “SPOUSE/PARTNER,” PROBE FOR
WHAT THAT PERSON’S WORK STATUS IS.

13

Research Questions
Q36 reference period changed to “currently.” Explore
for issues with providing response.
Q37 and Q38 have modified reference periods
(previously explored) and revised sets of items.
Probe for issues with the items.

Question 36 [Q68]
How easy or difficult was it for you to answer
Question 36?
IF DIFFICULT: What

made it difficult?

IF SELECTED “SELF-EMPLOYED/OTHER”

Tell me about your answer to this one.
IF NEEDED: Were

you answering “Selfemployed” or “Other”?
IF NEEDED: Was

there anything missing from
this list of forms of payment?
Question 37 [Q69]
How about question 37 – how easy or difficult
was it to answer this one?
IF DIFFICULT: What

made it difficult? Which
parts of the question were hard to answer?
Question 38 [Q73]
IF R DID NOT ANSWER ALL 8 OF THE ITEMS:

I noticed you didn’t answer [ITEM] – can you
tell me why you skipped that one?
PROBE ACROSS Q1/Q2 AND Q37/Q38:
We’ve asked you questions today about things that
may have happened since the start of the pandemic
to impact your household finances and your
mortgage situation.
Is there something significant that happened to
you during the pandemic that you haven’t had a
chance to tell me about yet?
AS NEEDED:

SCROLL BACK TO Q1, Q2, Q3, Q4 TO ALLOW QUICK
REVIEW OF THE QUESTIONS

THEN SCROLL BACK HERE TO Q37/Q38.

PROBE ON THINGS MENTIONED:

When did that happen?
Can you tell me more about how that affected
you? How did that affect household finances?

14

Research Questions
Q39 reference period changed from prior calendar
year to current. Instructions in parentheses are new.
Q40 question text is new. Explore any difficulty
answering, and R understanding of reference period.
Q41 has revised reference period. Probe for issues.
Q42 is the same as in 2020.

Question 39 [Q76]
How easy or difficult was it for you to answer
Question 39?
IF DIFFICULT: What

made it difficult?
Did you find the text in the parentheses helpful
or not helpful? Why?
Question 40 [Q77]
How easy or difficult was it for you to answer
Question 40?
IF DIFFICULT: What

made it difficult?

What does the term “pre-pandemic” mean to
you? When was that?
Question 41 [Q78]
How about the phrase “next year” – what does
that mean to you?
What timeframe does that represent to you?

Question 42 [Q84]
IF R DID NOT ANSWER ALL 4 OF THE ITEMS:
I noticed you didn’t answer [ITEM] – can you
tell me why you skipped that one?

CHECK EMAIL FOR ANY QUESTIONS FROM
THE OBSERVERS.

15

.

Respondent Story
Now, just tell me in your own words about the mortgage you were thinking about when you
answered the survey questions.
And tell me about the difficulties you were having with making mortgage payments, what
caused the difficulties, what kinds of things you did or people you talked to about those
difficulties, and what you did or are doing to help. I’d like to hear your story.
MAKE SURE TO CAPTURE AS MUCH DETAIL AS POSSIBLE ABOUT:


The mortgage R was thinking about when answering the survey questions.



Difficulties R had.



Cause of difficulties.



Action taken.



Help sought/solution to difficulties.



Mortgage still held.

(IF NEEDED) Probe to determine the following:


Did respondent answer about a mortgage held 18 months to 2 years ago? If not, why?



If R has/had more than one mortgage, how they decided which one to answer for, and
whether they answered for the same mortgage throughout the survey.



Exactly what the difficulty was (missed payment, late payment, stopped payment,
something else).



What caused the difficulty?



What action was taken to try and mitigate the difficulties?



How helpful or unhelpful was the loan servicer?



Was respondent talking about the most recent mortgage difficulty?

16

Closing
DOUBLE-CHECK EMAIL FOR ANY REMAINING QUESTIONS FROM THE OBSERVERS.

That is all the questions I have for you. Is there anything we haven't discussed that you would
like to mention?
DISCUSS ANY RESPONDENT COMMENTS.

VENDOR should be sending your $100 to you soon. Please reach out to them with any
questions. If you have other questions, our contact information is provided on that very first form
we looked at.
Please throw away everything else we sent you. IF ENVELOPE #2 NOT NEEDED, Envelope #2 was
something we sent you just in case we had technical difficulties. You can throw it away too.
STOP ZOOM RECORDING.

17