5. Cognitive Testing Protocol dated April 18, 2024

Survey of Mortgage Borrowers
Cognitive Interview Protocol
Notes for the Interviewer
•
•
•
•

Interviews will be conducted via Zoom.gov. We will not be using webcams.
Record the Zoom session. Ensure respondent names or identifying information are not
captured on the recording.
Respondents will be sent a packet of materials (see content list below) via FedEx with
the request not to open the envelopes included inside.
Monitor the chat for questions from observers.

Packet Contents
•
•
•

Consent Form
Envelope #1 contains the cover letter and full questionnaire.

Do not have the respondent complete the full questionnaire. They will complete the shortened
testing version of the questionnaire (in the form of a fillable PDF) on Zoom.

Envelope #2 contains the shortened testing version of the questionnaire.

Only have the respondent open Envelope #2 in case technical difficulties prevent use of the
fillable PDF in Zoom.

Introduction (5 minutes)
Hello, my name is ______________ and I work for Westat, a survey research company in
Rockville, Maryland. Thank you for taking the time to participate in this research study.
Westat is working with the Federal Housing Finance Agency and Consumer Financial Protection
Bureau to develop questions for a national survey of mortgage borrowers. Before surveys are
conducted, it’s important to try out questions with the help of people such as yourself.
It is important that the questions make sense, are easy to answer, and that everyone
understands the questions the same way. If you agree to take part in this study, we will ask you
to complete a questionnaire. While you complete the interview, and after you finish, I will ask
you some questions about the answers you gave, including what you think the questions were
asking or how you came up with your answers. There are no right or wrong answers. Our
purpose is not to compile information about you. Instead, your interview along with those of
others will show us how to improve these questions.
We need the materials we sent you for our interview. Do you have the FedEx package we sent?
•

•
•

IF NO AND THE MATERIALS ARE CLOSE BY, ASK RESPONDENT TO RETRIEVE THEM.
IF MATERIALS AREN’T CLOSE BY, PROCEED WITH INTERVIEW USING ELECTRONIC
VERSIONS OF MATERIALS (CONSENT, LETTER, FULL QUESTIONNAIRE).
MODIFY INSTRUCTIONS TO RESPONDENT AS NEEDED.

1

Informed Consent
Before we get started, there are a few things I should mention. This is a research project, and
your participation is voluntary. If you prefer not to answer any questions just say so and we’ll go
on to the next one. It’s also okay if you change your mind after starting and would rather not
participate.
All your answers, everything you say, will be kept confidential. We will not use your name or
other identifying information in any reports, though we may include quotes you provide in our
reports. The interview will take about an hour and you will receive $100. We will also need to
record the interview session on Zoom.gov. This helps me, so I can listen to what you are saying
and won’t have to take a lot of detailed notes while you are talking. It will also help when we
write up a summary of this interview. Zoom.gov will also record what happens on the computer
screen that we are sharing. We are not using webcams, so it will just record you clicking on
answers to questions. Only project staff will have access to the recordings and other project
materials. These materials will be destroyed once we have completed the project.
Finally, some of the researchers developing the questions are observing our interview today, to
learn if there are things that might need to be changed to improve the survey questionnaire.
There’s a sheet of paper in the FedEx envelope that has “Consent to Answer Questions to be
used in a Survey of Mortgage Borrowers.” It contains all of the things I just told you about your
rights in this interview. Please read it over and let me know when you finish. If you have any
questions, please ask.
ANSWER ANY QUESTIONS FROM THE RESPONDENT.
AFTER RESPONDENT FINISHES,

Do you agree to participate in this interview and to have the

interview recorded?
IF RESPONDENT ANSWERS NO, TERMINATE INTERVIEW.

The date and time is _______. Now that we are recording, let me
ask again, do you agree to participate in this interview? Is it okay with you if we record this
interview?

START ZOOM RECORDING.

2

Reactions to Survey Packet (5 minutes)
Research Questions


Would the respondent think the survey packet was junk mail, or would they open it and
read the letter?



If the respondent received this survey in the mail at home, would they be likely to
respond, or to throw it out?

Before we get to the survey itself, I’d like you to open Envelope #1. It contains a letter and
questionnaire. For the actual survey, this is what would be mailed to people. Unlike you, they
would not know ahead of time that they were getting it.
In addition to the letter and questionnaire, the envelope would contain a $10 bill as a token of
appreciation.
If you received this package in the mail at home, what would you do?
IF NEEDED:

read it, toss it, respond to the survey right away, set it aside to respond to later, etc.

What reactions, if any, do you have to what’s inside the envelope?

Cover Letter
Okay, now I’d like you to read the cover letter. Let me know when you’re finished.
What do you think of the letter? What, if anything, grabs your attention?

Survey Cover
Now I’d like you take a few moments to read the text on the cover of the stapled survey. Just
the first page—no need to flip it open. Let me know when you’re finished.
What do you think of the cover?
What, if anything, is confusing or unclear?

3

Concurrent Think-Aloud Completion of Questionnaire (30 minutes)
Your package contains the full questionnaire that we plan to mail to people in the sample for the
main study. Today, however, we’d like you to complete a shorter version that contains only
the questions we’d like to learn more about during this interview.
Now, I’d like you to complete the shorter version of the questionnaire on the computer. It’s here
on the screen, and I’m going to give you control so that you can answer the questions. You can
just move the mouse and click on the answer you want. If you decide to change your answer,
you can do that as well. You can scroll up and down to move around the survey.
You may notice that the questions are only on the left side of the page. That’s just to make it
easier to scroll through the survey during our interview today. When we mail the actual survey to
people, it will have two columns of questions side-by-side.
As I mentioned before, we’re interested in what you’re thinking while you’re filling out the survey.
So, I’m going to ask you to think out loud. It would be helpful if you could tell me about any
reactions—good or bad—that you have to the survey and the questions in it. You don’t have to
read the questions themselves out loud—just tell me what you’re thinking as you go through the
survey. I may sometimes interrupt you with a question.
WHILE THE RESPONDENT COMPLETES THE QUESTIONNAIRE, MAKE NOTES ON:
•

Any comments spoken aloud;

•

Any difficulty following skip items;

•

Any hesitation answering;

•

Any questions left blank, and

•

When respondent changes an answer.

IF RESPONDENT ASKS A QUESTION:

For now, please answer it the way you would if I wasn’t here

listening. I’m making a note of your question, and after you finish the survey, we’ll talk about it.

4

Questionnaire with Embedded Probes

Research Questions


Q1 [Q1]: We want to understand how they
applied the instructions under Q1.



Q6 [Q8]: Probe for whether Q6 was
confusing or difficult to answer, and for what
mortgage rates R thinks lenders are
currently offering.

Question 2 [Q2]
[NOTE WHETHER THEY READ THE INSTRUCTIONS
BELOW Q1.
AFTER THEY ANSWER Q2, PROBE:]

Do you have multiple mortgaged properties, or
just one?
[IF MORE THAN ONE MORTGAGED PROPERTY]

Please tell me about the mortgaged
properties you currently have.
When you answered question 2, which
property were you thinking about?

Question 6 [Q8]
[AFTER THEY ANSWER Q6, PROBE:]

Did you find this question to be confusing or
hard to answer?
[PROBE IF NOT ALREADY MENTIONED BY
RESPONDENT:]

What rates do you think mortgage lenders are
offering these days?
5

Research Questions


Q11 [Q19]: Probe for clarity.

Question 11 [Q19]
[AFTER THEY ANSWER, PROBE:]

In question 11, what does the word
“considered” mean to you?
[IF THEY ANSWERED YES TO Q11]

I see you answered “yes” to this question –
what were some of your considerations when
thinking about selling your property?

6

Research Questions


Q14 [Q22]: Probe for clarity.

Question 14 [Q22]
[AFTER THEY ANSWER Q14, PROBE:]

What does the phrase “decreased significantly”
mean to you?
Were there any parts of this question that were
confusing or hard to answer?

7

Research Questions


Q17 [Q25] and Q18 [Q26]: Probe for whether
the two year time horizon was easy or
difficult to remember.
How well do they remember what their
payment was two years ago?
If difficult, how well do they remember
what their payment was one year ago?

Question 17 [Q25]
How easy or difficult was it for you to answer
question 17 thinking about the past two years?
How well do you remember what your mortgage
payment was two years ago?

[IF NOT WELL]
What about your mortgage payment one year
ago – how well do you remember what that was?

Question 18 [Q26]
How easy or difficult was it for you to answer
question 18?
[IF DIFFICULT]

What made it difficult?

8

Research Questions


Q24 [Q32] and Q25 [Q33]: For respondents
who check one of the boxes in Q24 [Q32],
after they answer Q33 probe on:
(1) Whether it is the same or different
company from their homeowners
insurance, and
(2) Whether it is a separate payment from
what they reported for homeowners
insurance.

Question 24 [Q32] and Question 25 [Q33]
PROBE AFTER ANSWER Q25:
Is this separate policy provided by the same
company that provides your homeowners
insurance?

[IF YES/SAME COMPANY]
What about the payments for this separate
insurance policy – do you make those separately
from the payments for homeowners insurance
reported at Q21?

9

10

Research Questions


Q32 [Q40] – Probe for whether the two year time
horizon was easy or difficult to remember, and, if
difficult, how well do they remember what these
payments were one year ago?
How did they interpret ‘increased a lot’ and
‘increased a little’?



Q34 [Q42] – If they pause or seem unsure how to
answer, probe whether they are familiar with the term
escrow account?

Question 32 [Q40]
How easy or difficult was it for you to answer
question 32 thinking about the past two years?
How well do you remember what these payments
were two years ago? Are some of these easier to
remember than others?

[IF NOT WELL]

What about these payments one year ago – how
well do you remember what those were?
What do “increased a lot” and “increased a little”
mean to you – how big of a change would you
consider to be a lot or a little?

Question 34 [Q42]
IF R HESITATES, SEEMS UNSURE HOW TO
ANSWER, OR ANSWERS DON’T KINOW:

What does the term “escrow account” mean
to you?

11

Research Questions


Q36 [Q44] – If they check yes to the item about
force-placed insurance, probe to describe what
happened. Why did the servicer purchase insurance
for them?



Q37 [Q45] – If they pause or seem unsure how to
answer, probe for whether they have borrowed
against their equity in some other way beyond the
answer options provided. If it seems like they are
confused by the question, probe for whether they find
the term ‘borrowed against your equity’ to be clear or
confusing.

Question 36 [Q44]
IF R ANSWERS “YES” TO “SERVICER PURCHASED
INSURANCE FOR ME (FORCE-PLACED
INSURANCE)”:

Why did your servicer purchase force-placed
insurance for you? What happened?

Question 37 [Q45]
IF R HESITATES OR SEEMS UNSURE HOW TO
ANSWER:

Have you borrowed against your equity in
some other way beyond the answer options
provided?
IF THEY CONFUSED BY THE QUESTION:

What about the term “borrowed against your
equity” – do you find that to be clear or
confusing?

12

13

Research Questions


Q44 [Q54] and Q45 [Q55] – After they answer, probe
on what kinds of repairs they included and did not
include. Did they interpret the question to include or
exclude minor repairs or regular maintenance
issues?

Question 45 [Q54] and Question 46 [Q55]
AFTER R ANSWERS BOTH QUESTIONS, OR AFTER
Q45 IF R ANSWERS “NO” - PROBE:

For question 45, what kind of repairs were you
thinking about?
Did you include or not include minor repairs and
regular maintenance issues when answering?
FOR THOSE WHO ANSWER Q46:

Please tell me about the repairs you included in
your answer to question 46.

14

Research Questions


This section – Q48 [Q57) through Q58 [Q67]
If Q51 [Q60] is answered ‘own with a mortgage’,
probe on whether they answered the earlier
questions in the survey about this property or a
different one.
How did they choose which property they answered
the earlier sections of the survey about, and is that
the same property they are thinking about here?
(The skip logic in the survey and instructions in Q1
mean no one should check ‘own with a mortgage’
here.)
After completing the section, probe whether they own
any other properties not mentioned. How many?
Have any of them previously been their primary
residence?

Question 51 [Q60]
IF R ANSWERS “Own with a mortgage”:
Did you answer the earlier questions in the
survey about this same property, or a
different one?
How did you choose which property to
answer about at the start of the survey?

15

Research Questions


See previous page.

After Q58 [Q67]
Do you own any other properties that you
haven’t mentioned yet?
IF YES:
How many do you own?
IF 1:
Has this property previously been used as
your primary residence? When was that?
IF 2 OR MORE:
Have any of these properties previously
been used as your primary residence? When
was that?

16

Research Questions


After they answer Q62 [Q75] (or if they pause or
seem confused about Q60 [Q73] or Q61 [Q74]), ask
them to describe what they think the question is
asking and whether it was confusing or difficult to
answer.

During Q61 [Q73] and Q62 [Q74]
IF R PAUSES OR SEEMS CONFUSED BY THE
QUESTION:
Please tell me what you think this question
is asking, in your own words?
Is it confusing or difficult to answer?

After Q63 [Q75]
Please tell me what you think this question
is asking, in your own words?
Is it confusing or difficult to answer?

17

CHECK EMAIL FOR ANY QUESTIONS FROM
THE OBSERVERS.

18

.

Overall Impressions of Survey (5 minutes)
We’re interested in your general thoughts about the survey you just completed. Remember that this was a
shorter version used for testing – the full version has twice as many questions.

•
•
•

If you received this in the mail, would you consider responding to the survey?
What, if anything, influenced whether you would be likely to respond?
Were there questions that were hard for you to answer? [SKIM THROUGH QUEX FROM START TO
FINISH TO ADDRESS THESE AS NEEDED.]

Respondent Story (5 minutes)
Before we end the interview, I’d like you to tell me in your own words about the mortgage/property you
were thinking about when you answered the survey questions.
Please tell me about:

•
•
•

How the mortgage costs for your property have changed in the past two years,
Changes in the cost of insurance and fees for this property over the past two years, and
How mortgage interest rates have affected your thinking and decision-making about your property.
[IF MORE THAN ONE PROPERTY/MORTGAGE, PROBE:]
• How did you decide which mortgage/property to answer for?
• Did you answer about the same mortgage throughout the survey?

[MAKE SURE TO CAPTURE AS MUCH DETAIL AS POSSIBLE. PROBE WHEN R FINISHES TO ASK IF
ANYTHING ELSE R WANTS TO TELL US ABOUT THEIR MORTGAGE STORY.]

Closing (2 to 5 minutes)
DOUBLE-CHECK EMAIL FOR ANY REMAINING QUESTIONS FROM THE OBSERVERS.

Those are all of the questions I have for you today. Is there anything we haven't discussed that you would
like to mention? [DISCUSS ANY RESPONDENT COMMENTS.]
The organization that recruited you will send your $100 to you soon. Please reach out to them with any
questions. If you have other questions, our contact information is on the first form we looked at today.
Please throw away everything else we sent you. IF ENVELOPE #2 NOT NEEDED, Envelope #2 was something
we sent you just in case we had technical difficulties. You can throw it away too.
STOP ZOOM RECORDING.

19