Unmoderated Usability Testing of the Mortgage Closing Product

1

Task 6: Unmoderated Usability Testing of the Mortgage Closing Product
Purpose:

1. Validate UX Test findings
2. Assess effects of ePlatform on consumer confidence and knowledge.

Method:

300 participants will be recruited via opt-in panel to complete an unmoderated usability test. Respondents will have
closed a mortgage or refinanced within the previous six months or will have plans to purchase a home within the
next six months. They will interact with one of four websites and be asked about their knowledge and attitudes
regarding the mortgage closing process.
A. Generic website group (n=75): Will interact with a site unrelated to mortgages. Will allow us to tease out the
Internet interaction that may impact responses on post-questions.
B. “Ask” group (n=75): Will be instructed to explore the mortgage closing process. Will allow us to assess the
effects of the platform versus other information on the Internet.
C. Platform/no task group (n=75): Will be shown the e-closing platform but will not be forced to interact with it
– this group will be asked to freely “explore” the portal.
D. Platform/task group (n=75): Will interact with the e-closing platform and will complete the exact tasks that
participants in the UX study completed. This group will:
a. Be “forced to interact” with the platform and information provided to consumers – proxy for
knowledge checking in the real world, where a lender may ask consumers to check/confirm specific
aspects of their documents.
b. Validate UX findings

Overview (details follow):
1. Generic
scenario about
what participants
will see

2. Tasks/
Explore

Group A: Generic
Website
Imagine that you are
interested in
attending concerts
this summer, and
your friend has
recommended this
site to you
(ticketmaster.com).
Explore concerts
that are taking place
this summer.
Free exploration

Group B: Ask Site;
Mortgage Tasks
Imagine that you will
be closing on a
mortgage in three
days, and you have
some questions. Use
this site (askcfpb) to
explore the mortgage
closing process.

Knowledge is
available on the
Internet; what
consumers would do
without the improved
method of sharing
info

Group C: Platform; No
Tasks
Imagine that you will be
closing on a mortgage
in three days, and your
lender has sent you a
link to a portal that
contains your closing
documents. He has
asked you to review
your documents using
this portal. Please
review your documents.
Knowledge is available
to consumers; they are
not forced to interact

Group D: Platform;
Mortgage Tasks
Imagine that you will
be closing on a
mortgage in three
days, and your lender
has sent you a link to
a portal that contains
your closing
documents. He has
asked you to review
your documents using
this portal.
Knowledge is
available to
consumers; they are
“forced” to interact;
UX Validation

1010 N. Glebe Road, Suite 510 | Arlington, Virginia 22201 | www.forsmarshgroup.com

2
3. Consumer
Knowledge,
Empowerment,
Efficiency, and
ConsumerFriendliness
Questions

Baseline of
consumer
knowledge

Assess the effects of
forcing consumers to
interact with a
different site
regarding mortgages

Assess the effects of
merely providing
information to
consumers

Assess the effects of
forcing consumers to
interact with the
information

4. Final tasks:
Show mortgage
materials and ask
what respondents
would do next.

Control

Control

Test the effects of
having seen the
platform – does having
the information lead to
increased knowledge?

Test the effects of
having interacted with
the platform – does
this “forced
interaction” transfer
to a new task?

5. Debrief: Provide information about consumer rights at closing so respondents do not think they have rights that
they do not (because in Section 3, some incorrect rights will be planted).

Details:
Participants will be invited to participate, and once screened and accepted for participation, they will see
the following:
Thank you for participating in this study. You will see a website and will be asked to answer questions. This
entire session should last no more than 15 minutes.
Privacy Notice
The information you provide through your responses will assist the survey sponsor, the Consumer Financial
Protection Bureau (“CFPB”) develop services to better provide financial information to consumers.
The CFPB will not obtain or access any directly identifying information about study participants from the isay Panel. The agency will only obtain and access de-identified results and aggregated analyses of those
results. To review the Policy, please click here [hyperlink to the Paperwork Reduction Act below]
Your participation in this survey is also subject to the i-say privacy policy, available at http://isay.com/Privacy/PRIVACYPOLICY/tabid/167/Default.aspx
Paperwork Reduction Act [Text will be hyperlinked in the section above]
According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is
not required to respond to a collection of information unless it displays a valid OMB control number. The
OMB control number for this collection is 3170-0022. It expires on 12/31/2015. The time required to
complete this information collection is estimated to average approximately 20 minutes per response,
Responding to this collection of information is voluntary. Comments regarding this collection of information,
including the estimated response time, suggestions for improving the usefulness of the information, or
suggestions for reducing the burden to respond to this collection should be submitted to Bureau at the
Consumer Financial Protection Bureau (Attention: PRA Office), 1700 G Street NW, Washington, DC 20552,
or by email to PRA@cfpb.gov.

Participants will then be shown the scenario text above (Section 1), depending on the group they are in.
Section 2. Tasks: Group D Only
1010 N. Glebe Road, Suite 510 | Arlington, Virginia 22201 | www.forsmarshgroup.com

3
Imagine that you will be closing on a mortgage in three days, and your lender has sent you a link to a portal that
contains your closing documents. He has asked you to review your documents using this portal.
Your lender sent you private log-in information, and you have securely logged in. You are now on the main page of
the portal.
Task 1: Closing Documents
1a.
Using the site, find the purpose of the HUD-1 Settlement Statement. (click path, end page, time on task, and
accuracy/abandon data)
First Impressions
What are your first impressions of the homepage? (open-ended response)
Which documents appear to be most important? (multiple choice – list all documents that appeared on
page; if possible, display screen shot of the portal landing page)
1b.

Using the site, find the purpose of the Mortgage Note. When you find the purpose read it to yourself. (click
path, end page, time on task, and accuracy/abandon data)

1c.

Using the site, find the purpose of the Deed of Trust. When you find the purpose read it to yourself. (click
path, end page, time on task, and accuracy/abandon data)

Task 2: Comparing Information
2a.
You want to compare your HUD-1 Settlement Statement to your Good Faith Estimate (GFE). Pull the
documents up side-by-side. (click path, end page, time on task, and accuracy/abandon data)
2b.

Are there any discrepancies between these two documents? (yes, no, I don’t know) (click path, time on task,
and accuracy data)

2c.

[If 2b=yes] How did the costs on your HUD-1 Settlement Statement compare to your original Good Faith
Estimate (GFE)? (multiple choice)
Costs were higher on the HUD-1
Costs were lower on the HUD-1
There was no change
I don't know

•
•
•
•
2d.

[If 2b=yes] Using this site, find out if the discrepancies between these two documents are legal. (click path,
end page, time on task, and accuracy/abandon data)

Task 3: Understanding Terminology
3a.
Let’s say that you are confused about some of the terminology on the HUD-1 Settlement Statement. Using
the site, find out what Earnest Money means. (click path, end page, time on task, and accuracy/abandon
data)
3b.

Some terminology that you have seen for mortgages is confusing to you. Using the site, find out what APR
means. (click path, end page, time on task, and accuracy/abandon data)

Task 4: Contacting an Individual for Help
4.
You have looked around this tool, but you still have more questions. You want someone to answer your
questions. Navigate to the area of the site that can get your questions answered. (click path, end page, time
on task, and accuracy/abandon data)
Task 5: The Title Process & Taxes

1010 N. Glebe Road, Suite 510 | Arlington, Virginia 22201 | www.forsmarshgroup.com

4
5a.

Using the site, find more information about the title process. For example, when do you get your name on the
title of your new home? (click path, end page, time on task, and accuracy/abandon data)

5b.

Using the site, find out how property taxes on your loan are paid. (click path, end page, time on task, and
accuracy/abandon data)

Task 6: Assessing Changes in a Loan
6a.
You have a friend who recently closed on a home, and the amounts at closing were different than what she
expected. You want to make sure that doesn’t happen to you. Using the site, find out what can change from
your Good Faith Estimate (GFE), or loan offer, and what, by law, cannot. (click path, end page, time on task,
and accuracy/abandon data)
6b.

Using the site, show what would you do if you found an error in your HUD-1 Settlement Statement. (click
path, time on task, and accuracy/abandon data)

6c.

Using the site, find out what prepayment penalty in the Good Faith Estimate (GFE) means. (click path, end
page, time on task, and accuracy/abandon data)

6d.

Using the site, find out more information about Line 801 in the HUD-1 Settlement Statement. (click path,
end page, time on task, and accuracy/abandon data)

Task 7: E-signature
8a.
You have reviewed the HUD-1 Settlement Statement, and now you want to approve it. How would you use
this site to approve the document? (click path, end page, time on task, and accuracy/abandon data)

All participants will receive the following instructions:
The following section is about the mortgage closing experience.
Please read each item carefully, and answer each item honestly.
Consumer Knowledge
1. How confident are you that you have these rights at closing? (1 = strongly agree – 5 = strongly disagree;
randomize order of items) – All Groups
a. I can negotiate any of the closing costs at the closing meeting.
b. I can walk away if there is a difference between what I agreed to and what is on the documents and
the difference isn’t corrected.
c. I can contact someone who can help answer questions I have.
d. I can stop the closing process to ask clarifying questions to others at the closing table.
e. I can stop the closing process to call someone I know to ask clarifying questions.
f. I can sign all the documents even if I have not read them thoroughly.
g. I can sign some of the documents, but not sign the ones I have questions about, and I can still close.
h. I can sign the documents and pay the closing costs at a later date.
i. I can get the keys to my house, but finish signing the documents at a later date.
j. I can decide to send someone else to the closing, like a friend or family member who knows more
about the process than I do.
2. Please rate how much you agree with the following statements (1 = strongly agree – 5 = strongly disagree;
randomize order) – Groups B, C, and D Only
a. I have a clear understanding of the terms of my loan and how my payments will be determined.
b. I can find answers to my questions about the closing process.
c. The information on the website helps me understand the closing process.
d. The website makes me feel more comfortable with the closing process.
1010 N. Glebe Road, Suite 510 | Arlington, Virginia 22201 | www.forsmarshgroup.com

5
e. I have a clear understanding of the terms of my loan and how my payments will be determined.
f. I have a clear understanding of my rights and responsibilities as a homeowner, and the
consequences if I do not fulfill those responsibilities.
g. I have a clear understanding of the final closing costs on my HUD-1 Settlement Statement and the
justification for any differences from the quotes on my initial Good Faith Estimate (GFE).
Empowerment
3. How likely are you to do each of the following at closing: (1 = not likely at all – 5 = extremely likely;
randomize order) – All Groups
a. Negotiate if there is a difference between what you agreed to with your service provider and what is
on the documents.
b. Walk away if a difference between what you agreed to with your service provider and what is on the
documents isn’t corrected.
c. Contact someone who can help answer questions you have.
d. Stop the closing process to ask clarifying questions to others at the closing table.
e. Stop the closing process to call someone I know to ask clarifying questions.
f. Sign all documents without reading them thoroughly.
Efficiency/Consumer-Friendliness
Please rate how much you agree with the following statements (1 = strongly agree – 5 = strongly disagree;
randomize order) – All Groups
4. The mortgage closing process is…
Speed
a. Quick
b. Efficient
c. Speedy
Clarity
d. Simple
e. Easy
f. Straight-forward
Stressfulness
g. Tense
h. Tiring
i. Exhausting
5. How do you think the mortgage closing process would make you feel?
Empowerment
a. Powerless
b. Helpless
c. Vulnerable
Clarity
d. Confused
e. Ignorant
f. Puzzled
Stressfulness
g. Overwhelmed
h. Rushed
i. Anxious

1010 N. Glebe Road, Suite 510 | Arlington, Virginia 22201 | www.forsmarshgroup.com

6
Section 4. Final Tasks – All Groups

1. How can the costs on the HUD-1 change from the original quotes on the Good Faith Estimate (GFE)?
• Some of the closing costs cannot increase, some can only increase by 10 percent, and the rest can change
• Some of the closing costs cannot increase and the rest can only increase by 10 percent
• There are no limitations on how the closing costs can change
• None of the closing costs can change
• I don't know
2. What is the annual percentage rate (APR)?
• The cost of borrowing money based on a percentage rate of the principal.
• The amount of money that the borrower must pay each month.
• The cost of borrowing money, including interest rate, points, broker fees, and certain other charges.
• The total cost owed to the lender, including the principal payment and interest.
• I don't know
3. Does the consumer have the right to negotiate the costs on the HUD-1 at the closing meeting?
• The consumers can only negotiate the costs that increased by more than 10 percent at the closing meeting.
• The consumer can negotiate costs with the seller at the closing meeting, but not anyone else present at the
closing meeting, like the appraiser, loan officer, or settlement agent.
• The consumer cannot negotiate any of the closing costs at the closing meeting.
• The consumer can negotiate any of the closing costs at the closing meeting.
4. What is the primary purpose of the security instrument, which may be called the “Mortgage” or “Deed of Trust?”
a. It is the document that itemizes the costs that a borrower must pay at closing.
b. It is the document that puts up the borrower's house as collateral for the loan and defines
homeowner responsibilities to keep the home.
c. It is the document that shows the agreement between the borrower and the lender that the loan
exists.
d. It is the document that transfers the title of the home from the seller to the borrower.
e. I don't know
5. [Show HUD-1 and GFE] When a consumer receives his/her HUD-1 Settlement Statement before or at closing,
what should he/she do? (select all that apply)
a. Skim the HUD-1 Settlement Statement, as it cannot change substantially compared to the Good
Faith Estimate (GFE)
b. Compare to the Good Faith Estimate he/she received earlier and look for pricing discrepancies
c. It doesn’t matter, they can review it after closing
d. Sign the HUD-1 Settlement Statement
e. I don’t know
6. Imagine that you have a 30-year fixed-rate mortgage and that this is your HUD-1 Settlement Statement. You
have 20 years left on your mortgage, but you recently received a large lump sum of cash from an insurance
settlement. You want to use that money to pay off the remainder of your mortgage. Will you be charged anything
for paying off your loan early? (yes, no, I don’t know)
7. What does Earnest Money mean? (multiple choice)
a. Money that I receive after closing.
b. Money owed at closing.
c. Money I put down on the house for a deposit.
d. Costs and other fees for closing on a house.

1010 N. Glebe Road, Suite 510 | Arlington, Virginia 22201 | www.forsmarshgroup.com

7
8. Imagine that you are comparing Good Faith Estimates (GFE) from two different banks to see which is offering
you a better loan for the same home. Which bank is offering you a better loan? [show Good Faith Estimates
(GFE), side by side, have different origination costs. Participants should know from previous experience with the
portal to look at different charges and that a higher origination cost will be a bad deal for them].
a. GFE from bank 1
b. GFE from bank 2
Section 5. Debrief – All Groups
Thank you for completing this activity. Before you go, we wanted to share some information about rights people have
during the closing process.
• You can negotiate any of the closing costs at the closing meeting.
• You can always question if there is a difference between what you agreed to and what is on the documents
and the difference isn’t corrected.
• You can contact someone who can help answer questions you have.
• You can stop the closing process to ask clarifying questions to others at the closing table.
• You can stop the closing process to call someone you know to ask clarifying questions. You can always ask
questions, at any point in the process.
Thank you for participating.

1010 N. Glebe Road, Suite 510 | Arlington, Virginia 22201 | www.forsmarshgroup.com