Appendix I. KI Interview Protocol

Appendix I
Key Informant Interview Protocol

This page has been left blank for double-sided copying.

ACL process evaluation interview protocol

Interview protocol
Introductory statement and consent
Thank you for speaking with me today. My name is _____________ and I am with USAging. In
partnership with Mathematica, we are conducting a process evaluation of the Aging Network for the
Administration for Community Living (ACL). The purpose of this study is to learn about how the Aging
Network collaborates to improve the lives of older adults, and how it determines the effectiveness of these
efforts.
We have several goals for this discussion. We are interested in hearing about your organization’s
experiences collecting data to compare the value or benefit of program services relative to the costs; that
is, what is often referred to as “return on investment (ROI) analysis.” We would also like to hear about
how you have used the results of these analyses, and any lessons learned, challenges, or types of supports
that would be helpful. We expect our discussion to last about one hour. 1 Your participation is voluntary,
and you may skip any question. Your information will be kept private to the extent permitted by law. We
will not name individual staff or agencies in our study report.
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of
information unless such collection displays a valid OMB control number (OMB 0985-xxxx). Public
reporting burden for this collection of information is estimated to .25 hours per response, including time
for gathering and maintaining the data needed and completing and reviewing the collection of
information. The obligation to respond to this collection is voluntary.

Would it be okay if we recorded the interview? We take notes to capture everything you say but in case
we want to check something in our notes, it is useful to have a recording. We do not share recordings
outside of the study team and will destroy recordings when our project ends.
[If interviewee does not wish to be recorded, acknowledge their request and do not record the interview.
Otherwise, START audio recording.]
Do you have any questions before we begin? Okay, let’s get started.

I.

Background
1. To start, please introduce yourself and describe your role at [ORGANIZATION NAME]. What
are your key responsibilities?
a. How long have you worked in your current role? How long have you worked at your
organization? Have you previously held other roles at your organization?
2. We are interested in learning more about the partnerships [ORGANIZATION NAME] has with
other organizations. We understand that you partner with [TYPES OF ORGANIZATIONS].
Does your organization primarily have individual relationships with partners, or do you belong to
any networks, collaboratives or coalitions, or both?

1

Priority questions that should be asked are shown in bold.

I.3

ACL process evaluation interview protocol

a. For partnerships with individual agencies: What have your successful relationships with
individual agencies accomplished? [Probe: What service needs did you and your partners
help address?]
b. If part of a collaborative: What has the collaborative accomplished? [Probe: What service
needs did the coalition address?]
Now I would like to talk about your organization’s experience calculating the financial value of services,
including participating in or conducting return on investment (ROI) or cost benefit analyses, which
calculate the net benefit of an investment compared to its cost. Sometimes this means translating
something that is challenging to measure, such as quality of life, into quantifiable values. In our survey,
we asked about the services, costs, and benefits that are assessed. We want to learn more about how you
define each of those concepts, and also hear about your organization’s experiences and lessons learned.
To start, we’ll talk about your organization’s overall experience with ROI.
3. Why did your organization begin [CONDUCTING/PARTICIPATING IN] assessments to
determine the value or benefit compared to the cost of program services? [Prompts: Do any
of your funders or partners require or strongly encourage it? To support a business case to
potential partners? To advocate for more funding with state/local officials?]
4. We understand that your organization has assessed costs and benefits for [TYPES OF
SERVICES]. Is that correct? Why do you focus on these services?
a. Are there any services for which you do not conduct ROI analyses? Why or why not?
5. When your organization began conducting ROI or cost benefit analyses, what resources or data
system changes were required?
6. What other organizations do you partner with to assess costs and benefits, if any?
a. What are the roles of each organization in the ROI analysis? For example, does your
organization collect and analyze the data?

II.

Cost and benefit calculation

Next I am going to ask about how you think about and calculate costs and benefits. We are interested in
learning about how you typically think about and conduct ROI assessments. If it’s helpful, we can focus
on your most recent or largest analysis.
7. For both benefits and costs, what types or sources of data do you use? From whom do you
collect or with whom do you share data?
a. How are data analyzed? What specific software is used? Who does the analysis?
8. When you think about these financial calculations, from whose perspective do you typically
calculate costs and benefits? [Prompt: Your organization? The participants? Caregivers? Payers
(e.g. ACL, Medicaid, Medicare)? Society at large (e.g. taxpayers)?] Why?
9. When you calculate the value of benefits for these analyses, we understand that you
typically include [TYPES OF BENEFITS]. Is that correct? We would like to learn more
about each of these benefits.
Probe on WHY OR WHY NOT the organization considers the following benefits, including
how they monetize the benefit, which outcome measures are used and why (required by
funders? Following fidelity guidelines?), and whether there are barriers to measurement:

I.4

ACL process evaluation interview protocol

a. Improved management of chronic conditions (e.g. diabetes, high blood pressure, high
cholesterol)
b. Improved or maintained functional status (ADLs/IADLs)
c. Greater independence, or delaying or avoiding entry into long term care facilities
d. Increased socialization or reduced loneliness
e. Reduced use of costly health and social services (e.g. fewer avoidable hospital
admissions and ED visits)
f. Improvements in self-reported physical or mental health
g. Increased life expectancy
h. Improved quality of life
i.

Other (e.g. increased employment, consumer satisfaction, food security, caregiver ability
to retain employment or conserve time off)

10. When you calculate costs for these analyses, we understand that you typically include
[TYPES OF COSTS]. Is that correct? We would like to learn more about each of these
costs.
Probe on WHY OR WHY NOT the organization considers the following costs, including how
they monetize the cost and whether there are barriers to measurement:
a. Staff costs (e.g. salaries and fringe benefits, volunteer time)
b. Direct service costs (e.g. service contracts, support services, partner and provider service
costs)
c. Supply and equipment costs (e.g. supplies and/or leases on purchase of equipment)
d. Overhead and operating costs (e.g. overhead, facilities, utilities, indirect cost)
e. Development and maintenance of data systems

III.

Uses and lessons learned

Next we will discuss how your organization uses the results of ROI analyses and any lessons learned from
conducting or participating in such analysis.
11. How has your organization used the results of your ROI analysis? [Prompt: How have either
the results, or the measurement itself, affected your decisions to continue (or discontinue) or
expand programs? Have the results been used to demonstrate value to potential partners or
funders?]
a. If the analysis has identified any cost savings, to whom have the savings accrued?
[Prompt: Your organization? The participants? Payers (e.g. ACL, Medicaid, Medicare)?
Society at large (e.g. taxpayers)?]
12. Have you conducted the analyses more than once? If so, did you make any major changes in how
you conducted the analyses and if so why?
13. What has your organization learned from measuring ROI?
a. What worked well, or made it a success?

I.5

ACL process evaluation interview protocol

b. What were (or are) the biggest challenges? How did (or does) your organization address
those challenges?
14. What additional supports, if any, could your organization have used to strengthen your ability to
effectively measure ROI and to use the results to improve service delivery (or any other uses,
such as demonstrating value to partners and funders)?
15. Finally, are there any last comments or observations you would like to share regarding your
organization’s practices conducting ROI analyses?
Thank you! Those are all the questions I have for you today.
Would it be okay to contact you after today with a follow up question or two in case we missed
something in our notes?

I.6