Medicaid Agencies

Section 811 PRA Demonstration Evaluation – Phase I

Data Collection Instruments: Process Evaluation Interview Protocol – Medicaid Agency

105 Lock Street

Suite 207

Newark, NJ 07103
(973) 622-0900 (Phone)

(973) 622-0655 (Fax)

info@bctpartners.com (E-Mail)

www.bctpartners.com (Web)

Process Evaluation Interview Protocol – Medicaid Agency

State: ______________________________________________________

R name: ____________________________________________________

R affiliation: _________________________________________________

R phone: ___________________________________________________

R email: ____________________________________________________

BCT AND ABT INTERVIEWERS WILL COMPLETE AS MUCH AS THIS INTERVIEW PROTOCOL AS POSSIBLE IN ADVANCE OF THE SITE VISIT FROM AVAILABLE DATA SOURCES INCLUDING THE 2012 PRA APPLICATION FOR FUNDING, QUARTERLY GRANTEE REPORTS, AND HUD ADMINISTRATIVE DATA. INTERVIEWERS WILL ONLY ASK QUESTIONS THAT ARE NOT FILLED IN PRIOR TO THE SITE VISIT OR IF THE FILLED-IN RESPONSES REQUIRE UPDATES OR CLARIFICATION. Interviews will be conducted orally and respondents will not be asked to provide written responses to any interview query.

Thank you very much for taking the time to meet with us. BCT Partners and Abt Associates have been contracted by HUD to conduct an evaluation of the Section 811 PRA programs, specifically examining how the programs are being implemented across the 12 grantee states funded under the first round of demonstration grants. We will be talking with representatives from state Medicaid lead agencies, grantee agencies, and other partner organizations involved in the program.

The goal of the evaluation is to document the process by which the program is being implemented by grantee states, and ways in which the actual program differs from what was planned. We hope that you will be candid in the information you provide about the program. The information you provide is crucial to improving programs that provide housing and services for non-elderly people with disabilities. In addition, this information will be used to inform the implementation of additional rounds of Section 811 PRA grants.

While we will make every effort to protect your confidentiality, HUD staff will read our reports, and it is not possible to guarantee complete anonymity given the high level of HUD involvement in the PRA program effort. However, your name will not be used in reporting what we have learned during this onsite visit, and your responses will be combined with those of other grantees and their partners in other states.

Do you have any questions about the study before we begin?

I’m going to start by reviewing the responsibilities of the lead Medicaid agency based on the grant application, quarterly reports, cooperative agreement, and other materials we have on file. As I understand it, the Medicaid agency is (or was) responsible for:

[Interviewer checks the box prior to interview and lists responsibilities]

[If information is not available from materials, ask following questions and use responses to check boxes.]

• What was your role (if any) in planning for the Section 811 PRA Demonstration?

• In responding to the NOFA?

• In the implementation of the demonstration?

• A. Section 811 PRA Demonstration Application Planning

  • Overall Strategy for application

  • PRA Program Characteristics

• B. Partnerships and Organizational Structure

• C. Tenant Participation

  • Outreach and Referrals to the program

  • Pre-screening

  • Application and Eligibility

  • Referrals to units/properties

• D. Housing Transition Services

• E. Ongoing Support Services

• F. Program Monitoring, Reporting, and Oversight

A. Section 811 PRA Demonstration Application Planning

I would first like to understand your agency’s role in the state and how the decision was made to apply for the PRA demonstration.

Overall Strategy

1. What are the roles of the LEAD MEDICAID AGENCY in STATE?

2. What other programs does LEAD MEDICAID AGENCY operate that are targeted to (or otherwise serve) non-elderly low income persons with disabilities?

3. What were the reasons STATE decided to apply for the PRA Demonstration?

(Probes: obligations under Olmstead decisions; rebalancing goals; supplementing other existing programs, Money Follows the Person, state initiatives to support community-based housing and services, to meet advocacy goals, homeless assistance programs)

4. What state agencies were expected to be the key partners in the demonstration? How were key partner roles in the demonstration determined?

5. Within the agency(ies) administering Medicaid, which staff or divisions were expected to be involved?

6. What have been recent priorities or trends in STATE’s Medicaid and long-term services and supports systems?

(Probes: How have funding levels changed in the past few years? Have efforts been directed at particular populations, types of facilities, or population groups?)

7. How has the STATE engaged in rebalancing efforts; that is, moving Medicaid resources away from institutional settings and towards community living?

8. Has STATE expanded eligibility for Medicaid in the state as a result of ACA? If so, when did that happen? How has this affected housing and services options for non-elderly people with disabilities?

9. How did STATE expect the PRA Demonstration would fit in the system of housing and long term services and supports for non-elderly people with disabilities, including any Olmstead plans or settlement agreements?

[Before site visit, confirm existence of olmstead PLAN AND/OR settlement agreement and confirm on-site. IF state has a COURT ORDER/settlement agreement, site lead will request a copy of the agreement to review in advance and confirm responses to questions below.]

10. Does your state have a formal (i.e. written) or informal Olmstead Plan? Can you describe the major goals and activities included in the Plan? What is the role of your agency in meeting these goals? Has the Plan changed since its development and if so, can you tell me how it has changed?

11. If your state has an Olmstead settlement(s) or consent decree(s) in effect (list information for each court order/settlement in effect):

    1. What are STATE’s obligations for responding?

    2. What geographic area(s) and target population(s) are affected?

    3. What is the timeline for responding?

    4. How far along is the state in responding to the settlement requirements?

12. How will the PRA program assist STATE in meeting Olmstead settlement agreement requirements?

The following questions are related to the housing strategy, target population, and services decisions made ahead of the grant application. For these questions we would like you to answer based on the planning stage, even if some of these items were modified during implementation.

Housing Strategy

13. How did STATE decide how many units to apply for?

14. How did STATE decide the intended geographic location of PRA units?

• Specific need in geographic area

• Olmstead requirements

• Access to community amenities

• Access to services for target population

• Other:_______________________

Target Population and Support Services

15. [Confirm target population from application] Why did STATE decide on the target population(s) to be served?

16. What were expected to be the service needs of each of the target populations? How were these service needs determined?

17. What were the expected resources for addressing the service needs [CONFIRM INFORMATION FROM APPLICATION]:

    1. Through Medicaid? (E.g., MFP; Section 1915 (c ) Home and Community Based Services; Section 1915(I ) state plan amendments; Section 1115 Research and Demonstration Projects; Balancing Incentive Program)

    2. Through other federal resources? (HRSA, Healthcare for the Homeless, other?)

    3. Through state or local resources?

18. Does STATE have ways to pay for health and behavioral health services for PSH tenants who are not eligible for full Medicaid, such as a state-only health insurance program or a partial Medicaid package under a Medicaid waiver?

B. Partnerships and Organizational Structure

19. How many staff from your agency are involved in the PRA program and what roles do they play?

    1. What percentage of their time do they spend on the PRA program?

    2. How are their positions funded?

    3. How has staff turnover impacted the implementation of the PRA program?

20. Describe the partnership between your agency and the GRANTEE AGENCY.

    1. How long have the two agencies worked together and for what purpose?

    2. How were agency roles determined for the PRA demonstration when you developed the state’s response for the 2012 NOFA?

    3. What mechanisms are used for coordinating activities and resolving issues (e.g. regular meetings, periodic reports)?

21. What are the roles of advocacy groups (i.e., groups like Legal services, service provider organization, affordable housing coalitions, and/or non-profits organizations who advocate for increased and better services and/or housing for the target population(s)) in the program?

22. How do actual partners, roles, and relationships differ from what you expected?

    1. What changed and why?

    2. Did the inter-agency agreement sufficiently delineate the demonstration roles between the partner agencies? Did any changes require you to revise the inter-agency agreement developed for the NOFA application?

C. Participant Selection

We would like to understand how potentially eligible applicants learn about the PRA program, get referred, are pre-screened for eligibility, and (if determined eligible) are referred to available units. I’d like to talk about how this works in STATE, step-by-step, including what part of the process your agency is responsible for, and changes you have made from the approach described in your application. Also if you have a flow chart or other visual aid that helps me to understand your process, please provide that to me. Let’s start with outreach to potential applicants.

Target Population

23. Are target population members already enrolled in Medicaid?

24. Are PRA target population members who are covered by Medicaid in managed care plans, fee for service plans, or some of each? If both, what proportion of the target population do you estimate fall in each category?

25. [If STATE has expanded Medicaid eligibility under the ACA:] Are they people who were categorically eligible for Medicaid prior to the ACA, or part of the newly eligible population? If both, what proportion of the target population do you estimate fall in each category?

26. Is the population identified in the grant application similar to the population occupying PRA units? If not, why not?

Outreach and Referrals to the PRA Program

27. Describe the outreach activities that are performed to obtain potential program participants/tenants.

    1. Were outreach activities integrated into existing programs or services or were new outreach programs created for the PRA program?

    2. What staff are responsible for outreach?

    3. Where are staff based and how are they organized (e.g., in teams, single individuals with geographic territories to cover, by agency location, etc.)?

    4. How much of their time do these staff spend on PRA outreach, and how are their positions funded?

28. [If not already discussed] Do outreach methods differ for different target populations?

29. Have your organization’s outreach methods to potential applicants changed compared to what was proposed in the grant application? If yes, why did the methods change and what changes were made?

30. How many potential PRA applicants does your agency engage through outreach in a given month? To date?

31. How many potential PRA applicants have been referred to the program to date?

Pre-Screening for PRA Program Eligibility

32. Describe how PRA applicants are determined eligible for the PRA program.

1. Which agencies or organizations conduct eligibility? Do property owners conduct eligibility?

2. How are applicants determined to be an eligible member of the target population?

3. How do eligibility determinations differ by target population?

4. How are applicants determined to be eligible under the income requirements?

5. Do you consider other eligibility criteria at pre-screening in addition to income, disability, and receipt of services?

6. When does pre-screening happen? Who does it?

7. What tools do you use?

8. How many people have been (or generally are) screened out at this point?

9. Why are people screened out? What is the range of barriers encountered?

33. Are referrals and pre-screening part of a case management process so that PRA participants receive assistance from application through lease up? If yes, describe that process.

34. What happens once an interested applicant has been identified? How and to whom are they referred to complete an application?

35. Consider recent referrals to the PRA program. Is there a difference in the people being referred compared to the target populations initially identified? If yes, what are some reasons for any differences in actual referrals versus targeted populations?

36. How often do potential PRA applicants identified through outreach decide not to apply to the program? What are the reasons why?

37. Have the outreach and referral to the program activities worked as you expected? Have outreach activities resulted in enough eligible applicants to fill available PRA units?

38. What has worked well and what have been the challenges in outreach and referral? Would you consider this model successful?

Are there any operational improvements to outreach and referral methods that you would suggest?

Referral to PRA Units/Owners

39. Once a person is found eligible for the PRA program, are they placed on a waiting list? If not, why not?

If yes:

1. What agency(ies)/property(ies) maintain the list?

2. Is SocialServe or other program used for waiting list management?

3. How is priority determined?

4. How many PRA applicants are currently on the waiting list?

5. Does LEAD MEDICAID AGENCY provide any support to clients who are on the waiting list?

40. Once determined eligible for the PRA program, how do applicants learn about PRA units that are available for leasing?

    1. Does the STATE use a housing locator?

    2. Does the STATE use an online housing resource such as SocialServe or HousingLink? If yes please describe how the online resources are used by program staff and by applicants.

41. Describe the process of referring clients to available, appropriate PRA units/owners.

1. What factors (e.g. unit size, accessibility features etc.) are taken into consideration?

2. Which agencies/partners are responsible for matching the applicant’s requirements to an appropriate, available unit?

3. How many applicants are referred for each available unit? How many unit choices does an applicant have under the program?

4. Is the number of eligible applicants adequate to fill available units?

5. How long does it normally take between when a participant is determined eligible for PRA funding and occupancy in a PRA-funded unit?

6. Do you or another agency assist these applicants in requesting reasonable accommodations from the Owners?

42. What has worked well and what have been the challenges in tenant selection and referrals to PRA units/owners?

Would you consider this model successful?

Are there any operational improvements to the selection and referral methods that you would suggest?

Tenant Application and Eligibility Determination

These next questions ask about how PRA participants apply to and are determined eligible for the PRA program.

43. Describe the tenant application and eligibility determination process for a PRA unit (this may be performed at the PRA property site).

    1. Who conducts the application and screening process for a unit?

    2. [IF NOT ANSWERED ABOVE] How do potential applicants receive applications?

    3. How are applications submitted? (By whom? How are they delivered?)

    4. What agencies and staff process applications?

    5. What information and documentation must an applicant provide?

44. Does your agency learn when referred applicants are determined eligible for the PRA unit? If yes:

    1. How does your agency learn about this determination?

    2. How long does it normally take between an applicant referral and eligibility determination for a unit?

45. To date, how many applicants have been referred to the PRA units and by whom?

    1. How many referred applicants have been eligible for PRA units?

    2. How many referred applicants have been found ineligible for PRA units? Why?

46. What happens to applicants found ineligible? Are they referred to other programs or properties? If yes, which ones?

47. To date, how many applicants have withdrawn their application or have chosen not to sign a lease? Why?

48. Is there anything about the eligibility determination process that is different in implementation than you expected? If yes, please describe.

D. Supportive Services

These next questions ask about your agency’s role in providing services to tenants in PRA units. First we will ask about services provided during a participant’s transition to a PRA unit. Then we’ll ask about services provided as part of ongoing occupancy of tenants and the service providers and funding sources for both types of services.

[SITE LEAD WILL COMPLETE SERVICES TABLE BELOW WITH INFORMATION FROM GRANT APPLICATION AND OTHER RESOURCES AND WILL CONFIRM WITH THE RESPONDENT BY ASKING QUESTIONS BELOW.]

Service	Provider(s)	Funding Source(s)	Transition Services (Y/N)	Ongoing Services (Y/N)
Service Coordination
Intensive case management/Service coordination
Information and referral
Supports of participant direction (Support Broker)
Direct Services
Clinical services (physical therapy, occupational therapy, behavioral services, Assertive Community Treatment, etc.)
Counseling, options counseling
Peer counseling
Skilled nursing services
Person-directed goods and services
Home and Community Based Support Services
Conduct housing assessments to identify barriers to successful tenancy
Day supports
Home modifications for accessibility
Adaptive equipment and supplies
Family and caregiver services
Respite care for primary caregivers
Assisted living (care portion, not room & board)
Habilitation/skill building
Personal emergency response systems/crisis management
Personal/attendant care
Homemaker/chore services
Meals (congregate or home-delivered meals)
Self-Sufficiency Services
Employment counseling
Supported employment
Financial management services
Translation and other social services
Transportation
Vehicle repair/modification
Advocacy
Other:
Other:

[IF LEAD MEDICAID AGENCY PROVIDES OR COORDINATES TRANSITION SERVICES]

49. What transition services are provided or coordinated by LEAD MEDICAID AGENCY for new participants as they move into PRA units?

    1. What services are funded by Medicaid and under what authority (MFP; Home and Community Based Services; Section 1915(I) state plan amendments; Section 1115 Research and Demonstration Projects; Balancing Incentive Program)?

    2. Other than services funded through Medicaid, what other services are offered? What organizations provide services and how are they funded?

50. Do transition services differ based on target population?

51. Do transition services differ by type of disability (Probes: physical disability, mental health disability, intellectual or developmental disability.)

52. Once a participant has been matched to an available PRA unit, describe the process for moving the eligible applicant into the PRA unit. Which agency manages the transition? Does it differ by population?

    1. How does your organization determine service needs for PRA participants during the transition?

    2. How do PRA participants learn about services your organization provides while they move to PRA units?

    3. How do PRA participants meet move-in costs such as security deposit requirements (if applicable), furniture, etc.?

53. Are PRA participants assigned a service coordinator or case manager to help them through the transition to PRA units? If yes:

    1. What agency(ies) employ the service coordinators or case managers?

    2. How often do case managers meet or talk with participants?

    3. What assistance do case managers provide?

    4. For how long are transition services provided?

Ongoing Supportive Services

[IF LEAD MEDICAID AGENCY PROVIDES OR COORDINATES ONGOING SUPPORTIVE SERVICES]

54. After the tenant has moved in, what services are provided and coordinated by your LEAD MEDICAID AGENCY?

    1. What services are funded by Medicaid and under what authority (MFP; Home and Community Based Services; Section 1915(I) state plan amendments; Section 1115 Research and Demonstration Projects; Balancing Incentive Program)?

    2. Other than services funded through Medicaid, what other services are offered? What organizations provide services and how are they funded?

    3. Do services differ based on target population?

    4. Do services differ by type of disability (Probes: physical disability, mental health disability, intellectual or developmental disability.)

55. How do participants learn about and access services available to them from your organization after they move to PRA units?

56. Describe the process for determining a participant’s service needs. Is a service plan developed? After a service plan is developed, how often is progress checked?

57. Does your organization assign a service coordinator or case manager to work with them after they have moved into PRA units? Is the transition case manager the same person as the case manager for on-going supports? If yes:

    1. What is the role of the case manager?

    2. How often do case managers meet or talk with participants?

    3. How often do case managers meet or talk with property management?

58. Once the person is housed in the PRA unit, does the LEAD MEDICAID AGENCY do any follow up with the tenant?

    1. With property management?

    2. With the service providers?

    3. With the GRANTEE AGENCY?

59. How are disputes between tenants and property owners handled?

60. How are disputes between tenants and service providers handled?

61. What measures are in place to ensure that PRA tenants will continue to be successful tenants? (Probes: case management, referrals to services, tenant education)

62. What is the process for evicting a PRA tenant if necessary?

Services Coordination and Funding

63. Have the funding streams for services you planned to use been available, appropriate, and sufficient for the people you are serving? What is the likelihood of continued funding?

64. Are there services that are difficult to find, in short supply, or that funding streams do not cover? If yes, what are they?

65. What factors are affecting STATE’s ability to provide services to the target population?

66. Are there any sources of flexible “do what it takes” funding that can be used to cover gaps in what Medicaid or other programs cover?

67. Does receiving PRA rental assistance provide quicker or more flexible access to services (e.g. priority on wait list or more flexible process for enrollment)?

68. Consider all service delivery activities. What has worked well and what have been the challenges in transition services? Ongoing services? Service Coordination? Would you consider this model successful? Are there any operational improvements to the supportive service delivery methods that you would suggest?

E. Program Monitoring, Reporting, and Oversight

Next I would like to ask some questions about how the agency oversees and monitors the PRA program and coordinates with the other state agencies, including the grantee.

69. Describe how the LEAD MEDICAID AGENCY oversees and monitors the implementation and outcomes of the PRA program.

70. What works well about your agency’s oversight and monitoring practices? What could be improved, and how?

71. How effectively do owners and service providers coordinate:

1. During outreach and referral?

2. During transitions to housing?

3. Once participants are settled in housing?

72. What process and outcome measures do you monitor?

(Probes: Process measures include outreach, referrals, number applicants housed, service delivery. Outcomes include retention in housing, reduced use of institutional care, returns to institutional care, and health outcomes for tenants.)

73. Do you have targets or benchmarks you are trying to achieve in addition to the goals stated in your NOFA response? If so, what are they and what progress have you made toward them?

74. What data systems within your agency support oversight and monitoring? What systems do your partner agencies use?

75. Are there mechanisms for sharing data across PRA partner agencies? If so, please describe:

    1. What data are shared (outreach, referral, occupancy in unit, retention, service delivery, participant outcomes)?

    2. Who is responsible for compiling the data?

    3. How and with whom are data shared?

    4. How are the data used: record keeping and reporting, program management and improvement, sharing results with stakeholders/the public?

    5. Was data sharing difficult to achieve?

76. How effective are the existing systems for monitoring the program? What other data would be helpful?

F. Closing Questions

Thank you for taking the time to meet with me today and answer our questions about the PRA demonstration. I just have a few more final questions about your thoughts on the overall PRA demonstration and how it was implemented in your state.

77. Consider the program as implemented. How does progress of the demonstration to date compare with where you expected to be at this point?

78. In what areas of the program have you encountered the greatest successes in getting the demonstration up and running effectively and efficiently?

[interviewers should use list below to probe for areas with successes. ]

• Developing organizational structure for program and improving agency efficiency

• Successfully refining, revising, adapting program design

• Creating good partnerships

• Increasing staff capacity and skill levels

• Staff turnover

• Providing a more comprehensive level of service for target populations

• Increased quality of life, increased services, and/or more stable home environment for target population

• Eliminating gaps in services

• Increased communication among partners leading to better working relationships

• Increased communication with advocacy community

• Meeting Olmstead settlement requirements

• Increased interest in serving target population by developer community

• Good fiscal and programmatic oversight

• Getting complex program up and running smoothly

• Ability to navigate electronic data systems (explain:___________)

• Securing units/attracting property owners to program

• Outreach to identify potential tenants

• Timely referral to owners

• Ability to meet HUD program regulations (such as environmental review, definition of multifamily, 30-year use restriction, etc.) (explain:__________________)

• Other (explain: _________________________)

79. In what areas of the program have you encountered the greatest challenges in getting the demonstration up and running effectively and efficiently?

[interviewers should use list below to probe for areas with challenges. for each area identified, ask about what is being done to improve the situation]

• Developing organizational structure for program and improving agency efficiency

• Successfully refining, revising, adapting program design

• Creating good partnerships

• Increasing staff capacity and skill levels

• Staff turnover

• Providing a more comprehensive level of service for target populations

• Increased quality of life, increased services, and/or more stable home environment for target population

• Eliminating gaps in services

• Increased communication among partners leading to better working relationships

• Increased communication with advocacy community

• Meeting Olmstead settlement requirements

• Increased interest in serving target population by developer community

• Good fiscal and programmatic oversight

• Getting complex program up and running smoothly

• Ability to navigate electronic data systems (explain:___________)

• Securing units/attracting property owners to program

• Outreach to identify potential tenants

• Timely referral to owners

• Ability to meet HUD program regulations (such as environmental review, definition of multifamily, 30-year use restriction, etc.) (explain:__________________)

• Other (explain: _________________________)

[ask next three questions if not answered in above questions]

80. How effective are the PRA program partnerships? What would you like to do differently? Would you partner with the same agencies again?

81. What works well about your organizational structure? What would you do differently?

82. Do you think the program is targeting the right population based on local or state need?

83. What advice would you give to agencies like yours in other states about applying for the PRA Option? About implementing the PRA Option?

84. What suggestions would you make to HUD or HHS to improve the PRA option?

85. Is there anything else you would like to mention that we have not talked about yet?