State Housing Agency

Section 811 PRA Demonstration Evaluation – Phase I

Data Collection Instruments: Process Evaluation Interview Protocol – PRA Grantee

105 Lock Street

Suite 207

Newark, NJ 07103
(973) 622-0900 (Phone)

(973) 622-0655 (Fax)

info@bctpartners.com (E-Mail)

www.bctpartners.com (Web)

Process Evaluation Interview Protocol – PRA Grantee

State: ___________________________________________________________

R name: _________________________________________________________

R affiliation: ______________________________________________________

R phone: ________________________________________________________

R email: _________________________________________________________

BCT AND ABT INTERVIEWERS WILL COMPLETE AS MUCH AS THIS INTERVIEW PROTOCOL AS POSSIBLE IN ADVANCE OF THE SITE VISIT FROM AVAILABLE DATA SOURCES INCLUDING THE 2012 PRA APPLICATION FOR FUNDING, QUARTERLY GRANTEE REPORTS, AND HUD ADMINISTRATIVE DATA. INTERVIEWERS WILL ONLY ASK QUESTIONS THAT ARE NOT FILLED IN PRIOR TO THE SITE VISIT OR IF THE FILLED-IN RESPONSES REQUIRE UPDATES OR CLARIFICATION. Interviews will be conducted orally and grantees will not be asked to provide written responses to any interview query.

Thank you very much for taking the time to meet with us. As you know, BCT Partners and Abt Associates have been contracted by HUD to conduct an evaluation of the Section 811 PRA programs, specifically examining how the programs are being implemented across the 12 grantees funded under the first round of demonstration grants. We will be talking with representatives of each housing agency grantee as well as state Medicaid lead agencies and other partner organizations involved in the program.

The goal of the evaluation is to document the process by which the program is being implemented by grantees and ways in which the actual program differs from what was planned. We hope that you will be candid in the information you provide about the program. The information you provide is crucial to improving programs that provide housing and services for non-elderly people with disabilities. In addition, this information will be used to inform the implementation of additional rounds of Section 811 PRA grants.

While we will make every effort to protect your confidentiality, HUD staff will read our reports, and it is not possible to guarantee complete anonymity given the high level of HUD involvement in the PRA program effort. However, your name will not be used in reporting what we have learned during this onsite visit, and your responses will be combined with those of other grantees and their partners in other states.

Do you have any questions about the study before we begin?

I’m going to start by reviewing the responsibilities of the grantee agency based on your grant application, quarterly reports, cooperative agreement, and other materials we have on file. As I understand it, the Grantee agency is (or was) responsible for:

[Interviewer checks the boxES prior to interview and lists responsibilities, then confirms and revises onsite as appropriate.]

• Section 811 PRA Demonstration Planning

  • Overall Strategy for Application

  • PRA Program Characteristics

• Partnerships and Organization

• Property Selection

• Tenant Participation

  • Outreach and Referrals to the Program

  • Pre-screening

  • Application and Eligibility

  • Referrals to Units/Properties

• Supportive Services

  • Housing Transition Services

  • Ongoing Supportive Services

• Grant Administration and Fiscal Oversight

  • RAC Administration

  • HUD Data Systems

• Leveraged Vouchers and Other Resources

• Program Monitoring, Reporting, and Oversight

• Other_______________________________________________

A. Section 811 PRA Demonstration Planning

I would first like to understand your agency’s role in the state and how the decision was made to apply for the PRA demonstration.

Overall Strategy for the PRA Demonstration Application

1. What is the GRANTEE’s role in the state?

2. What other programs does GRANTEE operate? What other affordable housing programs (including loan programs, tax credit programs, etc.) does grantee administer that are exclusively or in part designed to serve non-elderly persons with disabilities?

[BREIFLY Confirm housing programs based on review of 2012 and 2013 PRA applications and grantee agency’s website. NOTE WHETHER PROGRAM SERVES PERSONS WITH DISABILITIES OR NON-ELDERLY PERSONS WITH DISABILITIES EXCLUSIVELY]

Housing Program	Program Size (# of units or households served)	Target Population

3. Why did STATE decide to apply for the PRA Demonstration?

(Probes: fill unmet need, augment housing for non-elderly disabled persons, and increase mainstream housing opportunities, to expand on existing programs.)

4. Are there other state agencies that administer housing programs for non-elderly people with disabilities in STATE? If yes, how was the decision made that this agency would be the Grantee?

5. What state agencies were expected to be the key partners in the demonstration? How were key partner roles in the demonstration determined?

6. What role did advocacy groups (i.e., groups like Legal Services, service provider organizations, affordable housing coalitions, and/or non-profit organizations who advocate for increased and better services and/or housing for the target population(s)) play during the application process and development of the PRA program?

7. What have been recent priorities or trends in STATE’s housing plans for non-elderly people with disabilities?

(Probes: How have funding levels changed in the past few years? Have efforts been directed at particular populations, types of facilities, geographic locations?)

8. How did STATE expect the PRA demonstration would fit in the system of housing and long-term services and supports for people with disabilities?

   1. How did the STATE expect the PRA demonstration to fit into STATE or local plans to end homelessness?

9. Will PRA funds or units fill a gap in terms of program type (rental assistance paired with services)? If yes, does this meet a need for:

   1. Specific target population

   2. Geographic area served

   3. Unit size

   4. Increase mainstream housing opportunities

   5. Other: ___________________________________

10. How does the PRA program help to meet your Fair Housing Goals or remove impediments to Fair Housing?

[Before site visit, confirm existence of olmstead PLAN AND/OR settlement agreement and confirm on-site. IF state has a settlement agreement, site lead will request a copy of the agreement to review in advance and confirm responses to questions below. ]

11. Does your state have a formal (i.e. written) or informal Olmstead Plan? Can you describe the major goals and activities included in the Plan? What is the role of your agency in meeting these goals? Has the Plan changed since its development and if so, can you tell me how it has changed?

12. If your state has an Olmstead settlement(s), court order or consent decree(s) list information for each settlement in effect:

    1. What are STATE’s obligations for responding?

    2. What geographic area(s) and target population(s) are affected?

    3. What is the timeline for responding?

    4. How far along is the state in responding to the settlement requirements?

13. How did the STATE expect the PRA demonstration to fit into the STATE’s Olmstead plan or (if applicable) settlement agreements?

PRA Program Characteristics

Now I would like to review the information in the 2012 PRA funding application and talk about the reasons behind the STATE’s PRA funding request. Please confirm the following characteristics of the PRA demonstration at the time of application.

[Interviewers will complete APPLICATION inFORMATION IN the table below prior to the site visit and confirm information WITH grantee. BEFORE COMPLETING ENTIRE INTERVIEw GUIDE, COMPLETE THE CURRENT/ACTUAL INFORMATION COLUMN IF CURRENT DATA INFORMATION DIFFERS FROM APPLICATION]

PROGRAM DATA	APPLICATION INFORMATION	CURRENT/ACTUAL INFORMATION
PRA Funds Requested*
PRA Units
Average Per Unit PRA Rental Subsidy
Rent levels (%AMI or FMR)
Grantee prioritizes target population? Yes/No
Target Populations (in order of priority (if grantee prioritizes)	1.	1.
	2.	2.
	3.	3.
	4.	4.
Types of Properties Targeted (check all that apply)	Existing properties New construction Substantially Rehabilitated properties Other: ______________________	Existing properties New construction Substantially rehabilitated Other:_________________
Source(s) of dedicated development financing for PRA units**
Targeted Location of Units	Statewide OR Specific geographic locations: ____________________________________ ____________________________________ ____________________________________	Statewide OR Specific geographic locations: ____________________________________ ____________________________________ ____________________________________
Targeted Property Characteristics (indicate number or percent of planned units by bedroom size. Also indicate # or percent of units for each accessibility type)	Bedroom size(s): _____0-BR 1-BR 2-BR 3-BR 4+-BR Proximity to transportation Proximity to services (list): _____________ ______________________________________ ______________________________________ Accessible for mobility-impaired:______ Accessible for hearing-impaired:______ Accessible for vision-impaired:_______ ______________________________ ______________________________ ______________________________	Bedroom size(s): _____0-BR 1-BR 2-BR 3-BR 4+-BR Proximity to transportation Proximity to services (list): ___________________________________________________ Accessible for mobility-impaired:______ Accessible for hearing-impaired:______ Accessible for vision-impaired:_______ __________________________ __________________________ __________________________

*If actual budget amount received differs from application request, enter in the Current/Actual Column

**Required by application

14. How did STATE decide on the target population to be served?

15. What were expected to be the service needs of each of the target populations? How were these service needs determined?

16. How did STATE decide how many units to apply for?

    1. So far, have the assumptions about subsidy levels held up during implementation?

    2. What is the actual number of units currently supported by the grant?

17. How did STATE estimate the amount of average expected rental assistance? How did STATE determine the rent levels (%AMI or %FMR)?

    1. What are the actual per unit rental assistance (subsidy) amounts? What is the actual tenant portion of the rent amounts?

    2. Can you provide the number of units and properties by each rent level (30%, 40%, or 50% of AMI or FMR)?

    3. So far, have the assumptions about average rental assistance held up during implementation?

18. How did STATE decide what type of properties (existing, new construction, substantially rehabilitated or some combination) would be the best, most expeditious method for meeting its PRA unit production goals?

19. [IF USING EXISTING PROPERTIES] How were the existing properties financed (e.g. LIHTC, HOME, Multifamily Bond Program)?

20. [IF DEVELOPING UNITS THROUGH REHAB AND/OR NEW CONSTRUCTION] What were the expected financing sources for new construction (e.g. LIHTC, HOME, Multifamily Bond Program)? Have those the financing sources been used? If not, why not? Which sources replaced them?

21. How did STATE decide the intended geographic location of PRA units?

• Specific need in geographic area

• Olmstead requirements

• Access to community amenities

• Access to services for target population

• Other:_______________________

22. Did you face any challenges in determining the PRA program characteristics or in putting together the grant application? If yes, what were they and how were they overcome?

B. Partnerships and Organizational Structure

Now I’d like to review the various roles that your organization, the Lead Medicaid Agency, and other organizations play in the PRA program. I’ll go through each of the program areas and ask you to confirm which organization(s) has roles in those areas.

[Interviewers will complete in the table below prior to the site visit and confirm information WITH grantee]

	Agency/Partner	Division	Staff Contact/Title
Property Selection
Establish policy (types of properties, set QAP priorities, etc.)
Market the PRA program to development community
Coordinate with capital funding rounds
Select properties to receive funding
Review capital projects
RAC administration
Oversee subsidy determination
Olmstead Plan Contact
Outreach and Referral
Market/outreach program to potential tenants
Refers applicants to program
Pre-screen applicants
Refers applicants to available properties/units
Takes applications
Determine applicant eligibility for unit
Maintains Waiting List
Service Provision
Service coordination
Case management
Service agency (1)
Service agency (2)
Service agency (3)
MFP Program (or Similar) Contact
Administration/Management
Program administrator
Management/Oversight
Finance
Program evaluation
Data Systems
Applicant referrals to program
Applicant referrals to units
Participant tracking (eligibility, tenure, etc.)
Unit tracking
Leveraging commitment
Public Housing Authorities
Location of PRA units 1
Location of PRA units 2
Location of PRA units 3

*May be done at the property so indicate “at property”

23. How many staff from your agency are involved in the PRA program and what roles do they play?

    1. What percentage of agency staff time is committed to the PRA program?

    2. How are staff positions funded?

    3. Has staff turnover impacted the implementation of your program? Explain.

24. Describe the partnership between the GRANTEE AGENCY and LEAD MEDICAID AGENCY.

1. How long have the two agencies worked together and for what purpose?

2. How were agency roles determined for the PRA demonstration when you developed the state’s response for the 2012 NOFA?

3. What mechanisms are used for coordinating activities and resolving issues (e.g. interagency committees, regular meetings, periodic reports)?

25. How do actual partners, roles, and relationships differ from what you expected?

What changed and why?

26. Did the inter-agency agreement sufficiently delineate the roles between the partner agencies?

    1. Did any changes require you to revise the inter-agency agreement since execution?

27. Is there anything you would change about the partnership structure?

    1. If yes, what would you change and why?

    2. Do you think you would use this partnership structure for other efforts in the future?

C. Property Selection

Now I would like to review the property selection process, specifically how a property owner learns about and obtains PRA funding. But first I’d like to update or confirm the information we have on the status of occupied units.

28. What is the total number of PRA-assisted units under lease at this time? [Confirm most recent information from quarterly grantee progress report.]

29. How many total RACs have been executed?

30. What is the total number of units under a RAC agreement at this time?

31. How many units were expected to be filled during the first year of the grant (i.e. since the execution of the Cooperative Agreement (CA) with HUD in 10/14)? During the entire grant period? Is STATE on schedule (according to CA) to meet those goals? If not, why not?

32. For the occupied PRA units or those expected to be occupied soon, what proportion of properties are existing properties, new construction, and newly renovated properties? For properties expected to be occupied soon, what are the expected dates that the units will be available? [COMPLETE TABLE BELOW.]

Property Type	Number of Units under RAC Agreement	Number of PRA Units Occupied to Date	Number of PRA Units in Pipeline* to be Occupied	Expected Occupancy Date(s) of Pipeline Units
Existing Properties
New Construction
Renovated Properties
Other: ______________

*Pipeline is defined as units still under development/construction but where there is an anticipated completion and occupancy date

33. Are the property types as you expected? If not, what changed and why?

[Ask the following questions FOR EACH TYPE OF PROPERTY IDENTIFED FOR THE PRA program based on THE INFORMATION ABOVE]

34. Can you explain how owners and their properties eligible for rental assistance under the PRA program are identified?

    1. How do property owners, managers, or developers learn about the program?

    2. What marketing or outreach is done to encourage participation? Has this changed since program’s inception? If so why?

35. Describe how interested owners are selected to receive rental assistance under the PRA program.

    1. What is the application and approval process and who is involved?

    2. What are the selection criteria for properties and owners?

    3. How were the selection criteria determined?

    4. Have the criteria changed since the 2012 PRA grant application? If yes, how so and why?

    5. Did grantee provide any incentives to participate in PRA program such as giving a preference to receive tax credits, or impose requirements in order to receive financing?

Has this changed since the program’s inception? If so, why?

36. Which sources of development funding/financing (i.e. tax credits, bond financing, state trust funds, etc.) were instrumental in creating the units that are now part of the PRA program?

1. Have state programs or other resources helped bring more properties and units into the program?

2. How much funding/financing from these sources has been obligated or used to produce properties used in the 811 PRA program?

37. What is the timeline for selecting properties/units? Explain any differences in the expected timeline from the actual schedule for property selection and unit availability.

38. To date, how many owners have applied for Rental Assistance Contracts (RAC)? How many units and properties does this represent?

39. To date, how many owners have been selected to receive a Rental Assistance Contract (RAC)? How many units and properties does this represent?

40. To date, how many owners were determined ineligible to receive rental assistance under the PRA program? Why?

41. Have PRA units under construction/development have been completed according to schedule for use in the PRA program? If not, why not?

42. Has STATE been able to attract sufficient properties and units to meet STATE’s PRA program goals? If not, why not?

43. Has STATE been able to attract units with specific characteristics targeted for PRA units, such as property or unit size, accessibility features, services or amenities onsite or nearby, or the experience of the developer or owner? If no, why do you think this has been a challenge?

44. Do you provide vacancy payments to owners? If so, have you made any vacancy payments so far?

45. Is there anything about the property selection process that is different than you expected? If yes, please describe. What changes were made to the process?

46. Would you describe the property selection process as planned and/or implemented as successful? Why or why not?

47. Are there any operational improvements to the property selection methods that you would suggest?

D. Participant Selection

We would like to understand how potentially eligible applicants learn about the PRA program, get referred, are pre-screened for eligibility, and (if determined eligible) are referred to available units. I’d like to talk about how this works in STATE, step-by-step, including what changes you have made from the approach described in your application. Also, could you provide me with any flow charts or other visual aids that will help me understand your processes? Let’s start with outreach to potential applicants.

Outreach, Referrals and Pre-Screening to Determine Program Eligibility

Outreach to Service Providers

48. Describe how the program disseminates information to service providers and other agencies to make sure they are aware of this resource.

49. Have the STATE’s methods of information dissemination about this resource to service providers or other agencies that serve the target population changed compared to what was proposed in the grant application? If yes, why did the methods change and what changes were made?

Outreach to Potential Program Applicants, Referral to Program

50. Describe the outreach activities that are performed to obtain potential program participants/tenants.

    1. Were outreach activities integrated into existing programs or services or were new outreach programs created for the PRA resource?

    2. What agencies and staff are responsible for outreach to clients?

    3. Where are staff based and how are they organized (e.g., in teams, single individuals with geographic territories to cover, by agency location, etc.)

    4. How much of their time do these staff spend on PRA outreach, and how are their positions funded?

51. Have outreach methods to potential applicants changed compared to what was proposed in the grant application? If yes, why did the methods change and what changes were made?

52. Do outreach methods differ for different target populations?

53. How many potential PRA applicants are engaged through outreach in a given month? To date?

54. How many potential PRA applicants have been referred to the program to date?

55. How does _________ AGENCY (identify agency) coordinate referrals to the program with the LEAD MEDICAID AGENCY and other PRA partners?

    1. Is there a centralized agency or staff person to whom all applicants are referred?

    2. Is there a referral list? Who maintains it? How often is it updated?

56. Consider recent referrals to the PRA program. Is there a difference in the people being referred to the program compared to the target populations initially identified? If yes, what are some reasons for any differences in actual referrals versus targeted populations?

57. Describe the application process to the PRA program.

    1. How do potential applicants receive applications?

    2. How are applications submitted? (By whom? How are they delivered?)

    3. What agencies and staff process applications?

    4. What information and documentation must an applicant provide?

58. How often do potential applicants identified through outreach decide not to apply to the program? What are the reasons why?

Pre-Screening for PRA Program Eligibility

[INTERVIEWER: GRANTEES GENERALLY HAVE A 2-STEP ELIGIBILITY PROCESS: AN INITIAL ELIGIBILITY DETERMINATION FOR THE PRA PROGRAM AND THEN A 2^ND ELIGIBILITY DETERMINATION FOR THE UNIT THE APPLICANT WILL BE OCCUPYING (OFTEN DICTATED BY THE PROGRAMS INVOLVED SUCH AS PRA, LIHTC, ETC.]

Now we want to learn more about your step 1 eligibility screening process, which determines initial eligibility into the PRA Program. We will call this step “pre-screening.”

59. How is pre-screening (i.e. eligibility for the PRA program) conducted and which partner(s) is responsible for this activity?

1. How are applicants determined to be an eligible member of the target population?

2. How do eligibility determinations differ by target population?

3. How are applicants determined to be eligible under the income requirements?

4. Do you consider other eligibility criteria at pre-screening in addition to income, disability and receipt of services?

5. When does pre-screening happen?

6. What pre-screening tools are used?

7. How many people have been found ineligible for the PRA program at pre-screening?

8. What are the reasons that people are found ineligible for the PRA Program? What is the range of barriers encountered?

9. Do you have a case management process that starts with the referral and pre-screening processes so that PRA participants receive assistance from referral through lease up? If yes, describe that process.

60. Is there anything about the pre-screening (initial PRA program eligibility) process that is different in implementation than you expected? If yes, please describe.

61. What has worked well and what have been the challenges of the pre-screening process? Would you consider this model successful? Are there any operational improvements to the process that you would suggest?

62. Does your agency learn when individuals referred to the program are determined eligible for the PRA program? If yes:

    1. How does your agency learn about this determination?

    2. How long does it normally take between a referral and eligibility determination?

63. Have the outreach and referral to the PRA program activities worked as you expected? Have outreach and referral activities resulted in enough eligible applicants to fill available PRA units?

64. What has worked well and what have been the challenges in the outreach and referral processes? Are there any operational improvements that you would suggest for any of these activities? Would you consider this model successful?

Referral to PRA Units/Owners

65. Once a person is found eligible for the PRA program, are they placed on a waiting list? If not, why not? If yes:

    1. What agency(ies)/property(ies) maintain the list?

    2. Is SocialServe or other program used for waiting list management?

    3. How is priority determined?

    4. How many PRA applicants are currently on the waiting list?

66. Where do clients stay/live while on a waiting list? Does the STATE provide any bridge funding for rent subsidies prior to PRA funding being available? If yes, describe how bridge funding is provided and who provides the funding.

67. Once determined eligible for the PRA program, how does the referring agency and the applicants learn about PRA units that are available for leasing?

    1. Does the STATE use a housing locator? If yes, please describe how the locator works.

    2. Does the STATE use an online housing resource such as SocialServe or HousingLink? If yes, please describe how online resources are used by program staff and by applicants.

68. Describe the process of referring clients to available appropriate PRA units.

1. What factors (e.g. unit size, accessibility features, etc.) are taken into consideration?

2. Which agencies/partners are responsible for matching the applicant's requirements to an appropriate, available unit?

3. How do potential applicants apply for a specific unit or property?

4. How many applicants are referred for each available unit? How many unit choices does an applicant have under the program?

5. Is the number of eligible applicants adequate to fill available units?

6. How long does it normally take between when a participant is determined eligible for the PRA program and occupancy in a PRA-funded unit?

Tenant Application and Screening/Eligibility

These next questions ask about how PRA participants apply to and are determined eligible for a unit within the PRA program.

69. Describe the tenant application and screening process for a unit (this may be performed at the PRA property site)

    1. Who conducts the application and screening process which determines eligibility for the unit?

    2. What agencies/partners and staff process applications?

    3. What information and documentation must an applicant provide?

70. To date, how many applicants referred to a PRA unit have been found eligible for PRA units? How many found ineligible? Why?

71. What happens to applicants found ineligible for a PRA unit? Are they referred to other programs or properties? If yes, which ones?

72. To date, how many applicants have withdrawn their application for a PRA unit, fallen out of the application process or have chosen not to sign a lease? Why?

73. Once a participant has selected an available PRA unit, describe the process for moving the eligible applicant into the PRA unit.

74. Once the person is housed in the PRA unit, does the GRANTEE or partner do any follow up with the tenant?

    1. With property management?

    2. With the service providers?

75. Has there been unit turnover so far? If yes, how many units have been vacated and for what reasons?

76. Describe the process if a PRA tenant wants to/needs to transfer to another unit. Have any PRA tenants transferred from one unit to another? If yes, why did the tenant transfer units?

77. Describe the process if a PRA-assisted unit turns over.

    1. How do referring agencies learn of the vacancy?

    2. What is the applicant referral process for newly vacant units (e.g., property waiting list, centralized waiting list, other)?

    3. Is follow-up with the tenant who left the unit conducted by grantee or partner agency?

78. What has worked well and what have been the challenges in the processes of referring PRA applicants to units and in PRA tenant occupancy? Are there any operational improvements that you would suggest? Would you consider this model successful?

79. Is there anything about the screening eligibility determination process that is different in implementation than you expected? If yes, please describe.

80. What has worked well and what have been the challenges of the screening eligibility determination process? Would you consider this model successful? Are there any operational improvements to the process that you would suggest?

E. Supportive Services

The following questions ask about services provided to tenants in PRA units. First we will ask about services provided during a participant’s transition to a PRA unit. Then we’ll ask about services provided as part of ongoing occupancy of tenants.

Transition to PRA Units

81. What transition services are available for new participants as they move into PRA units (e.g. assistance with moving, obtaining furniture and house goods, tenant education, housekeeping skills, setting up utilities, accessibility modifications)?

    1. How are transition service needs determined?

    2. Who provides transition services?

    3. How are these services funded? (Probes), Medicaid and other HHS programs, other federal resources; state or local resources.)

    4. Do services differ based on target population?

    5. Do services differ by type of disability (Probes: physical disability, mental health disability, intellectual or developmental disability.)

83. How do participants learn about services available to them while they move to PRA units? Are participants assigned a service coordinator or case manager to help them through the transition to PRA units?

84. How do participants access services during transition to PRA units?

85. How do PRA participants meet the security deposit requirements?

Ongoing Supportive Services

86. After the tenant has moved in, what services are available to PRA participants (e.g. case management, development of individual service plans, access to HCBS, ADL, employment services)?

    1. How are ongoing supportive service needs determined?

    2. Who provides services?

    3. How are these services funded? (Medicaid and other HHS programs, other federal resources; state or local resources.)

    4. Do services differ based on target population?

    5. Do services differ by type of disability (Probes: physical disability, mental health disability, intellectual or developmental disability.)

87. How do participants learn about services available to them after they move to PRA units? Are participants assigned a service coordinator or case manager?

88. How do participants access services after they have moved into PRA units?

89. What measures are in place to ensure that PRA tenants will continue to be successful tenants (Probes: case management, referrals to services, tenant education).

90. What is the process for evicting a PRA tenant if necessary?

91. What is the process used by the property manager for handling urgent or crisis service needs?

92. How are disputes between tenants and property owners handled?

93. How are disputes between tenants and service providers handled?

94. Consider the entire model of delivering services to tenants – transition services, ongoing supportive services, and dispute resolution. What has worked well and what have been the challenges? Would you consider this model successful? How would you improve this process?

F. Grant Administration and Fiscal Oversight

These next set of questions pertain to the administration of PRA grant funds and how the grantee agency executes Rental Assistance Contracts with the property owners after the properties are selected for funding.

95. Have any units or owners been found ineligible for PRA rental assistance after selection into the PRA program? If yes, why?

96. Describe the move-in, annual and Uniform Physical Condition Standards (UPCS)* inspection processes and timeline. *UPCS is performed every three years

    1. Which agencies and staff conduct inspections of PRA units?

    2. Of the potential PRA units that have been inspected, what percent of units passed after first inspection? What percent passed overall?

    3. Has the inspection process led to any delays in releasing PRA funds?

97. Describe the process for executing Rental Assistance Contracts (RACs).

    1. Which grantee staff are involved? Are other agencies involved? What are their roles?

    2. How long does the process to execute a RAC take?

    3. Have there been any challenges in executing RACs?

98. Describe the process for requesting PRA RAC funds from HUD and releasing funds to owners.

    1. Which agencies and staff are involved?

    2. How long does the process take?

    3. Have there been any challenges in releasing PRA funds?

99. Which partner is responsible for entering TRACS data? Have they encountered any issues entering tenant data into TRACS? If yes, please describe.

100. Have you or other partners encountered any issues with any other HUD data system during the implementation of this grant? If yes, please describe.

101. What types of assistance has your agency needed from HUD to implement the grant? Has your agency received the assistance you needed from HUD? If not, what assistance do you need?

G. Leveraged Vouchers and Other Resources

These next few questions address the leveraging of the Housing Choice Vouchers and other resources as indicated in the grant application.

102. Your grant application indicated that you proposed to leverage [insert number of HCVs set aside iN application] Housing Choice Vouchers for non-elderly disabled persons. Is that right? What other resources were leveraged (Housing Choice Vouchers, Project-Based Vouchers, Public Housing, other)?

1. Did the vouchers or units come from state resources or from local PHAs? Describe the process of obtaining the commitment from PHAs.

2. Have there been any changes to that unit commitment? Is the agency on track to achieve its commitment? Have you had trouble accessing any promised resources?

3. How many vouchers or other housing resources have been set aside for this population to date?

103. Your grant application indicated that GRANTEE also proposed to leverage [insert other leveraged resources here] for the PRA program. Have there been any changes to that goal?

     1. If yes, why?

     2. How were the resource amounts determined?

     3. Were there any changes in the amount expected of services funding/resources, housing resources, or other resources? What were the reasons for the change?

H. Program Monitoring, Reporting, and Oversight

Next I would like to ask some questions about how the agency oversees and monitors the PRA program and coordinates with the other state agencies.

104. Describe how the GRANTEE AGENCY oversees and monitors the implementation and outcomes of the PRA program.

     1. How does GRANTEE monitor the work of its own divisions and staff?

     2. How does GRANTEE monitor the work of LEAD MEDICAID AGENCY?

     3. How does GRANTEE monitor the work of other partner agencies?

105. What process and outcome measures do you monitor for the PRA program?

106. Do you have targets or benchmarks you are trying to achieve? If yes, what are they and what progress have you made toward them? Are the goals different from what was stated in the application? If yes, what changed and why?

107. What data systems within your agency support oversight and monitoring? What systems do your partner agencies use?

108. Are there mechanisms for sharing data across PRA partners/agencies? If yes, please describe:

     1. What data are shared (outreach, referral, unit selection and occupancy, retention, service delivery, participant outcomes)?

     2. Who is responsible for compiling the data?

     3. How and with whom are data shared?

     4. How are the data used: record-keeping and reporting, program management and improvement, sharing results with stakeholders/the public?

     5. Did you encounter any data-sharing difficulties?

109. How effective are the existing systems for monitoring the program?

What other data would be helpful?

110. What works well about your agency’s oversight and monitoring practices? What could be improved, and how?

I. Closing Questions

Thank you for taking the time to meet with me today and answer our questions about the PRA program. I just have a few more final questions about your thoughts on the overall PRA program and how it was implemented in your state.

111. Consider the program as implemented. How does progress of the program to date compare with where you expected to be at this point?

112. If you responded to Rating Factor 5 in the PRA grant application, can you provide a status update on progress made in implementing replicable or innovative processes?

113. In what areas of the program have you encountered the greatest successes in getting the program up and running effectively and efficiently?

[interviewers should use list below to probe for areas with successes.]

• Developing organizational structure for program and improving agency efficiency

• Successfully refining, revising, adapting program design

• Creating good partnerships

• Increasing staff capacity and skill levels

• Staff turnover

• Providing a more comprehensive level of service for target populations

• Increased quality of life, increased services, and/or more stable home environment for target population

• Increased mainstream housing opportunities and housing choice

• Eliminating gaps in services

• Increased communication among partners leading to better working relationships

• Increased communication with advocacy community

• Meeting Olmstead settlement requirements

• Increased interest in serving target population by developer community

• Good fiscal and programmatic oversight

• Getting complex program up and running smoothly

• Ability to navigate HUD electronic systems (explain:___________________)

• Securing units/attracting property owners to program

• Timely RAC payments to owners; successful use of HUD system

• Outreach to identify potential tenants

• Timely referral to owners

• Ability to meet HUD program regulations (such as environmental reviews, definition of multifamily properties, 30-year use restriction, etc.) (explain:__________________)

• Other (explain:_________________________)

114. In what areas of the program have you encountered the greatest challenges in getting the program up and running effectively and efficiently?

[interviewers should use list below to probe for areas with challenges. for each area identified, ask about what is being done to improve the situation]

• Developing organizational structure for program and improving agency efficiency

• Successfully refining, revising, adapting program design

• Creating good partnerships

• Increasing staff capacity and skill levels

• Staff turnover

• Providing a more comprehensive level of service for target populations

• Increased quality of life, increased services and/or more stable home environment for target population

• Eliminating gaps in services

• Increased communication among partners leading to better working relationships

• Increased communication with advocacy community

• Meeting Olmstead settlement requirements

• Increased interest in serving target population by developer community

• Good fiscal and programmatic oversight

• Getting complex program up and running smoothly

• Ability to navigate HUD electronic systems (explain: ______________________)

• Securing units/attracting property owners to program

• Delays in RAC payments; difficulty with systems

• Outreach to identify potential tenants

• Timely referral to owners

• Ability to meet HUD program regulations (such as environmental review, definition of multifamily, 30-year use restriction, etc.) (explain:__________________)

• Other (explain: _________________________)

[ask next three questions if not answered in above questions]

115. How effective are the partnerships in the PRA program? What would you like to do differently? Would you partner with the same agencies again?

116. What works well about your organizational structure? What would you do differently?

117. Do you think the program is targeting the right population based on local need?

118. What advice would you give to agencies like yours in other states about applying for the PRA Option? About implementing the PRA program?

119. What suggestions would you make to HUD or HHS to improve the PRA option?

120. Is there anything else you would like to mention that we have not talked about yet?