SIF PFS OMB Part A_02.05.16

Pay for Success (PFS) is an innovative model that ties funding for an intervention to the achievement of its outcomes or impact in the community. PFS offers a way to catalyze philanthropic and private sector investments to deliver better outcomes, enabling government or other payors (for example, school districts or hospitals) to pay only for outcomes achieved – that is, to pay only for what works. PFS increases investments in effective social interventions by changing the way government allocates and invests its resources.

The PFS strategy typically involves a number of stakeholders not always found in the traditional government-service financing model. Philanthropies, high net-worth individuals, financial institutions, or others provide the investment to cover the cost of a service provider delivering an intervention. An independent party evaluates the impact of the service delivered and determines whether outcome goals were met. Depending upon the achievement of established milestones, then government agencies or other payors provide investors with “success payments,” which may include the principal and a modest return as agreed upon by parties involved. Often the government entity contracts with an intermediary to coordinate the PFS project. The intermediary’s responsibilities may include raising capital from investors, facilitating contract negotiations and agreements between partners (including defining outcome goals), overseeing project implementation, and commissioning evaluation activities.

In 2014, the Corporation for National and Community Service (CNCS) initiated the Pay for Success program as part of its Social Innovation Fund (SIF). The goal of SIF PFS program is to build capacity of and provide support to local and state governments and community organizations to engage in Pay for Success activities. The SIF PFS program includes two types of grants: (1) feasibility assessment/capacity building grants to assess feasibility of PFS projects and to build capacity of local and state governments and community organizations to engage in PFS activities, and (2) transaction structuring grants to financially support the negotiation of PFS financial transactions and bring PFS projects to implementation.

In the program’s first year, CNCS awarded feasibility assessment/capacity building grants to seven organizations (mostly large nonprofit and academic organizations). The goal of these grantees is to provide assistance to local and state government agencies and community organizations to build partnerships (between transaction coordinators, funders, evaluators, and payors¹), to explore issue and policy areas for PFS projects, develop project logic models, select target populations, identify evidence-based interventions, and conduct other activities needed to initiate a PFS project. These activities are expected to (1) increase the internal capacity of governments, other payors, and nonprofits to engage in PFS projects and outcomes-based contracting; (2) attract capital to high-performing organizations that can effectively address challenges in low-income communities; and (3) ultimately provide a return on investment to both investors and payors. These grantees held competitions to identify subrecipients, which could be state and local governments, service providers, and/or other organizations that were interested in exploring the feasibility of implementing PFS projects.

CNCS also awarded one transaction structuring grant to a large community development financial institution (CDFI). The goal of transaction structuring assistance is to catalyze PFS projects where feasibility is either confirmed or in the process of being established by local or state government entities. The current transaction structuring grantee has allocated 80 percent of its grant funds as a pass-through to support the activities (e.g., salaries, fees) of transaction coordinators (called service recipients) to create a partnership structure for the PFS project, negotiate the financial transaction of the project, and ensure that the project is ready to be implemented.

The logic model presented in Exhibit A.1 illustrates the purpose of the SIF PFS program and its associated theory of change. The logic model serves as a demonstration of the SIF PFS program’s inputs, activities, outputs, and outcomes that can be expected—and should be measured. The logic model also provides a depiction of the various stakeholders involved in a PFS project and their roles in the process.

All SIF PFS program grantees received grants for up to three years and were required to select at least one of CNCS’s core focus areas (youth development, economic opportunity, and healthy futures) to use when selecting subrecipients and service recipients. As of the date of this submission, the eight grantees selected a total of 55 subrecipients/service recipients. CNCS expects to award two additional cohorts of feasibility assessment/capacity building and transaction structuring grantees in 2016 and 2017. These grantees will also be expected to select subrecipients and service recipients to engage in exploring and developing PFS projects. Currently, SIF/PFS funding supports PFS project development but stops short of PFS project implementation or contract execution.

As a new funding strategy, very little is known about best practices for developing and implementing PFS projects. As such, it is important to assess the SIF PFS program, both to support CNCS’ management and oversight responsibilities and to advance the larger goal of learning from these new evidence-based initiatives. To meet the need for information on the implementation and effectiveness of the SIF PFS program, CNCS has contracted with Abt Associates to support CNCS’s Office of Research and Evaluation to implement a Process Evaluation of the SIF PFS program. The Process Evaluation is designed to elicit and synthesize information not only on the effectiveness of the SIF PFS program, but also on the opportunities and challenges facing future PFS efforts. The goal of the SIF PFS Program Process Evaluation is to provide actionable information to CNCS and to key stakeholders (grantees, subrecipients, service recipients, funders, and evaluators). Collectively, these insights will contribute to building the evidence base and operational knowledge needed to continue to advance this innovative evidence-based strategy.

Exhibit A.1. SIF PFS Program Logic Model

A2. Purposes and Uses of the Collected Information

The SIF PFS Process Evaluation is designed to increase the evidence and knowledge base associated with the SIF PFS program. Results of the evaluation will be used by staff at CNCS to strengthen PFS programming and document the effects of the SIF PFS program on the development of PFS projects across the country. The evaluation will inform policy and funding decisions at CNCS and will also generate actionable information on best practices that will reduce learning curves for current and future PFS stakeholders (grantees, subrecipients, service recipients, funders, and evaluators).

Although the evaluation is designed to provide information about the activities and progress of grantee and subrecipients/service recipients involved in the SIF PFS program, results from the evaluation are expected to be of interest and use to other agencies and organizations that are interested in, or actively engaged in PFS-related activities.

A.2.1. Research Questions

The SIF PFS Program Process Evaluation is designed to address four key evaluation questions:

How effective have the SIF PFS program grantees been to date in completing the activities and deliverables outlined in their proposals?
What have been effects of the SIF PFS program to date on the pipeline of PFS projects, the capacity of organizations to implement PFS projects, access to capital to fund PFS projects, and the catalyzing of PFS projects that may be sustained in the future?
What have been the key lessons learned to date in terms of (1) strengths and weaknesses of the SIF PFS program and potential areas of improvement, (2) the role of SIF in advancing PFS by complementing state, local, philanthropic, and private PFS activity and its potential role in the future, (3) other social finance options available, and (4) ancillary benefits of PFS that accrue to subrecipients?
What is the current status of the PFS model(s) within the larger discussion of public policy and service provision and what is the outlook for the future based on experiences and lessons learned to date?

A.2.2. Data Collection Plans

The primary approach to data collection is a self-administered survey of all 2014, 2016, and 2017 grantees and their subrecipients/service recipients. The surveys will be designed using FluidSurveys, a Section 508 compliant online survey development and deployment tool. Since FluidSurveys is compatible with mobile web browsers, respondents will be able to complete the survey on a smartphone or tablet if they so choose. The surveys will be kept brief, and will take approximately 20 minutes per response. The surveys will be conducted on a longitudinal basis in the spring of 2016, 2017, and 2018 and will include all grantees and subrecipients/service recipients selected that year. Copies of the surveys are presented in Appendix A.1

In each round of survey administration, all respondents will be asked a core set of questions focused on general survey topic areas such as organizational background, PFS experience and involvement, and PFS staffing. In addition to the core questions, tailored topics and questions will be asked separately of the grantees and subrecipients/service recipients. For example, the grantee survey will include questions about SIF PFS program information and technical assistance/deal structuring approaches utilized. Alternatively, the subrecipient/service recipient survey will include questions about responses to grantees’ RFPs and progress on specific project activities and capacities. Exhibit A.2 summarizes the general topics and subtopics for each survey and provides examples of the types of data variables that are included in the surveys.

In addition to the surveys, the evaluation will include less structured and informal qualitative data collection activities such as review of grantee documents provided by CNCS (e.g., grantee applications and progress reports), telephone discussions with expert stakeholders (PFS implementers, federal agency staff, funders/investors, and policy experts) to obtain a more in-depth understanding of particular PFS issues, and site visits to the eight 2014 grantees to learn more about their PFS plans and activities. These additional data collection activities will focus on the specific grantee, respondent, and stakeholder experiences and activities. Individual

Exhibit A.2. Survey Topics and Data Variables

General Survey Topics	Grantee Survey Data Variables	Subrecipient/Service Recipient Data Variables
Organizational Background/Staffing	staffing	type of organization, location, years in operation, staffing
PFS Experience and Involvement	involvement with PFS	involvement with PFS
SIF PFS Program Information	PFS approach	N/A
PFS Subrecipient/ Service Recipient Selection	RFP timeframe and marketing activities, RFP requirements, pass-through funding	how respondent learned about RFP opportunity, RFP(s) responded to, use of funds received (if applicable)
Feasibility Assessment/Capacity Building Assistance or Transaction Structuring Approach	assistance model and expected duration, activities and materials planned and completed, focus areas of assistance	N/A
SIF PFS Activities	N/A	matching fund requirements, project plans
Feasibility Assessment Progress	responsibility for feasibility assessment, primary feasibility considerations, number of projects in each stage of feasibility assessment, reasons projects determined not feasible (if applicable)	feasibility assessment progress, reasons projects determined not feasible (if applicable)
Transaction Structuring Progress	transaction structuring progress, reason transaction was not completed (if applicable)	N/A
Subrecipient/Service Recipient Project Progress	N/A	detailed information on each PFS project's activities and progress
Reflection on the SIF PFS Program	Capacity building, lessons learned	Capacity building, lessons learned, needs, usefulness of assistance received
Perceptions of SIF PFS Program	satisfaction with various aspects of SIF PFS program	N/A

discussion guides will be developed with issues and discussion points relevant to each respondent’s particular focus area and experience with SIF PFS program and project activities. In no case will these discussions collect standardized or identical data from more than nine respondents of the same type. For this reason, these additional data collection activities are not included in this submission.

A3. Use of Information Technology and Burden Reduction

Several efforts have been made to minimize the burden on respondents. In designing the survey, each question was carefully considered to avoid duplication of data available from other sources. The burden will be further minimized by making sure that participants only receive questions that are relevant to them; this will be accomplished by employing skip patterns.

As indicated above, the survey will be administered primarily online using FluidSurveys. Online administration will lessen the burden on participants by allowing them to complete the survey at a time and in a place that is most convenient to them. Additionally, respondents will have the option of completing the entire questionnaire at once, or coming back to it if they are interrupted, or need to consult with colleagues to provide an accurate response (e.g., budget information). Using an online survey format also reduces burden by eliminating the time and effort required for respondents to return a hard-copy survey.

Upon request, respondents without Internet access will be given the option of completing a hard-copy version of the survey. However, it is estimated that only a very small number of respondents will need or desire this option.

A4. Efforts to Identify Duplication

The survey does not duplicate a current data collection activity and there is no information currently available to answer all research questions of the evaluation.

A5. Methods to Minimize Burden on Small Entities

The information collection is not expected to involve small businesses. Some proportion of subrecipients and service recipients may be small nonprofit entities; however, their participation is necessary to meet the study objectives and answer the research questions. The information being requested for the surveys in this study has been held to the minimum required for the intended use. The information requested will not have a significant economic impact on the small entities, as the surveys will require approximately 20 minutes to complete.

A6. Consequences of Less Frequent Collection

Without this data collection, CNCS will not be able to measure or assess the implementation and effectiveness of the SIF PFS program or gauge the effectiveness of grantees and subrecipients/service recipients in developing PFS projects. Reducing the survey data collection from three annual administrations to a single administration would prevent CNCS from measuring the progression of PFS activities and project development over time. It would also preclude CNCS from incorporating future rounds of grantees and subrecipients/service recipients into the evaluation.

A7. Special Circumstances Influencing Collection

This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.

A8. Comments in response to the Federal Register Notice and Consultation outside the Agency

A 60-day public comment Notice was published in the Federal Register (Volume 80, No. 238, pp. 76848 - 76849) on December 11, 2015 (Appendix A.2). The comment period ended February 9, 2016. No comments were received.

SIF PFS program senior leadership and the Contracting Officer’s Representative (COR) at CNCS were consulted for input on the design and content of the surveys. In addition, a pilot test of the surveys and administration protocols was conducted with two grantees and four subrecipients/ service recipients. The pilot tests and revisions to the instruments based on the pilot are described in Part B.4. Tests of Procedures or Methods to Be Used.

A9. Explanation of Any Payment or Gift to Respondents

No payments, gifts, or other financial/in-kind incentives will be given to respondents for participation in the survey.

A10. Assurance of Confidentiality Provided to Respondents

Measures will be taken by the contractor to remove key personal identifiers on surveys prior to data analysis so that responses for a grantee or subrecipient/service recipient cannot be linked to the specific individual who completed the survey. In addition, all analyses, summaries, or briefings will be presented at the aggregate level and therefore will not include individual respondents’ names. However, due to the uniqueness of specific respondent organizations and the small number of respondents in the 2016 data collection, it may be possible to identify a specific organization. Therefore, assurance of privacy cannot be provided to respondents.

As noted above, the survey will be administered in an online format using FluidSurveys. Respondent email addresses will be uploaded into FluidSurveys and each respondent will be emailed an invitation to the survey with a survey link embedded. All data exported from FluidSurveys will be stored on a contractor’s computer that is protected by a firewall that monitors and evaluates all attempted connections from the Internet. Once the project is completed, all private data on each respondent will be deleted, though it should be noted that the contractor maintains backup tapes that are not amenable to the deletion of particular files. The entire database will be encrypted so that any data stored will be further protected. Finally, access to any data with identifying information will be limited to only contractor staff directly working on the surveys.

A11. Justification for Sensitive Questions

No questions of a sensitive nature will be asked in the grantee or subrecipient/service recipient surveys. The surveys will collect information about the grantee and subrecipient/service recipient organizations and their activities and progress in the implementation of SIF PFS program activities.

A12. Estimate of Annualized Hour Burden and Costs to Respondents

The survey data collection will take place annually for three years. Grantees and subrecipients/ service recipients will be surveyed during the years in which they are involved with SIF PFS program activities. Since the 2016 and 2017 grantees and their subrecipients/service recipients have not yet been selected, the number of respondents in each year are projections based on preliminary discussions and planning activities currently taking place within CNCS and are subject to change. The exact number of respondents will depend on the number of new grantees funded by the SIF PFS program in 2016 and 2017 and the number of subrecipients/service recipients that each grantee selects to work with during each year of their grant.

The estimated 20-minute time burden per survey participant was based on a pilot test of the survey instruments with two grantees and four subrecipients/service recipients. The estimated hourly cost in respondent time was calculated based on the median total compensation for the President, CEO, or Executive Director of the eight 2014 grantees and a sample of their subrecipients/service recipients.² The total response burden and costs for this information collection are provided in Exhibit A.3. The estimated burden and costs shown in the exhibit are the totals for all categories of respondents for each survey year and the annualized burden and costs over the three-year data collection period.

Exhibit A.3: Estimated Respondent Burden for Survey Data Collection Activities

Respondent Type	Number of Respondents	Respondents per Survey Round			Total Responses Over Three Years	Average Responses per Year	Average Response Burden per Year (Hours)^a	Average Cost per Year^b
Respondent Type	Number of Respondents	2016	2017	2018	Total Responses Over Three Years	Average Responses per Year	Average Response Burden per Year (Hours)^a	Average Cost per Year^b
2014 Grantees	8	8	8		16	5.33	1.78	$151.11
RFP 1 Subrecipients/Service Recipients	55	55	55		110	36.67	12.22	$733.33
RFP 2 Subrecipients/Service Recipients	55	55	55		110	36.67	12.22	$733.33
2016 Grantees	4	4	4	4	12	4.00	1.33	$113.33
RFP 1 Subrecipients/Service Recipients	28		28	28	56	18.67	6.22	$373.33
RFP 2 Subrecipients/Service Recipients	28			28	28	9.33	3.11	$186.67
2017 Grantees	6		6	6	12	4.00	1.33	$113.33
RFP 1 Subrecipients/Service Recipients	42			42	42	14.00	4.67	$280.00
RFP 2 Subrecipients/Service Recipients^c	0				0	0.00	0.00	$0.00
Total Burden	226	122	156	108	386	128.67	42.89	$2,684.44

^a Estimated time burden is 20 minutes per response.

^bEstimated hourly rate is $85 for grantee respondents and $60 for subrecipient/service recipient respondents.

^c The 2017 RFP 2 subrecipients/service recipients are expected to be selected in fall 2018, after the last round of data collection has been conducted.

A13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

The telecommunications costs of the surveys are considered part of customary and usual business practices. The only cost to respondents will be the time it takes to respond to the survey. The survey will not involve any additional cost burden to respondents or record-keepers, other than that described above.

A14. Estimated Annualized Cost to Federal Government

The estimated cost to the Federal Government for the first year of SIF PFS Process Evaluation survey data collection activities is $52,584 -- this includes activities unique to the first year of data collection such as survey development, pilot testing activities, and web programming of surveys. This is the cost of CNCS’s Federal contractor, Abt Associates, for the first round of survey data collection activities associated with this submission. Each subsequent round of survey data collection will cost approximately $13,459. The total cost to the Federal Government for the SIF PFS Program Process Evaluation’s survey data collection is $66,043 over a three-year period.

A15. Reason for Change in Burden

There are no changes. The surveys described in this supporting statement comprise a new information collection.

A16. Plans for Tabulation and Publication of Results and Project Time Schedule

This section contains the plans for the project timeline and dissemination and the data analysis.

A.16.1 Project Timeline and Publication Plans

The schedule for information collection, data analysis, reporting, and dissemination of results is shown in Exhibit A.4.

Exhibit A.4: Project Activities and Time Frame

Project Activity	Time Frame
Year 1 Survey
Data collection	May-June 2016
Data analysis	June-July 2016
Interim Report preparation	August-September 2016
Dissemination	October 2016
Year 2 Survey
Data collection	May-June 2017
Data analysis	June-July 2017
Interim Report preparation	August-September 2017
Dissemination	October 2017
Year 3 Survey
Data collection	May-June 2018
Data analysis	June-July 2018
Interim Report preparation	August-September 2018
Dissemination	October 2018

A.16.2. Analysis Plan

The analysis plan for the SIF PFS program survey data is based on the research questions listed in Section A.2.1. As stated earlier, the PFS field is new and the current evidence base on these programs is limited. Our analyses will provide the actionable information that is needed by CNCS, and the PFS field overall, to understand and improve implementation of the PFS model in future grant programs.

The data collected from the grantee and service recipient/subrecipient surveys will be descriptively analyzed in order to document the experiences of SIF PFS program grantees and subrecipients/service recipients. This analysis will compare and synthesize results across each group of respondents to form descriptions of “typical” or “composite profiles” of experiences of grantees and subrecipients/service recipients throughout the exploration and implementation of PFS projects. Specifically, these analyses will examine implementation activities in relation to the key research topics of building interest in PFS models, supporting readiness assessment efforts, and designing PFS partnerships and projects. In subsequent years of the surveys, the longitudinal aspect of surveys will be utilized to gain a better understanding of the key elements and timelines common to a successful PFS project.

Data Cleaning and Analysis Plan

Data set up and cleaning. Once the survey administration period has closed, the data will be downloaded from FluidSurveys’ secure website and cleaned, applying any post coding as needed for the analysis. Data files will be produced in a single restricted data set.

Response rates and nonresponse bias analysis. The data analysis will include the calculation of response rates (per OMB’s Standards and Guidelines for Statistical Surveys) for the overall sample and for all relevant subgroups to identify any limits of the study’s representativeness to SIF PFS program grantees and subrecipients/service recipients. If the response rate is below 80 percent, a nonresponse and response bias analysis for individual survey items will be conducted.

Data visualization. Prior to conducting any significance testing, histograms and scatterplots will be used to visually assess the distributions of the variables to be analyzed. These assessments will provide insight into possible patterns in the data as well as indicate potential outliers. Outliers will be identified on a case-by-case basis, and because they may exert an inordinate effect on group means given the small sample sizes of the grantee group, these data points may be censored or excluded as needed to ensure the accuracy of group statistics.

Analysis for reporting. Basic descriptive analyses (including frequencies, percentages, means, and standard deviations) will be calculated. Response frequencies (i.e., counts and percentages) for each item and the mean, median, mode, standard deviation, and minimum and maximum values for every survey question will be calculated. To the extent possible, subgroups will be compared to test for differences between groups (e.g. using t-tests or chi-squared tests). Additional analyses will be determined on an as needed basis, as additional hypotheses or areas for exploration emerge over the course of the analysis.

A detailed description of the plans for analysis of the survey data along with information on weighting due to unit or item nonresponse is presented in Part B of this submission.

A17. Display of OMB Expiration Date

The OMB approval number and expiration date will be displayed on the surveys that are completed by participants for this study.

A18. Exceptions to Certification Statement

CNCS does not request an exception to the certification of this information collection.

1 “Payor” is the preferred PFS term for the entity that is ultimately responsible for paying investors proportional to the agreed amount based on the level of measureable impact achieved. In many cases, the pay for success payor is a federal, state, or local government agency.

2 Total compensation was collected from IRS Form 990s for nonprofit organizations and public employee salary databases for governmental organizations.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Wilson, Alicia
File Modified	0000-00-00
File Created	2021-01-24

SIF PFS OMB Part A_02.05.16