Supporting Statement OUP Evaluation Part A 110211

Supporting Statement

Evaluation of the Department of Housing and Urban Development’s Office of University Partnerships’ University Programs

Part A: Justification

1. Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

The Department of Housing and Urban Development (HUD) is conducting an evaluation of four grant programs funded through HUD’s Office of University Partnerships (OUP). The four OUP programs are: Historically Black Colleges and Universities (HBCU); Hispanic-Serving Institutions Assisting Communities (HSIAC); Alaskan Native/Native Hawaiian Institutions Assisting Communities (ANNHIAC); and Tribal Colleges and Universities Program (TCUP).

These programs were designed to encourage and expand the growing number of partnerships formed between colleges and universities and their communities. Program grants are used to fund community development activities in disadvantaged communities and to encourage minority-serving colleges and universities to contribute their technical expertise, organizational capacity, and resources to local community development efforts.

There has been no prior evaluation of the outcomes or impacts of activities funded through OUP grants, which have an average annual value of $25 million. Therefore, this evaluation will be the first to systematically document program outcomes and to explore how factors such as partnership structure or the types of activities completed with grant funds affect outcomes for OUP grants. In addition, the study will help the Office of University Partnerships better understand the challenges that grantees face in implementing grant-funded activities. The results of the evaluation will assist the Department in designing grant programs in the future.

This request is for data collection through a web survey and telephone interviews with 67 OUP grant recipients. The web survey instrument and telephone interview protocol are provided in Appendices 5 and 6, respectively. Together, the web survey and telephone interviews will be used to collect in-depth information about the activities funded with OUP grants. The web survey will be used to develop a comprehensive list of activities undertaken by grantees, and to document the partners and additional funding used to support OUP-funded activities. The telephone interviews will focus on two non-trivial activities per grantee (an activity will be considered non-trivial if more than 20 percent of grant funds from a given OUP grant were dedicated to it). The evaluation team will use the telephone interviews to collect more detailed information on the goals, accomplishments, and beneficiaries of the activities, as well as the partnership structures used to implement funded activities.¹

This data collection activity is authorized under [12.U.S.C. 1701z-1], which reads as follows: 

                     TITLE 12--BANKS AND BANKING

                      CHAPTER 13--NATIONAL HOUSING

Sec. 1701z-1.Research and demonstrations; authorization of appropriations; continuing availability of funds

       

 The Secretary of Housing and Urban Development is authorized and directed to undertake such programs of research, studies, testing, and demonstration relating to the mission and programs of the Department as he determines to be necessary and appropriate.

The information sought supports the U.S. Department of Housing and Urban Development’s strategic goal to promote decent affordable housing and strengthen communities.

2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

HUD has contracted with Abt Associates Inc. and its subcontractors, Econometrica and ACKCO, to conduct the evaluation of HUD’S Office of University Partnerships’ University Programs. HUD seeks to evaluate the outputs and outcomes of OUP-funded grants. This study has four primary goals: 1) to assess the types of activities undertaken with OUP grant funds; 2) to describe the outcomes of these projects for the participating universities and the populations and communities they serve; 3) to provide HUD with a deeper understanding of the outcome data grantees have collected and for which they are able to report; and 4) to further HUD’s understanding of the challenges that grantees face in implementing their grants.

The data collection methodology and instruments for this evaluation have been designed to gather specific information on achieved program outcomes, recognizing that outcome data may not be available for many of the grant activities that colleges and universities have undertaken. To collect the information necessary for this study, the Department will implement a web survey and conduct telephone interviews with staff members from a sample of 67 of the 102 OUP minority-serving institutions that have received at least one OUP grant between fiscal years 2005 and 2008. Selected grantees will represent all four OUP university programs.

The first step in data collection will be the web survey, which is expected to take grantees approximately 20 minutes to complete. The second part of the process will be to complete a telephone interview with each grantee in the sample. The interview will focus on two non-trivial activities implemented by the grantee during the study period.² The activities of focus for each grantee’s interview will be randomly selected prior to the beginning of data collection (sampling methodology is described in greater detail in Part B, Question 1).

After the web survey has been completed and prior to the telephone interview, the interviewer will send an email to the grantee summarizing the topics to be discussed during the telephone interview (see Appendix 3.3–Telephone Interview: Follow-up Email Prior to Telephone Interview). The email will also include a customized list of outcomes that correspond to the activities selected for the telephone interview. The hope is that advance notice about the evaluation’s key outcomes will increase the likelihood that all available outcome data for the targeted activities will be reported during the interview. Once this information has been sent, the telephone interview will be conducted.

The results of the data collection efforts will provide HUD and OUP with a greater understanding of OUP grantees’ accomplishments. Additionally, the results will allow HUD to understand the challenges grantees face during implementation and to analyze whether there is sufficient data to assess program effectiveness under a future evaluation. If the evaluation finds that there is insufficient data, it will outline needed revisions to grantee reporting in order to more rigorously evaluate the effect that OUP grant funds have on the communities and populations served by grantees.

The item-by-item justifications for the web survey and telephone interview instruments are provided in Appendix 1. As explained in Part B, Item 2, Abt developed two versions of the telephone interview instrument, Format A and Format B. The two versions allow grantees to talk about their grant-funded activities either as independent activities or as activities co-located within a multi-purpose center or campus facility. For ease of presentation, Appendix 1 uses the questions found in Format A. The questions in Format B are nearly identical; the primary difference is the number of times certain questions are repeated in order to capture the range of activities implemented as part of the center or facility.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.

The proposed data collection will make extensive use of electronic mechanisms and will use information technology to reduce respondent burden. Web-based surveys are now widely used in social science research of the type being proposed for this evaluation. The recipients have the ability to complete the survey using the web at their convenience. The survey will contain instructions, help screens, definitions and other aids to guide respondents through the process.

Improved information technology will be used in the data collection in the following ways:

• To establish the sampling frame and recruit grantees to participate in data collection, clarifying the best person to complete the web survey and telephone interview;

• To contact grantees via email about the telephone interviews and provide them with a list of topics to be discussed so that they can locate necessary information before participating in the interview;

• To collect basic grant information through the web survey in order to shorten the telephone interview by avoiding questions better suited to a written form (e.g., partner list, funder list) and streamlining the telephone interview to skip unnecessary questions (e.g., if a grantee reports no additional funding was used to implement the project, Abt would not ask about the proportion of project costs that were funded by OUP).

All grantees are associated with accredited academic institutions and should be generally comfortable with automated communication and find it easy to communicate with data collectors.

4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.

While information on OUP grant outcomes cannot be found in known datasets, there are three sources of current grantee data that could potentially be used in this evaluation: the OUP database, grantees’ annual narratives, and grantees’ quarterly financial reports. Each is described below.

The OUP grantee database includes data from grantee’s original proposals such as the name of the principal investigator, planned activities, and overall budget request. It also includes information about subsequent grant amendments. Abt Associates will utilize this available administrative data as a first step in the data collection process. This data will be used to populate questions 1.1 and 1.2 of the web survey and to randomly select the activities that will be the focus of the telephone interviews (see Part B, Question 2 for additional information). The database does not provide information on which of the planned activities were implemented, nor does it describe intended partners nor allocation of grant funding by activity.

The second source of data, OUP grantees’ annual report narratives, includes information about grantees’ implementation progress. These narratives answer general questions about project progress but do not provide the depth or detail needed to meet the purposes of data collection. For example, these reports do not go into the nature of the partnerships that grantees form with local organizations, nor do they systematically report outcomes for each activity undertaken.

The third source of data is grantees’ quarterly reports. These reports detail OUP grant spending but do not provide information about other funding sources or cost per activity. Further data collection through the more detailed OUP web survey and telephone interviews is necessary to collect this information.

Given the limited overlap between the existing data sources and the evaluation’s data needs, Abt Associates must conduct the proposed data collection effort.

5. If the collection of information impacts small businesses or other small entities (Item 5 of OMB Form 83-I), describe the methods used to minimize burden.

Colleges and universities are included under Educational Services (NAICS 611310). The size standard for Educational Services (Colleges, Universities, Jr. Colleges and Professional Schools) is $7.0 million in average annual receipts (see 13 CFR § 121). Most colleges and universities have average annual receipts of over $7.0 million; however, since the population being targeted could be designated as minority institutions, most (if not all) respondents could be considered small disadvantaged institutions.

HUD does not foresee any barriers to providing responses due to their status as small institutions. The entities expected to respond are familiar with HUD’s community development activities and requirements. HUD reviewed, evaluated and scored the applications received from these institutions for the OUP program based on the criteria in the Notice of Funding Availability (NOFA).  In awarding OUP funds, HUD acknowledged that the entities accepted into the program have a staff with the “relevant knowledge and skills” to undertake program activities; and the staff is able to “clearly identify outcomes oriented goals with an ability to evaluate and measure performance for each activity proposed under the program.” Given that the prospective respondents have been accepted into the program under the standards noted in the NOFA, HUD is confident that they are capable of easily answering the questions posed. The information sought in this collection refers to their unique experience in the program. The instruments use language that is familiar and therefore easily understandable by the respondents. In addition, the instruments are designed to minimize the burden on the respondents. They contain specific instructions to the interviewer to facilitate explanations and definitions of technical terms, and other related assistance to guide respondents through the interview process.

6. Describe the consequences to the Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

One of the primary objectives of this evaluation is for HUD to obtain information on OUP program outcomes. This will enable HUD to improve and broaden its framework for the program. Failure to collect this information would impair HUD’s ability to assess the effectiveness of the OUP program. This is a one-time collection of information. This data collection will take place over a two- month period, following OMB approval.

7. Explain any special circumstances that require the collection to be conducted in a manner:

• requiring respondents to report information to the agency more often than quarterly;

• requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;

• requiring respondents to submit more than an original and two copies of any document;

• requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years;

• in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;

• requiring the use of a statistical data classification that has not been reviewed and approved by OMB;

• that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or

• requiring respondents to submit proprietary trade secrets, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.

There are no special circumstances that would require this information collection to be conducted in any of the ways listed.

8. If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency’s notice, required by 5 CFR 1320.8(d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden.

Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported. Consultation with representatives of those from whom information is to be obtained or those who must compile records should occur at least once every 3 years—even if the collection of information activity is the same as in prior periods. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained.

In accordance with the Paperwork Reduction Act of 1995, the US Department of Housing and Urban Development published a 60-day notice in the Federal Register on March 22, 2011. The docket number is FR-5486-N-06 and the document number is FR Doc. 2011-6607. The Federal Register notice appeared on pages 15990 and 15991. The authority is Section 3506 of the Paperwork Reduction Act of 1995, 44 U.S.C, Chapter 35 as amended. The notice provided a 60-day period for public comments, and comments were due by May 23, 2011. A copy of the text from the notice is shown in Appendix 6. One comment was received in response to this notice. The author of the comment presented his views on the OUP program as a whole, rather than this particular data collection. Therefore, the comment had no bearing on the data collection and no changes were made in response.

The web survey and telephone interview instruments have been developed and are being implemented with the assistance of Abt Associates Inc., the prime contractor for the Evaluation of HUD’s OUP Programs, and key members of the study team. All of these individuals are experts in conducting this type of research. (Members of the Abt team who reviewed the protocol and instruments are included in Part B.)

9. Explain any decisions to provide payments or gifts to respondents, other than remuneration of contractors or grantees

HUD will not provide any payments to grantees.

10. Describe any assurance of confidentiality provided to respondents and the basis for assurance in statute, regulation, or agency policy.

The data collection effort assures respondents that the grantee-level data will be treated as proprietary. Participants will be assured that their data will be kept private to the extent allowed by the law. Abt has established stringent procedures and safeguards for securing and protecting the information collected against inappropriate disclosure or release . The report that Abt prepares for HUD will not identify the responses of individual grantees in order to protect the privacy of respondents. Participation in the survey is voluntary.

The invitation to participate in the study will contain a signed letter from HUD’s Deputy Assistant Secretary for Policy Development and Research that provides assurance about the legitimacy of the research and endorses the study’s privacy and its voluntary nature (see Appendix 2).

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

The proposed information collection does not contain any questions of a sensitive nature.

12. Provide estimates of the hour burden of the collection of information. The statement should:

* Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample (fewer than 10) of potential respondents is desirable. If the hour burden on respondents is expected to vary widely because of differences in activity, size, or complexity, show the range of estimated hour burden, and explain the reasons for the variance. Generally, estimates should not include burden hours for customary and usual business practices.

* If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens in Item 13 of OMB Form 83-I.

* Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage rate categories. The cost of contracting out or paying outside parties for information collection activities should not be included here. Instead, this cost should be included in Item 13.

The hour burden estimate is based on pre-tests conducted by Abt Associates with four grantees.

Additionally, Abt Associates factored into the burden the fact that up to three individuals may need to be consulted in order to gather all of the information needed for the web survey and the telephone interview. The burden estimates include 5 extra minutes for the web survey and 10 extra minutes for the telephone survey, taking into account that the key respondent may want to gather information from multiple people when completing the web survey or telephone interview. These increases are modest for two reasons. First, for the web survey, all of the requested information should be available in project files, even if the respondent is not personally familiar with all grants in question. Second, for the telephone interview, Abt staff will assume primary responsibility for contacting additional respondents and gathering any information that the initial primary contact cannot provide.

Exhibit A-1. Estimated Annualized Respondent Burden

Form	(A) Number of Respondents	(B) Average Time to Coordinate Response (minutes)	(C) Average Time to Complete (minutes)	(D) Frequency	(E) Total Burden (Hours) (A*(B+C)*D)/60	Total Cost ($) (E*$60)
Web survey	67	5	20	1	27.92	1675.00
Telephone survey	67	10	45	1	61.42	3685.00
TOTAL		15	65	1	89.33	$5,360.00

The cost burden estimate is based on a quote by a Community Development Specialist in a HUD Field Office who reported that the respondent who would most likely answer the questions makes, on average, $60 per hour including overhead.

13. Provide an estimate of the total annual cost burden to the respondents or record-keepers resulting from the collection (excluding the value of the burden hours in Question 12 above).

The data collection effort involves no recordkeeping or reporting costs for respondents other than those described in item A.12 above.

14. Provide estimates of annualized costs to the Federal government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information. Agencies may also aggregate cost estimates from Items 12, 13, and 14 in a single table.

There are no additional costs to the Government beyond the expense of $243,633 budgeted in the contract with Abt to conduct the survey and telephone interviews. There are no other marginal costs to the Federal Government for this data collection.

15. Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-I.

This request for clearance does not involve a change in burden due to any program changes or adjustments. It concerns a new data collection not previously submitted to OMB for review.

16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

Upon OMB approval, the information generated by the OUP web survey and telephone interviews will be used to develop a research report on the outcomes of OUP grants.

The following sections list the study’s research questions, provide a sampling of planned table shells to illustrate expected tabulations, summarize the final report’s structural outline, and present the evaluation’s timeline.

Research Questions

The study’s research questions are designed to provide HUD with a better understanding of the nature of the activities conducted with OUP funds, their outcomes, and the challenges that grantees faced in implementing the grant activities. Exhibit A-2 shows the main topics of the evaluation and the research question associated with each topic.

Exhibit A-2. Research Question for the Report on Outcomes of OUP Grants

Research Topic	Research Questions
Characteristics of Activities Funded Through OUP	RQ1a: What are the characteristics of institutions that receive OUP grants? RQ1b: What are the long-term goals of OUP-funded projects? RQ1c: What is the expected timeframe for OUP-funded projects? RQ1d: What activities did grantees plan to implement? RQ1e: Who are the beneficiaries of OUP-funded projects? RQ1f: What other funding sources are used to support OUP projects? RQ1g: How do grantees decide which activities to fund and who will be the beneficiaries?
Role of Partner Institutions and Organization in Activities Funded Through OUP	RQ2a: What partnerships were created to support grant implementation? RQ2b: What are the partners’ roles, paying particular attention to the role of the university or college grantee?
Outputs of Selected Activities Funded Through OUP	RQ3a: What activities were completed? RQ3b: What outputs have been realized?
Outcomes of Selected Activities Funded Through OUP	RQ4a: What outcomes have been realized for the communities, populations served and/or university partners? RQ4b: To what extent did OUP funding produce successful community development partnerships?
Factors that Influenced Whether or Not Project Produced Planned Outcomes	RQ 5a: How does partnership structure, especially the role of the university or college, influence the outputs and outcomes grantees were able to accomplish? RQ5b: How does grant purpose (i.e., goals and activities) influence the outputs and outcomes grantees were able to accomplish? RQ5c: How does institution type influence the outputs and outcomes grantees were able to accomplish? RQ5d: How does receipt of multiple grants influence the outputs and outcomes grantees were able to accomplish? RQ5e: What challenges do grantees and their partners face in implementing the program?

Tabulations

After the web survey and telephone interviews have been completed, information gathered from the survey and interviews will be tabulated. The web survey will gather basic information about all activities, while the telephone interview will focus on two randomly selected activities undertaken by the grantee. If one of the randomly selected activities is part of a larger multi-purpose center or campus facility, the evaluation will also report about any additional activities implemented as part of the center or facility.³

The analytical techniques to be used will not be complex. They will be descriptive and will look at data such as the following:

• Distribution of activities by program and activity type

• Percent of activities with measurable outcomes by activity type

• Percent of activities with measurable outcomes by program type

• Outputs and outcomes by activity type

• Additional sources of funding used to support OUP-funded activities

• Common partnership structures

Information from the surveys will be presented using tables similar in format to the shells shown in the exhibits below. As described in Section B, the sample will be stratified between the different OUP grant types. Results will be shown for each stratum.

Exhibit A-3. Distribution of Activities by Program and Activity Type (Illustrative Shell)

	ANNHIAC	HSIAC	HBCU	TCUP	Total
Housing: New construction and rehabilitation	#	#	#	#	#
Public Facilities: New construction and rehabilitation	#	#	#	#	#
Training and Education	#	#	#	#	#
Planning and TA	#	#	#	#	#
Housing Assistance	#	#	#	#	#
Business Development	#	#	#	#	#
Other	#	#	#	#	#
Total	#	#	#	#	#

Exhibit A-4. Outputs and Outcomes by Activity—Employment Training (Illustrative Shell)

	Range	Median
Number of individuals trained by project	#	#
Percent of trained individuals placed in employment	%	%
OUP funds per project (in thousands)	$	$
OUP funds per individual trained (in thousands)	$	$
OUP funds per individual employed	$	$

Exhibit A-5. Partnerships (Illustrative Shell)

	ANNHIAC	HSIAC	HBCU	TCUP
Average number of partners per activity	#	#	#	#
Percentage of partners by partner type:
Community development corporations	%	%	%	%
Community action program (CAP) agencies	%	%	%	%
Neighborhood organizations	%	%	%	%
Faith-based organizations	%	%	%	%
Government agencies	%	%	%	%
Other	%	%	%	%
Average length of partnership (years)	#	#	#	#

Reporting

The analyses and tabulations of data from the OUP web survey and telephone interviews will be presented in a report format. A summary-level outline of the report is shown in Exhibit A-6. The final report will be published by PD&R using guidelines developed for research purposes. PD&R typically disseminates studies and publications through HUDUSER. HUDUSER policies and procedures comply with all applicable Information Quality Guidelines. HUD Information Quality Guidelines were published November 18, 2002. See: http://www.hud.gov/offices/adm/grants/qualityinfo/fr4769n02-final.pdf.

Exhibit A-6. Draft Outline of Report on Outcomes of OUP Grants

Chapter Topic	Chapter Description
Chapter 1: Introduction	Provides an overview of OUP programs, discusses the purpose of study. Reviews study methodology and provides characteristics of study sample.
Chapter 2: Characteristics of Grantees and Partners	Describes grantee institutions and partner organizations
Chapter 3: Characteristics of Activities Funded	Describes funded activities and sub-activities, as well as project goals and beneficiaries. Discusses activity duration/timeframe and sources of funding.
Chapter 4: Program Outputs and Outcomes	Provides an overview of outputs and outcomes by activity or project type. Includes case studies of select OUP grants.
Chapter 5: Factors Affecting Program Outcomes	Discusses activity and project goals as well as grant amount and other sources of funding. Describes characteristics of grantees and partners and discusses challenges in implementation.
Chapter 6: Conclusions and Policy Implications	Summarizes study findings and provides an overall assessment of program success. Discusses opportunities for program improvement and provides recommendations for further evaluation.

Timeline

Upon approval from OMB of the evaluation’s instruments, data collection will begin immediately, followed by analysis and drafting of a final report. Assuming that OMB approval is received by early September, data collection will occur in September and October 2011. Analysis of the web survey data will begin in early November 2011 and analysis of data from all sources will end in February 2012. HUD will review first and second drafts of the report by March 11 and May 9, 2012, respectively. The final report will be delivered by May 23, 2012. In total, data collection, analysis, and delivery of final reports will last approximately nine months.

17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons why display would be inappropriate.

HUD is not seeking an exemption.

18. Explain each exception to the certification statement identified in Item 19, "Certification for Paperwork Reduction Act Submissions," of OMB Form 83-I.

This submission describing data collection requests no exceptions to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).

1 In most cases, grantees will be interviewed about two activities. In the case where a randomly selected activity is implemented within a multi-purpose center, the telephone interview will capture data on all activities related to that center or facility, not just the randomly selected activity. The decision to use this approach is discussed in Part B, Question 2.

2 Again, in the case where a randomly selected activity is implemented within a multi-purpose center, the telephone interview will capture data on all activities related to that center or facility, not just the randomly selected activity. The decision to use this approach is discussed in Part B, Question 2.

3 As explained in Part B, Question 2, the evaluation team decided to ask grantees to report on all activities co-located in a specific center or facility because the pre-test interviews showed that grantees perceive their “center” or “facility” efforts as complete projects. To ask about one activity and not the rest of a center’s activities could be frustrating or confusing to the grantee. Additionally, a more comprehensive approach provides the evaluators with a more complete understanding of the interconnectedness of grant activities.

Abt Associates Inc. Part A: Justification 13