Asfr Evaluation Plan

NOFO Simplification Evaluation Plan

Goal: Reduce the burden associated with applying for grant opportunities for qualified applicants, especially those representing and/or providing services to underserved communities.

Evaluation Plan:

1. Objective: Simplify and reorganize NOFO content to help qualified applicants easily access, read, and understand NOFO objectives, program and application requirements, and instructions so that they submit strong applications.

   1. Secondary objective: Standardize applicable NOFO content and provide NOFO writing guidance so grantors can create clear and concise NOFOs more efficiently and receive strong submissions from qualified applicants.

2. Metrics:

From Objective	Metrics
Accessible and usable NOFO	Amount of time spent applying for funding compared to baseline
	% Compliance to 508 requirements
	Ease of NOFO navigation
	# of applications received from applicants representing and/or providing services to underserved communities
	Abandonment rate (# of subscribers and workspaces versus # of applicants)
Readable NOFO	Grade level reading score
	Efficiency/speed: Length of time to read specific NOFO components
	Word count
	Line length (optimal = 50 to 75)
Comprehensible NOFO	Cloze test scores that measure comprehension
	# and types of questions received by program points-of-contact and help desk
	Rating of NOFO ease
	Rating of various individual components
Applicants are qualified and applications are strong	% of disqualification factors and application errors
	% of applicants representing and/or providing services to underserved communities
	% of awards going to applicants representing and/or providing services to underserved communities
	Average score of applications
	Applicant rating of experience/satisfaction on Likert scale
	Grantor rating of experience/satisfaction on Likert scale

3. We’ll make and test the following changes to the NOFOs to determine if we’ve met our objective:

• Reduce the total amount of words and pages in a NOFO

• Eliminate jargon and write NOFOs in plain language at or below a 10^th grade reading level

• Use common naming conventions for the NOFO sections and components

• Reduce the number of steps to apply, including submission requirements & forms

• Adopt a welcoming tone of voice

• Add digital bookmarks for easier wayfinding

• Use codified UX/graphic design principles, including optimal line lengths, breadcrumbs, section titles, typographic hierarchy, and visual reinforcements like iconography

• Adhere to 508 compliance

• Standardize boilerplate language across NOFOs

4. Evaluation (methods paired with metrics from above)

1. Applicant surveys post-submission

Evaluating NOFO prototypes

All applicants will receive an opt-in survey following the submission of their funding opportunity application. By including all applicants, and not just those who replied to the prototype funding opportunities, we can collect baseline metrics.

N = 1000 (70% of NOFO prototype applicants, at least 40% of general NOFO applicants; outreach through GrantSolutions (if possible))

Metrics:

1. Amount of time spent on application (need a longer conversation around what is baseline; potentially asking this question to all NOFO applicants, not just to those who applied for the prototype awards)

2. Rating of overall comprehension

3. Rating of overall cognitive overload

4. Rating of overall NOFO experience/satisfaction

5. Rating of ease of NOFO

6. Write-in sections to learn why applicants had a positive or negative experience; why it did or did not take a long time to put together an application.

2. Moderated applicant interviews

Evaluating NOFO prototypes

Applicants who interacted with one of the four prototypes will be interviewed by evaluation team members about their experience using the NOFO and applying to the funding opportunity. The questions will be the same or like the questions in the survey, but with follow up threads to capture deeper qualitative insights.

N = 200; outreach through Program Leads and through opt-in question on survey.

Metrics:

1. Amount of time spent on application

2. Rating of overall comprehension

3. Rating of overall cognitive overload

4. Rating of overall NOFO experience/satisfaction

5. Rating of ease of NOFO

6. Qualitative: Why applicants had a positive or negative experience; why it did or did not take a long time to put together an application.

3. Moderated usability tests with potential applicants

Evaluating NOFO individual components

Moderated usability tests will be conducted by evaluation team members to assess the effectiveness of individual NOFO components and capture quantitative and qualitative insights from participants. They will include cloze tests to measure comprehension, and comparative usability tests to measure the effectiveness of various designs.

• Cloze tests

  • NOFO prototype program summary paragraphs

  • NOFO prototype eligibility paragraphs

  • NOFO prototype registration instructions

  • NOFO prototype submission requirements

  • All of the above re-written in a more conversation or formal tone

• Comparative Usability Tests

  • One-pagers

  • Submission instructions page

  • Breadcrumb tools

  • Divider pages

  • Matching requirement formulas

  • Table designs

  • Page with iconography versus one without

  • Formatting (landscape vs portrait)

N = 300 potential applicants; outreach using Program Leads as credible messengers/connectors to individuals and trade associations and setting up pop-up testing sites at grant conferences.

Metrics:

1. Cloze test scores

2. Rating of NOFO individual components

3. Qualitative: Why users had a positive or negative experience with the individual components.

Analyzing descriptive measures of NOFO characteristics

Evaluating NOFO prototypes

Descriptive statistics of different features of NOFO prototypes using the metrics below. Compare against the previous year’s versions of these NOFOs (baseline) to measure how they have changed.

E.g., (using example numbers) for NOFO prototypes, the average reading score is grade level 10. This is compared to grade level 13 in the previous year for these same programs, which is a 25% drop.

Metrics:

4. Grade level reading scores

5. Word counts

6. PDF accessibility reports

Analyzing Quantitative measures of the outcomes of NOFO changes

Evaluating NOFO prototypes

Descriptive statistics of program outcomes for NOFO prototypes using the metrics below. Report absolute and comparative measures (using the previous year’s NOFOs as the baseline for comparative measures). Comparative measures should come with the caveat that many other factors, aside from NOFOs, affect program outcomes.

E.g., (using example numbers) for NOFO prototypes, 80% of applicants meet eligibility criteria. This is compared to 70% in the previous year for these same programs, which is a 14% increase.

Metrics:

7. % of applicants that meet eligibility criteria

8. % of applicants that represent underserved communities

9. # and types of questions received (need to develop a baseline that’s more than anecdotal)

10. Average score of applications

11. Abandonment rate