Supporting Statement Part B_REVISED_FINAL

Supporting Statement for the Our Town Implementation Study, Part B

Table of Contents

B.1 Respondent Universe and Sampling Methods 3

B.2 Procedures for the Collection of Information 3

B.3 Methods to Maximize the Response Rates and to Deal with Nonresponse 4

B.4 Test of Procedures or Methods to be Undertaken 7

B.5 Individuals Consulted on Statistical Aspects & Individuals Collecting and/or Analyzing Data 8

Table of Attachments

Attachment A: Communication Materials

Attachment B: Web Survey Instrument

Attachment C: Cognitive Testing Report

Attachment D: Final Study Plan

B.1 Respondent universe and sampling methods

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The potential respondent universe for Phase II of the study is the 381 current and past grantees who received funding from the Our Town program of NEA. Grantees of the Our Town program include local government agencies and nonprofit organizations in local urban, rural, and tribal communities throughout the U.S. The survey is a census; thus, the “sample” is also all 381 current and past grantees. The targeted response rate for the collection as a whole is 80 percent following OMB guidelines, the threshold where potential biases are acceptably small. However, it should be noted that since a survey of this population has not been conducted before, it is difficult to accurately estimate the response rate.

Table 1. Web Survey Respondents

Data Source	Respondents	Response Universe	Estimated Response Rate
Our Town Web Survey	Our Town current grantees	177	80%
Our Town Web Survey	Our Town past grantees	204	80%
TOTAL		381	80%

B.2 Procedures for the collection of information

Describe the procedures for the collection of information, including

• statistical methodology for stratification and sample selection,

• estimation procedure,

• degree of accuracy needed for the purpose described in the justification,

• unusual problems requiring specialized sampling procedures, and

• any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The information collection for Phase II of the Our Town Study will consist of a one-time web survey census of all current and past Our Town grantees. No special sampling, stratification, or estimation procedures are anticipated.

B.3 Methods to maximize the response rates and to deal with nonresponse

Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

The contractor will administer the web survey to all current and past Our Town grantees that did not complete cognitive testing of the survey instrument (381 grantees). Figure 1 provides an overview of the process the contractor (2M Research) will follow to administer the survey, as well as the target number and percentage of grantees that are in the response, nonresponse, or ineligible categories of the total grantee population at each step. First, the contractor, on behalf of the NEA, will mail a letter on NEA letterhead to the contact listed on the grantee’s final descriptive report (for past grantees) or the contact listed on the grantee’s application (for current grantees) that invites grantees to take the survey and introduces the contractor. This initial letter from the NEA is anticipated to be a highly effective strategy to increase response rates. Next, the contractor will follow up with grantees by phone and with an email that provides a link to the survey. As Figure 1 shows, it is anticipated that several of the emails the contractor sends out will be returned as undeliverable. The contractor will attempt to correct email addresses during the follow-up phone call and the invitation letter includes language that asks the recipient to email or call the contractor if there is a better respondent for the survey. If the point of contact (POC) no longer works at the grantee organization and no alternative respondent is identified, or the grantee organization itself is no longer in operation, the contractor will consider the grantee ineligible to take the survey and drop that grantee from the survey sample.

Figure 1. Web Survey Flowchart

Note. Figure 1 contains the target sample sizes and response rates at each step in the survey data collection process. The final numbers and percentages will depend on how many grantees are ineligible to take the survey, and how many respond to the survey.

The target response rate within 2 weeks of the initial email containing the survey link is 40 percent. It is anticipated that about 5 percent of grantees will decline to take the survey at this stage. The contractor will cease contact with grantees that decline the survey and add them to the nonresponse sample. It is also anticipated that about 2 percent of grantees will be found ineligible in the first 2 weeks of data collection.

Two weeks after sending the initial email with the link to the survey, the contractor will begin to email and call grantees to remind them to take the survey. Each grantee will be called up to 3 times. In addition to the initial reminder email, the contractor will continue to email automated reminders to grantees that have no activity on the web survey for 1 week or more. The target response rate by week 8 of data collection (6 weeks after the first email reminder) is 40 percent. Again, it is anticipated that about 1 percent of grantees will decline the survey, and about 1 percent will be found ineligible (see Figure 1).

Next, 8 weeks after the initial email with the link to the survey, NEA will send an email reiterating the importance of the survey to grantees that have not responded; this email will include a link to the survey. It is anticipated that this email will have a significant impact on the response to the survey because grantees are more likely to respond to NEA than to the contractor. The target response rate from the NEA email is 40 percent. It is also anticipated that about 1 percent of grantees will refuse the survey at this point in the process.

Finally, 9 weeks after the initial email, the contractor will call the grantees that have still not responded to the survey. The target response rate for the final grantee calls is 15 percent (72 percent remain nonresponses, and 13 percent identify as ineligible).

The process outlined in Figure 1 shows the steps to arrive at the target response rate of 80 percent for the web survey. With 381 grantees in the total response population, the contractor would reach this target with 64 percent (or 245 grantees) of the total population of grantees completing the survey, 16 percent of the total population refusing (or not responding to) the survey, and 20 percent of the total population identifying as ineligible to take the survey.

In the event that our target response rate of 80 percent is not met, the contractor will conduct a nonresponse bias analysis to ensure there are no significant differences in the response rate of different types of respondents. The nonresponse bias analysis, if applicable, will analyze the response rate based on the following grantee characteristics: (1) the type of lead organization (government agency or nonprofit), (2) the status of the grant (current or past), (3) the budget of the project, and (4) grantee urbanicity (urban/rural/tribal). The nonresponse bias analysis can identify any difference between the total population and survey response sample in terms of these characteristics. If the contractor finds a significant difference, the contractor will then weight the survey data to be more representative of the total population. For example, if the nonresponse analysis finds that the proportion of past grantees in the survey sample is half of what the proportion is in the total population, the contractor would weight the responses of past grantees twice as much as current grantees in the survey sample to compensate. The survey weights will correct for any difference in the characteristics between the grantee population and the survey response sample, as well as reduce bias in the survey. The contractor will perform this step, if needed, prior to conducting any of the analyses of the survey data that we describe below.

B.4 Test of procedures or methods to be undertaken

Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The contractor tested the web survey with seven Our Town grantees in April to June 2018. All seven grantees completed the web survey in a pre-test mode that allowed them to provide comments on the survey questions. In addition, five of the seven grantees completed a 30-minute cognitive interview after testing the survey. The objectives were to detect issues of usability, clarity, and readability in the survey instrument, as well as to estimate the average time to complete the survey. Changes were made to the survey instrument following completion of cognitive testing. The Cognitive Testing Report can be found in Attachment C.

B.5 Individuals consulted on statistical aspects & individuals collecting and/or analyzing data

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Table 2. Individuals Consulted

Name	Title (Project Role)	Organizational Affiliation and Address	Phone Number
Persons conducting the data collection
Debra Holden	Project Director	2M Research 1521 North Cooper Street, Suite 600 Arlington, TX 76011	470-481-6776
James Murdoch	Project Manager and Lead Analyst	2M Research 1521 North Cooper Street, Suite 600 Arlington, TX 76011	817-856-0869
NEA staff consulted
Patricia Moore Shaffer	Deputy Director | Research and Analysis	National Endowment for the Arts 400 7th Street SW | Washington, DC 20506	202-682-5535
Jenifer Hughes	Director | Design and Creative Placemaking	National Endowment for the Arts 400 7th Street SW | Washington, DC 20506	202-682-5547
Katherine Bray-Simons	Grants Management Specialist | Our Town	National Endowment for the Arts 400 7th Street SW | Washington, DC 20506	202-682-5499
Melissa Menzer	Program Analyst | Research & Analysis	National Endowment for the Arts 400 7th Street SW | Washington, DC 20506	202-682-5548