APP Justification Part A

Evaluation and Learning for IMLS's Applying Promising Practices for Small and Rural Libraries (APP) Program

PART A. JUSTIFICATION

A.1. Necessity of the Information Collection

IMLS’s Applying Promising Practices for Small and Rural Libraries (APP) program funds projects that enable small and rural libraries to serve their communities in three practice areas: (1) Digital Inclusion (promoting digital literacy, providing internet access, and enabling community engagement through civic data and civic technology); (2) Community Memory (engaging local communities in the collection, documentation, and preservation of their local histories, experiences, and identities); and (3) Transforming School Library Practice (nurturing school libraries to serve as hubs for self-directed, inquiry-based learning and positioning school library professionals as integral instructional partners to classroom teachers). In addition to making project grant awards, IMLS also supports Communities of Practice (COP) for the grantees in each of the three practice areas, with each COP led by a mentor organization that facilitates communication between grantees, provides expert guidance, and builds grantee capacity in their practice area.

The APP program furthers IMLS’s Strategic Plan 2018-2022 with libraries working to address the needs of their local communities by enabling access to digital content and collections, fostering innovation and learning, and aiming to improve overall community well-being. Across the country, small and rural libraries focus on a wide range of activities within these areas and serve communities with diverse cultural and socio-economic backgrounds, but many of these libraries have similar capacity-building needs, working with small teams and constrained resources. The APP program is intended to help meet these needs, and this evaluation provides a means to assess its first two years of operation in informing future direction. In the process, the evaluation will also serve to inform the library profession as a whole by promoting an understanding of what works and what does not work in building the capacity of small and rural libraries across the United States.

Partners for Public Good (PPG) is conducting this grant program evaluation to help IMLS and other stakeholders better understand the efficacy of different approaches used by APP mentor organizations to improve the capacity of small and rural library grantees to design and implement community projects, the extent to which engagement in a COP contributes to this capacity building, and the extent to which the COP participants (small and rural library grantees) are able to apply newly gained skills to their Digital Inclusion, Community Memory, and Transforming School Library Practice projects. This evaluation will take a close look at the initial implementation of the APP program for each of the COPs with comparisons between and across Cohort 1 (Sept. 2019-Aug. 2021) grantees and Cohort 2 (Sept. 2020-Aug. 2022) grantees.

The evaluation will also take a detailed look at IMLS’s recruitment processes for the APP program in order to understand the extent to which any changes in those processes influenced differences in the starting capacities of Cohort 1 and Cohort 2 grantees and to provide insights for future recruiting efforts.

Finally, as the current implementation of the APP program takes place during the time of the COVID-19 global pandemic, the evaluation will look at the extent to which these circumstances influence the ability of mentor organizations and grantee libraries to implement their activities as originally planned. In the process, it will seek to uncover ways in which grantees were able to adapt to environmental changes.

In addition to IMLS as the grant maker, other primary audiences for this evaluation include the grantees/COP participants (the APP program’s participating small and rural libraries), other funders of libraries and archives, library practitioners, and public policy makers at local, state, and federal levels.

A.2. Purposes and Uses of the Data

This evaluation will collect primary and secondary data to understand how the APP program builds the capacity of small and rural libraries through a mentor-led, COP-centered model. Using a theory of change for the program as a basis for testing (see Appendix F), the investigation will pursue these goals:

1. Understand the extent to which a third-party mentor organization model is effective in building the capacity of small and rural libraries to engage with their communities, and identify best practices in using such a mentor organization model.

2. Understand the extent to which a cohort-based model is effective in building the capacity and connecting small and rural libraries, and identify best practices in using such a cohort model.

3. Understand the extent to which COVID-19 disrupted the planned project and capacity-building activities of the APP program, and how program activities were adapted.

4. Understand the extent to which capacity-building gains are sustained and identify any additional supports needed by small and rural libraries in supporting their communities in the three practice areas of Community Memory, Digital Inclusion, and Transforming School Library Practice.

5. Identify any aspects of the recruitment process and other factors that may have influenced different incoming characteristics between libraries in Cohorts 1 and 2.

Achieving these goals will allow IMLS to test the efficacy of grant making that couples capacity building using cohort-based grant making model with funding for grant awards to inform the future direction of the APP program, as well as to provide insights that can be applied to other IMLS programs, particularly those operating in small and rural communities. These findings additionally are intended to help other funders seeking to support small and rural libraries. Finally, given the disparities of resources for small and rural libraries¹ as compared to larger and more urban-centered libraries, this evaluation has the potential to showcase how resources provided to small and rural libraries can be maximized and how these libraries can be better integrated into the larger library field.

To achieve these goals, Partners for Public Good (PPG) will use a mixed-methods design incorporating quantitative data from surveys and document review with qualitative interview data. Appendix H provides tables that map the evaluation questions to the overarching goals and provide details on the data sources and methods that will be used to answer those questions. Information derived from the data collected will be shared with the grantees, mentor organizations, and IMLS with the intent to both learn from the project for future capacity-building efforts and to inform possible mid-stream changes to the capacity-building programming.

A.3. Use of Information Technology

All survey data in this evaluation will be collected online via SoGoSurvey. The survey instrument includes the instructions, the questionnaire, and the PPG contact information in the event a respondent requires assistance. Key terms are defined in the survey instructions, as well as throughout the survey, as appropriate.

Interviews will be conducted remotely by phone or audio teleconference, and a conference line or Zoom audio link will be provided for the interviewees’ convenience. Interviews will be recorded for use exclusively by the PPG evaluators.

All documents (program descriptions, data collection tools, notes, raw data, recorded interviews, etc.) related to this evaluation will be kept in a secure digital format, accessible to only the PPG evaluation team.

A.4. Efforts to Identify Duplication

The APP program is a new program model for IMLS in the sense that it features a mentor-led, capacity-building curriculum, alongside a group of peers, for small and rural libraries that have received project grants within the practice areas of Community Memory, Digital Inclusion, and Transforming School Library Practice. There is no other data collection being conducted, nor has one been conducted, that duplicates the efforts of this proposed study, and we are not aware of any other studies that demonstrate what it effective in building the capacity of small and rural libraries.

A.5. Methods Used to Minimize Burden on Small Businesses

The small and rural libraries that will be involved in this evaluation will engage with data collection at three points across a two-year period. Each data collection point is voluntary. Average survey times will be kept to 20 minutes or less, with the majority of questions programmed as optional within the survey. Surveys will be open for a minimum of two weeks, and that timeframe may be extended if needed. Libraries will be invited to participate in one, hour-long interview at the completion of their program. Evaluators will be available to schedule these phone interviews at the convenience of the participating libraries. Additionally, evaluators will coordinate closely with mentor organizations and IMLS to understand any other potential requests of cohort members so that the overall burden on the members is minimized.

Unsuccessful applicants that did not receive a grant in the first cohort, and then opted not to apply in the next round of funding will be invited to participate in one online survey. Their participation is voluntary, and the average survey time length will be kept to 20 minutes or less, with the majority of questions programmed as optional within the survey. The survey will be open for a minimum of two weeks, and that timeframe may be extended if needed.

Mentor organizations have expressed an interest in participating in the evaluation and learning from its findings. They will be asked to participate in four interviews (two per cohort) across a three-year period. Each interview will be one hour long, conducted by phone, and scheduled at the convenience of each mentor organization. Additionally, in each quarter without a previously scheduled interview, evaluators will hold a brief 30-minute check-in with each mentor organization so that any programmatic changes (such as responses to COVID-19 or other environmental considerations) are captured and integrated into the evaluation. Where two or more representatives are being interviewed from one organization, the evaluators will be flexible if scheduling constraints determine that it is more convenient for the organization to conduct two separate interviews. Evaluators will make every effort to maximize the time spent with mentor organizations to collect only high-priority, relevant data.

A.6. Consequences of Less Frequent Data Collection

This evaluation will include voluntary data collection across a three-year period (two years per cohort with a one-year overlap). This study will provide IMLS with insights as to the effectiveness of a mentor-led, cohort-based capacity-building model for small and rural libraries. Without this evaluation, IMLS and other funders will have no data on the effectiveness of this type of model to help inform future programming and resource allocation for this population of libraries. Other funders and supporters of libraries, as well as small and rural libraries themselves, will similarly be at a loss for what works well in building the capacity of libraries to better engage and serve their communities.

Conducting data collection less frequently or with fewer participants would reduce the validity and generalizability of the findings and would impede the efforts of IMLS and other funders to understand the most effective models for building the capacity of small and rural libraries. It would also deprive APP grantees and mentor organizations of key learnings related to building their own capacities as mentors and mentees. Removing mid-point data collection in the form of interviews of the mentor organizations and surveys of the grantees would harm the ability to understand disruptions caused by COVID-19 and adaptations occurring in response, including shaping conversations within and across the COPs. This mid-point data collection will supplement the general learning evolved from the pre-test/post-test data collected at the beginning and end of the project.

A.7. Special Circumstances

The proposed data collection activities are consistent with the guidelines set forth in 5 CFR 1320.6 (Controlling Paperwork Burden on the Public-General Information Collection Guidelines). There are no special circumstances that require deviation from these guidelines.

A.8. Consultations Outside the Agency

IMLS published a 60-Day Notice of Proposed Information Collection Request for Comments on the Federal Register on November 4, 2019 (84 FR 59422-59423). Written comments had to be submitted to the Office of Grants Policy and Management, Institute of Museum and Library Services on or before December 31, 2019. One comment was received and responded to. IMLS published a 30-Day Notice of submission for OMB review, comment request, on October 5, 2020 (85 FR 62774).

A.9. Payments or Gifts to Respondents

No payments or gifts will be given to respondents.

A.10. Assurance of Confidentiality

During each primary data collection, respondents will be notified of the following:

• the purpose of the data collection and data collection methods being used;

• that all data collected will be confidential and only reported in aggregate, de-identified form; and

• that participation in the data collection is voluntary.

Access to the raw data collected will be granted only to PPG team members actively involved in the APP evaluation.

PPG uses the cloud-based platform Dropbox to store its files. Dropbox is ISO, U.S. Privacy Shield, SOC 1, 2, and 3 reports, CSA STAR and is HIPPA/HITECH compliant². We also have access to a local secured server that can only be accessed through an encrypted VPN portal. These systems are accessible only to PPG employees.

PPG will protect personally identifiable information (PII) by storing names and other PII separate from the response data collected. PPG will assign linking codes to data elements for each organization and individual respondent (e.g., CMC1P1 to represent the first participant for the first cohort of the Community Memory Community of Practice). The list collating identification numbers with names will be stored in a separate secured folder to which only PPG employees actively involved in the evaluation will have access.

Data sets provided to IMLS at the end of the study will not contain any PII—such as name or address of respondents or their organizational affiliation—that could permit disclosure or identification of respondents, directly or by inference. PPG will destroy all personally identifiable information at the end of the study.

A.11. Justification for Sensitive Questions

No sensitive questions will be asked of study participants as seen in the appendices for the proposed instruments. All questions will be related to the services provided or not provided as a part of the APP program and participants’ views on the extent to which those services contribute to enhancing their work within small and rural libraries.

A.12. Estimates of Hour Burden to Respondents/Table

The evaluation team takes respondent burden seriously and has designed the evaluation with an eye towards minimizing respondent burden without sacrificing data quality. Grantee organizations (i.e., the small and rural libraries participating in the APP program) will be asked to contribute a total of 2 hours each to the evaluation over a 2-year period. Mentor organizations (the organizations selected by IMLS to provide capacity-building support to each Community of Practice within the APP program), who have expressed interest in actively learning from the evaluation throughout its implementation, will be asked to contribute a total of 21 hours each to the evaluation over a 3-year period (e.g., 7 hours per year). IMLS staff members that were involved in APP program recruitment will contribute 6 hours to the evaluation (1 hour per staff member), and APP applicants that were not accepted into Cohort 1 and that did not reapply for Cohort 2 will be invited to commit 10 minutes each to the evaluation. The table below outlines the estimated response times and burden hours for each data collection tool used with each respondent set in the evaluation.

Table A.12 Respondent Burden by Data Collection Tool

Data Collection Tool	Respondent Sets	Number of Respondents	Estimated Response Time	Total Burden hours
Baseline Grantee Capacity Survey – Cohort 1 grantees for each of the 3 Communities of Practice	Grantee organization (library)	37	20 minutes	12.33 (740 minutes)
Mid-point Grantee Capacity Survey – Cohort 1 grantees for each of the 3 Communities of Practice	Grantee organization (library)	37	20 minutes	12.33 (740 minutes)
Endpoint Grantee Capacity Survey – Cohort 1 grantees for each of the 3 Communities of Practice	Grantee organization (library)	37	20 minutes	12.33 (740 minutes)
Endpoint Grantee Interviews – Cohort 1 grantees for each of the 3 Communities of Practice	Grantee organization (library)	37	1 hour	37
Midpoint Mentor Organization Interviews – For each of the 3 Communities of Practice regarding the midpoint of Cohort 1 grantees	Mentor organization	9	1 hour	9
Quarterly Mentor Organization Check-in Interviews (3x/year per cohort)	Mentor organization	9	30 minutes per interview X 6 interviews over 2 years	27
Endpoint Mentor Organization Interviews – For each of the 3 Communities of Practice regarding the endpoint of Cohort 1 grantees	Mentor organization	9	1 hour	9
Baseline Grantee Capacity Survey – Cohort 2 grantees for each of the 3 Communities of Practice	Grantee organization (library)	17+	20 minutes	5.67 (340 minutes)
Mid-point Grantee Capacity Survey – Cohort 2 grantees for each of the 3 Communities of Practice	Grantee organization (library)	17+	20 minutes	5.67 (340 minutes)
Endpoint Grantee Capacity Survey – Cohort 2 grantees for each of the 3 Communities of Practice	Grantee organization (library)	17+	20 minutes	5.67 (340 minutes)
Endpoint Grantee Interviews – Cohort 2 grantees for each of the 3 Communities of Practice	Grantee organization (library)	17+	1 hour	17
Midpoint Mentor Organization Interviews – For each of the 3 Communities of Practice regarding the midpoint of Cohort 2 grantees	Mentor organization	9	1 hour	9
Endpoint Mentor Organization Interviews – For each of the 3 Communities of Practice regarding the endpoint of Cohort 2 grantees	Mentor organization	9	1 hour	9
IMLS staff interviews	IMLS staff member	6	1 hour	6
Non-returning Applicant survey	Applicant organization (library)	72	10 minutes	12 hours (720 minutes)
Total Hours				189 Hours

A.13. Estimates of Cost Burden to Respondents

There is no cost burden for Cohort 1 and Cohort 2 grantees because time required to participate in data collection is incorporated into their APP grant program projects.

The estimated Cost Burden for Non-Returning applicants is $591.60 (see Appendix G).

.

A.14. Estimates of Cost to Federal Government

The cost to IMLS for staff time is estimated at $11,010 and the total cost of the cooperative agreement to PPG for the evaluation is $616,028. The total cost to the federal government is $627,038 (see Appendix G).

A.15. Reason for Program Changes or Cost Adjustments

This is a new submission. There are no program changes or cost adjustments.

A.16. Project Schedule

Evaluation will begin within 30 days of OMB approval. Note: C1 = Participation of Cohort 1 grantees for each of the three Communities of Practice, C2 = Participation of Cohort 2 grantees for each of the three Communities of Practice.

Table A.16 Project Schedule

	Q1	Q2	Q3	Q4	Q5	Q6	Q7	Q8	Q9	Q10
Introductory meetings with mentors and IMLS staff
Baseline Grantee Capacity Survey	C1 C2
Mid-point Grantee Capacity Survey				C1 C2
Endpoint Grantee Capacity Survey								C1 C2
Endpoint Grantee Interviews								C1 C2
Midpoint Mentor Organization Interviews				C1 C2
Quarterly Mentor Organization Check-in Interviews (3x/year per cohort)
Endpoint Mentor Organization Interviews
APP program document review
Webinar observation
Web-portal observation/review
Convening observation
IMLS staff interviews	C2
Non-returning Applicant survey	C2
Recruitment document review	C2
Interim presentation & making meaning					C1 C2
Final report development
Action planning meeting with IMLS
Final convening								C1 C2
Lessons Learned document for the field

A.17. Request to Not Display Expiration Date

We are not requesting an exemption from the requirements to display the expiration date for OMB approval. All data collection materials and documentation will include the OMB approval number and expiration date. 

A.18. Exceptions to the Certification

No exceptions to the certification statement apply to the Accelerating Promising Practices for Small and Rural Libraries (APP) program evaluation. 

1 The Institute of Museum and Library Services. 2019. Public Libraries in the United States Fiscal Year 2016. Washington, DC

2 https://www.dropbox.com/business/trust/compliance

15