Measures that Matter -- Assessing Public Libraries’ Activities Related to Workforce Development
Part B. Supporting Statement
Measures that Matter (MtM) is a collaborative project between IMLS and Chief Officers of State Libraries (COSLA) to improve the library field’s ability to collect, report, and communicate public library outcomes and contributions to the community. COSLA hired Mt. Auburn Associates to pilot a research effort to provide insights into how to measure library activities related to workforce development, a starting component in a broader effort to support public library system’s (“libraries”) capacity to measure their role in promoting community wellbeing.
The pilot project is designed to 1) develop a framework that articulates libraries’ roles in workforce development; 2) explore libraries’ design and delivery of library workforce and development services, engagement with partners, and outcome assessment; and 3) identify potential tools to measure the contribution of libraries on workforce development outcomes to help build library capacity to develop innovative programming.
To investigate the proposed research questions, noted in the table below, Mt. Auburn Associates plans to conduct ten case studies of public library systems engaged in innovation related to workforce development or business development service delivery and measurement. The proposed case studies are intended to build hypotheses related to how these library systems organize around these activities independently or in conjunction with other partners, their role in the broader workforce development ecosystem, and the feasibility and utility of program measurement and communication. These case studies are intended to be illustrative of good examples of workforce and economic development service design and delivery to inform future research and practitioner dialogue.
The following table outlines the research question and their relationship to the case study interview questions. Please see Appendix I: Case Study Interview Protocols for the full set of questions.
Table 1. Research Questions by Responding Case Study Protocol Question
Research Question |
Case Study Questions (See Appendix I) |
|
System-Level Library Staff Interviews Q1, Q2, Q3, Q4, Q5, Q7, Q8
Outlet-Level Library Staff interviews Q1, Q2, Q3, Q4, Q7, Q8
Workforce and business development service provider interviews Q1, Q2, Q3, Q4, Q5, Q7
Civic leaders from business community and public sector interviews Q1, Q2, Q3, Q4, Q5, Q7 |
|
System-Level Library Staff Interviews Q1, Q2, Q3, Q5
Outlet-Level Library Staff interviews Q1, Q2, Q3, Q5
Workforce and business development service provider interviews Q1, Q2, Q3, Q6
Civic leaders from business community and public sector interviews Q1, Q2, Q3, Q5, Q6 |
|
System-Level Library Staff Interviews Q2H, Q6
Outlet-Level Library Staff interviews Q2I, Q6
Workforce and business development service provider interviews Q1D, Q6
Civic leaders from business community and public sector interviews Q6 |
|
All interviews |
B1. Respondent Universe
There are 9,237 public library systems operating in the United States, with many having multiple branches or outlets or approximately 17,000 in total. 1 This study is not focused on this entire universe of libraries but instead focuses on a subset determined to be “influential.” The universe of “influential” libraries is defined by Mt. Auburn Associates as having been reported in a national journal article, book, or blog post as having a noteworthy approach to meeting the workforce or business development needs of patrons; and/or include libraries nominated by the Project Advisory Committee for notable implementation and/or innovation in delivering or measuring workforce or business development services. We estimate the universe of influential public library systems to equal 85.
The case studies will be focused on each public library system to support learning about how workforce and business development services are integrated and managed across outlets. The case studies will have a particular focus on how public library systems think about measuring outcomes and how individual outlets within each system are able to contribute to efforts to deliver services and track outcomes. This is important and unique because the literature review and discussions with the Project Advisory Committee revealed uneven documentation about the quality of these programs, how they are managed across systems, and limited details about programmatic components at the outlet level. In-depth case studies will help build a more rigorous understanding of how these types of efforts are established, how they are designed and implemented across a variety of structures, why measurement may be a barrier, and what hypotheses are emerging that could be more systematically studied in the future.
B2. Potential Respondent Sampling and Selection Methods
Mt. Auburn Associates intends to complete ten case studies in this pilot with each case study based on a public library system. Due to variations in the willingness of the libraries to participate in the case study process, 15 systems will be selected from the universe in order to have a back-up site with similar characteristics.
The selected sample is not intended to provide representative findings that can be extrapolated to the broader population of all 9,237 public library systems nor even the 85 identified as influential. Rather, it will identify sites from the 85 influential systems that include a range and variety of contexts and implementation strategies to understand in more depth specific research questions and uncover potential hypotheses for further testing. Using qualitative methodology, generalization will draw from generated theory, not from a statistical derivation.
The selected case study sites will be drawn using a purposive sample, applying the following criteria:
Diversity of administrative structures, including selection of single outlet systems, multi-outlet systems with less than ten outlets, and multi-outlet systems that have more than ten outlets.
Diversity of state policy contexts. The literature review found that some states have developed specific policies that encourage the inclusion of public libraries in their workforce development systems. The case study sites will include communities in such states where the literature review found specific policies or programs that support library engagement in workforce and business development, as well as include sites in states that do not have targeted efforts in this area.
Variety of workforce and business development programs, including sites that have robust workforce development programs, sites that have robust business development programs, and sites that are doing both workforce and business development. The data on the types of programs will be based on review of secondary and grey literature.
Variety of administrative entities, including sites that that are operated by a municipality or county, sites that are part of a special library district, and sites that are run as nonprofit entities.
Geographic size, including sites that are in rural communities, small and midsized cities, and larger cities or metro regions.
Once selected, the team will plan to interview a variety of stakeholders at each site. The following table outlines the case study activity and participant type by case study activity.
Table 2. Proposed Interview Respondents per Case Study Site
Case study activity |
Participant Information |
Internal sources |
2-8 library staff representing outlet(s) if a multi-outlet system 2-4 library staff representing systems
*Mt. Auburn Associates will do between 4-12 interviews of library staff, with the distribution customized to the relevant structure (e.g. single outlet vs. multi-outlet systems, staffing sizes) |
External sources |
4–6 external stakeholder interviews in each case study |
Total participants for all ten sites |
130 individuals* |
* Mt. Auburn Associates expects to interview 130 respondents across the ten case sites. The composition of internal and external sources may vary somewhat across sites based on local context.
The exact number of library staff and external stakeholders in this case study universe is not fully known since some smaller systems may not have access to the full range of participants noted above. According to the IMLS Public Libraries Survey, the average number of librarians per library system is sixteen (16). With this assumption, there are approximately 1,360 librarians that are involved in “influential” sites. Within this universe, we will be interviewing 4-12 library staff in each of the ten case study sites for a total of an estimated 80 librarians, or about 6 percent of the librarians involved in influential systems. At each case site, we expect a range of 2-8 staff working at the system level and another 2-4 working in the system’s outlets.
The number of “external” stakeholders in the universe of influential systems is less certain that the library staff, but the assumption is that it will vary significantly based on contextual factors in each case study site. These external stakeholders involve: workforce and business development “partner” organizations engaged in active collaboration with libraries; organizations providing workforce or business development services in the region but not engaged in formal library partnership; and civic leaders involved in business organizations, economic development intermediary organization, or the public sector. We plan to survey 4-6 such external stakeholders in each case study site, for an estimated 50 external stakeholders in total across the ten case study sites.
B3. Response Rates and Non-Responses
To maximize the likelihood that a system will agree to participate, the research team will work collaboratively with COSLA in the outreach process. Mt. Auburn Associates expects the following planned strategies to encourage case study participation and mitigate non-response:
Responsive outreach to proposed case study sites to explain the study and describe its utility.
As part of the introductory messaging, explain the relevance of study content and expected products to case study participants to encourage interest.
Assigned Mt. Auburn case study liaisons for clear and fast communication over the course of participation.
Reasonable burden on case study stakeholders for interview participation.
To encourage individual stakeholder participation in case study interviews, Mt. Auburn Associates will work with primary case study contacts to identify relevant staff and partner organizations to participate in planned interview data collection. Based on the long experience of the evaluation team in administering similar workforce and business development studies, Mt. Auburn Associates expects nearly 100% participation from 8-18 respondents per case site using on our focused outreach approach and replacement strategy. When Mt. Auburn Associates does introductory calls with the libraries to identify the interviewee pool, it will develop a list with the understanding that the stakeholders are likely to participate but will ask for secondary options as well. In the event that an interviewee does not respond to requests, Mt. Auburn Associates plans to ask the secondary stakeholders to reach our expected number of respondents.
Mt. Auburn Associates will send an introductory email to all proposed respondents to introduce the project and describe the opportunity and estimated burden level of their participation. It will attempt to primarily schedule interviews in-person while on site, however, to accommodate individual respondents and reduce response burden, it may schedule virtual or phone interviews that meet their availability.
Mt. Auburn Associates expects that these strategies will limit any need for case study sites to drop out of the process, however, we will have five back up case study sites should one site need to back out. Additionally, if individual stakeholders need to drop out of the interview process for any site, Mt. Auburn Associates will use a chain-referral process to identify stakeholder nominees to infill for any dropouts. Select case study sites may also not have enough interviewees to participate as expected because of size. In these cases, Mt. Auburn Associates will note the adjustments accordingly in the text of the case study.
To facilitate ongoing participate in the case study process, Mt. Auburn Associates will identify a “lead” team liaison and a “second” support staff. This team will be consistent throughout the process and will be involved in the baseline research, case study design and planning, all interviews with their site, including an on-site visit, data analysis, and writing up the final case study.
The case study process will start with introductory phone calls with system leads to further the understanding of the library, to identify any potential challenges related to the case study process, and to identify both internal and external stakeholders who system leaders believe would be critical to interview.
Based upon the call, as well as the baseline research, each site liaison will develop a potential list of interviews in the following categories:
Internal stakeholders: The internal interviews will be with staff from the library system, including leadership at both the system and outlet levels, individuals involved directly in workforce or business development activities, and communication and technology related staff.
External stakeholders: The external interviews will focus on stakeholders in the workforce and business development ecosystem and civic leaders in the library’s service area as noted in Table 2 above.
B4. Tests of Procedures and Methods
The Mt. Auburn team has developed interview protocols based on common practices. The interview questions were developed from information gathered during the literature review phase and are structured to reflect the way in which the library field tends to approach workforce or economic development activities. The Project Advisory Committee also completed a review and revision process of the case study interview protocols to verify relevance of questions with project scope and ensure clear, accessible language.
The research team will meet throughout data collection and analysis process to discuss emerging themes. The team will increase inter-rate reliability by directing different researchers to interpret the same data. Mt. Auburn expects to complete interview coding and analysis on an ongoing basis over the course of case study implementation.
Baseline data collection. Prior to doing any interviews, the Mt. Auburn team will conduct baseline research using secondary and grey literature on the communities in which the libraries are located. Areas of research will include a demographic and economic overview, a description of the library system, an assessment of State policy and programs that support library workforce or business development engagement, and an overview of the workforce development and business development ecosystems in the community.
Case study implementation. Mt. Auburn will use semi-structured protocols to guide stakeholder interviews that explore the purpose, implementation, challenges, and successes of libraries’ provision of workforce and business development services. The team expects to engage diverse library and workforce representatives to understand service delivery choices, scale of provision, degree of collaboration across systems, challenges to implementation, measurement activities, and critical success factors.
Interview data analysis. The team will capture in-person and telephone interviews with notetaking and a rough transcript via audio recording and the Otter transcription application with interviewee agreement. Mt. Auburn expects to complete interview coding and analysis on an ongoing basis over the course of case study implementation. The team will use NVivo coding software to analyze transcripts and will develop a coding analysis plan and structure based on the research questions. Using the coding analysis plan, Mt. Auburn will pull relevant sections from the narrative drafts of interview notes and aggregate interview narratives by coding scheme/research questions to analyze for key themes and emerging trends across sites, as well as variation based on site characteristics. The team will increase data validity by exploring inter-rate reliability and directing different researchers to interpret the same data. The research team will meet throughout data collection and analysis process to discuss emerging themes.
Data synthesis: Data analysis and synthesis will result in the following research products:
A final report that includes case studies on each of the library systems included in the research phase, as well as a synthesis report summarizing key themes and findings emerging from the case studies and identifying related hypothesis that could be further tested through future research efforts.
Three targeted research briefs for distribution to the library and workforce development field, including: 1) Innovative practices in libraries’ approaches to workforce development and business development; 2) Making the library workforce development case to your community; and 3) Workforce and business development outcome measurement tools.
B5. Contact Information.
The agency responsible for receiving and approving contract deliverables is:
Office of Research and Evaluation
Institute of Museum and Library Services
955 L’Enfant Plaza North, SW
Suite 4000
Washington, DC 20024
Point of Contact: Emily Plagman, Expert Consultant, P: 202-653-4763, eplagman@imls.gov
The organization responsible for case study design, data collection, and data analysis is:
Mt. Auburn Associates
408 Highland Avenue
Somerville, MA 02144
C: (516) 526-2011
Point of Contact: Beth Siegel, President and Co-Founder, P: 617-625-7770, bsiegel@mtauburnassociates.com
1 https://www.imls.gov/research-evaluation/data-collection/public-libraries-survey
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Emily Plagman |
| File Modified | 0000-00-00 |
| File Created | 2021-11-11 |