IMLS Museums for America Evaluation Study

OMB Package

IMLS Museums for America Evaluation Study

Contract 2009-057

.

March 19, 2010

OMB Package

IMLS Museums for America Evaluation Study

Contract 2009-057

Submitted to:

Carlos Manjarrez

Office of Policy, Planning, Research & Communication
Institute of Museum and Library Services
1800 M Street NW, 9th Floor
Washington, DC 20036-5802
Phone: 202-653-4647
Fax: 202-653-4611
email: cmanjarrez@imls.gov

Submitted by:

RMC Research Corporation

1000 Market Street, Building 2

Portsmouth, NH 03801

Phone: 603.422.8888

Fax: 603.436.9166

email: aapley@rmcres.com

March 19, 2010

Contents

Supporting Statement for Paper Work Reduction Act Submissions

Overview 1

Purpose 1

Methodology 1

Data Collection 2

Pilot Testing 3

Timetable 3

Appendices 5

Appendix A 6

Online Survey Outline 7

Online Survey Flow Chart 9

Online Survey Questions 10

Appendix B 33

Case Study Outline 34

Case Study Protocols 36

Justification A & B separate documents

Supporting Statement for Paper Work Reduction Act Submissions

Overview

This document has been prepared to support the clearance of procedures and data collection instruments to be used in the Institute of Museum and Library Services’ (IMLS) study of its Museums for American grant program. This study falls under the IMLS contract 2009-057.

Part A, “Justification,” and Part B, “Information Collection Using Statistical Methods,” respond to instructions in SF93. Appendices contain copies of the instruments and descriptions of procedures for which clearance is sought.

Purpose

The Institute of Museum and Library Services (IMLS) is the primary source for federal support for the nation’s libraries and museums (federal law 20 U.S.C. Section 9171). Museums for America (MFA) is the Institute’s largest grant program for museums and is designed to support projects and ongoing activities that build museums’ capacities to serve their communities. The program serves all museum types (aquariums, arboretums, botanical gardens, art museums, youth and children’s museums, general museums, historic houses and sites, history museums, nature centers, natural historical and anthropology museums, planetariums, science and technology centers, specialized museums, and zoological parks), with the exception of for-profit and federally funded institutions. The Museums for America grant program supports high-priority activities that advance an institution’s mission and strategic goals. From 2004 to 2007 IMLS solicited applications and awarded grants in three categories: Serving as Centers of Community Engagement, Sustaining Cultural Heritage, and Supporting Lifelong Learning. More recently awards are being made in refinements of the earlier categories: Engaging Communities (Education, Exhibitions, and Interpretation); Building Institutional Capacity (Management, Policy, and Training); and Collections Stewardship. 

In response to its legislative authority to conduct analyses on the impact and effectiveness of museum and library services (20 U.S.C. Chapter 72, Section 9108), IMLS proposes to assess the effectiveness and impact of its Museums for America grant-making program. Data from the study will be used to inform IMLS of 1) applicant and grantee profiles, 2) the applicants’ view of the application process, 3) funded project and implementation activities, and 4) organizational and community impacts from funded projects that have completed. Evaluation findings will contribute to enhancing the MFA application process and the grant-making itself. In addition, the study results will highlight the overall accomplishments and impacts the grant program has on the museum field.

Methodology

RMC Research Corporation has designed a two-part data collection effort. The first is an online survey. The survey will solicit information from all applicant institutions (approximately 1,800 institutions) that applied for Museums for America grants from 2004 to 2009. Questions will include perceptions of outreach, grant requirements and assistance, application outcomes for an institution, and the role of IMLS funding. Additional survey questions will solicit data on the project implementation process from all grantees during this period (approximately 850 institutions). These items will include questions about partnerships formed and grant activities. A further set of questions on the impacts of grantee activities will be addressed to the 2004 – 2007 grantees with completed projects (approximately 350 institutions). If a grantee’s project was completed three or more years ago, the grantee will be asked if there were any long-term impacts on the institution or community from the funded project (approximately 250 institutions). See Appendix A for the online survey outline, flow chart, and survey design.

Six case studies will be conducted at selected MFA grantee sites. These case studies will provide a nuanced view of the impacts of Museums for America grantee activities through the close study of six museums with completed MFA projects. Qualitative data gathering at these institutions will allow for targeted and specific inquiries into the factors associated with impacts both within the institution and in the surrounding community. These data will provide an opportunity to investigate the connections among institutional and community characteristics, program implementation choices, and impact outcomes.

The case studies will be based on interviews and focus groups conducted with museum administrators, program staff, and participants as appropriate to each program, accompanied by extensive review of grant applications, annual reports and other documentation, media coverage, and first-hand observations. Two-person teams of RMC staff will conduct each site visit, anticipated to take one-and-a-half days to complete, with extensive pre-visit planning contact, and follow-up telephone interviews as needed.

Case study site visit selection will occur in close consultation with the IMLS. It is anticipated that two studies will be conducted in each of the three grant categories (Serving as Centers of Community Engagement, Sustaining Cultural Heritage, and Supporting Lifelong Learning). In seeking maximum variability among case studies, selection criteria may also include institution size, location (geographic and urban or rural), audience(s) served, and exemplary successes. RMC intends to conduct three (one from each grant category) of the six site visits as pilot cases prior to OMB approval. See Appendix B for the case study outline and protocols.

Because of staff turnover and the potential for new staff’s unfamiliarity with the MFA application process and in order to secure adequate representation to examine the application process by institution size and type, the entire universe of about 1,800 MFA applying institutions will be solicited for the online survey, rather than a representative sample. A survey of all applicants will allow the Institute to garner valuable information about the application process (including its potential value to non-funded museums in clarifying their mission or strengthening their organizations). All 850 institutions awarded grants from 2004 to 2009 will be asked additional questions concerning project implementation, and a subset of 350 grantees with grants completed will be asked questions about the impacts of their grant-related activities. These latter questions will focus on the institutions’ perceived impact of the grants on their operations and their communities.

Data Collection

Online Survey

With the approval of the survey by OMB and the completion of the online forms and database, RMC will begin the implementation of the online survey. Respondents will complete the surveys online using a code and password. The sequence will be as follows:

1. IMLS will send out an announcement via letter of the study including the importance of participation to all Museums for America applicants.

2. RMC will send invitations to Museums for American applicants via postcards addressed to the current director; invitations will direct recipients to the url of the online survey form and contain the code and password to access the survey.

3. Email reminders will be sent to all non-responding applicants.

4. To ensure a high response rate of completed project grantees within the desired analytic characteristics (e.g., institutional types and sizes), follow-up phone calls will be made to non-responding grantees.

5. Completed surveys will be automatically saved and logged into RMC’s secure database.

6. Any outlying or questionable survey data will be verified via email or phone to ensure data integrity.

Case Studies

RMC Research Corporation has developed a menu of case study protocols in order to choose the most appropriate depending on the type of project a grantee undertook. Case study data will be analyzed as they are collected in order to identify the range of responses for each indicator.

Measures to ensure reliability of the qualitative data include: a) consistent note-taking, b) exposure to multiple and different situations using triangulated methods, and c) comparing emerging themes with findings from related studies. Validity will be addressed as follows:

• Face validity (i.e., whether observations fit a plausible frame of reference);

• Internal validity through pre-testing instruments, rigorous note-taking and methods, peer debriefing, and member checks or participant verification;

• External validity through “thick description” and comprehensive description of our methods so that others can determine if our findings compare with theirs.

Pilot Testing

In order to minimize any technical problems and to clarify minor wording or language issues that could cause confusion, RMC will pilot-test the online survey with four applicants and four grantees. The pilot testing will take place after the online survey has been approved by the OMB. In order to avoid duplication of effort, data collected through pilot testing will be used as survey data.

Three pilot case studies are proposed prior to conducting another three site visits after OMB approval.

Timeline

The following chart details the evaluation timeline by task.

Task Description

Jan '10

Feb

Mar

Apr

May

Jun

Jul

Aug

Sep

Oct

Nov

Dec

Jan '11

Develop Evaluation Instruments and OMB package

Develop survey and case study instruments

X

X

Draft OMB documents

X

Submit OMB draft package to IMLS

X

Revise and submit final package to IMLS

X

OMB Submission and Review

X

X

X

X

X

Develop Online Survey & Database

Obtain grantee data and contact information from IMLS

X

X

Create data base of grantees and grants and contact info

X

Develop online survey forms

X

X

X

Pilot respondents to test online version

X

Finalize forms and database

X

Collect Data

Conduct Pilot Case Study Visits (n=3)

X

X

X

Conduct Case Study Visits

X

X

Prepare and contact grantees for online survey announcement

X

Collect online survey data from applicants and grantees

X

X

Send reminders to complete online survey

X

Collect & verify data in database

X

X

Analyze Data

Analyze existing administrative applicant and grantee data

X

X

Analyze survey data for preliminary findings

X

Review preliminary findings with IMLS

X

Conduct final analyses

X

Preparation of Final Deliverables

Write draft report

X

X

Write data file documentation

X

Submit draft report and documentation to IMLS for review

X

Revise report and documentation

X

Design and produce copies of exec summary

X

X

Submit final report, exec summary, and data files to IMLS

X

Send exec summary to responding grantee institutions

X

IMLS Museums for America Evaluation Tasks and Timeline

Appendices

Appendix A

Online Survey Outline

Online Survey Flow Chart

Online Survey Questions

Appendix B

Case Study Outline

Case Study Protocols

Appendix A

Online Survey Outline

Online Survey Flow Chart

Online Survey Questions

MFA Online Survey Outline

March 18, 2010

NOTE: Large institutions serving as a parent to multiple organizational units that have applied to the MFA program will be identified. Each organizational unit will be considered a separate, unique entity. Each organizational unit will be asked to participate in the survey and respond to its MFA experiences. The population of MFA applicants from stand-alone museums and separate organizational units are referred to as museum below.

PART A: ALL APPLICANTS from 2004-2009(~1800 museums)

I. Introduction and Instructions

II. Museum Background

• All Pre-Populated Records of Application History \ Knowledge of each application

• Institutional Type, governance, maturity

• Experience with other grant programs

III. Overall Application Process

• Application Process Experience

• Application Process Impacts

PART B: OVERALL UNFUNDED MFA APPLICATION REACTIONS (~ 1400 museums)

IV. Overall Unfunded Project Experience

• Selection and Award Process

• Reapplication History

PART C: FUNDED GRANT PROJECT ACTIVITIES and STATUS (~850 museums with open or closed projects)

V. Individual Grant Project Descriptions

• Pre-Populated Record of Funded Grant\ Knowledge of the project

• Grant Project Background

  • Partnerships

  • Activities

  • Exhibitions

  • Collections

  • Technology and Online Resources

  • Organizational Development

  • Groups Served by the Project

• Status of Activities

  • Open Projects

  • Closed Projects

PART D: IMPACTS and REFLECTIONS (~350 museums with 2004-2007 closed grants)

NOTE: This section on individual funded project impacts will provide data for identifying patterns of impacts across analytic characteristics such as grant categories, the size of grants, and institution types. In addition, project impacts will be examined by the different types of activities the institution engaged in during the project. The project impact data are NOT intended to trace any direct causal influence from each project activity.

VI. Individual Grant Project Impacts

• Project Impacts

  • Organizational

  • Community

  • Impacts due to Partnerships

• Sustainability and Reflection Questions

PART E: LONG TERM IMPACTS (< 250 museums with projects completed 3 or more years ago)

VII. Individual Grant Project Long-Term Impacts

• Organizational, Community, and Unexpected Impacts

MFA Evaluation

Online Survey Flow Chart

Each institution/organizational unit will be asked to complete at least two parts of the online survey. The number of parts will depend on the status of their applications (any unfunded, any funded), the timing of project awards (<2008, >=2008), and how long ago the project was completed (3 or more years ago). The chart below presents the sections of the survey a respondent will fill out according to the examples provided.

	Examples of: Institution/Organizational Unit MFA application and funding completion status
Target Respondents Online Survey Parts	1 or more MFA applications; none were funded	3 MFA applications; 2 unfunded; 1 funded and currently in process	3 MFA applications; 1 unfunded; 1 funded 2006 completed < 3yrs ; 1 funded 2008 completed 2010	2 MFA 2004-2006 applications; 2 funded; 2 completed 3+ yrs. ago
Part A All Applicants Background Other Grant Programs Overall Application Process Overall Application Impacts	X	X	X	X
Part B Any Unfunded Applicants Overall Unfunded Application Experience	X	X	X
Part C Funded MFA Grantees Earliest Individual Project Background Status		X Project #1	X Project #1
Part D Funded 2004-7 Closed Earliest Individual Project Impacts and Reflections			X Project #1	X Project #1
Part E Funded 2004-7 Closed 3+yrs Earliest Individual Project Long-term impacts				X Project #1
			Return to:	Return to:
Part C Funded MFA Grantees Next Earliest Individual Project Background Status			X Project #2	X Project #2
Part D Funded 2004-7 Closed Next Earliest Individual Project Impacts and Reflections				X Project #2
Part E Funded 2004-7 Closed 3+yrs Next Earliest Individual Project Long-term impacts				X Project #2

Online Survey Questions

Museums for America Evaluation

March 18, 2010

PART A: MUSESUM BACKGROUND and OVERALL MFA APPLICATION PROCESS

COMPLETED BY ALL ELIGIBLE MFA APPLICANTS from 2004-2009

Introduction and Instructions

[Introduction to come: including discussion of survey purpose, and confidentiality]

IMLS will assist in wording of confidentiality

(INFO) = additional definition of terms in pop-up form

Museum Background

This section will be structured at the museum (institution\organizational unit) level (n~1800)

1. Pre-Populated Record <Fields> To Appear On The Online Survey

1.1 Are you knowledgeable about each of the following MFA applications?

 <Institution/Museum Name>  <OrgUnit>

All MFA applications submitted to IMLS by first submit date to last

Knowledgeable about?		Fiscal Year	Project Title	Grant Category	Current Status	Contact Name	Contact Title
No	Yes
No	Yes
No	Yes
No	Yes

2. Museum Background

2.1 Which of the following most closely describes your museum? (select one)

	Aquarium		Historic House/Site		Science/Technology Museum
	Arboretum/Botanical garden		History Museum		Specialized Museum (INFO)
	Art Museum		Natural History/Anthropology Museum		Zoo
	Children’s /Youth Museum		Nature center
	General Museum (INFO)		Planetarium		Other; please specify:_____________________

(INFO) A museum with collections representing two or more disciplines equally (e.g., art and history)

(INFO) A museum with collections limited to one narrowly defined discipline (e.g., textiles, stamps, maritime, ethnic group, tribal)

2. Which of the following most closely describes your museum’s governance? (select one)

	College, university or other academic entity		Federal		Local (county or municipal)
	Non-profit, non-governmental organization or foundation		State
	Native American Tribe/Native Hawaiian Organization

2.2.a. If your museum is governed by a college, university, or other academic entity, which of the following most closely describes your governance? (select one)

	Community college		Private four-year college or university
	Historically Black college or university		State four-year college or university
			Other; please specify: ___________________

  

2. What year was the museum first open to the public? ______

2. How many people have held the museum’s leadership role (e.g., director, curator, president) over the past seven years (since 2004)? ______

2. How many current staff members does the museum have?

___ full-time employees

___ part-time employees

___ unpaid staff (volunteers, docents, Board members)

2. Other Grant Programs

3.1 How many times has your museum applied for any IMLS grant in the last 12 months? ______ _ Don’t Know

3.2. Has your museum applied for any OTHER IMLS grants listed below?

3.2. If yes, were you awarded a grant?

3.3. If yes, was it before or after your MFA application?

Other IMLS Grant Programs	3.2. Did your institution apply?			3.3. If yes, were you awarded a grant?		3.4. If funded, was it before or after MFA application?
	No	Yes	Don’t Know	No	Yes	Before	After
a. 21st Century Museum Professionals	Skip to b		Skip to b	Skip to b
b. Connecting to Collections: Statewide Implementation Grants	Skip to c		Skip to c	Skip to c
c. Connecting to Collections: Statewide Planning Grants	Skip to d		Skip to d	Skip to d
d. Conservation Assessment Program	Skip to e		Skip to e	Skip to e
e. Conservation Project Support	Skip to f		Skip to f	Skip to f
f. Museum Assessment Program	Skip to g		Skip to g	Skip to g
g. National Leadership Grants	Skip to h		Skip to h	Skip to h
h. Museum Grants for African American History & Culture	Skip to i		Skip to i	Skip to i
i. Native American/Native Hawaiian Museum Services,	Skip to j		Skip to j	Skip to j
j. American Heritage Preservation Grants	Skip to 3.4		Skip to 3.4	Skip to 3.4

3.4 Has your museum applied for any federal agency grants other than to IMLS?

No

Yes

Don’t Know

III. Application Process

4.Application Process Experience 

In order to improve the MFA application process, IMLS is interested in your experience(s) with the process.  Please take into account all MFA applications your museum has submitted to IMLS.      

4.1. The IMLS Museums for America grant program requires museums to commit to a 1:1 cost sharing for the proposed project. Between 2004 and 2009, did your institution ever CONSIDER applying for a MFA grant but DID NOT APPLY due to the cost sharing requirement?

No

Yes

Don’t Know

4.2. For the MFA application(s) the museum did submit to IMLS, did the cost-sharing requirement have an impact on any of the following with respect to your proposed project(s):

	No	Yes	Don’t Know
Amount of time to raise the cost sharing funds
Type of MFA grant applied for
Duration of the proposed project
Scope of the proposed project
Staffing costs of the proposed project
Non-labor costs of the proposed project

4.3 Which of the following description(s) apply to the person(s) who wrote your MFA application(s)?

(select all that apply)

	Dedicated staffed grant writer (paid or unpaid)		Administrative Leadership (e.g., Executive Director, Head Curator, President)
	Hired grant writer(s) as consultant		Other staff member(s) (e.g., program coordinator)
	Other: please specify________________________		Don’t Know

4.4 Were you aware of the IMLS outreach activities listed below?

4.5 If yes, did you participate in any of the activities?

4.6 If yes, how helpful were the activities in completing your MFA application?

	4.4. Aware		4.5. If aware, did you participate		4.6. If participated, how helpful was the activity?
	No	Yes	No	Yes	Not at all helpful	Somewhat helpful	Very helpful
a. Information session(s) at conferences/meetings	Skip to b		Skip to b
b. Information from national, regional, or state associations regarding MFA application	Skip to c		Skip to c
c. Audio conference calls prior to application deadlines	Skip to d		Skip to d
Id. ndividual counseling through phone calls, emails, or in-person visits	Skip to 4.7		Skip to 4.7

4.7. Were you aware of the following MFA resources posted on the IMLS website?

4.8. If yes, did you utilize any of the resources?

4.9. If yes, how helpful were the resources in completing your application?

	4.7 Aware		4.8 If aware, did you utilize?		4.9 If yes, how helpful was the activity?
	No	Yes	No	Yes	Not at all helpful	Somewhat helpful	Very helpful
a. Sample MFA applications and tips on IMLS website	Skip to b		Skip to b
b. Outcome-based evaluation resources	Skip to 4.10		Skip to 4.10

10. To what extent do you disagree or agree with the following statements? 

Topic	Statement	Strongly Disagree	Disagree	Agree	Strongly Agree	Don’t Know or Not Applicable
Access to MFA grant information	Information about the MFA grant opportunities was easy to obtain.
Clarity of MFA application	The guidelines and requirements were clear.
	Instructions on budsget (e.g.;direct, indirect costs, cost-sharing) were clear.
	The evaluation requirements were clear.
	Instructions on submitting an application were clear.
	The MFA grant-review process was clear.
Grants.gov	Downloading the Grants.gov application from the website was easy.
	Submitting the online applications through the Grants.gov was easy.
	IMLS staff was helpful in answering questions about using the Grants.gov website.
	Submitting an application by hand and mail was easier than using grants.gov

4.11 Including yourself, has anyone on your staff served as a:

	No	Yes	Don’t Know
Field reviewer for the MFA grant program
Grant review panelist for the MFA grant program

5. Application Process Impacts

5.1. IMLS is interested in hearing if the application process itself, regardless of funding, had any effects on your museum. To what extent did the application process prompt your museum to : 

	Not at All Effected	Somewhat Effected	EffectedA Lot	Don’t Know
Create a mission/strategic plan for the first time
Revise your mission/strategic plan
Increase awareness of different programming
Increase awareness of your resources/strengths
Reframe your project in order to fit into the categories in IMLS guidelines
Adopt new ways of integrating outcomes-based planning in the project development and evaluation processes
Explore new external partnerships
Collaborate among departments/staff
Explore new technologies
Improve your institution's ability to apply for other (non IMLS) funding
Other; please specify:

5.2. Is the MFA application process, more difficult (more complex, requires more time), the same or less difficult (less complex, requires less time) than other types of grant applications?

	More difficult	Same	Less difficult	Don’t Know
Other Federal Grants
Other Government Grants (State, Local, County, etc)
Private Grants

3. Were you aware that IMLS DOES NOT track reapplications, therefore each application, whether a reapplication or not, is treated individually and on its own merits?

No

Yes

1. How likely is it that your museum will apply for another Museums for America grant in the future?

Not at all likely

Somewhat Likely

Very Likely

Don’t Know

5.4.a If not at all likely, why not? (Check all that apply)

	Already have an application or project in process
	Our Mission or strategic is no longer current
	Unavailable resources to plan and write an application
	Unavailable or inefficient resources to carry out a project if awarded
	Finding funds/resources for cost sharing is too difficult
	Funding is not needed at this time
	Don't see ourselves as competitive/ discouraged by previous failed applications
	The application process requires hardware and/or software not available to us
	Other; please specify: ____________________________________________

2. What recommendations do you have for improving the MFA application process?

SYSTEM NOTE : IF MUSEUM HAD ALL APPLICATIONS FUNDED Skip to Part C for individual funded project level.

SYSTEM NOTE: IF MUSEUM HAD ANY APPLICATIONS UNFUNDED Continue with Section B.

PART B: OVERALL UNFUNDED MFA APPLICATION REACTIONS

COMPLETED BY ALL INSTITUTIONS with ANY UNFUNDED MFA APPLICATIONS

IV. Unfunded MFA Project(s) (all institutions with any unfunded MFA applications)

• Structured at the institution\organizational unit level (n=1400)

• Unfunded project questions ask about overall experience (not app specific)

6. Application Experience

1. To what extent do you disagree or agree with the following statements:

	Strongly Disagree	Disagree	Agree	Strongly Agree	Don’t Know or Not Applicable
Our application(s) was/were given full and fair consideration.
The reviewers’ comments were useful for redesigning the project.
IMLS staff was helpful in providing feedback.

2. Did your museum do any of the following with any of the unfunded MFA applications:

Did you:	No	Yes	Don’t Know
redesign an unfunded project and resubmit the application to the IMLS-MFA program?
If YES, did you receive IMLS-MFA funding for the redesigned project?
resubmit a MFA unfunded project to a different IMLS grant program?
If YES, did you receive IMLS funding from a non-MFA grant program?
submit a MFA unfunded project to a non-IMLS funder?
If YES, did you receive funding from the non-IMLS funder?

6.2.a. If your museum did not receive any outside funding for unfunded MFA projects, were you able to execute any version of a MFA unfunded project?

No

Yes

Don’t Know

SYSTEM NOTE:IF NO FUNDED GRANTS - EXIT

Thank you for taking the time to reply to this survey.

Please hit the SUBMIT button if you are done.

SYSTEM NOTE:IF ANY FUNDED GRANTS continue with Part C at the individual funded project level.

PART C: FUNDED GRANT PROJECT ACTIVITIES and STATUS

COMPLETED BY ALL MFA GRANTEES (open and closed projects)

Structured at the museum\project level (n ~ 850)

Pre-populated fields for an individual funded project.

SYSTEM NOTE: Table below will be presented at the top of each new survey page for Section C, D, E

540 museums had 1 funded grant; 160 had 2;

42 had 3-5 THOSE WILL BE ASKED TO ANSWER QUESTIONS ON ONLY 2 awards – respondent will chose which ones based on knowledge

Knowledgeable about?		Grant#/Total Grants	Fiscal Year	Project Title	Grant Category	Current Status	Contact Name	Contact Title
No	Yes

V. Individual Grant Project Descriptions

7. Grant Project Background

7.1. Please identify the types of partner organizations (INFO involved in this MFA project.

(INFO) Partner organizations are defined as any outside organization which expended cash or in-kind resources on the project. In-kind resources include s any payment made in the form of goods and services, rather than cash.

	Other Museums; please identify museum type _(DROP DOWN MENU)_________________________
	Library

Government organizations
	State government
	Local or County government
	City Government
	Other; please describe ___________________________________

Non-Government Organizations/Community Organizations
	Community Health facility (hospital, mental health facility, health clinic, etc)
	Family Services Organization (day care, YMCA, family services center, etc)
	Youth Organization (Boys and Girls Club, 4H Club, Afterschool Program)
	Senior Services
	Arts and Culture Organization
	Legal Services Organization (family court, legal aid organization, etc)
	Local Media
	Civic Organization (Kiwanis, Chamber of Commerce, etc)
	Local Business
	Other Non-Profit Organization; please describe_________________________________
	Other; please describe

Education
	Elementary School
	Secondary/High School
	4 Year College or University
	Community College
	Other; please describe

Policy and Research Organization; please identify ______________

Online/ Technology Partner; please identify ______________________

	Other; please describe _____________________________________
	NO partner organizations were\are involved

7.2. Indicate which of the following types of activities (Programming, Exhibitions, Collections, Technology and Online Resources, or Organizational Development) your museum conducted or is in the process of doing as part of this MFA project.

Programming Activities– If this MFA project included programming activities, please indicate for each activity whether the museum developed a new activity, OR enhanced an existing activity (e.g., expanding for different age groups, upgrading materials) OR continued an existing activity.

	Select one - Radio button for selection
	Develop new programs or materials	Enhance existing programs or materials	Continue existing programs or materials
Concerts
Film festivals
Live Performances
Broadcasts
Demonstrations and Workshops
Lectures
Community discussion groups
Education programs at our institution
In-school programs
Afterschool programs
Curriculum guides
Interpretive programs or materials
Classes or institutes
Training sessions
Conferences
Internships, mentoring or apprenticeships opportunities
Other; please describe

No Programming Activities

B. Exhibitions (select all that apply)

	Upgrade/expanded current exhibit, including making current exhibit accessible (e.g., multi-languages, Braille)
	Develop concept for new exhibit
	Research new exhibit
	Plan new exhibit
	Fabricate a new exhibit
	Create traveling exhibition
	Other, please describe

No Exhibitions

C. Collections (check all that apply)

	Digitize collections
	Convert non-digital content to digital content
	Repurpose digital content (INFO)
	Create new digital content

(INFO) Utilized digital content in a different way (e.g., used for education programs)

If the MFA project involved digitizing the museum’s holdings, what portion of the collections did/would the project cover? (select one only)

			Less than 10% of collections
			Between 10-25% of collections
			Between 25%-50% of collections
			Between 50% and 75% of collections
			Over 75% but less than 100% of collections
			100% of our collections
			Not Sure
	Inventory collections
	Move collections
	Create public collections finding guides
	Create new collections management guidelines/procedures
	Implement new collections management system
	Other; please describe

No Collection Activities

D. Technology and Online Resources (select all that apply)

	Update or create new website
	Create online access to collections records or information
	Purchase technology equipment (e.g., computers, digital cameras, scanner, voice recorder)
	Consolidate multiple databases
	Develop searchable online database
	Upgrade, purchase or install new software (e.g., new collections management software)
	Create audio tour
	Create online exhibition
	Develop high-tech interactive exhibition
	Other; please describe

No Technology Activities

E. Organizational Development, including Staffing Resources (select all that apply)

	Provide staff, volunteer and/or docent training, including in use of new technology and online resources
	Hire full-time staff (e.g., program coordinator, director, curator, educator)
	Hire part-time staff (e.g., program coordinator, director, curator, educator)
	Hire a consultant (e.g., interpretive, planning, education, exhibition, web)
	Contract for services (fabrication, design, security, etc)
	Train interpreters, volunteers or docents
	Create or expand interpreter, docent or volunteer program
	Support a research and evaluation program, including conducting surveys
	Develop key management plans (e.g., comprehensive interpretive plan, emergency plans, collections management plans)
	Other; please specify

No Organizational Development

7.3. Which of these groups were\are directly served by this project? (select all that apply)

	Paid staff; please describe
	Unpaid staff (docents, interns, volunteers); please describe
	Adult learners
	Seniors
	Parents
	Youth
	Teachers
	Students
	Administrators
	Professionals
	Residents of particular neighborhood or community/geographic area within a community
	Policymakers
	Persons with disabilities
	Low income citizens
	Specific racial/ethnic communities
	Families
	Not group-specific
	Other; please specify

7.4 Age groups served by your grant project (select all that apply)

	Preschool
	Grades K-5
	Grades 6-8
	Grades 9-12
	Post high school/Young adults (18-25)
	Adults (26-64)
	Seniors (65 and older)
	Multi-age/Family
	Other, please specify

8. Status of Activities

Status of activities –the list of activities that grantees check off in the previous section will be populated to this section.

OPEN GRANTS ONLY

8.1 What is the completion status of the project’s activities?

Example:

	Completion Status
<populated with specified activities>	Will not do as planned	Still in progress	Completed	Don’t Know
Afterschool programs
Curriculum guides
Train interpreters or docents
Establish a docent program and recruit docents
Implement a research and evaluation program, including conducting surveys

8.2. Were any other activities conducted or planned?

Yes

No

If yes, please list activities and current status of each below

	Completion Status
Other activities:	Will not do as planned	Still in progress	Completed

CLOSED GRANTS ONLY

8.1 What is the completion status of the project’s activities?

Example:

<populated with specified activities>	Did not do as planned		Completed	Don’t Know
Afterschool programs
Curriculum guides
Train interpreters or docents
Establish a docent program and recruit docents
Implement a research and evaluation program, including conducting surveys

8.2. Were any other activities conducted or planned?

Yes

No

If yes, please list activities and current status of each below (e.g., Completed, Still In Process

Other activities:	Did not do as planned	Completed	Don’t Know

SYSTEM NOTE/NOT DISPLAYED:

IF this project is in progress OR recently completed AND the museum has no additional MFA granted projects EXIT

OTHER COMMENTS:

As a thank you for your time and interest in participating in this study, IMLS would like to send you a copy of the Museums for America Grant Evaluation Executive Summary.

If interested please fill out the following information:

Name:

Title:

Organization:

Address: City State Zip

Thank you for taking the time to reply to this survey.

Please hit the SUBMIT button if you are done.

SYSTEM NOTE/NOT DISPLAYED:

If this project is completed AND funded under the 3 earlier grant categories, THEN respondent will continue on to Section D: Impacts and Reflections.

If this project is in progress OR recently completed AND the museum has more MFA granted projects THEN the respondent returns to Section C and is asked questions about the next funded project.

PART D: IMPACTS and REFLECTIONS

COMPLETED BY MFA GRANTEES WITH COMPLETED GRANTS UNDER THE EARLIER 3 GRANT CATEGORIES 2004-2007

Structured at the museum\project level (n ~ 350 with ~ 22 with 2 completed projects)

VI. INDIVIDUAL Grant Project Impacts

9. Project Impacts

1. Please review the list of impacts this MFA project may have had on the organization or community.

Select each impact resulting from the MFA project.

ORGANIZATIONAL IMPACTS (select all that apply)

Programming
	Grant-funded programming continued beyond grant
	Sustained grant-funded programming
	Did not sustain grant-funded programming
	Sought additional funding to sustain grant-funded programming
	Developed related programming to expand value of grant-funded program
	Expanded our range of possibilities for programs or exhibitions
	Other; please describe:
	No changes in programming/not applicable to grant

Organizational Capacity
	Enhanced staff capacity in program development
	Enhanced staff capacity in creating new kinds of exhibitions
	Enhanced staff capacity in reaching new or larger audiences
	Enhanced staff capacity in forming external partnerships
	Enhanced staff capacity in working internally across departments
	Enhanced staff capacity in using outcomes based evaluation
	Improved staff cohesion and commitment to mission
	Greater alignment of staff responsibilities with mission
	Greater board involvement
	Helped institution fulfill its mission (e.g., meet certain benchmarks)
	Increased ability to attract outside funding
	Decreased ability to attract outside funding
	Other; please describe:
	No effect on our organization/not applicable to grant

COMMUNITY IMPACTS (select all that apply)

Audiences
	Reached new audiences (e.g., youth, families, minorities)
	Increased commitment by existing audiences
	Changes in regular audience participation
	Increased audience access (e.g., expanded hours, mobile programming)
	Other; please describe:
	No effect on audiences/not applicable to grant

Community Profile
	Changed institutional identity
	Strengthened museum’s public image
	Raised institution’s prestige in its community
	Increased local media coverage of institution’s activities
	Increased visibility of institution as a center of community learning
	Other; please describe:
	No change to our profile/not applicable to grant

Partnerships
	Improved skills in building partnerships
	Strengthened ongoing partnerships
	Developed new partnerships
	With schools or Institutes of Higher Education
	With youth organizations
	With other museums/historical societies
	With other community organizations
	Developed new types of partnerships
	Decreased ability to attract partners
	Increased ability to attract more partners
	Other; please describe:
	No effect on partnerships/not applicable to grant

IMPACTS DUE TO THE PARTNERSHIP(S) (select all that apply)

	Brought in new audiences
	Brought in new resources not normally available to our organization
	Brought in new staff, paid or unpaid
	Brought in new volunteers or memberships
	Raised community awareness of our organization
	Other; please describe:
	No effect from partnerships/not applicable to grant

9.2. Are there any other impacts not covered above? If yes, Please describe

__________________________________________________________________________

9.3. If the grant project enabled new programming, to what extent has your organization been successful in continuing the programs?

Not at all successful

Somewhat successful

Very successful

Not applicable

9.4. If the grant enabled you to bring in new audiences, to what extent has your organization been successful in sustaining these new audiences?

Not at all successful

Somewhat successful

Very successful

Not applicable

9.5. Was this MFA grant from IMLS larger, smaller, or about the same as other grants you may have received?

Larger

Smaller

About the same

Not applicable

9.6. Were the MFA grant funds sufficient to complete the planned project activities?

No

Somewhat

Yes

Don’t Know

SYSTEM NOTE/NOT DISPLAYED :

IF this project ended less than 3 years ago AND the museum has no additional MFA granted projects EXIT

OTHER COMMENTS:

As a thank you for your time and interest in participating in this study, IMLS would like to send you a copy of the Museums for America Grant Evaluation Executive Summary.

If interested please fill out the following information:

Name:

Title:

Organization:

Address: City State Zip

Thank you for taking the time to reply to this survey.

Please hit the SUBMIT button if you are done.

SYSTEM NOTE/NOT DISPLAYED:

If this project was completed 3 or more years ago THEN respondent will continue on to Section E: Long-term Impacts.

If this project is in progress OR recently completed AND the museum has more MFA granted projects THEN the respondent returns to Section C and is asked questions about the next funded project.

PART E: LONG-TERM IMPACTS

COMPLETED BY grantEEs WHERE PROJECT WAS completed 3 or more years ago

VII. Individual Grant Project Long-Term Impacts

10.1. Please describe any long-term impact(s) that the MFA project has had on your organization.

10.2. Please describe any long-term impact(s) that the MFA project has had on your community or community relationships.

10.3. Please describe any UNEXPECTED long-term impact(s) that the MFA project has had on your organization, community or community relationships.

SYSTEM NOTE/NOT DISPLAYED :

IF this project is the only/last one completed under the 3 earlier grant categories THEN GO TO EXIT

If there is another MFA project completed under the 3 earlier grant categories THEN the respondent returns to Section C and is asked questions about the next funded project.

EXIT

OTHER COMMENTS:

As a thank you for your time and interest in participating in this study, IMLS would like to send you a copy of the Museums for America Grant Evaluation Executive Summary.

If interested please fill out the following information:

Name:

Title:

Organization:

Address: City State Zip

Thank you for taking the time to reply to this survey.

Please hit the SUBMIT button if you are done.

Appendix B

Case Study Outline

Case Study Protocols

Case Study Outline

Museums for America Evaluation

Overview

Case studies will provide a nuanced view of MFA impacts through close study of a select group of completed MFA grants. These case studies will be based on a series of qualitative data collection activities conducted at selected institutions, and will allow for targeted and individual inquiry into the factors producing project impacts, both within the institution and in the community.

The case studies will be based on interviews and focus groups with museum administrators, program staff, and participants as appropriate to each program, as well as extensive review of grant applications, annual reports and other documentation, media coverage, and first-hand observations. Case studies will provide an opportunity for investigating the connections among institutional and community characteristics, program implementation choices, and impacts.

Two-person teams of RMC staff will conduct each site visit. Each visit is expected to take one-and-a-half days to complete, with extensive pre-visit planning contact, and follow-up telephone interviews as needed.

Site Selection

Six case studies will be conducted, and will include a mix of each of the three MFA grant categories (Serving as Centers of Community Engagement, Sustaining Cultural Heritage, and Sustaining Lifelong Learning). In seeking maximum variability among case studies, selection criteria may include:

• A range of project activities

• Institution size, type and location variety

• Project director and staff still employed at the museum

Selection of the case study sites will be made in close collaboration with the IMLS. As part of this process, RMC will review relevant grantee reports. RMC will make the final selection based on recommendation of 10-12 projects/institutions by IMLS program officers.

Site selection will conclude with a discussion between IMLS and RMC about the significant factors leading to the particular selections, i.e. the successes or outstanding features of each of the selected grant experiences.

Visit Preparation

Pre-visit discussions will be conducted with the project director and/or other appropriate institution representation to identify optimal interview and focus group subjects and to develop a schedule for the visit that will use the available time to maximum effect.

Possible on-site interviews (approximately six per site) will be conducted with the institution’s project director, executive director, and management or administrative person, project staff, partners, and/or audience members. These may include both paid and unpaid staff and volunteers. Focus group subjects will be drawn, again as appropriate, from community and audience members, education and public programming staff, volunteer staff and interns, community activists, and representatives of school and other partner organizations. RMC will also review available documentation of program effects, such as attendance patterns and histories.

Protocols

RMC will develop protocols for the following categories of interviewees:

• Project Manager(s)

• Museum Leadership

• Project Staff/Educators

• Partners

• Community/Audience Members

The interview and focus group protocols will be tailored to the specifics of the institution and grant project prior to each site visit.

Case Study Protocols

Project Manager Interview

Interviewee Name: _______________________ Title: ______________________

Institution: ___________________________ Date:_________ Interviewer: ____

Application Process

1. Could you share with us the history of this project? How did the idea originate?

2. How did you decide to apply for the IMLS MFA funding? Did you apply more than once? If you applied more than once, did you get feedback and was it helpful?

3. How would you characterize the application process compared with other grants you have applied for? How does it compare to other Federal grants you’ve applied for?

4. Did you find the application process valuable? If yes, what was most valuable about it (e.g. what did you learn about your institution, project development, etc.)?

5. What was most challenging about completing the application?

6. How did you approach the cost-sharing requirement?

Project Implementation and Sustainability

1. To what extent was the project successful in advancing your institution’s mission? How does this project align with your institution’s mission?

2. What aspects of the project were most successful?

3. What challenges did your institution encounter in implementing the project?

4. What was the impact or value to your institution of partnerships developed through the project? Have these partnerships been maintained, and if so, how?

5. What does sustainability mean to you in the context of this project? (e.g., exhibits or programming continue, new audiences engaged, continuation of new offerings, access to collections or other resources, etc.) How was this project or component parts supported after the grant period?

Community Impacts

1. How did the target audience and/or your existing audience respond to the program? What trends did you observe over time?

2. How has this project changed the museum’s reputation in the community?

3. How has the community’s involvement with your museum continued or grown as a result of the program?

Organizational Impacts

1. In what areas, if any, has the capacity of your staff and/or institution been improved (e.g., growth in knowledge, skills, efficiency, programming)? To what extent were these capacity changes the result of new staff, professional development, strategic planning, budgeting changes, or new technologies?

2. To what extent, if any, did the project support the development of new relationships within the institution, such as across departments?

3. Have there been any outcomes of this improved capacity? If so, what? (e.g. additional new programs or initiatives)

4. Did the project result in any changes in how individuals in the organization perceive the institution’s mission and/or value to the community?

Planning and Evaluation

4. To what extent were the outcomes-based planning and evaluation requirements familiar to your or your institution?

5. Do you view planning and evaluation differently as a result of this project? Please explain your response.

6. To what extent are you integrating outcomes-based planning and evaluation in subsequent projects?

Summary

1. Overall, what would you say is the most important impact of having received this grant from IMLS?

2. Were there any surprises or other impacts you did not expect or foresee?

3. How would you describe the value of the MFA program for institutions like yours?

4. Any other comments?

Project Staff

Interviewee Name: _______________________ Title: ______________________

Institution: ___________________________ Date:_________ Interviewer: ____

1. Describe your involvement with this project. What was your role?

2. How would you describe the implementation of this project? Did it go smoothly?

3. What new skills or knowledge did you gain from this project?

   1. How have you been able to use these skills or knowledge in subsequent projects?

4. Did the grant have an impact in how you do your work? If so, explain.

5. Are there any other ways in which the grant changed how you do your job, your responsibilities, or improved your ability to do your job?

6. To what extent were you involved in evaluation aspects of the project?

   1. If you were involved in these, were the IMLS requirements valuable?

   2. What challenges did you face in meeting the evaluation requirements? How might these requirements better fit the needs of the MFA projects?

   3. What did you learn from the experience?

7. Are there other ways in which this grant affected your institution’s ability to serve the public?

   1. What kind of community feedback have you received about the project?

8. What other aspects of the project have been sustained since the grant period ended? Please describe. How were these projects supported after the grant period?

Summary

9. Overall, what would you say is the most important impact of having received this grant from IMLS?

10. Were there any surprises or other impacts you did not expect or foresee? Please describe.

11. How would you describe the value of the MFA program for institutions like yours?

12. Any other comments?

Museum Leadership Interview

Interviewee Name: _______________________ Title: ______________________

Institution: ___________________________ Date:_________ Interviewer: ____

Overall Impacts

1. Please describe the value of this grant for your institution in terms of your mission (e.g., offerings, audience, efficiency).

   1. How does this grant relate to what you consider your institution’s strength in serving the community?

   2. How does this grant relate to what you consider its greatest challenges?

2. What aspects of the project were most successful?

3. What challenges did your institution encounter in implementing the project?

4. How has this project impacted what your institution does?

   1. [If a conservation grant, probe about conservation practices after grant. What changed?]

   2. [If an outreach grant, probe about outreach practices after grant. What changed?]

   3. Etc.

Community Impacts

5. Did this project change the museum’s reputation in the community? If so, how?

6. How has the community’s involvement with your museum continued or grown as a result of the program?

7. Has the museum’s level of engagement with the community changed over time? If so, how? To what do you attribute that change?

IMLS Grantee

8. Are you regular applicants to IMLS or other federal funders? Was this the first IMLS grant your institution has received or one of many? First MFA grant?

9. How important was receipt of this funding for the institution?

   1. How did the size of the grant compare with other grants received?

   2. What kind of prestige is conveyed by receipt of the grant?

   3. Were you able to leverage the funds or the prestige in additional ways?

10. In what ways did the application process or the implementation of the grant impact how you understand your mission or how you understand how you can achieve it?

11. How would you describe the value of the MFA program for an institution like yours?

12. What challenges have you faced in sustaining the project?

13. What new opportunities for sustaining the project have emerged?

Planning and Evaluation

14. To what extent were the outcomes-based planning and evaluation requirements familiar to your or your institution?

15. Do you view planning and evaluation differently as a result of this project? Please explain your response.

16. To what extent are you integrating the outcomes-based planning and evaluation by IMLS in subsequent projects? Please describe current efforts.

Community Member Interview

Interviewee Name: _______________________ Title: ______________________

Institution: ___________________________ Date:_________ Interviewer: ____

1. How were you connected to the IMLS MFA grant project (attended exhibit or program, community partner, volunteer or docent at the museum, etc.)?

2. Have you had any dealings with the museum in the past? If yes, what types of activities or programs did you conduct with the museum?

3. What did you like most about this project?

4. What did you learn from your involvement in this project?

5. What value did this project provide to the community?

6. Did the project have an impact on your impression of the museum? If yes, please describe.

7. Did the project have an impact on your attendance at museum events or involvement in museum activities?

8. How would you characterize the relationship between the museum and the community?

9. Do you know of other community groups that participate in programming with the museum? If so, what types of activities are they engaged in?

10. Has the museum’s level of engagement with the community changed over time? If so, how? To what do you attribute that change?

Partner Interview

Interviewee Name: _______________________ Title: ______________________

Institution: ___________________________ Date:_________ Interviewer: ____

1. What was your institution’s role in the MFA project? To what extent were you involved in developing the project? To what extent were you involved in the grant application process for this project?

2. What was your interest in participating in this project? In what ways was this project valuable for your institution?

3. Was this the first time you worked with the museum? If so, did this project represent a change in the types of collaboration you have engaged in previously?

4. How was the partnership structured?

5. How has your relationship with the museum changed as a result of this project?

6. Have you continued to work together as a result of the project? If no, why not? If yes, is this work on the same project or on a new project?

7. Do you anticipate future partnerships with the museum? If so, what kinds of partnerships you would be interested in pursuing? What aspects of the partnership would you continue? Do differently?

8. How would you characterize the relationship between the museum and community organizations such as yours?

9. Do you know of other organizations that participate in programming with the museum? If so, what types of activities are they engaged in?

10. Has the museum’s level of engagement with the community changed over time? If so, how? To what do you attribute that change?