2017FINALSupportingStatement_3095_0067_AASLH

Supporting Statement

NARA Visitors Study

Submitted by:

National Archives and Records Administration

Contact:

Michael Hussey

National Archives

700 Pennsylvania Ave., NW

Washington, D.C. 20408

(202)-357-5469

Fax (202)357-5926

Michael.Hussey@nara.gov

Date:

May 30, 2017

A. Justification

Circumstances Making the Collection of Information Necessary 1

Purpose and Use of the Information 1

Use of Information Technology and Burden Reduction 2

Efforts to Identify Duplication and Use of Similar Information 2

Impact on Small Businesses or Other Small Entities 2

Consequences of Collecting the Information Less Frequently 2

Special Circumstances 3

Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency 3

Explanation of Any Payment or Gift to Respondents 3

Assurance of Confidentiality Provided to Respondents 4

Justification for Sensitive Questions 4

Estimates of Hour Burden Including Annualized Hourly Costs 4

Estimate of Other Total Annual Cost Burden to Respondents or Recordkeepers 5

Annualized Cost to the Federal Government 5

Explanation for Program Changes or Adjustments 5

Plans for Tabulation and Publication and Project Time Schedule 6

Reasons Display of OMB Expiration Date is Inappropriate 6

Exceptions to Certification for Paperwork Reduction Act Submissions 7

B. Collection of Information Employing Statistical Methods

Respondent Universe and Sampling Methods 8

Procedures for the Collection of Information 18

Methods to Maximize Response Rate and Deal With Nonresponse 12

Test of Procedures or Methods to be Undertaken 12

Individuals Consulted on Statistical Aspects and Individuals Collecting and Analyzing Data 12

Appendix

CASRO Code for Respondent Confidentiality 15

Timeline 18

1. Circumstances Making the Collection of Information Necessary

As a part of the National Archives and Records Administration’s (NARA) role to exhibit records and other documentary materials as cited in USC 2109, NARA has created the National Archives Museum. In order to meet the requirements of NARA's strategic plan, specifically the long-range performance target 5.1, it is essential to develop a mechanism to benchmark the performance of the National Archives Museum in relation to other history museums. The National Archives Museum includes the Public Vaults, a Special Exhibition Gallery, Records of Rights Gallery, the William H. McGowan Theater, a learning center, a redesigned Web site, and the preexisting Charters of Freedom exhibit. The purpose of the National Archives Museum is to teach visitors the value of records as part of America. All of these venues are available to the public at no cost. Through access to these components, the goal of the National Archives Museum is to encourage visitors to enjoy the information provided in the National Archives and find a personal connection to American democracy, appreciating the progress made by individual and government contributions.

2. Purpose and Use of the Information

In order to measure whether the National Archives Museum is successfully achieving its goals, as well as to determine if any modifications need to be made, NARA will conduct a visitors study among those who have visited the National Archives Museum using the American Association of State and Local History (AASLH) customer survey form. This will be done by administering a 12-minute questionnaire to a random sample of those exiting the National Archives location in downtown Washington, DC. Among the information to be collected are:

• Overall impact;

• Expectations;

• Presentation;

• Logistics;

• Motivation;

• Demographic profile; and

• Learning experience.

NARA proposes to conduct this study once every two years or twice in a year if there is a significant change to one or more of the exhibits or an addition of a new exhibit. NARA’s goal is to enroll in the AASLH customer survey program for two survey periods 1. To enroll by June 2018 in order to administer the questionnaire beginning on July 16, 2018 and 2. To enroll by June 2020 in order to administer the questionnaire beginning on July 13, 2020. The goal in these two wave of surveying will be to collect 200 completed questionnaires each. For each survey period, this data will then be compiled, cross-tabulated and analyzed by AASLH/Center for Non-Profit Management Nashville (CNM) in order to assist NARA in determining the National Archives Museum’s success in achieving its goals.

3. Use of Information Technology and Burden Reduction

Respondents will complete the survey by filling out a three page paper questionnaire. Those who are unable to complete the questionnaire themselves will have the option of having the survey read to them by a trained interviewer. When possible, questions have been pre-coded in order to make questions as simple as possible for respondents to complete.

4. Efforts to Identify Duplication and Use of Similar Information

A survey effort was administered in 2004 and 2005 when the National Archives Museum first opened (3095-0061 – NARA Visitors Study) and in 2008, 2010, 2014, and 2016. These surveys will be the fifth and sixth opportunity to gauge visitors’ impressions of these venues and continue to make a comparison to the benchmark information. Additionally, by joining AASLH’s customer survey program, the National Archives Museum will benefit by learning how it compares to other similar museums in the survey program.

5. Impact on Small Business or Other Small Entities

No small businesses or small entities will be involved in this study.

6. Consequences of Collecting the Information Less Frequently

Because the National Archives Museum opened in September 2004, the data collection in December 2004, March 2005, and July 2005 provided a baseline to measure the success of the National Archives Museum. Subsequent waves of research have helped to reaffirm the results garnered from the previous survey efforts and measure the impact of any adjustments made to the National Archives Museum based on the findings. NARA chose July 2018 through October 2018 and July 2020 through October 2020 since historically there is a higher volume of visitors during those months.

7. Special Circumstances

The only special circumstance occurring in this information collection effort is in regards to 5 CFR 1320.5 d (2) (ii). Respondents will be asked to complete the questionnaires on-site. Having respondents complete the questionnaire immediately after they have visited the National Archives Museum will ensure that the visit is clear in the respondent’s memory as well as help maintain a higher response rate than might otherwise be possible with other methodologies.

Otherwise this data collection effort fully complies with 5 CFR 1320.5.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency

A Federal Register Notice was published on June 14, 2017 (82 FR 27289). We received no comments.

The last survey of the National Archives Experience was conducted in the summer of 2016. It is typical in the museum field to survey customers every two to three years to get feedback and gauge customer service. NARA is committed to maintaining and improving customer service where needed.

NARA initially consulted with three museum experts regarding AASLH’s customer survey program and form: Cherie Cook, Project Director, Performance Management and Standards Programs, AASLH; Lin Ezell, Director, National Museum of the Marine Corps; and Joyce Jensen, Education Director, U.S. Naval Undersea Museum. Ms. Cooke reviewed the survey questions and procedures in order to ensure that the survey instrument and methodology used would adequately fulfill NARA’s needs. When NARA contacted Ms. Ezell, she was in the process of reviewing the customer survey form to use at her museum and subsequently joined the survey program. Ms. Jensen joined the AASLH survey program and has used the survey. She thought very highly of the program and found that it has provided the U.S. Naval Undersea Museum with valuable feedback and comparison studies.

9. Explanation of Any Payment or Gift to Respondents

No payment or gift is provided to respondents for this information.

10. Assurance of Confidentiality Provided to Respondents

Respondents will not be asked on the questionnaire to provide their name, mailing address, telephone number, or e-mail address. AASLH and NARA abide by the Counsel of American Survey Research Organizations’ (CASRO) Code of Standards and Ethics for Survey Research. Responses are subject to the Freedom of Information Act. However, participation is voluntary and responses are anonymous.

11. Justification for Sensitive Questions

No questions commonly understood to be of a sensitive nature (i.e., pertaining to religious, sexual or other private attitudes or behavior) are included in the survey. When distributing the questionnaires, survey administrators will inform potential respondents that completing the questionnaire is voluntary.

12. Estimate of Hour Burden Including Annualized Hourly Costs

The number of respondents for the survey is estimated to be 200 for both survey periods 1. between July 2018 and October 2018 and 2. between July 2020 and October 2020. The estimated total hour burden is 40, based on each form taking no more than 12 minutes to read and complete. The 12 minutes per form results from previous survey experience at NARA. The annualized hourly cost is estimated to be $2064 based on a GS-8 step 5 ($53,508annually) of the U.S. federal salary scale at a rate of $51.28 per hour and $0.86 per minute ($0.86 x12= $; 200 x $10.32= $2064).

13. Estimate of Other Total Annual Cost Burden to Respondents or Recordkeepers

There are no costs to respondents other than their time to complete the survey. Respondents are not required to keep records to support their response.

14. Annualized Cost to the Federal Government

The contractor’s annualized cost of administering Archives I Visitors Study will be approximately $4,800 in FY 2018 and in FY 2020. The contractor’s price includes the survey template, analysis of the responses, benchmark comparisons, and storage of the data. NARA’s additional costs include: printing of survey forms ($150); pencils ($15); clipboards ($75); nametags ($25); return envelopes ($25); shipping ($100); and travel for on-site presentation and training in Nashville, TN ($800) per person.

15. Explanation for Program Changes or Adjustments

There are no changes in burden.

16. Plan for Tabulation and Publication and Project Time Schedule

AASLH/CNM will keypunch and tabulate all data. At the end of the wave of research, AASLH will provide NARA with a detailed report and executive summary analyzing relevant findings (including what was previously discussed in A.2), and providing explanations and recommendations. AASLH/CNM will analyze the data using a variety of techniques (e.g., t-tests, correlations, discriminate analysis, ANOVA) as is deemed appropriate. The results from this research will help NARA determine the effectiveness of current exhibits in the National Archives Museum, as well as identify any changes that need to be made. AASLH/CNM will also recommend any additional analysis that may be necessary based upon the results of this research.

Study results will be used to improve the value of the National Archives Museum for customers. Reports, articles, and presentations will be developed, as appropriate, to share survey findings with NARA offices and stakeholders. The survey instrument is proprietary material and may not be shared or reproduced without permission from AASLH/CNM. Information from the surveys, including individual narrative comments from the completed surveys, also will be used to illustrate performance accomplishments and supplement the various performance measures NARA gathers and publishes in its annual performance and accountability report.

A tentative timeline for the wave of research is included in the Appendix. If, for whatever reason, the field period is changed, production and reporting dates will be adjusted accordingly.

17. Reasons Display of OMB Expiration Date is Inappropriate

All questionnaires will have an expiration date in the upper right hand corner to meet 5 CFR 1320.8(b) requirements. The questionnaire will display an OMB control number.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

The proposed data collection does not require any exceptions to the certification statement in Item 19 of OMB Form 83-I.

1. Respondent Universe and Sampling Methods

Visitor information for the National Archives states that there are between 1.1 million to 1.3 million visitors per year. The AASLH performance management program recommends at least 200 completed questionnaires. This will result in at most a ±5.8% margin of error at the 90 percent Confidence Level. Standard errors are shown for various percentages in the table below.

Table 1: Standard Error

If the percentage found is around:	50%	40% or 60%	30% or 70%	20% or 80%	10% or 90%	1% or 99%
Then, the standard error, in percentage points would be:
Total Sample (n =200)	±5.8	±5.7	±5.3	±4.6	±3.5	±0.4

2. Procedures for the Collection of Information

We will distribute the survey instrument 5 times per month over a four month period. A total of 10 surveys will be completed per day. At 10 surveys per day, the total data collection period for each wave will take approximately 1 month. In addition, the 1 month period will allow for no one event to overly impact the results. For example, if there is a march or protest scheduled for one of the days during the data collection, it might impact the person interviewed on that day. However, with 15 additional interviewing days, that subset of data will not skew the final results. The table below illustrates the proposed data collection period, with the hours reflecting the National Archives’ schedule.

July 2018	7/16 Mon.	7/17 Tues.	7/18 Wed.	7/19 Thurs.	7/20 Fri.			Total Interviews
	11am-3:00pm 10 surveys	1:00pm -5:00pm 10 Surveys	11am- 3:00pm 10 surveys	1:00pm -5:00pm 10 Surveys	11am-3:00pm 10 surveys			50
August 2018		8/13 Tues.	8/14 Wed.	8/15 Thurs.	8/16 Fri.	8/17 Sat.		Total Interviews
		1:00pm -5:00pm 10 Surveys	11am- 3:00pm 10 surveys	1:00pm -5:00pm 10 Surveys	11am-3:00pm 10 surveys	1:00pm -5:00pm 10 Surveys		50
September 2018	9/17 Mon.	9/18 Tues.	9/19 Wed.	9/20 Thurs.			9/23 Sun.	Total Interviews
	11am-3:00pm 10 surveys	1:00pm -5:00pm 10 Surveys	11am- 3:00pm 10 surveys	1:00pm -5:00pm 10 Surveys			11am- 3:00pm 10 Surveys	50
October 2018	10/21 Mon.	10/22 Tues.	10/23 Wed.	10/24 Thurs.	10/25 Fri.			Total Interviews
	1:00pm -5:00pm 10 Surveys	11am- 3:00pm 10 surveys	1:00pm -5:00pm 10 Surveys	11am- 3:00pm 10 surveys	1:00pm -5:00pm 10 Surveys			50

July 2020	7/13 Mon.	7/14 Tues.	7/15 Wed.	7/16 Thurs.	7/17 Fri.			Total Interviews
	11am-3:00pm 10 surveys	1:00pm -5:00pm 10 Surveys	11am- 3:00pm 10 surveys	1:00pm -5:00pm 10 Surveys	11am-3:00pm 10 surveys			50
August 2020		8/10 Tues.	8/11 Wed.	8/12 Thurs.	8/13 Fri.	8/14 Sat.		Total Interviews
		1:00pm -5:00pm 10 Surveys	11am- 3:00pm 10 surveys	1:00pm -5:00pm 10 Surveys	11am-3:00pm 10 surveys	1:00pm -5:00pm 10 Surveys		50
September 2020	9/21 Mon.	9/22 Tues.	9/23 Wed.	9/24 Thurs.			9/27 Sun.	Total Interviews
	11am-3:00pm 10 surveys	1:00pm -5:00pm 10 Surveys	11am- 3:00pm 10 surveys	1:00pm -5:00pm 10 Surveys			11am- 3:00pm 10 Surveys	50
October 2020	10/19 Mon.	10/20 Tues.	10/21 Wed.	10/22 Thurs.	10/23 Fri.			Total Interviews
	1:00pm -5:00pm 10 Surveys	11am- 3:00pm 10 surveys	1:00pm -5:00pm 10 Surveys	11am- 3:00pm 10 surveys	1:00pm -5:00pm 10 Surveys			50

Simple random sampling, a probability sampling technique, will be used. Survey administrators will attempt to distribute questionnaires to each individual exiting the National Archives according to the distribution time table. This should result in a sample reflective of typical National Archives visitation. All respondents must be 18 years of age or older.

People will be approached in the order that they present themselves. Once 10 visitors have agreed (+1 or 2 extras), no more questionnaires will be collected in that time period. From the extras, a few questionnaires are then available to replace any that fail the quality control criteria (less than ½ completed or a respondent giving all identical answers). Those envelopes containing extras will be labeled as such. The entire group of surveys collected during that time will be banded together and appropriately labeled as to day, date, and time. By approaching visitors this way, there is no reliance on a survey administrator’s judgment or counting. (There is always a bias introduced when judgment is involved no matter what measures are taken.) Informal feedback collected by CNM has revealed that overall at least 1 in 4 visitors participate when asked. In many cases, it’s nearly all.

CNM reports the data as collected and does not try to compensate for those who refuse to participate. While in the analysis CNM calculates and presents the mean ratings, it emphasizes how a given organization is doing as it compares to other organizations – by budget size, geographic location, etc. and over time, of course, when a museum is conducting a second, third, etc. iteration. CNM also focuses on differences across respondent types (such areas as gender, age group, or weekday vs. weekend visits, etc.). That way NARA will learn how it is doing relative to other organizations and across respondent groups. This allows for the best allocation of resources for performance management.

NARA will use its trained volunteers, interns, and staff to distribute the questionnaires as necessary. Survey administrators will work on-site in pairs, with one staff member functioning as a supervisor. They will track the number of completed questionnaires. They will also keep track of non-response rates by using and appropriately filling out all necessary information on a survey log. All survey administrators will be trained on this particular data collection process on-site before their first shift. This training will be conducted by NARA staff. The training procedures will include: appropriate dress, appropriate location within the venue for distribution, brief script for verbal approach, handling refusals, actual survey distribution, etc. During these training sessions, supervisors and survey administrators will be given the training manual which will provide the procedures for properly intercepting visitors as they exit the National Archives and contact information for NARA staff.

Supervisors will be trained how to properly troubleshoot for potential issues that may arise. Supervisors will be responsible for:

• monitoring the survey administrators;

• serving as primary contact with NARA managers, if necessary; and

• accessing back-up survey administrators, if necessary.

Prior to each day’s distribution, survey kits will be assembled containing the following:

• Questionnaires;

• Identification badges;

• Pencils and clipboards.

NARA photograph badges will be worn by supervisors and survey administrators taking part in this project. These badges will allow them to be recognized by NARA and security personnel. Once each day’s distribution is finished, completed questionnaires will be bundled and dated and set aside for shipment later to AASLH/CNM. CNM’s responsibilities will include the following:

• Each survey will first be checked for completeness. Those not at least 50% complete will be put aside and not used.

• Each survey first will be checked to ensure that all answers are not the same. If they are, they are put aside and not used.

• Each survey will then be edited. Questions with more than one answer will be recorded as ‘missing.’

• Each survey will have an I.D. number and the date collected to help with internal organization and data processing.

AASLH/CNM will input and clean the survey data. All data will be edited, cleaned, and tabulated in-house. Survey data will be analyzed by Statistical Package for Social Sciences (SPSS), a data processing software package. In order to ensure that data is accurate, the following checks will be used:

• The keypunched data will be checked using SPSS (a data processing software package). Any inconsistencies will be flagged and the related questionnaires checked to determine what can be done to correct the problem.

• At the completion of data entry, AASLH/CNM will review frequencies of the data to assess whether there are any out-of-range values, invalid values, or incorrectly answered/skipped questions. If any of the above are noted, AASLH/CNM will check the questionnaires and determine how best to resolve the problem.

3. Methods to Maximize Response Rates and Deal With Nonresponse

As part of the training process, survey administrators will be trained in means to reduce refusals (e.g., speaking clearly, explaining the reasons for the study, and being prepared to answer respondent’s questions). Past experience at NARA shows that the response rate is estimated at about 80%.

4. Test of Procedures or Methods to be Undertaken

This is a proven instrument. The AASLH performance management program was piloted with 10 institutions each year for 3 years. Since the pilot stage, the program was formally launched in 2006 and 200 additional museums have taken part.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Debora Wilcox, Director of Evaluation, CNM Nashville

Lin Ezell Director, National Museum of the Marine Corps

Joyce Jensen, Education Director, U.S. Naval Undersea Museum

Appendix

CASRO Code of Standards and Ethics
for Survey Research

Responsibilities to Respondents

A. Confidentiality

1. Since individuals who are interviewed are the lifeblood of the Survey Research Industry, it is essential that Survey Research Organizations be responsible for protecting from disclosure to third parties--including Clients and members of the Public--the identity of individual Respondents as well as Respondent-identifiable information, unless the Respondent expressly requests or permits such disclosure.

2. This principle of confidentiality is qualified by the following exceptions:

a. A minimal amount of Respondent-identifiable information will be disclosed to the Client to permit the Client: (1) to validate interviews and/or (2) to determine an additional fact of analytical importance to the study (including the practice of appending Client-owned database information to the Survey Research Organization’s data file as an analytic aid). Where additional inquiry is indicated, Respondents must be given a sound reason for the re-inquiry; a refusal by Respondent to continue must be respected.

Before disclosing Respondent-identifiable information to a Client for purposes of interview validation or re-inquiry, the Survey Research Organization must take whatever steps are needed to ensure that the Client will conduct the validation or recontact in a fully professional manner. This includes the avoidance of multiple validation contacts or other conduct that would harass or could embarrass Respondents. It also includes avoidance of any use of the information (e.g., lead generation) for other than legitimate and ethical Survey Research purposes or to respond to Customer/Respondent complaints. Assurance that the Client will respect such limitations and maintain Respondent confidentiality should be confirmed in writing before any confidential information is disclosed.

Where Respondent-identifiable data is disclosed to clients so that the Survey Research Organization may analyze survey data in combination with other respondent-level data such as internal customer data, respondent-level data from another survey, etc., it is understood that the information will be used for model building, internal (Survey Research Organization) analysis, or the like and not for individual marketing efforts and that no action can be taken toward an individual respondent simply because of his or her participation in the survey. To assure Client compliance, the Survey Research Organization must obtain written confirmation from the Client before releasing any data. (A suggested CASRO Client agreement clause is available.)

Further, with respect to such research uses as Database Segmentation and/or Modeling (see preceding paragraph), specific action(s) may not be taken toward an individual Respondent as a result of his/her survey information and participation beyond those actions taken toward the entire database population group the Respondent by chance has been selected to represent. In order for such specific action, the following two elements must be met:

The Respondent has first given his/her permission to do so, having been told the general purpose and limitations of such use; and

The research firm has obtained a written agreement from the Client assuring that no other use will be made of Respondent-identifiable information.

Predictive equations which integrate a segmentation scheme into a Client database may be applied so long as no action is taken toward an individual Respondent simply because of his or her participation in the survey. Respondents must be treated like all other individuals in the database according to the segment(s) to which they belong or have been assigned.

b. The identity of individual Respondents and Respondent-identifiable information may be disclosed to other Survey Research Organizations whenever such organizations are conducting different phases of a multi-stage study (e.g., a trend study). The initial Research Company should confirm in writing that Respondent confidentiality will be maintained in accordance with the Code.

c. In the case of research in which representatives of the Client or others are present, such Client representatives and others should be asked not to disclose to anyone not present the identity of individual Participants or other Participant-identifying information except as needed to respond, with the Participant's prior specific approval, to any complaint by one or more of the Participants concerning a product or service supplied by the Client.

3. The principle of Respondent confidentiality includes the following specific applications or safeguards:

a. Survey Research Organizations' staff or personnel should not use or discuss Respondent-identifiable data or information for other than legitimate internal research purposes.

b. The Survey Research Organization has the responsibility for insuring that Subcontractors (Interviewers, Interviewing Services and Validation, Coding, and Tabulation Organizations) and Consultants are aware of and agree to maintain and respect Respondent confidentiality whenever the identity of Respondents or Respondent-identifiable information is disclosed to such entities.

c. Before permitting Clients or others to have access to completed questionnaires in circumstances other than those described above, Respondent names and other Respondent-identifying information (e.g., telephone numbers) should be deleted.

d. Invisible identifiers on mail questionnaires that connect Respondent answers to particular Respondents should not be used. Visible identification numbers may be used but should be accompanied by an explanation that such identifiers are for control purposes only and that Respondent confidentiality will not be compromised.

e. Any Survey Research Organization that receives from a Client or other entity information that it knows or reasonably believes to be confidential, respondent identifiable information should only use such information in accordance with the principles and procedures described in this Code.

f. The use of survey results in a legal proceeding does not relieve the Survey Research Organization of its ethical obligation to maintain in confidence all Respondent-identifiable information or lessen the importance of Respondent anonymity. Consequently, Survey Research firms confronted with a subpoena or other legal process requesting the disclosure of Respondent-identifiable information should take all reasonable steps to oppose such requests, including informing the court or other decision-maker involved of the factors justifying confidentiality and Respondent anonymity and interposing all appropriate defenses to the request for disclosure.

National Archives

March 10, 2018

Estimated Project Time Line

(n=200)

June 1, 2018

Registered with AASLH

July 16 – Oct. 25, 2018

Survey distribution

November 2018 – March, 2018

Data Processing and Analysis

April 2019

Receive Electronic Report and On-site Meeting in Nashville Presentation and Training with AASLH/CNM

National Archives

March 9, 2020

Estimated Project Time Line

(n=200)

June 1, 2020

Registered with AASLH

July 13 – Oct. 23, 2020

Survey distribution

November 2020 – March, 2021

Data Processing and Analysis

April 2021

Receive Electronic Report and On-site Meeting in Nashville Presentation and Training with AASLH/CNM