Justification B VA Education Program Research Final 10.2015

Development of a Historic Resources Educational Program

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

NCA understands that Section B is required for most research collections, program evaluations and other collections containing surveys, censuses, or otherwise employing statistical estimation techniques such as sampling or imputation, or other statistical estimation technique.

This data collection effort is a survey of a very small universe of directors and educational directors who develop and/or make decisions about educational programs in libraries; museums; and national associations and local organizations with interests related to war, veterans, preservation and history. This data collection has following characteristics:

• The study is formative and exploratory in nature. Its purpose is to provide initial direction to NCA as it develops preliminary conceptual frameworks and educational programs that can be used by institutions and organizations that serve the public and are interested in using programs developed by NCA. The research will be used to more clearly define target markets and preliminary program topics and delivery methods.

• For some categories, we will send invitations to directors or educational directors of all of the organizations identified. When used, sampling will be purposive rather than random. For example, libraries and museums related to war and veterans will be given preference because they are most likely to represent the population of interest – that is, the potential target market for NCA educational programs.

• No statistical estimation will be used. Respondents will be portrayed simply as “respondents to the survey,” with no claim that they represent a particular defined universe. Their opinions will be portrayed as the “input of interested parties” rather than the opinions of a greater universe.

• Analysis and interpretation of the results will show percentages graphically in order to show the general magnitude of the results. However, percentages will not be cited in text. Instead, we will use terms like “most” or “few.”

Thus, this data collection effort is essentially qualitative in the sense that (1) we seek to reach the individuals in organizations that have some interest in the topic and can give us the input needed to design an educational program that fits their needs and (2) there will be no claim that the results represent a defined population with a particular degree of precision.

1. Provide a numerical estimate of the potential respondent universe and describe any sampling or other respondent selection method to be used. Data on the number of entities (e.g., households or persons) in the universe and the corresponding sample are to be provided in tabular format for the universe as a whole and for each stratum. Indicate expected response rates. If this has been conducted previously include actual response rates achieved.

	Description of Stratum	Max. # Of Organizations In Population*	Goal: Responses	Estimated Response Rate
1	Libraries	817	50	TBD
2	Museums, Historical Societies	261	50	TBD
3	National and Community Organizations	1,438	100	TBD
	TOTAL	2,515	200	TBD

*Some, perhaps many, of these organizations will be eliminated after we receive lists and vet them.

Since this is the first effort to survey this population on this topic, we do not have a precedent for making an estimate of the response rate.

1. Describe the procedures for the collection of information, including:

Sampling and Stratification and Unusual problems Requiring Specialized Sampling Procedures

NCA and its contractor, ADRG, is working with Dun & Bradstreet, to compile lists of the types of organizations described above, which comprise the strata for this study. As noted earlier, our methodology does not include traditional survey sampling, which would inappropriate for such a small and highly targeted audience.

In some cases, we will send email invitations to directors and/or educational directors in all of the institutions or organizations identified in a category. In others, we will purposeful selections. Libraries and museums with “war,” “veterans,” or “preservation” in their names would have higher probability of selection. Certain national organizations, such as the American Legion and the American Historical Society, will always be selected.

Screening. To assure that we are reaching the intended audience, we will screen respondents to make sure that they make or share in decisions about educational programs for their organization. This screening question will further reduce the number of qualified respondents.

Estimation Procedure

No estimation procedures will be used for this survey.

Degree of Accuracy Needed

In survey research, the term “accuracy” refers to the margin of sampling error (MOSE), which is based on the assumption of random sampling. Since no random sampling is planned for this study, MOSE should not be used. We believe that the planned number of at least 200 responses is adequate to obtain the directional information required by NCA as it develops preliminary conceptual frameworks for educational programs. Also, given the relatively small number of potential respondents, 200 respondents is a practical goal for this study.

Less Frequent Data Collection to Reduce Burden

This is a one-time data collection effort.

Methods to Maximize the Response Rate and Deal with Issues of Non-Response

We plan to take the following steps to maximize response.

• Send out email invitations in batches and monitor response closely

• Send up to four email reminders to non-respondents. Email invitations and reminders are attached at the end of this document.

• Since Dun & Bradstreet has telephone numbers for most potential respondents, we will call (1) those for which we do not have an email address and (2) those that have not responded to the email invitation.

• If we cannot reach the respondent by phone, we will leave a message.

• If we reach the respondent by phone, we will give them the opportunity to be interviewed by a live interviewer.

Analysis of Non-Response

The Dun & Bradstreet lists will include information about each institution or organization in terms of size, location and other key variables. At the conclusion of the data collection effort, we will analyze this data to assess differences between respondents and non-respondents and the implications of these differences for the representativeness of the overall results.

Normally, we would use the results of this type of analysis to weight the data. However, there is a complication in this case. The email invitations are very clear about the purpose of the study. It is possible that potential respondents that are not involved in educational programming and/or do not use historical resources will simply screen themselves out of the study rather than opening the study and being screened out when they answer the screening question. We expect to resolve this issue by close examination of the pattern of the analysis of nonresponse. For example, are there systematic patterns of non-response? Do certain types of organizations tend to opt out of the survey?

Justification: Adequacy of the Research for the Intended Uses

This study is formative research that is exploratory in nature. The target audience (the effective population) is small, highly specialized and not easily defined.

The primary purpose is to give direction to the development of preliminary conceptual frameworks and the design of programs that institutions and organizations can use for educational purposes.

This data collection effort is essentially qualitative in the sense that (2) we seek to reach the individuals in organizations that have some interest in the topic and can give us the input needed to design an educational program that fits their needs and (2) there will be no claim that the results represent a defined population with a particular degree of precision.

However, this survey will have a broader scope and provide much more information than would focus groups or other strictly qualitative research. For example, it will allow us to assess interest among different general types of organizations/institutions.

Testing

As a pretest of the online survey, we send out a small number of invitations and will analyze the first few responses to detect any problems and make changes if necessary. In addition, at the end of the survey, we will ask respondents to rate the interest level and difficulty of the survey and to provide any comments.

We will also analyze the first few responses to the telephone survey and make changes if necessary before proceeding. Comments made during the interview will be noted and analyzed, as well as ratings and an open-ended question asking for comments at the end.

In our experience, the most common change is elimination of certain questions to keep the questionnaire within the prescribed length. Since we do expect to make major changes, these “test” respondents will be part of the survey database and, thus, do not constitute a separate collection.

Individuals Consulted and Those Who Will Work on the Study:

American Directions Research Group (ADRG)

Rebecca C. Quarles, PhD, Project Manager (703) 567.7655

Jerry Karson, Data Collection Director (863) 709.0553

Page 4