Cognitive Interview Form

Request for Approval under the “Generic Clearance To Conduct Pre-Testing of Surveys” (OMB Control Number: 3137-0125)

TITLE OF INFORMATION COLLECTION: IMLS National Museum Survey Cognitive Interviews

PURPOSE:

In 2021, the Institute of Museum and Library Services (IMLS) awarded a contract to the American Institutes for Research (AIR) to develop and pilot a National Museum Survey (NMS). The NMS pilot is planned to run in 2023 and will gauge the proposed collection’s long-term feasibility as part of the Federal Statistical System. As part of the survey development process, AIR will conduct 24 cognitive interviews in two rounds of 12 interviews each with likely respondents to the future NMS to test and improve draft survey items.

Cognitive interviewing involves a set of techniques (e.g., think-aloud method, verbal probes) that enable researchers to analyze how participants understand survey questions and whether the questions measure what they were intended to measure.¹ Conducting cognitive interviews to test draft NMS survey questions will ensure that the final survey questions are easier to understand, less burdensome, more relevant to respondents, and result in higher quality data overall. The total burden will not change for respondents during the testing period, but the research will evolve iteratively as needed to address findings from the cognitive interviews.

DESCRIPTION OF RESPONDENTS:

Participants will be museum² administrators whose museums are members of the American Alliance of Museums and/or the American Association for State and Local History. To be eligible for the interviews, administrators’ museums must (a) be government or nonprofit entities, (b) serve the public from a fixed physical location, (c) be open at least 120 days per year, and (d) have staff and/or volunteers whose hours make up at least one full-time equivalent.

TYPE OF COLLECTION: (Check one)

[ ] Customer Comment Card/Complaint Form [ ] Customer Satisfaction Survey

[ ] Usability Testing (e.g., Website or Software) [ ] Small Discussion Group

[ ] Focus Group [X] Other: Cognitive Interviews

CERTIFICATION:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low burden for respondents and low cost for the Federal Government.

3. The collection is noncontroversial and does not raise issues of concern to other Federal agencies.

4. The results are not intended to be disseminated to the public.

5. Information gathered will not be used for the purpose of substantially informing influential policy decisions.

6. The collection is targeted to the solicitation of opinions from respondents who have experience with the program or may have experience with the program in the future.

Name:. Jacob Soffronoff.

To assist review, please provide answers to the following question:

Personally Identifiable Information:

1. Is personally identifiable information (PII) collected? [X] Yes [ ] No

2. If yes, will any information that is collected be included in records that are subject to the Privacy Act of 1974? [ ] Yes [X] No

3. If yes, has an up-to-date System of Records Notice (SORN) been published? [ ] Yes [ ] No

AIR will collect personally identifiable information (PII) through the online sign-up form for recruitment and scheduling purposes only. During recruitment, each museum administrator who completes the online sign-up form will provide their name, email address, and institution name. AIR will use this information to (a) help ensure that administrators who complete the form do not receive subsequent reminder emails and (b) schedule participants for the interviews. AIR will maintain the recruitment lists and responses to the online form in a secure database separate from interview notes. Recruitment lists will be destroyed after recruitment is complete.

Interview notes will not contain PII, and the interviews will be recorded for research purposes only. Online form data, interview recordings, and notes will be stored on AIR’s secure data servers for the duration of the study and destroyed after the final report is submitted. All materials will be accessible only through password-protected laptops with disk encryption, and only AIR research team members working on the cognitive interviews will have access. AIR will not provide IMLS with any PII collected as a part of this study, and all PII will be destroyed by AIR following the project’s reporting.

Gifts or Payments:

Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to participants? [ ] Yes [X] No

NOTE: The IMLS-AIR team will not provide remuneration to participants in the form of monetary incentives. Instead, the online form will ask eligible participants if they would like IMLS to recognize their museums (not individuals) publicly for their contributions to the NMS development process. After the testing is complete, AIR will provide a list of museums to IMLS to be publicly recognized in a blog post about the NMS development process. The list includes museums whose administrators both (a) participated in the interviews and (b) gave permission to IMLS to publicly recognize their museums. AIR will not provide IMLS with the names of, or any other information relating to, museums that do not provide this permission.

BURDEN HOURS

Category of Respondent	No. of Respondents	Participation Time (Hours)	Burden
Private sector (online form only)	54	0.1	5.4
State, local, or tribal governments (online form only)	18	0.1	1.8
Private sector (online form + cognitive interview guide)	18	1.0	18.0
State, local, or tribal governments (online form + cognitive interview guide)	6	1.0	6
Total	96		31.2

FEDERAL COST: The estimated cost to the Federal Government for this information collection is $12,320.20, which includes an estimated 176 hours of staff time for five IMLS employees of differing GS levels.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents

1. Do you have a customer list or something similar that defines the universe of potential respondents and do you have a sampling plan for selecting from this universe? [X] Yes [ ] No

If the answer is yes, please provide a description of both below (or attach the sampling plan)? If the answer is no, please provide a description of how you plan to identify your potential group of respondents and how you will select them?

AIR will collaborate with two subject matter experts (SMEs) who lead professional museum associations to recruit participants from their membership lists. The SMEs will provide a convenience sample of 400 museum administrators for recruitment. The institutions included on the list will vary in size, discipline, and location to help ensure diversity from across the museum field. The information included in this SME-supplied sampling frame will be maintained by AIR, destroyed following recruitment, and will not be shared with IMLS.

AIR will recruit by sending the solicitation to the full list of museum administrators, which is necessary to achieve recruitment goals and to help ensure diversity in the final sample of participants. This strategy was successful when recruiting participants from this same population for the focus groups that AIR conducted previously as part of the National Museum Survey development process. Based on that experience, we anticipate that 18% (n = 72) of the administrators on the list will complete the online sign-up form and that 70% (n = 50) of their museums will meet the eligibility criteria for inclusion in the interviews. During recruitment, AIR will review responses to the online sign-up form on a rolling basis and purposefully select 24 eligible museum administrators who represent museums of varying sizes, disciplines, and locations for the interviews. If a selected participant is no longer available at their selected time, cancels, or does not show up for their interview, AIR will attempt to reschedule or replace that individual to meet recruitment goals.

Administration of the Instrument

1. How will you collect the information? (Check all that apply)

[X] Web-based or other forms of social media

[ ] Telephone

[ ] In-person

[ ] Mail

[X] Other, Explain – Virtual interviews via Zoom

2. Will interviewers or facilitators be used? [X] Yes [ ] No

Please make sure that all instruments, instructions, and scripts are submitted with the request.

Instructions for completing “Generic Clearance To Conduct Pre-Testing of Surveys” (OMB Control Number: 3137-0125)³

TITLE OF INFORMATION COLLECTION: Provide the name of the collection that is the subject of the request. (e.g. Comment card for soliciting feedback on xxxx)

PURPOSE: Provide a brief description of the purpose of this collection and how it will be used. If this is part of a larger study or effort, please include this in your explanation.

DESCRIPTION OF RESPONDENTS: Provide a brief description of the targeted group or groups for this collection of information. These groups must have experience with the program.

TYPE OF COLLECTION: Check one box. If you are requesting approval of other instruments under the generic, you must complete a form for each instrument.

CERTIFICATION: Please read the certification carefully. If you incorrectly certify, the collection will be returned as improperly submitted or it will be disapproved.

Personally Identifiable Information: Provide answers to the questions. Note: Agencies should only collect PII to the extent necessary, and they should only retain PII for the period of time that is necessary to achieve a specific objective.

Gifts or Payments: If you answer yes to the question, please describe the incentive and provide a justification for the amount.

BURDEN HOURS:

Category of Respondents: Identify who you expect the respondents to be in terms of the following categories: (1) Individuals or Households; (2) Private Sector; (3) State, local, or tribal governments; or (4) Federal Government. Only one type of respondent can be selected per row.

No. of Respondents: Provide an estimate of the Number of Respondents.

Participation Time: Provide an estimate of the amount of time (in minutes) required for a respondent to participate (e.g. fill out a survey or participate in a focus group)

Burden: Provide the Annual burden hours: Multiply the Number of Respondents and the Participation Time then divide by 60.

FEDERAL COST: Provide an estimate of the annual cost to the Federal government.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents. Please provide a description of how you plan to identify your potential group of respondents and how you will select them. If the answer is yes, to the first question, you may provide the sampling plan in an attachment.

Administration of the Instrument: Identify how the information will be collected. More than one box may be checked. Indicate whether there will be interviewers (e.g. for surveys) or facilitators (e.g., for focus groups) used.

Submit all instruments, instructions, and scripts are submitted with the request.

1 Tourangeau, R, Rips, L. J., and Rasinski, K. (2000). The psychology of survey response. Cambridge, England: Cambridge University Press.

2 The term “museum” is used here to include the range of institution types identified in IMLS regulations, 2 C.F.R. 200 part 3187.

3 Adapted from “Generic Clearance for the Collection of Routine Customer Feedback”

6