2023 ASR Feasibility Test-- Supporting Statement A

Feasibility Testing of Online Annual Survey of Refugees Mode

OMB Information Collection Request

0970 - 0355

Supporting Statement

Part A - Justification

MARCH 2023

Submitted By:

Office of Refugee Resettlement

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Sara Tompkins

A1. Necessity for the Data Collection

This information collection request is to conduct a feasibility test for implementing an online version of the Annual Survey of Refugees (ASR), to be conducted by the Urban Institute under contract with the Office of Refugee Resettlement (ORR) under the Administration for Children and Families (ACF), HHS. Urban Institute (UI) will work with the contractor, SSRS, and Research Support Services (RSS) on administration of the feasibility test. SSRS is a survey firm that has been UI’s partner in administering the ASR since 2016, and RSS is a research firm specialized in multilingual research with expertise working with immigrant and refugee populations and cognitive interviewing.

The pretest is necessary to assess the effectiveness of the survey instrument in an online setting and to determine whether an online component can be included in future ASR projects.

Background

ORR funds an annual survey of newly arrived refugee households to collect information on refugees’ progress toward self-sufficiency and integration during their first five years of living in the United States. The survey data are used to meet reporting requirements for ORR’s Annual Report to Congress and inform refugee resettlement policy, by providing a unique source of nationally representative information on refugees’ experiences of adjustment to life in the United States. The ASR collects information on a range of information about refugees, including language proficiency, training and education, employment and income, public assistance, social connections, and access to health care. The annual survey is conducted by telephone in up to twenty languages.

ORR is interested in collecting evidence to inform potential transition to a multi-mode version of the survey, in which respondents could participate either by telephone or online.

This feasibility test is intended to be a first step in a three-phase testing design to inform ORR’s decision making about permanently integrating an online mode to ASR data collection. If resources are made available in the future, the initial feasibility test will be followed by larger-scope testing phases: a mode test with a sample of 50 to 100 to test for mode effects; and a field test with a sample of 250 to 500 done alongside a future ASR administration in which ASR telephone-only results will be compared to multi-mode sample results.

Legal or Administrative Requirements that Necessitate the Collection

There are no legal or administrative requirements that necessitate the collection. This feasibility test data collection is being taken at the discretion of the agency.

A2. Purpose of Feasibility Test and Data Collection Procedures

Overview of Purpose and Approach

The objectives of this feasibility test are to identify questionnaire and procedural issues in an online setting and to identify solutions to those problems. This includes revising the questionnaire so that respondents can more accurately comprehend and respond to questions in an online environment as opposed to a telephone interview. If additional testing steps provide strong evidence for an online component, it could be a complement to the larger ASR project and provide an opportunity to both lower overall costs to the federal government and to reach populations that traditionally are less likely to complete telephone surveys.

RSS will recruit twenty refugees¹ who have not participated in a prior telephone ASR to take an online version and participate in cognitive interviewing with a trained interviewer. Cognitive interviews and online surveys will be done in four languages – Ukrainian, Dari, Arabic, and Kiswahili - with five respondents participating in each language. The interviews will be conducted virtually via an online platform (Zoom); the respondent will share their virtual screen with the interviewer as they click through the online survey, so that the interviewer can observe their process. The interviewer will then conduct a cognitive interview in which respondents will share their reflections on how easy or hard it was to answer the questions and what cognitive or other challenges they perceived as they completed the online survey.

Key Questions

The ASR online feasibility test will seek to determine whether an online version of the ASR can be implemented in the future. As respondents will be answering the same questions as on the telephone version of the ASR, no additional research questions are generated by this project.

Study Design

The study design was developed to maximize insights within the constraints of the limited budget and time line. A purposive sample of refugees will be drawn the Refugee Arrivals Data System (RADS), which provides the sampling frame for the basic ASR. A purposive sample is not a random sample, and it is not intended to make inferences about the population, but does ensure the proper mix of respondents are selected for the cognitive testing. RSS interviewers with linguistic and cultural competence for each of four study languages will recruit and collect data from refugees who will be taking an online version of the survey and participating in a cognitive interview. This process will generate evidence on the experience of taking an online version of the ASR, and will inform remaining rounds of testing of an online mode.

Processes for Information Collection

Data collection efforts under this information collection request will involve a combination of completed online surveys and cognitive interviews with refugees.

Survey completions. Respondents will complete the full ASR in an online mode, as opposed to a telephone interview. The online survey data will not be analyzed.

Cognitive interviews of refugees. Interviewers will work directly with respondents as the respondents complete the online version of the ASR by observing them in a Zoom call. Interviewers will provide information about the study, go through the informed consent process, and obtain consent. Interviewers will observe participants’ expressions as they complete through the survey and take hand-written notes on their observations. After a section of questions, the interviewer will stop the participant and ask them some targeted probing questions about the survey section they just completed, including any challenges to survey completion. The probes will be aimed at eliciting respondent interpretation of specific words or full survey questions, to make sure survey questions were being interpreted as intended. Notes will be stored securely in locked drawers or boxes. Audio and video recording will be downloaded from Zoom and stored securely; the video recording is required so that the interviewer can review facial expressions and body language reactions to the survey. However, the recordings will be deleted as soon as analysis is complete.

The entire process is voluntary and will take around 75 minutes to complete.

A3. Improved Information Technology to Reduce Burden

This feasibility test is intended to collect information about how refugees navigate an online survey in terms of comprehension, burden and timing. The results of this test will help determine whether an online component to the ASR is justifiable in future iterations, with the goal of reducing burden on respondents and well as saving costs to the federal government.

A4. Efforts to Identify Duplication

No similar data are available from other sources on any potential differences between online and telephone survey modes for the refugee population.

A5. Involvement of Small Organizations

RSS is a small organization and will implement the feasibility test.

A6. Consequences of Less Frequent Data Collection

As a request to conduct a feasibility test, this issue is not applicable. The feasibility test will only occur once.

A7. Special Circumstances

There are no special circumstances for the proposed data collection efforts.

A8. Federal Register Notice and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection request to extend approval of the umbrella generic with minor changes. The notice was published on January 28, 2022, (87 FR 4603), and provided a sixty-day period for public comment. ACF did not receive any comments on the first notice. A second notice was published, allowing a thirty-day period for public comment, in conjunction with submission of the request to OMB. ACF did not receive any comments on the second notice.

Consultation with Outside Experts

No consultations have taken place with experts outside of the project team.

A9. Incentives for Respondents

Participants will receive a $40 token of appreciation for participating in the survey and the cognitive interview. RSS will email digital gift cards to participants after the interviews.

A10. Privacy of Respondents

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law.

As specified in the contract, UI and UI’s partners shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. A Human Subjects Protection plan has been reviewed and was approved by UI’s Institutional Review Board (IRB) on February 23, 2023. The Contractor will ensure that employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.

As specified in the contract, the Contractor shall use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; and establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. All data, including audio and video recordings of Zoom sessions, will be destroyed at the end of the feasibility test study. All hand-written notes will also be destroyed after the interviewer writes a summary of their observations.

A11. Sensitive Questions

There are no sensitive questions in this data collection in terms of divulging risky or illicit behaviors. However, some refugees may consider some questions to be sensitive, such as questions about experiences about living in a refugee camp, income and barriers to employment, health, and the experiences of their children. To the extent that these are perceived as sensitive by this population, interviewers have been trained to reassure subjects of the privacy of their reported data under federal law.

A12. Estimation of Information Collection Burden

Burden and Cost Estimates

The estimated burden to respondents is 30 burden hours. Respondents will be asked to participate in one survey and cognitive interview that is estimated to take an average of 75 minutes to complete, and 5 additional minutes for a telephone screening call when they are recruited for participation. We plan to collect information from 20 refugee respondents.

The estimated annual cost to respondents to this feasibility test is calculated using the U.S. Department of Labor federal minimum wage of $7.25.² To account for fringe benefits and overhead, the rate was multiplied by two, totaling $14.50 per hour. The estimated cost to respondents per hour is $14.50, times 30 hours, for a total annual cost of $435. (https://www.bls.gov/oes/current/oes_stru.htm)

Instrument	Total Number of Respondents	Total Number of Responses Per Respondent	Average Burden Hours Per Response	Total Burden Hours	Average Hourly Wage	Total Annual Cost
ASR/Online feasibility test	20	1	1.5	30	$14.50	$435
Total Burden and Cost Estimates:				30		$435

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The total cost for the feasibility test (including respondent incentives) is funded by a government contract totaling $111,836.

A15. Change in Burden

This is for an individual information collection under the OMB information collection request (0970-0355).

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

Survey and data collection will begin following OMB approval, scheduled for May through June 2023. Data from the cognitive interviews will be analyzed and summarized by mid August 2023, to inform an internal memoranda to ORR which will share the findings and provide plans for remaining two testing phases.

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

Attachments

• ASR survey protocol with cognitive interview probes

• Letter of introduction

• Telephone recruitment script

1 The allocated budget for this project limits participation to twenty respondents.

2 U.S. Department of Labor, Minimum Wage, https://www.dol.gov/general/topic/wages/minimumwage (last visited February 24, 2022).

8