Annual Survey of Refugees
OMB Information Collection Request
0970 - 0033
Supporting Statement Part B –
Statistical Methods
September 2023
Submitted By:
Office of Refugee Resettlement
Administration for Children and Families
U.S. Department of Health and Human Services
Respondent Universe and Sampling Methods
Respondent Universe and Target Population
The Annual Survey of Refugees (ASR) is an annual cross-sectional study, measuring the characteristics and experiences of refugee individuals and households entering the United States in the previous five fiscal years. This request covers the following administrations of the ASR: ASR 2023 (to be fielded spring 2024), ASR 2024 (to be fielded spring 2025), and ASR 2025 (to be fielded spring 2026).
The size of the U.S. refugee population varies by federal fiscal year, according to an allowable ceiling set by the President of the United States and required pre-arrival processing by the U.S. Department of State. The 2023 Annual Survey of Refugees, which will introduce an updated survey instrument (Instrument 2), will cover refugee households and their constituent individuals entering the U.S. in the time period of FY2018-FY2022. Table B1 presents the universe of refugee entrants by fiscal year for FY 2018 through FY 2022.
Table B1. Admitted Refugee Individuals by Federal Fiscal Year
Federal Fiscal Year |
Refugee Individuals |
2018 |
22,496 |
2019 |
30,000 |
2020 |
11,814 |
2021 |
11,411 |
2022 |
25,465 |
Total |
101,186 |
Source: US Department of State Office of Admissions, Refugee Processing Center, accessed 9/7/2023 |
|
Sampling Method
The
sampling frame for the ASR is the Office of Refugee Resettlement
(ORR) Refugee Arrivals Data System (RADS), which contains demographic
information about the universe of refugee entrants to the United
States. Each year, a sample of refugee Principal Applicants (PAs)
will be drawn from RADS. The PA is the individual whose case formed
the basis for the household’s administrative claim to refugee
status. This individual is typically also the head of the household.
The PA responds to the survey on behalf of all eligible adult in
their household. The methodology for drawing the sample is as
follows:
Contractors, in consultation with ORR, will analyze administrative data on refugee arrivals to determine which language groups to include in the survey administration. The goal is to maximize coverage of the population while minimizing the logistical challenges of serving small language groups. In 2021, the ASR was offered in 20 languages (including English), covering over 77% of eligible refugee entrants from the focal fiscal years. To achieve complete coverage of the remaining 23% of refugees would require the addition of about 200 languages.
Contractors
will draw a stratified sample from the universe of primary
applicants in eligible language groups. The principal stratum is a
three-category “arrival cohort” (arrived in prior fiscal
year; arrived two or three years ago; arrived four or five years
ago). Within arrival cohort, data are further stratified by
geographic sending region, language, age group, gender, and
household size. These are proportionate strata, ensuring the
resultant sample is representative of the eligible refugee
population.
The survey will secure 500 completed interviews from each principal stratum; new arrivals are over-sampled to ensure statistical power to accurately capture the important first year of transition as well as detect meaningful differences between the three cohorts. We expect the maximum margin of error (MOE) at the 95 percent level of confidence to be +/- 4.8 percentage points for estimated percentages of refugees arriving in FY2022 and +/- 6.8 percentage points for percentage estimates of refugees arriving in a given year between FY2018 and FY2021.
A
replicated sample design is used for sample management. Within each
arrival cohort, the stratified sample is randomly partitioned into
30 smaller “snapshots.” This strategy allows the
contractor to monitor the sample release and response production
closely, maximizing response rate within arrival cohorts while
securing the targeted number of completed surveys per cohort in the
12-week fielding period.
Projected Response Rate for 2023
Survey administrations in 2021 and 2022 provide the best available estimates of expected response rate for future ASR administrations. In 2021, the overall response rate was 19 percent. Non-response was largely driven by difficulty locating respondents due to insufficient contact information (21 percent of the sample) or unanswered calls after the maximum number of attempts (55 percent of the sample). Conditional on successful contact, 79 percent of sampled households completed a survey. As in 2021, the 2022 response rate was driven by the inability to update contact information but did increase to 20 percent.
Successfully securing responses to telephone surveys is a broad challenge in contemporary survey research and is also consistently a challenge in the administration of the ASR. To respond to this known challenge, in 2023, ACF elected to explore the feasibility of additional survey modes. Results are still being analyzed.
Beginning in fall 2019, there has been a systematic review of recent surveys of refugee populations in the U.S. and abroad, with a goal of identifying how survey mode is related to response rates and demographic response bias in this highly mobile, educationally- and linguistically-diverse population. The review supports recommendations for potential future improvements to the ASR that could result from incorporating additional survey modes, such as text message and/or web, particularly for ASR questions of key policy interest. Based on all of the work between 2019 and the 2023 feasibility testing, ACF is considering recommendations based on the strength of evidence for potential improvements in data quality and resource efficiency, and in context of the overall resources available for this data collection effort. Should the findings of the 2023 field scan and available resources support changes to the ASR administration protocol, ACF will submit proposed changes to OMB for approval.
Procedures for the Collection of Information
Design of Conceptual Framework
ASR questions are intended to collect information about factors that social science research suggests affect refugee resettlement outcomes, in order to help ORR better understand the experiences of the population it serves and increasing the practical utility of the data collection. Domains in the survey include household demographics; experiences before arrival; human capital; economic self-sufficiency; social connection; well-being and the receiving community; health; and children and schools. To enable the contextualization of data from the ASR, particularly for key measures reported in the Annual Report to Congress, such as labor force participation and use of public benefits, survey items were drawn from nationally representative studies of the U.S. population whenever possible.
Statistical Methodology for Stratification and Sample Selection
To ensure quality of sampling selection and stratification, additional statistical methods and procedures are implemented before and during the fielding process.
Prior to beginning subject tracing, the study team will conduct sample validation exercises to ensure that the stratified random sampling and the replicate partitioning procedure performed as intended and are representative of the intended inferential population.
During the fielding process, the replicate sample release procedure allows for close monitoring, adaptation, and continuous learning. Throughout the field period, the contractor will produce weekly summaries of survey progress by key demographic groups, in order to monitor sample representativeness and redouble efforts to secure participation from underrepresented populations as necessary.
Tracing of Respondents
The most recent contact information contained in ORR administrative data is collected by the U.S. State Department 90 days after the principal applicant’s arrival in the United States. Sampled individuals will be submitted to location tracing, in attempts to update their contact information in preparation for the survey administration.
Consistent with prior rounds of administration, the contractor will seek updated contact information for sampled respondents using the National Change of Address system and TransUnion Batch Lookup. Based on past experience with this survey’s administration, these are the most comprehensive, relevant sources of updated information, as travel loans from the U.S. State Department for refugees’ arrival in the United States are reported to the TransUnion credit bureau. Given the lack of U.S. credit history for recently-arrived individuals, other attempted batch lookup processes have proven ineffective.
Advance Mail-out
The contractor will prepare and send an Introduction Letter and Postcard (Instrument 1) to each of the potential respondents, which reflects the updated time estimate for survey completion and the $2 token of appreciation to update participant’s information. This letter introduces the survey and provides means to contact the research team with updated telephone information via pre-paid postal mail, email, or telephone. If a potential respondent does not update their information upon receipt of the letter, the contractor relies on the most up-to-date telephone number available from tracing efforts. Since implementing the $2 token of appreciation, the response rates have stopped declining and have started to improve as a result of updated contact information for respondents.
Interviewing
ASR data collectors use culturally sensitive interview methods, including matching interviewer and interviewee by gender and language, and avoiding calls on major religious days and holidays.
Telephone interviews will be conducted in the Principal Applicant’s preferred language, using computer-assisted telephone interviewing (CATI) protocol to ensure accurate and complete data collection. Interviewers will attempt to contact each selected refugee up to 10 times before final disposition is listed as “unable to contact.”
The study team will also:
Provide training to the interviewers in the conduct of these interviews in order to reduce interviewer error prior to interviewing. Interviewers will receive a thorough explanation of each survey question and identify logical and acceptable responses to questions; be briefed on their commitment to privacy; familiarize themselves with the flow and the CATI application, and then be evaluated to ensure an acceptable command of all concepts and technical aspects involved in the interview process. They will also be trained on handling respondent distress should that arise (which is a very rare occurrence in the ASR).
Provide ongoing monitoring of interview quality, including live listening to a sample of calls. Stronger interviewers will be assigned more difficult cases to maximize data quality.
Download and review data tables from CATI system, including frequency tests to identify any erroneous anomalies.
Estimation Procedures
The ASR data are used to produce national estimates of key household and individual characteristics for official external release in ORR’s Annual Report to Congress. Additional analyses provide descriptive information to inform internal discussions for program management and improvement. All point estimates are accompanied by corresponding measures of error using 95% confidence intervals.
Analytic weights are used to produce statistically valid point estimates and calculate statistical uncertainty that accounts for clustering of individuals within households (for person level statistics). In 2018, the nonresponse/post-stratification adjustment was developed by first conducting a Chi-square Automatic Interaction Detector (CHAID) Analysis to identify the factors most associated with survey response. The factors that emerged from this analysis (which are also available in RADS administrative data) were then used for the nonresponse weighting consistently for the most recent fieldings of the ASR. We do not anticipate changes to the weighting procedure for 2023.
The ASR data are not used to produce projections of future values.
Data Handling and Analysis
The ASR is administered via CATI with pre-programmed acceptable value ranges and skip patterns to mitigate data entry errors. As described previously, during the fielding period, the contractor periodically downloads and reviews data tables from the CATI system, including frequency tables, to identify and investigate any potentially erroneous anomalies.
As part of survey post-processing, the contractor compares respondent-provided household roster data to administrative data from ORR to ensure that only eligible refugees (arriving during the specified time period) are included in tabulations about refugee adults in the household.
The
primary statistical product from the ASR is a set of tabulations
included in ORR’s Annual Report to Congress. ORR presents the
ASR results alongside other quantitative and qualitative information
sources to fulfill its Congressionally-mandated reporting
requirements, per the Refugee Act of 1980. In ORR’s Annual
Report to Congress, ASR data are used in descriptive (bivariate)
tabulations.
The analytic code that generates the
contractor’s tabulations is returned to ACF as a deliverable of
the contract, to facilitate reproducibility.
Data Use
ORR’s Annual Report to Congress includes a description of data quality and limitations in the text of the report and technical footnotes accompanying each table. The report also includes a brief technical appendix that provides additional information important to the interpretation of survey data.
A restricted use analytic file and detailed technical documentation are produced for ACF’s internal use and will support the expansion of in-house analysis of the survey data.
ACF is interested in making data collected through federal contracts available for secondary research use. To this end, a public use analytic file from the ASR have been and will continue to be generated to enable use by external researchers. The study team will properly protect respondent privacy while generating these files. These files are archived annually on the website of ICPSR at the University of Michigan. The ICPSR archiving procedure sets standards for the form and content of technical documentation and codebooks necessary to support secondary analysts’ understanding of the data’s production and its limitations. The ASR public use technical documentation also includes sample analytic code and detailed instructions to assist secondary researchers in the correct application of analytic weights to generate representative point estimates and correctly measure statistical uncertainty.
Burden Reduction
In the spring of 2019, a working group of staff social scientists from ORR, the Office of Planning, Research, and Evaluation (OPRE), and HHS’s Office of the Assistant Secretary for Planning and Evaluation (ASPE) conducted an internal review of pre-test survey items to identify candidates for streamlining. The group assessed the contractor’s findings on the scientific validity of measures from the pretest and additionally considered the policy and programmatic relevance of candidate survey items. This analysis resulted in the reduction of survey questions.
During
the redesign efforts the number of questions asked for each adult
member of the household roster and the interviewer instructions for
rostering were streamlined to further reduce burden. Demographic data
is collected about all household members, any age, up to 15 people.
Moreover, to increase data quality from proxy reporting, PAs are only
asked economic questions for family-member
eligible refugees
aged 16+ (rather than all eligible refugees aged 16+, as is current).
Since the release of the 2020 version, this reduced burden for
households where non-family members are in residence.
In the summer of 2023, ORR conducted a feasibility test to collect evidence to inform potential transition to a multi-mode version of the survey, in which respondents could participate either by telephone or online. The ASR instrument was tested from June through August 2023 (approved under ACF’s Generic Clearance for Pre-Testing, OMB #: 0970-0355). The contractor recruited twenty refugees1 who had not participated in a prior telephone ASR to take an online version and participate in cognitive interviewing with a trained interviewer. Cognitive interviews and online surveys were conducted in four languages – Ukrainian, Dari, Arabic, and Kiswahili - with five respondents participating in each language. This feasibility test collected information about how refugees could navigate an online survey in terms of comprehension, burden, and timing. Analysis is ongoing, but the results of this test will help determine whether an online component to the ASR is justifiable in future iterations, with the goal of reducing burden on respondents and well as saving costs to the federal government. If a multi-mode version of the survey is deemed a desirable option based on this work, ACF will submit a request to OMB to update this request to reflect this update. Assuming the changes would be specific to change in mode and not substantive content changes, this request would be submitted as a nonsubstantive change request.
Methods to Maximize Response Rates and Deal with Nonresponse
Maximizing Response Rates
The primary strategy for maximizing response rates while securing the target number of completed surveys will be the replicated sampling strategy, outlined above. This allows the contractor to closely follow response rate by arrival cohort and release further sample into the field if production within a cohort is lower than expected. We offer tokens of appreciation as an additional avenue to assist with an adequate response rate. We propose to continue to provide a token at two time points. A pre-participation amount of $2 will be sent to each of the potential respondents with the Introduction Letter and Postcard (Instrument 1). A post-participation tokens of appreciation $40 will then be provided upon completion of the survey (Instrument 2). See section A9 of Supporting Statement A for additional information about the proposed tokens of appreciation.
Non Response
Analysis
of survey- and item-level non-response bias is a routine task in the
production of ASR analytic files, survey weights, and data
documentation. The contractor takes advantage of ORR administrative
(RADS) data on the universe of refugees eligible for the survey to
assess non-response bias on observable characteristics.
At the completion of data collection, the contractor calculates analytic weights to enable nationally representative point estimates and the calculation of statistical uncertainty that accounts for clustering of individuals within households. These weights include a base (sampling) weight reflecting the refugee household selection probability. Because sample allocations of each cohort are managed separately, selection probabilities vary by the size of the arrival cohort population and amount of the sample released into the field. Weights also include a post-stratification adjustment to correct for differential non-response across cohort and demographic subgroups, aligning the data to known population distributions taken from ORR administrative data. Both household level (i.e., PA) and person level analytic weights are developed for the ASR.
Identifying Measurement Error
ACF employs a continuous quality improvement framework during ASR survey administration. In the past, this effort has resulted in qualitative efforts to understand and reduce measurement error, such as debriefing field interviewers following the survey administration, and using that information to update item translations and interviewer training materials to improve the comparability and quality of data across the ASR’s many languages and cultural groups. Since the new survey instrument will require fresh translation into the survey’s languages, we anticipate employing similar quality improvement processes over the course of this approved data collection.
Generalizability of Results
This study is intended to produce national estimates of the characteristics and experiences of refugee households and refugee individuals aged 16 and older entering the United States in the previous five fiscal years. As described earlier, the 2022, 2021 and 2020 ASR were offered in 20 languages, covering approximately 77% of refugee entrants each year. Households speaking less-frequent languages (typically less than 1 % of the total refugee population) are intentionally excluded from the sample, and this limitation to national representativeness is noted in all technical documentation, written products, and digital products associated with the data collection.
Test of Procedures or Methods to be Undertaken
Survey Pre-Testing
As noted throughout this justification package, the ASR was redesigned in 2020. As part of these efforts, a variety of pre-testing and feedback efforts were engaged (as described in the previously approved request package). The resulting ASR was used in the 2020 ASR and 2021 ASR administration, implemented in the spring of 2021 and 2022.
The ASR included with this revision request primarily uses the same questions from approved in 2020, but also includes additional questions developed for the Operation Allies Welcome (OAW) Afghan Supplement Survey (OMB# 0970-0613). Newly developed survey questions specific to the Afghan population were tested in October 2022 by the contractor with four staff members from two organizations, Services Navigator from the Coalition for Humane Immigrant Rights (CHIRLA) and Women for Afghan Women. Staff members had significant expertise resettling Afghan evacuees and providing support related to housing, social services, benefits, and legal support. Each staff member provided direct feedback on questions, including phrasing and themes. The questionnaire has been reviewed by ORR Afghan language and culture experts as well.
Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
Dr. Saunji Fyffe
Social Science Research Analyst
ACF Office of Refugee Resettlement
Mary E. Switzer Building, 5th Floor
330 C Street, SW, Washington, DC 20201
Lauren Griffin
Social Scientist, VPD Government Solutions
ACF Office of Planning, Research, and Evaluation
(202) 729-8048
Sara Tompkins, Esq.
Monitoring and Evaluation Program Lead
ACF Office of Refugee Resettlement
Mary E. Switzer Building, 5th Floor
330 C Street, SW, Washington, DC 20201
Kenneth Tota
Director, Refugee Programs
ACF Office of Refugee Resettlement
Mary E. Switzer Building, 5th Floor
330 C Street, SW, Washington, DC 20201
Dr. Joseph Wantz
Research and Evaluation Lead
ACF Office of Refugee Resettlement
Mary E. Switzer Building, 5th Floor
330 C Street, SW, Washington, DC 20201
Attachments
Attachment A – Overview of changes proposed to the ASR
Instrument 1 – Introduction Letter and Postcard (2023 Proposed)
Instrument 2 – ORR-9 Annual Survey of Refugees (2023 Clean)
1 The allocated budget for this project limits participation to twenty respondents.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |