Mode Testing of Online Annual Survey of Refugees Mode
Generic for Pre-testing of ACF Data Collection Activities
Supporting Statement
Part A - Justification
OCTOBER 2024
Submitted By:
Office of Refugee Resettlement
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
A1. Necessity for the Data Collection
This information collection request is to conduct a mode test for implementing an online version of the Annual Survey of Refugees (ASR) (OMB #0970-0033). The test is necessary to assess the effectiveness of the survey instrument in an online setting and to determine whether an online component can be included in future ASR projects.
Background
ORR is interested in transitioning to a multi-mode version of the survey, in which respondents could participate either by telephone or online. This proposed data collection will inform those efforts. Specifically, this mode test is intended to be a second step in a three-phase testing design to inform ORR’s decision making about permanently integrating an online mode to ASR data collection:
Phase one included an initial feasibility test (conducted in summer 2023, as approved under 0970-0355: Feasibility Testing of Online Annual Survey of Refugees Mode).
Phase two is the focus of this request and includes a mode test with a sample of up to 100 to test for mode effects.
Finally, Phase three will include a field test with a sample of 250 to 500 done alongside a future ASR administration in which ASR telephone-only results will be compared to multi-mode sample results (future information collection request to be submitted).
The Urban Institute under contract with. Urban Institute (UI) will work with the contractor, SSRS on administration of the mode test. SSRS is a survey firm that has been UI’s partner in administering the ASR since 2016 and has expertise working with immigrant and refugee populations using multilingual data collection tools.
There are no legal or administrative requirements that necessitate the collection. This mode test data collection is being taken at the discretion of the agency.
A2. Purpose of Mode Test and Data Collection Procedures
Overview of Purpose and Approach
The objectives of this mode test are to identify questionnaire and procedural issues in an online setting and to identify solutions to those problems. This includes revising the questionnaire so that respondents can more accurately comprehend and respond to questions in an online environment as opposed to a telephone interview. The mode test will develop a revised online version of the ASR instrument that addresses the issues identified during the feasibility testing in phase one of this approach. This revised instrument will be tested with a random sample of refugees who were not part of a previous ASR, inviting sampled respondents to participate in an online-only version. If additional testing steps provide strong evidence for an online component, it could be a complement to the larger ASR project and provide an opportunity to both lower overall costs to the federal government and to reach populations that would prefer to complete online surveys. It may offer a quicker and less burdensome option for some respondents. The mode test will gather insights to finalize phone and online instruments to inform the third phase of testing.
Key Questions
The ASR online mode test will seek to determine whether an online version of the ASR can be implemented in the future. This will include understanding any issues participants have completing the online version of the survey.
Study Design
The study design was developed to maximize insights within the constraints of the budget and timeline. The sampling frame will be the same Refugee Arrivals Data System (RADS) admissions data that is used to select the main ASR sample. The mode test will include a random sample of the target population, which includes 18+-year-old refugee PAs arriving in the U.S. during the past 5 years and who speak one of the six mode test languages (Arabic, Kiswahili, Ukrainian, Dari, Spanish, and Sgaw Karen). This mode test will use a similar stratification strategy that is used with the ASR, beginning with a three-tiered arrival cohort: (1) refugees arriving 1.5-2.5 years ago, FY2023; (2) refugees arriving 2.5-4.5 years ago, FY2021-2022; and 3) refugees arriving 4.5-6.5 years ago, FY2019-FY2020. Within cohorts, PAs will be independently sampled via proportionate stratified sampling. An overall sample size of about 2,000 refugees will be selected and their addresses updated through tracing activities. This sample will be divided into 20 equal-sized replicates of 100 people each. Letters of invitation will initially be sent to four replicates (i.e., 400 people), and additional batches of 100 will be invited depending on the take-up we see of people taking the online survey. When we see 100 online surveys have been completed, we will stop releasing additional replicates and inviting additional batches. Working the sample by replicate will allow us to only work as much sample as is needed to complete the 100 online surveys; we expect that we will not need to send out invitations to the full sample of 2,000.
The online survey administration will last about 8 weeks for each sample release. 8 weeks allows enough time for a four-tiered mailing design - with an invitation letter, a sealed reminder postcard, a reminder letter, and a final postcard (see table 2). Depending on response, a second sample release would occur 2 weeks after the initial sample release, increasing the length of the overall field period to 12 weeks. If a target of 100 is not yet reached, a third sample release will occur in week 8, and the overall data collection field period would be about 16 weeks long.
Processes for Information Collection
Data collection efforts under this information collection request will involve respondents being invited by mail to consider participating, and their completing online surveys.
The outreach materials will consist of a four-tiered mailing design with an invitation letter, a sealed reminder postcard, a reminder letter, and a final postcard, all of which will be translated into the 6 mode test languages. Like with the ASR, the invitation letter will contain a $2 preemptive token of appreciation. All mailings will contain a link and QR code to the web survey, a unique passcode for the respondent to use to access the web instrument, and information on how respondents can contact support if they have any questions. Respondents will complete the full ASR in an online mode, as opposed to a telephone interview. The online survey will begin with several screening questions to ensure the sampled individual is indeed taking the survey.
The quality of the online survey data and characteristics of participation will be assessed for methodological purposes, but the survey findings will not be analyzed to take away findings on outcomes. The team will assess paradata on the demographic and linguistic characteristics of those who choose and choose not to participate, paradata1 on how many reminders were needed and duration between outreach and participation, survey completion times, item refusals, data quality checks to assess the quality of online completion such as attention checks, as well as explicit feedback questions on mode preference. All refugees who complete the online survey will receive a $30 gift card, like respondents to the telephone-based ASR.
The entire process is voluntary, and we expect that it will take an average of 45 minutes to complete the online survey.
A3. Improved Information Technology to Reduce Burden
This mode test is intended to collect information about how refugees navigate an online survey in terms of comprehension, burden, and timing. The results of this test will help determine whether an online component to the ASR is justifiable in future iterations, with the goal of reducing burden on respondents and well as saving costs to the federal government.
A4. Efforts to Identify Duplication
No similar data are available from other sources on any potential differences between online and telephone survey modes for the refugee population.
A5. Involvement of Small Organizations
None
A6. Consequences of Less Frequent Data Collection
As a request to conduct a mode test, this issue is not applicable. The mode test will only occur once.
A7. Special Circumstances
This request is for a mode test using the currently approved version of the ASR. The information will inform the ways the ASR may be administered in the future and specific responses to data elements will not be analyzed. As such, the survey included for testing purposes is the currently approved version of the ASR and does not reflect the recent updates to Statistical Policy Directive 15: Standards for Maintaining, Collecting, and Presenting Federal Data on Race and Ethnicity (SPD 15; 89 FR 22182). ACF plans a thorough assessment of these updated requirements as part of the agency’s Action Plan efforts and will address updates in request specific to the ASR (OMB #0970-0033) through a future request, either through a nonsubstantive change before the current expiration date of October 31, 2026 or through the next revision request, dependent on timing.
Similarly, ACF acknowledges updated guidance for sexual orientation and gender identity (SOGI). ACF also plans to update the related question in the survey based on recent guidance from OMB.
A8. Federal Register Notice and Consultation
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection request to extend approval of the umbrella generic with minor changes. The notice was published on June 20, 2024 (89 FR 51888) and provided a sixty-day period for public comment. ACF did not receive any comments on the first notice. A second notice was published on August 26, 2024 (89 FR 68444), allowing a thirty-day period for public comment, in conjunction with submission of the request to OMB. ACF received one comment during the second comment period in support of the proposed revisions to the umbrella generic.
No consultations have taken place with experts outside of the project team.
A9. Tokens of Appreciation for Respondents
Participants will receive a $30 token of appreciation for participating in the survey, as well as a $2 preemptive token of appreciation sent to all sampled individuals. SSRS will email $30 digital gift cards to participants after the surveys are completed. In order to provide a fair test of an online version, we need to offer the same tokens of appreciation that are provided in the current telephone-only ASR; here we provide the exact same amounts, a $2 preemptive and $30 post-survey token of appreciation. In both the telephone-only and mode test ASR, we include a $2 preemptive token and a post-completion token because of the strong research evidence that shows that such measures increase response rate.2
A10. Privacy of Respondents
Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law.
As specified in the contract, UI and UI’s partners shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. A Human Subjects Protection plan will be reviewed and approved by UI’s Institutional Review Board (IRB) in October 2024. The Contractor will ensure that employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.
As specified in the contract, the Contractor shall use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; and establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. The contractor will also abide by the Data Security and Monitoring Plan, which describes the guidelines for ensuring that personally identifiable and sensitive information is kept private at all times.
A11. Sensitive Questions
There are no sensitive questions in this data collection in terms of divulging risky or illicit behaviors. However, some refugees may consider some questions to be sensitive, such as questions about experiences about living in a refugee camp, income and barriers to employment, health, and the experiences of their children. Information in the informed consent script will reassure respondents of the privacy of their data, and they also have the option to skip questions they do not want to answer.
A12. Estimation of Information Collection Burden
Burden and Cost Estimates
We plan to collect information from 100 refugee respondents. Respondents will be asked to participate in one survey that is estimated to take an average of 45 minutes to complete.
The estimated annual cost to respondents to this mode test is calculated using the U.S. Department of Labor federal minimum wage of $7.25.3 To account for fringe benefits and overhead, the rate was multiplied by two, totaling $14.50 per hour. (https://www.bls.gov/oes/current/oes_stru.htm)
Instrument |
Total Number of Respondents |
Total Number of Responses Per Respondent |
Average Burden Hours Per Response |
Total Burden Hours |
Average Hourly Wage |
Total Annual Cost |
ASR/Online mode test |
100 |
1 |
.75 |
75 |
$14.50 |
$1,088 |
A13. Cost Burden to Respondents or Record Keepers
There are no additional costs to respondents.
A14. Estimate of Cost to the Federal Government
The total cost for the mode test is funded by a government contract providing costs for data collection of $105,566.
A15. Change in Burden
This is for an individual information collection under the OMB information collection request (0970-0355).
A16. Plan and Time Schedule for Information Collection, Tabulation and Publication
Survey and data collection will begin following OMB approval, scheduled for March through April 2024. Data from the online survey will be analyzed and summarized by June 2024, to inform an internal memorandum to ORR which will share the findings and provide plans for the final testing phase, the field test.
A17. Reasons Not to Display OMB Expiration Date
All instruments will display the expiration date for OMB approval.
A18. Exceptions to Certification for Paperwork Reduction Act Submissions
No exceptions are necessary for this information collection.
Attachments
Invitation letters and reminders
ASR online instrument
1 Paradata are a by-product of data collection, so may also include things such as keystroke or mouse-
click history for internet self-response surveys.
2 See Rao, Neomi. 2020. “Cost Effectiveness of Pre- and Post-Paid Incentives for Mail Survey Response.” Survey Practice 13 (1). https://doi.org/10.29115/SP-2020-0004
3 U.S. Department of Labor, Minimum Wage, https://www.dol.gov/general/topic/wages/minimumwage (last visited February 24, 2022).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | OPRE OMB Clearance Manual |
| Author | DHHS |
| File Modified | 0000-00-00 |
| File Created | 2024-12-24 |