Supporting Statement B - 22072020 - Final

Supporting Statement B

Survey of Unmanned-Aircraft-Systems Operators

B. Statistical Methods

1. Describe the potential respondent universe.

Universe and Sample Size

The respondent universe consist of all small (less than 55lbs and greater than 0.55lbs) Unmanned-Aircraft-Systems (UAS) operators within the United States. The source of operators contact information, both electronic mail and physical mail, are the registries for commercial operators under Title 14, Part 107 of the Code of Federal Regulation and recreational operators under Section 349 of the FAA Reauthorization Act of 2018.

The survey uses a census approach of registered UAS operators. The survey samples all registrants within the two registries. As such, 1,324,437 recreational registrants and 409,525 commercial registrants are expected to receive requests to participate in the survey for a total of 1,733,962 registrants in 2020. The registries are expected to grow in 2021 and 2022, increasing total sample to 1.9 and 2.0 million, respectively.

Response Rates

Two pilot questionnaire have been conducted previously. One questionnaire was directed at commercial UAS operators in 2017 and the other was directed at recreational UAS operators in 2019.¹ Both used a similar online questionnaire as the instrument and used emails from the Part-107 and Section-349 registries as the sample frame. In both cases, the questionnaires were voluntary and anonymous. In addition, only one email was sent to the registrant. The questionnaires did not include any type of follow up, reminder, community engagement, alternative mode of delivery, or media campaign.

The response rate for the pilot questionnaires and the proposed survey is calculated following the guidelines provided by the American Association for Public Opinion Research (AAPOR). Under these guidelines, the partial and completed questionnaires are divided by the total number of eligible UAS registrants in the sample. The following formula describes the calculation of the response rate.

RR = (C+P)/[(C+P) + (NR+INS+PMR)]

Where RR = Response Rate

C = Completed Surveys

P = Partially Complete Surveys

NR = No Response

INS = Insufficient Information

PMR = Bounced Email

The request to participate in the commercial questionnaire was sent by email to 89,379 registrants of which 7,614 responded by completing the questionnaire, an 8.5% response rate. Similarly, the request to participate in the recreational questionnaire was sent by email to 262,659 registrants of which 15,482 responded, a 5.9% response rate. This would suggest the proposed survey would have an aggregate response rate of around 7%.

However, in the recreational questionnaire, 23.8% of emails bounced and 45.6% of emails remained unopened. This suggests that cleaning the registry of inaccurate emails, using different modes of delivery, such as physical mail, and conducting awareness campaigns can drastically increase the response rate. Using this information, the survey is expected to have a responses rates of 6% for recreational registrants and 8% for commercial registrants. However, with the inclusion of reminders, follow-ups, community engagement, alternative mode of delivery, and media campaigns into the survey, the response rate could surpass 50%.

Given the sample size and a response rate of 6% and 8% for recreational and commercial registrants, respectively, the survey is expected to have 79,466 recreational respondent and 32,762 commercial respondents for a total of 112,228 responses. However, responses could be as high as 866,981 if the higher response rate is achieved.

2. Describe the procedures for the collection of information.

Estimation Procedure

National-level statistics are calculated as the aggregates of the responses since the entirety of the two registries are sampled. Profile of flight behavior is developed as a set of distributions, constructed from survey responses. Point estimates are derived from these distributions.

County-level estimates of airspace activity are calculated by aggregating within-county responses. When responses in a county are fewer than 30, responses from within-state, adjacent counties are used as donor observations assigned through a matching algorithm utilizing county economic and population data. Flight behavior is divided by land area to yield an estimate of UAS activity over the county. In addition to increasing the power of a county estimate, this process also ensures the privacy of operators by effectively blurring the data in counties with few operators.

Frequency of the Survey

In order to leverage economic and population data from other government agencies, the survey is conducted on an annual basis. This frequency should minimize burden while provided optimal predictive power.

3. Describe methods to maximize response rates.

The Sample Frame and Defining the UAS Universe

Several steps are taken to ensure the registry is current and accurate. During an initial awareness campaign, see below under Maximize Response Rate, a bounce test study of the registry is conducted to ensure the email addresses in the registry are current. Bounced emails are analyzed to correct any domain entry errors and retested. All email addresses that cannot be corrected are collected for a post survey study using alternative means of contacting the registrant, either by phone or postal mail. The post survey study serves to correct the information in the registry and to determine if this process injects significant sampling error into the study.

In addition to the bounce test, the registries are cleaned to eliminate duplications and other errors. Since the recreational registry is by users, the registry entry should be unique. Any duplication of identifier, such as address, email address, or phone number; are normalized to determine if they are duplications or separate individuals within a household or organization. However, we expect recreational registration rules to change from operator registration to aircraft registration in the foreseeable future, which would require additional cleaning of the recreational registry.

Part 107 requires that each asset/aircraft have its own registration. As such, users are often duplicated in the commercial/Part 107 registry. All registrations with duplicate address, emails address, phone number, or organization’s name are normalized to identify unique registrants.

Maximize Response Rate

The survey design has several procedures to maximize the response rate. These procedures include data collection protocols, combining survey data with supplementary data, and outreach with stakeholder organizations.

Awareness Campaign:

The survey starts with a pre-survey awareness campaign. This campaign entails emails sent to all registrants in both registries and messages on social media to warn registrants of the coming survey. The emails serve two purposes. First, it provides an accurate count of registrants with inaccurate emails as well as the number of registrants who are reading the emails. This information helps to correct any mistakes in the registry and update email subject lines to maximize open rates. Second, it provides information to the registrant about when to expect the email as well as why completing the survey is important to both them and the national as a whole.

Invitation Emails:

Invitation emails contain a link to the survey along with information regarding the importance of the survey. The invitation email acts as a cover letter to the survey instrument. The email contains the seal of the FAA and the signature from a trusted authority at the FAA. The email includes the logos of all participating sponsor organizations. The email explains how the information given will determine future investment in UAS as well as inform future rule making. Invitation emails also contain an email and phone number to answer any questions about the survey.

Follow-up Emails:

Follow-up emails are remainders to complete the online questionnaire. The follow-up are send to all sampled registrants initially. These email are similar to the invitation emails. Subsequent follow-up emails are only sent to registrants who have not completed the survey online. These emails implore the registrant to complete the questionnaire and restate how their responses are used for investment and rulemaking in their industry and hobby. A last chance email is send a week before as well as two days before the close of the survey. This emails clearly states the last opportunity to contribute and how the information can be misrepresented without everyone’s response.

Thank-you Emails:

The week after the closing of the survey, thank-you emails are sent to all those who have completed the questionnaire. The email contains an expression of thanks along with the importance of the responses collected.

Social Media Campaign:

During the survey, a social-media campaign is conducted, asking individuals who have registered with the FAA to look for the invitation emails. This includes messages on webinars, post of Facebook and Twitter, and banners on the FAA DroneZone website. These messages implore registrants to search for the invitation email and check junk boxes for FAA communication.

Collaborate with UAS Organizations:

As part of the pre-campaign, community-based UAS organizations and UAS trade groups are connected directly. These organizations are given a preview of the questionnaire and asked to discuss the survey with their communities. Suggestions from these organizations are collected for alteration of the survey during the renewal process. Organizations who are satisfied with the survey design are allowed to put their logo in the invitation email.

Imputation for Item Non-Response

All questionnaires with missing responses are kept if the core flight behavior questions are answered, that is questions 2 and 3. All missing questions us the remaining responses and geographic data to predict the responses to unanswered questions in the questionnaire. Give that many questions are on a type of likert scale, an ordered probit is used to regress the remaining responses and geographic information on questions with responses. The estimated model is then used to predict missing responses.

Non-Response Bias Estimations

As part of survey, a post-survey non-response study will be conducted. A stratified random sample of non-responders are contacted using the phone number and physical mail from the two registries. The questionnaire is conducted along with supporting questions regarding reasons for not responding. These answer and geographic information are compared to the responses and geographic information from the survey to determine if there are systematic differences in responses.

4. Describe tests of procedures and methods to be undertaken.

For estimating county level flight behavior, some counties are expected to have few than 30 responses. These counties are match with other within-state counties using probability matching. Respondents from matched counties are randomly assigned to the county without 30 responses until the threshold condition is met.

5. Provide the names of consultants and the person who will collect and analyze the information.

The survey is conducted by the Office of Aviation Policy and Plans (APO) in the Federal Aviation Administration (FAA). The following individuals are collecting and analyzing the information from the survey:

Supervisor Economist: Michael Lukacs 202-267-9641 michael.lukacs@faa.gov

Economist: Dipasis Bhadra 202-267-9027 dipasis.bhadra@faa.gov

Economist: Gavin Ekins 202-267-4735 william.g.ekins@faa.gov

1 The commercial questionnaire was conducted under OMB Control #2105-0573 while the recreational questionnaire was conducted under OMB Control #2120-0772.