General Aviation Survey Non-Response Studies

General Aviation Survey

Non-Response Studies

The following is extracted from the 1999 survey report, paragraph A.5.1

The section briefly discusses the history of NR studies for the GA Survey, which may be useful. You'll notice that interest in a nonrespondent telephone survey resurfaced after response rate dropped from 70% to 50% along with an increase in the number of postmaster returns. Some caution must be exercised in interpreting these figures because we do not know how the response rates were calculated during those survey years. If the response rate did not retain "undeliverable mail" (postmaster returns) in the denominator, then the response rate will be inflated. We can document calculation of the RR since we began conducting the survey - and we error on the conservative side - but we don't have any documentation on how the previous contractor computed RRs. Similarly, I don't think we have any documentation or reports on the earlier telephone NR surveys.

"Adjustments Based on a Telephone Survey of Nonrespondents

From the conduct of the first General Aviation Activity (GAAA) Survey in 1977 through the 1990 Survey year, the survey data were not adjusted to account for nonrespondents (aircraft owners selected as part of the survey sample but who chose not to complete and return the form). This is because telephone surveys of nonrespondents conducted in 1977, 1978 and 1979 did not show any significant differences or inconsistencies between respondents’ and nonrespondents’ replies. In 1980, the telephone survey was discontinued as a cost-saving measure.

The GAATA Survey response rate has fallen from over 70 percent prior to 1980 to the 50 percent range in most years since 1983, and the number of postmaster returns has greatly increased. Therefore, the FAA decided to conduct a telephone survey of nonrespondents to the ratio of active aircraft and inactive aircraft between mail respondents and telephone respondents. Nonresponse adjustment factors derived from these survey results have been applied to the GAA Survey up through 1995. In 1997, a telephone survey of nonrespondents to the 1996 GAATA Mail survey was conducted. In 1998, a telephone survey of nonrespondents to the 1997 GAATA Mail survey was conducted. This survey showed significant differences between respondents and non-respondents to the mail survey. This information was used to correct 1998 estimates for nonresponse bias. The results of this telephone survey have also been integrated into the 1991 through 1997 surveys to estimate more accurately active aircraft and hours flown.

In 2000 this telephone survey was again conducted nationally to ask non-respondents about active aircraft and hours flown. However, although the methodology of the non-respondent survey is sound, it does not allow for correct implementation because only a small percent of non-respondents can be located. Furthermore, an analysis of the estimates from the telephone survey show great variability over the years compared to the much more stable mail survey estimates. The difficulty in implementing the survey and variability of telephone estimates resulted in the finding that the telephone survey was not a constructive addition to the estimates. Therefore, the telephone non-respondent survey was not used to adjust 1999 estimates and its implementation has been discontinued as of this time."

The conclusion at the time was that the adjustment can do more harm than good. (This problem is not uncommon for NR studies of this type.)

In 2004, the contractor began studying other ways to reduce non-response bias. These have not yet been executed, nor are they published in a formal report. The possible proposals were included in the original response to question #10, and are repeated in their entirety here.

"Maximizing the response rate is the first step towards reducing nonresponse bias.

Post-stratification Adjustments for Nonresponse

Further steps to understand the extent of nonresponse bias and develop adjustments to correct for bias are possible using the Civil Aviation Registry. The Registry could be used to examine correlates of survey participation and develop post-stratification weights that adjust for nonresponse bias. For nonresponse adjustments to be effective in reducing nonresponse bias, the variables used in the adjustment must be related both to the probability of participation and to outcome variables of interest (e.g., flight activity). This approach is limited to examining and adjusting for bias only insofar as it is reflected in variables that are observed (i.e., measured and available on the Registry). Careful analysis is necessary because post-stratification adjustments can also introduce additional bias.

The Civil Aviation Registry is the source of information for the sample frame for the GA Survey. It contains several variables that could be used to examine differences between respondents and nonrespondents. To be useful, the variables selected should be related to key outcomes of interest, such as active/not active, hours flown, and possibly how flown (e.g., on-demand Part 135 activity vs. general aviation). We would work with the FAA to improve our understanding of aircraft activity and how aircraft characteristics may be predictive of activity. At first glance, the following variables may prove useful:

Year of manufacture of the aircraft

o This variable is limited insofar as it contains a great deal of missing data. It may be possible to supplement this information by using make-model-serial number and matching to the Aircraft Blue Book. In theory, this can be done but it would be a manual process.

State or region in which the aircraft is registered

o A dichotomy such as Alaska vs. other may be a useful comparison, particularly given the importance of aviation activity in Alaska for a range of commercial activities and access to outlying regions. Further differentiation may be less useful, in part because the location in which the aircraft is registered may not match the location in which it is flown or based.

Aircraft type

o This includes Registry information that is used to classify aircraft by engine type and number of engines – Fixed Wing Piston – Single engine, Fixed Wing – Turboprop, Turbine Rotorcraft, Amateur, Light-sport, etc.

Last Activity Date on the Registry Files

o Usefulness is limited insofar as the “activity date” can reflect any action on the record, ranging from sale of aircraft or renewal of registration to a name change, address update, or spelling correction. Nonetheless, one would expect that an aircraft owner (or his/her agent) filing any kind of updates with the Registry is likely to be related to aircraft activity and survey response.

Registry Status

o Aircraft without a valid registration are removed from the sample frame prior to survey execution. However, other statuses, which may be predictive of survey response and flight activity, remain eligible for the survey. These include statuses such as Postmaster Return of Triennial Aircraft Registration, aircraft registered to manufacturers under a dealer certificate, registration pending, or sale reported. (Note: PMR status of more than 10 years or sale reported more than 5 years ago are excluded from the survey sample frame.)

On-Demand Part 135 Status (from OPSS database)

o We also draw upon the FAA’s OPSS’s database to execute the sample design. This database allows us to identify which aircraft on the Registry are certificated to fly on-demand Part 135, which is a sample stratum. This information is reliable and could be used to compare responders and nonresponders.

It is possible to conduct analyses in which we compare respondents and nonrespondents on these or other variables. This could be done periodically (e.g., every 2 years) assuming resources are available to perform the analyses. This type of analysis has not been done for several years (some investigation in 2004). An extensive nonresponse study was conducted in 1999 with a telephone follow-up study of nonresponders. That study provided evidence that the majority of sampled aircraft owners for whom we had good contact information responded to the survey (80 to 90%); most nonresponders did not receive the survey."

Since understanding the effects of non-response bias is an important element of this survey's size and response rates, we will align resources to study and, if necessary, compensate for any effect. We will include the results in the methodology appendix of the annual report and provide them to OMB.