Part B SAFE Program Supporting Statement 07022020

Supporting Statement

Safety Awareness, Feedback, and Evaluation (SAFE) Program

B. Statistical Methods

1. Describe the potential respondent universe.

The below table contains the universe of potential respondents. Each potential respondent, as a member of a stakeholder group, is required by federal regulations to hold a current certificate issued by the FAA Flight Standards Service. The required returns are based on a 50/50 split in the population on characteristics of interest 3% target precision and 95% confidence levels. In previous surveys administered to these populations, the response rate from Commercial and Airline Transport Pilots was 40.1%, General Aviation Pilots was 37.4%, Repair Station Operators was 54.5%, Aviation Maintenance Technicians and Repairers was 31.1%, and Air Carrier Operations Managers was 42.9%. Because respondents will be offered options in accessing the survey (i.e., personal computer, mobile device, or print), we are basing the drawn sample size on a 41% response rate, the average response rate obtained in the last survey response cycle. We expect at most that 10% of respondents will request a paper version.

			Stakeholder Group
		Commercial and Airline Transport Pilots	General Aviation (left column) and Sport Pilots (right column)		Repair Station Operators	Aviation Maintenance Technicians and Repairers	Air Carrier Operations Management
Population Size (2018 data)		262,025	224,404	6,513	4,801	327,384	1,895
Required Returns* (95% confidence level, 3% margin of error)		1063	1063	918	873	1064	683
Drawn Sample Size (adjusted for 41% response rate)		2593^	4832^		2129^	2595^	1666^
	^Based on 1064, the largest of the computed sample sizes for populations of 200,000+.

2. Describe the procedures for the collection of information.

A simple random sample will be drawn from the sampling frames provided by the FAA system of records for certified pilots, mechanics, and air carriers. The SAFE program standard for precision in the sample data is 3 percentage points from the population value with a 95% confidence level that the estimate is within the margin of error.

Stakeholder groups are on a 4-year data collection cycle, which minimizes the burden on respondents. Further, a mix-mode survey approach is being used, so respondents can choose the most convenient way to submit a survey--via a personal computer (PC), mobile device, or on paper. Thus, a unimode survey design is being used for each stakeholder survey to present respondents with identical experiences (e.g., item sequence, type, and wording) no matter which administration mode they choose.

3. Describe methods to maximize response rates.

Methods to maximize response rates:

• A mixed-mode survey methodology is being used to allow respondents the convenience of choosing how (PC, mobile device, paper) and when they complete the survey, within the opening and closing dates.

• Notices will be published in trade magazines and posted on professional organizations websites, to alert potential respondents of the upcoming survey.

• An invitation from the FAA Flight Standards Service to participate will be mailed out to potential respondents, once the survey is online. The invitation will detail the purpose of the survey; the importance and benefit of providing feedback to improve service quality and delivery; directions for accessing the survey in each mode; the date the survey closes; and contact information for tech support and questions.

• Returns will be tracked based on a unique identifier; so that reminder letters from the FAA Flight Standards Service sent out 2 weeks and 4 weeks after the survey opens will only go to those potential respondents that have not submitted a survey.

• At 2 weeks before the survey closes, a final reminder letter from the FAA Flight Standards Service will be sent to those who have submitted a survey.

Methods to assess generalizability:

• Non-response bias analyses will be conducted to determine if respondents differ systematically from non-respondents. The two groups will be compared on proportions of relevant characteristics found in the sampling frame data (e.g., type of certificate, years certified, region operating out of, and size of aviation operation). Proportional differences greater than 15% will be examined in relation to impact on survey outcomes to determine if it is necessary to contact non-respondents to minimize bias. Foremost, the analysis will reveal potential limitations of the survey results.

• Comparisons of responses will be made among survey modes while controlling for respondent characteristics to test for response biases before compiling the final dataset.

• Weighted sample procedures will be used to mitigate bias if respondents differ significantly from the population.

4. Describe tests of procedures and methods to be undertaken.

A thorough item review process will be employed to include reviews by subject matter experts to ensure proper use of terminology and comprehensiveness of response options. As well, the electronic survey modes will undergo verification and validation to ensure that the items and response options properly display and correctly respond to input, and that the output is accurately stored.

5. Provide the names of consultants and the person who will collect and analyze the information.

The survey design and analysis will be conducted under the direction of Dr. Jamie Barrett [Jamie. barrett@faa.gov or (405) 954-6836], Engineering Research Psychologist, located at the Civil Aerospace Medical Institute, in Oklahoma City.

The survey will be distributed and the data will be collected by a survey contractor, Cherokee Nation 3-S. Their point of contact will be Janine King (Janine.ctr.king@faa.gov).