Beyond_Compliance SS Part B_07-13-20_clean

_{Department of Transportation}

Office of the Chief Information Officer

Supporting Statement B

Beyond Compliance

Part B. Collection of Information Employing Statistical Methods

1. CIRCUMSTANCES REQUIRING INFORMATION COLLECTION

Section 5222 of The Fixing America’s Surface Transportation Act (FAST Act)(Pub. L. 114-94, 129 Stat. 1312, Dec. 4, 2015)(49 U.S.C. 31100 note) requires the Federal Motor Carrier Safety Administration (FMCSA) to allow recognition, including credit or an improved Safety Measurement System (SMS) percentile, for motor carriers that:

1. Install advanced safety equipment;

2. Use enhanced driver fitness measures;

3. Adopt fleet safety management tools, technologies, and programs; or

4. Satisfy other standards determined appropriate by the Administrator.

The purpose of this study is to identify carriers with safety programs that go “beyond compliance” and to gauge their opinions on the perceived effectiveness of various safety program practices and technologies so that they may be incorporated into the design of a Beyond Compliance program. In the context of this study, compliant carriers are those whose safety performance allow them to avoid being prioritized for intervention by FMCSA via the SMS. This study investigates motor carriers who take actions that allow them to do more than simply avoid FMCSA intervention, but instead operate at the highest levels of safety performance relative to their peers. The four actions referenced above are indicative of motor carriers that have taken steps to achieve safety programs that go “beyond compliance.” The Beyond Compliance program would incentivize motor carriers to incorporate these actions into their safety programs by offering formal recognition, including credit or an improved SMS percentile.

The FAST Act also requires the FMCSA Administrator to carry out the Beyond Compliance provisions through, the following, to include among other things:

1. Developing a process for identifying elements of technology and safety programs as a basis for recognition.

2. Seeking input from stakeholders.

3. Authorizing utilizing a third party for a monitoring program.

4. Providing a report to Congress.

The data being collected for this study consists of responses from a select group of motor carriers on what they perceive to be the most effective technologies, programs, and policies for achieving safe operations. The input received from motor carriers will supplement information already gathered from in-depth interviews on motor carrier safety programs with nine high-performing carriers. This study is an exploratory analysis that relies on input from carriers that exhibit safe operations exceeding industry averages as indicated by driver out-of-service (OOS) rates, vehicle OOS rates, and crash rates. To identify these carriers, the study will utilize existing data from the Motor Carrier Management Information System (MCMIS) database.

Study Objectives: The objective of the study is to identify the elements of high-performing carriers’ safety programs that all carriers might be incentivized to adopt via a Beyond Compliance program.

To achieve these objectives, the study will:

1. Identify high-performing carriers in terms of safety performance.

2. Use the opinions of high-performing carriers to gauge the relative effectiveness of those safety technologies, programs, and policies for achieving safe operations that go beyond compliance.

2. DESCRIBE POTENTIAL RESPONDENT UNIVERSE AND ANY SAMPLING METHOD TO BE USED

The potential respondent universe is limited to only those motor carriers with safety performance records that are superior to industry averages. Those carriers will be identified using data available in MCMIS (i.e., DOT-reportable crash rates, driver OOS rates at roadside inspections, and vehicle OOS rates at roadside inspections). Only those carriers that perform near the top quartile across all three categories will be potential participants. Furthermore, those high-performing carriers are stratified into three size-based strata according to the number of power units they own: small (9 or fewer power units), medium (10–99 power units), and large (100 or more power units).

Carriers must have had at least three driver inspections and at least three vehicle inspections to be included in the sample. This will ensure the carrier’s safety performance is based on sufficient data.

Based on a review of the data available in MCMIS, carriers eligible for the survey will be those that meet the criteria listed above and that have:

• Driver OOS rates less than or equal to 5 total inspections with a driver OOS violation per 1,000 total driver inspections;

• Vehicle OOS rates less than or equal to 90 total inspections with a vehicle OOS violation per 1,000 total vehicle inspections; and

• Reportable crash rates less than or equal to 10 reportable crashes per 1,000 total power units.

These thresholds are near the top quartile of safety performance for large, medium, and small carriers, and they were chosen so that no carrier size group would be eliminated by the study’s participation criteria.

As of November 2018, there are 24,152 carriers that meet the above criteria.¹ Those carriers are divided into three strata: strata 1, large carriers, contains 126 observations (n₁ = 126); strata 2, medium carriers, contains 2,717 observations (n₂ = 2,717); and strata 3, small carriers, contains 21,309 observations (n₃ = 21,309). Of the 21,309 carriers in this population, a total of 225 will be sampled randomly across each strata (X₁ + X₂ + X₃ = 225) using the following sampling sizes:

• Large (X₁ = 75 carriers, selected at random).

• Medium (X₂ = 75 carriers, selected at random).

• Small (X₃ = 75 carriers, selected at random).
  
  

The reason for equal sample sizes (X₁ = X₂ = X₃ = 75) across the strata and for not sampling proportionate to their fraction of the overall population is to improve representation across the Medium and Large carrier size categories. The number of motor carriers in the Large and Medium groups are much smaller than the Small carrier group. If they are not sampled at higher rates relative to their proportions of the overall population of high-performing carriers, the study could potentially miss important insights that are specific to those groups. In addition, Medium and Large carriers are likely to have more firsthand experience with the range of safety technologies that are being investigated as part of this study.

Once the survey is administered and the results collected, the mean responses will be examined in order to determine if there are variations in response by strata (i.e., carrier size). If so, then the project team weight the results using the proportion of carriers by size in the population of high-performing carriers. The mean responses will also be examined within the strata (for example, 10 – 20, 21 – 50, and 50 – 99 for Medium carriers) to determine if there are variations in response. This will help to reduce the variance in the mean carrier scores across safety program elements. In this manner, the project team will be able to collect enough data to analyze carrier responses both within and across the strata.

Despite that some carriers will not respond to the survey request, the project team believes that we will be able to achieve a 50 percent response rate. This belief is based on the following factors: the observed response rates achieved by other researchers investigating the motor carrier industry (discussed in greater detail in section 3.2); the significant outreach effort that the team intends to perform as part of this information collection; and, the opportunity to help shape an FMCSA program that provides incentives to carriers. The 225 randomly selected carriers will be asked to participate in the study, with a goal of 112 responses.

3. DESCRIBE PROCEDURES FOR COLLECTING INFORMATION, INCLUDING STATISTICAL METHODOLOGY FOR STRATIFICATION AND SAMPLE SELECTION, ESTIMATION PROCEDURES, DEGREE OF ACCURACY NEEDED, AND LESS THAN ANNUAL PERIODIC DATA CYCLES

   1. Hypotheses

The hypothesis associated with the study can be stated as:

Carriers that exhibit a high level of safety performance share similar experiences and utilize similar tools, technologies, practices, and policies that enable them to achieve that level of performance.

2. Information Collection Procedures

As described in Section 2, high-performing carriers will be identified using data available in MCMIS. Preliminary results show that 24,152 carriers are eligible to participate in the study. Of those carriers, 225 will be randomly selected across the three strata (75 from each of the three strata based on carrier size) for participation in the study. Using the R programming language, carrier IDs as given in the MCMIS database will be sorted in ascending order within each strata and the positions of those IDs in those lists will be recorded. Using a random number generator for a uniform distribution also in the R programming language, a total of 225 random variates (75 in each strata) will be generated distributed across the three strata. The random uniform distributions will have ranges of (1 ... n₁), (1 … n₂), and (1 … n₃) according to the size of each strata. The carrier ID indices that correspond to those random variates identify the carriers that are randomly selected within each strata to participate. The study team will then contact these 225 randomly selected carriers via email—using contact information for carrier representatives currently stored in MCMIS—and request their participation in the study. The project team will also reach out to carriers using professional contacts within the industry/industry associations, through e-mail and phone when necessary/possible, as needed. In addition to the initial outreach by email to the selected carriers, the project team will also send email reminders as necessary to encourage them to complete the survey.

A total of 225 carriers will be sampled with the goal of achieving a targeted sample size of 112 assuming a response rate of 50 percent. As discussed in greater detail in section 3.5, 112 is the targeted sample size because it will yield an acceptable margin of error for the mean carrier scores for the various potential elements of a motor carrier safety program, which is the statistic of interest in this study. Given that this information collection represents but one of multiple factors that will be considered in the design of a Beyond Compliance Program (among them in-depth carrier interviews and a literature review of motor carrier safety technologies and practices), it is not necessary to increase the sample size with the goal of reducing the margin of error beyond that which is achievable with the current proposed sample size. Also, the information being collected (motor carrier perceptions on the effectiveness of safety program elements) changes fairly rapidly as new technologies are introduced in the market and new practices emerge in response to changes in the business environment. This information collection simply serves as a starting point for a Beyond Compliance program that will necessarily change and evolve over time.

The assumed response rate is based on rates achieved in past studies of the motor carrier industry and the project team’s belief that the opportunity to inform the development of a new FMCSA program will provide incentive to participate. Response rates for motor carrier surveys vary over a wide range and are likely influenced by a number of factors including the length of the survey, the nature of the questions being asked, and the sponsoring agency, among others. For example, the pilot test for the California Vehicle Inventory and Use Survey yielded a response rate of 2.6 percent.² However, a study of motor carrier safety adoption patterns in the U.S. conducted a survey of large carriers that yielded a response rate of 50.55 percent.³ Other research efforts including a study of motor carrier satisfaction with an online credentialing and tax payment system in Kentucky⁴ and a study of motor carrier satisfaction with electronic data interchange (EDI) technology⁵ yielded response rates of 19 and 47.1 percent, respectively. For this study, the research team is anticipating a higher response rate, given the opportunity it provides carriers to inform the development of a new FMCSA program and the targeted solicitation efforts of the research team. To increase the response rate, the research team will engage industry groups, such as the American Trucking Associations, National Tank Truck Carriers, the National Private Truck Council, State trucking associations, and others to make potential participants aware of the study. This will also help to identify specific individuals within motor carrier companies that may be best able to provide informed answers to the survey.

The 225 selected carriers will be asked (by email) to complete the survey. The primary format of the survey will use the analytic hierarchy process (AHP) framework, which will work by presenting participants with potential elements of a safety program and allowing them to systematically compare those elements. In addition to the AHP-based component, the survey will also include a component that asks carriers to provide information on how they measure safety performance, how they measure regulatory compliance performance, and what technologies and business practices they deploy to improve safety and regulatory compliance performance. The format for this component of the survey will be a combination of checklists and free-form response.

The survey results will then be analyzed to determine the safety program elements that were most frequently scored the highest across participants for the AHP component and most frequently cited for the checklist and free-form response component. The resulting information will reveal the elements of safety programs that motor carriers believe to be the most effective for achieving safety. These elements will be reviewed and may be incentivized in a Beyond Compliance program.

The specific information that will be collected is described in Table 1.

Table 1. Information to be collected.

Item	Description
Motor Carrier Name	The name of the motor carrier.
Motor Carrier/USDOT Number	The USDOT number of the motor carrier.
Place of Business Address	The primary address of the motor carrier.
Point of Contact Name	The name of the person who will serve as the point of contact for completing the survey.
Phone Number	The phone number of the person who will serve as the point of contact for completing the survey.
Email Address	The email address of the person who will serve as the point of contact for completing the survey.
Safety Performance Measures	The measures of effectiveness a carrier uses to assess safety performance.
Regulatory Compliance Measures	The measures of effectiveness a carrier uses to assess regulatory compliance.
Normalizing Variables	The variables a carrier uses to establish crash rates and/or out-of-service rates (e.g. miles, number of power units, etc.).
Focus Crash Types	The types of crashes, if any, that a carrier focuses on when performing safety performance analyses (e.g., rear-end, jackknife, etc.)
Crash Causal Factors	The factors a carrier considers when attempting to determine the cause of a crash (e.g., speed, distraction, fatigue, etc.).
Technology Solutions	The technologies a carrier has deployed in order to improve safety and regulatory compliance performance (e.g., rollover stability, lane departure warning, etc.).
Business Practice Solutions	The technologies a carrier has deployed in order to improve safety and regulatory compliance performance (e.g., rewards for safe driving, post-crash/incident training, etc.).
Relative Scores for Safety Program Elements	The carrier’s opinion of the relative importance of a safety program element expressed as a numeric value.

3. Information Collection Tools

Information will be collected via an electronic survey that consists of two components that utilize different survey designs. One component of the survey will gather information on the measures of effectiveness that carriers use for safety performance and regulatory compliance. Carriers are asked to provide information on the technologies and business practices they deploy to improve performance for their identified performance measures. The design of this component will consist of a combination of checklists and free-form response. An example of the design for this component of the survey is provided in Table 2.

Table 2. Example of Data Collection on Safety Measures of Effectiveness.

What safety measures of effectiveness do you track (check all that apply)?

□ Crash rates

□ Preventable crash rates

□ DOT Recordable crash rates

□ Preventable DOT Recordable crash rates

□ Critical events (e.g., hard braking, aggressive driving, speeding, motorist complaint calls, etc.)

The other component of the survey, which will use the analytic hierarchy process (AHP) framework, will determine the degree to which carriers prefer one technology or business over another for improving safety performance. The AHP is a tool for dealing with complex decision-making that employs a series of structured, pairwise comparisons in which respondents must express a preference for one alternative over another according to various evaluation criteria. In addition to the primary information on carrier measures of effectiveness for safety and regulatory compliance, technology and business practice solutions for improving safety and regulatory compliance performance, and the expressed opinions of carriers on various technologies and business practices collected in the AHP-based component of the survey, demographic information on the respondents will also be collected (i.e., motor carrier name, motor carrier and/or U.S. Department of Transportation [USDOT] number, place of business address, and phone number) as shown in Table 1.

The first step in implementing the AHP is to define the goal of the decision-making process. In our model, the goal is as follows:

• Determine the elements of a safety program that best represent a carrier operating at a superior level of safety performance, thereby considered “beyond compliant” with minimum safety standards.

The second step in implementing the AHP is to define the evaluation criteria or factors by which achievement of that goal is assessed. In the context of this study, these consist of the various tools, technologies, and practices available to motor carriers as parts of their safety programs. Based on the results of the literature review and the nine in-depth carrier interviews already conducted as part of this study, the project team identified seven broad safety program areas that are thought to be potentially key to a successful safety program. As shown in Table 3, as illustrated by examples, these are:

1. Advanced Safety Equipment;

2. Fatigue Management;

3. Driver Training;

4. Hiring Practices;

5. Data Analytics;

6. Safety Culture; and

7. Safety Incentives/Discipline.

Within the context of the AHP, the research team aims to use industry representatives to reveal the level of importance of each of these categories to contributing to safe operations via a series of pairwise comparisons. Furthermore, the comparisons will also capture the relative importance of the specific program elements within each category.

Table 3. Safety program elements exceeding regulatory requirements.

Safety Program Areas	Advanced Safety Equipment	Fatigue Management	Driver Training	Hiring Practices	Data Analytics	Safety Culture	Safety Incentives/ Discipline
Safety Program Elements	Rollover Stability Collision Avoidance Lane Departure Warning Video-Based Safety Monitoring Blind Spot Monitoring	Sleep Disorder Screening Sleep Disorder Compliance Monitoring Active Sleep Disorder Program Internally Modified Hours-of-Service Rules for Daytime and Nighttime Driving	Finishing Training Sustainment Training Post-Crash/Incident Training Simulation-based Training	Pre-Employment Screening Drug Testing Physical Functions Testing Qualifying Road Test	Predictive Analytics for Safety Performance Data Driven Risk Assessment	Safety as a Core Corporate Value Attempt to Measure Safety Culture	Rewards for Safe Driving Discipline for Unsafe Driving

The third step in implementing the AHP is the formulation and completion of the pairwise comparisons. Through pairwise comparisons of the safety program elements, local weights are calculated. Carrier opinions on the importance of safety program elements are expressed in terms of numeric valuations of one element relative to another, according to Saaty’s (1980) nine-point scale. As an example, when comparing the importance of Rollover Stability to Safety as a Core Value, a carrier may indicate that Safety Culture is strongly more important than Advanced Safety Equipment. To denote this opinion, the carrier will assign a value of 7 to Safety Culture as indicated by the underlined, emboldened “7” in Table 4.

Table 4. Example of Data Collection on Carrier Opinions.

Safety Program Element	Score	Safety Program Element
Rollover Stability	9 7 5 3 1 3 5 7 9	Safety as a Core Value

4. Analysis Methodology

The survey, which utilizes the AHP framework, will result in a score matrix of the various safety program elements. These scores reflect the perceived importance of the safety program elements of the carriers that complete the survey. There are several steps involved in the computation of these scores, the first of which is the computation of the weights of the different evaluation factors.

To compute the weights for the different factors, the AHP creates a pairwise comparison matrix A. The matrix A is an m x m real matrix, where m is the number of evaluation criteria considered. Each entry a_jk of the matrix A represents the importance of the jth criterion relative to the kth criterion. If a_jk > 1, then the jth criterion is more important than the kth criterion, while if a_jk < 1, then the jth criterion is less important than the kth criterion. If two criteria have the same importance, then the entry a_jk is 1.

The relative importance between two criteria is measured according to Saaty’s (1980) numerical scale from 1 to 9 (shown in Table 5), where it is assumed that the jth criterion is equally or more important than the kth criterion. The phrases in the “Interpretation” column of the table are used to translate the decision-maker’s qualitative evaluations of the relative importance between two criteria into numbers.

Table 5. Relative AHP scores.

Value of ajk	Interpretation
1	j and k are equally important
3	j is slightly more important than k
5	j is more important than k
7	j is strongly more important than k
9	j is absolutely more important than k

Once the matrix A is built, the normalized pairwise comparison matrix, A_norm, is derived by making equal to 1 the sum of the entries on each column. This is accomplished by computing each entry of the matrix A_norm as .

Finally, the criteria weight vector is developed by averaging the entries on each row of A_norm as .

The matrix of alternative scores is an n x m real matrix, S. Each entry, s_ij, of S represents the score of the ith option with respect to the jth criterion. In order to derive the scores, the pairwise comparison matrix B^(j) is first developed for each of the m criteria, j=1, …, m. The matrix B^(j) is an n x n real matrix, where n is the number of options evaluated. Each entry of the matrix B^(j) represents the evaluation of the ith option with respect to the hth option with respect to the jth criterion. If > 1, then the ith option compared to the hth option, while if <1, then the ith option is worse than the hth option. If two options are evaluated as equivalent with respect to the jth criterion, then the entry = 1. The entries and satisfy the following constraint:

.

Also, for all i. The evaluation scale introduced in Table 4 will be used to translate the decision makers’ pairwise comparisons into numbers.

The AHP applies to each matrix B^(j) the same two-step procedure described for the pairwise comparison matrix A; that is, it divides each entry by the sum of the entries in the same column and then averages the entries on each row. The results of these calculations are the score vectors s^(j), where j=1, …, m. The vector s^(j) contains the scores of the evaluated options with respect to the jth criterion.

Finally, the score matrix S is obtained as S = [s⁽¹⁾ … s^(m)]. Once the weight vector, w, and the score matrix, S, have been computed, the AHP obtains a vector, v, of global scores by multiplying S and w such that v = S · w.

5. Sample Size and Margin of Error

The purpose of the information collection is to gauge carriers’ opinions on the perceived effectiveness of various safety program practices and technologies so that they may be incorporated into the design of a Beyond Compliance program. This supplements information already collected during a series of nine in-depth, one-on-one interviews and information collected through a review of the motor carrier safety literature. Also, the information being collected (opinions on the perceived effectiveness of safety program elements) changes fairly rapidly as new technologies are introduced in the market and new practices emerge in response to changes in the business environment. Thus, this information collection simply serves as a starting point for a Beyond Compliance program that will necessarily change and evolve over time. Furthermore, there are no prior similar studies to indicate an expected population mean or level of variance for the data being collected. Because of these reasons, the target sample size of 112 carriers proposed for this information collection is believed to be sufficient for meeting the need of obtaining preliminary data.

4. DESCRIBE METHODS TO MAXIMIZE RESPONSE RATE AND TO DEAL WITH THE ISSUES OF NON-RESPONSE

To increase the level of participation in the pilot program, the project team will conduct direct outreach to high-performing motor carriers that are likely to have had experience with the specific safety programs examined in this study and to various industry groups. These include groups such as the American Trucking Association, the National Tank Truck Carriers, and the National Association of Small Trucking Companies, among others. By engaging these stakeholder groups, the project can increase awareness of the study throughout the motor carrier industry and the number of pre-approved carriers that participate in the study.

The research team will also increase the response rate by offering an online webinar and video that explains the evaluation design. As opposed to solely relying on written instructions to explain to participants how to complete the survey, the project team believes it would be useful to conduct an information session via webinar or a pre-recorded video so an example can be provided and any questions answered. The webinar would be conducted multiple times and participants would be given the option to select the one that best suits their schedules.

In addition to the initial outreach by email to the selected carriers, the project team will also send email reminders. The email reminders will help to boost the response rate by encouraging carriers that have not completed the survey to do so. Given that motor carriers are undoubtedly busy with the day-to-day activities of operating a business, email reminders are essential to helping the project team achieve the targeted response rate.

Taking steps (such as those described above) to increase the response rate is one method of reducing non-response bias. Despite these efforts there is still potential for non-response bias. To address this, the project team will examine the non-responses to determine if those carriers disproportionately share certain demographic characteristics (e.g., fleet size, employment size, cargo types handled, geography, etc.) that would indicate non-response bias. If present, the original carrier responses will be compared to weighted responses (i.e., weighted according to the proportion of the population of high-performing carriers that shares the same demographic characteristic as the non-respondents) to determine which is the most accurate measure.

5. DESCRIBE TESTS OF PROCEDURES OR METHODS TO BE UNDERTAKEN

There are no planned pilot tests of procedures or methods.

6. PROVIDE NAME AND TELEPHONE NUMBER OF INDIVIDUALS WHO WERE CONSULTED ON STATISTICAL ASPECTS OF THE INFORMATION COLLECTION AND WHO WILL ACTUALLY COLLECT AND/OR ANALYZE THE INFORMATION

FMCSA is sponsoring this information collection. The FMCSA point of contact is:

Nicole Michel

1200 New Jersey Avenue, SE

Washington, D.C. 20590

202-366-4354

nicole.michel@dot.gov

The contact persons at MaineWay Services and Cambridge Systematics are:

Gene Bergoffen

P.O. Box 166

Fryeburg, ME 04037

207-935-7948

bergoffen@roadrunner.com

Dr. Christopher Lindsey

730 Peachtree Street NE, Ste. 500

Atlanta, GA 30308

404-460-2602

clindsey@camsys.com

Don Osterberg

P.O. Box 166

Fryeburg, ME 04037

207-935-7948

don.osterberg@gmail.com

7. References

Saaty, T.L. (1980). The Analytic Hierarchy Process, McGraw-Hill, New York.

1 This estimate is based on an November 2018 analysis of the MCMIS database.

2 Jeong, K., Tok, A., Ritchie, S. G., & Park, J. (2016). California Vehicle Inventory and Use Survey: Pilot Study Insights. Transportation Research Record: Journal of the Transportation Research Board, (2547), 32-40.

3 Cantor, D., Corsi, T., & Grimm, C. (2006) Safety Technology Adoption Patterns in the U.S. Motor Carrier Industry. Transportation Journal, (45) 3, 20-45.

4 Langley, R. & Grossardt, T. (2004). 2003 Motor Carrier Survey. Report No. KTC04-03/SPR263-02-1F. Kentucky Transportation Center. https://uknowledge.uky.edu/cgi/viewcontent.cgi?article=1218&context=ktc_researchreports.

5 Crum, M., Premkumar, G., & Ramamurthy, K. (1996). An assessment of motor carrier adoption, use, and satisfaction with EDI. Transportation Journal, 44-57.