Part B FRA WFD Survey

SUPPORTING JUSTIFICATION – Part B
Workforce Development Survey; OMB No. 2130-NEW
Form Number FRA F 6180.158
1. Description of sampling method to be used.
This study is specifically focused on deriving information related to workforce issues in the
Railroad industry. The sampling (analysis) unit are the organizations involved in railroad
industry workforce development, the railroad workforce and organizations impacting the
railroad workforce. The observation units are officials in these organizations that have
knowledge of the role that their organizations play in determining and impacting workforce
issues. In view of the very qualitative nature and the specific requirement of the study, and
the involvement of groups of different organizations, mixed sampling percepts are adopted in
determining the sampling approach.
We adopt an expert sampling methodology. Expert sampling - a type of purposive sampling
technique was chosen since the study needs knowledge from individuals that have particular
expertise to respond to survey items. We developed stratified sampling approach to determine
the subgroups within the target population.
The total population of the railroad industry is extensive, reaching across many classes of rail
operations and including professional organizations and academia. Class I rail organizations
employ thousands although the number of organizations is very small. In contrast Short Line
and Regional rail is comprised of many organizations employ a small number of people. Also
included in the survey strata are labor unions, associations and academia.
From the thousands of personnel associated with the railroad industry this survey will focus
on a study population of those organizations that exercise or influence workforce
development activities. Specifically, these study populations include organizations from:
•
•
•

•

Class I Passenger Railroads
o Large passenger railroads with annual carrier operating revenues of $433.2
million in 2011 dollars
Class I Freight Railroads
o Large freight railroads with annual carrier operating revenues of $433.2
million in 2011 dollars
Short Line and Regional Railroads
o Regional (Class II) with annual operating revenues greater than $37.4 million
in 2011 dollars
o Short-Line (Class III) with annual operating revenues less than $20 million in
1991 dollars
Academia
o Learning institutions with dedicated curriculum and training programs for the
railroad industry
1

•
•

Labor Unions
o Union organizations with a specific focus on railroad workforce membership
and railroad programs
Professional Associations
o Associations with a specific focus on railroad workforce membership and the
railroad industry

These six groups of organizations define the strata for this study. From this strata, a sample of
the organizations will be purposively selected to represent the railroad community. The
survey population of interest was identified through working with management
representatives and research to pinpoint organizations with workforce development expertise
and experience within their organization. These organizations have been identified for their
experience and expertise to provide the qualitative data needed. Short-line corporations (Class
II and Class III) were included in the population under study.
The targeted organizations have the necessary experience and exposure to workforce
development activities to provide valuable information regarding the efficiency and
effectiveness of ongoing activities and provide input on opportunities and threats that should
be addressed. FRA, therefore, believes this is the best sample from which to draw this
qualitative data. We believe this group to be very knowledgeable about the railroad
workforce issues that are under study and they will be responsive and cooperative.
The scope for this is survey includes capturing qualitative data to understand the current status
of workforce development activities across defined parameters. These parameters or survey
items were developed by FRA to encompass the wide range of processes that comprise the
workforce development lifecycle. The below list represents the Survey Items for this survey.
•
•
•
•
•
•
•
•

Workforce Population
Aging Workforce
Workforce Diversity
Workforce Development Data
Industry Image Considerations
Training in the Industry
Work/life Balance
Technology Adoption

Using the study population as a starting point the survey team was able to develop the
sampling frame using the parameters to define which organizations to include. The primary
rule of inclusion used to determine the survey frame was organizations that were expected to
have knowledge and experience with the survey items. Understanding that the survey groups
vary significantly in size and total organizations represented, the survey will be fielded using
stratification techniques to ensure each group is fairly represented. The sampling frame
consists of an extensive list of organizations and subject matter experts from across the
industry.
2

FRA intends to collect this data every two years so the data will reflect the current state of the
rail industry as it relates to its workforce. The qualitative data FRA will collect via this survey
is not intended to draw inferences to matters outside the design of the survey items. In this
sense we are adopting a design based inference approach. The FRA has chosen an electronic
survey instrument for data collection. A survey instrument will allow for the collection of
consistent data across years and the data the FRA wishes to collect can best be collected with
the use of the electronic survey instrument.
The FRA has carefully considered the benefits and disadvantages associated with an
electronic data collection methodology. A web-based survey methodology may exclude
individuals that may be less comfortable with technology. However, given the nature of work
the preponderance of the survey population performs, technology challenges in taking the
survey are not anticipated. For this data collection effort, FRA made several assumptions
regarding sampling size:


The sample population consists of organizations involved in workforce
development activities from across the railroad industry, associations, unions
and academia. Duties regularly require work with personnel to support the
full workforce development lifecycle, ranging from recruiting and training to
retention and succession planning.



The sample population adequately represents all the strata comprising the
population as determined by the expert opinion of the survey team.



Data from the sample will be used for qualitative purposes only. No
quantitative analyses will be conducted on sample data.

The survey does not include questions regarding age, years in craft or primary work region.
The survey questions are designed for and targeted to a very specific population with
workforce development expertise, in order to increase the quality of the responses. The
demographics of the respondents are not considered relevant to the quality of the responses or
the representation of the workforce development community.
The survey population includes representatives from railroad organizations, academia, labor
unions and railroad associations. This broad spectrum approach is intended to ensure a
comprehensive representation from all facets and interests in railroad workforce.
Understanding that these entities have differing policies, goals and cultures, the survey
questions are slightly tailored to align to their perspective.
Although any given group to be surveyed may represent a small percentage of the total
population of railroad entities, survey sample sizes are consistently applied across all survey
groups to present a fair representation of all groups.
3

It is anticipated that approximately 50% of those invited to participate in the survey will
respond. This number is based on interaction with the stakeholder community and FRA’s
previous experience in soliciting data from these groups. The calculated sample was adjusted
to reflect the invitations needed to obtain the desired sample size given a 50% response rate.
If after the first year of this survey data collection effort it is determined that actual response
rates are much higher or lower than anticipated, FRA will adjust its assumed response rates
and sample sizes accordingly.
To address non-responsive groups, if any specific strata achieve less than a 30% response, the
survey team is prepared to send the survey to additional organizations to increase the survey
response rate for that survey group. This will ensure all organization groups are fairly
represented in the response pool.
2.

Description of procedures for information collection, including statistical
methodology for stratification and sample selection.
Survey data will be collected once every two years. FRA has considered the use of less
frequent data collection periods and determined data collection should occur every two years
to support the publication of results to industry and to establish trends as appropriate. If
proposed collection of information were not conducted or conducted less frequently, the
ability for FRA to assess the status of the industry workforce requirements and challenges
would be diminished.
Trustworthy qualitative data collected over a period of years is needed to ascertain the extent
of the workforce problems and to inform the design and development of potentially effective
programs, and to realign activities as needed.
The survey design limits the scope of the data collected to specific survey items and therefore
limits the ability to draw inference beyond the scope of the workforce related topics being
studied. This approach best fulfills the intent of the survey by providing specific information
about targeted workforce related issues. However, in the future, the approach may be changed
if a wider set of information is needed or the FRA desires to extrapolate inferences to other
workforce issues not currently covered in the survey.
Using a purposive sampling technique (nonprobability approach), the survey team has
developed the sample survey respondent organizations using the sample frame, based on their
experience and influence within the railroad industry and the workforce development domain.
The judgment of the research team was required to identify organizations that have subject
matter expertise relative to the survey items to be studied.
The survey frame consists of the organizations from across the industry that the survey team
deemed most likely to provide a valuable response. The frame consists of 318 organizations
4

across the selected strata. These strata represent the railroad industries’ organizations with
workforce management and/or development expertise. These are organizations that provide
for and support workforce development activities for their respective agencies and
organizations. Academia representatives provide the training and partnership with industry to
identify skills gaps and to develop and deliver the education needed to support the railroad
industry. Railroad associations focus on specific railroad issues and concerns while capturing
and highlighting workforce related needs and trends. FRA, therefore, believes these strata are
the best population from which to draw a sample for this data collection effort.
Of the 318 organizations shown in the table below a total of 91 invitations will be sent. Based
on the size and span of the Class 1 organizations we will attempt to survey 100% of the
population. Although the number of Short Line and Regional organizations is large, each
organization is much smaller. Therefore, we believe a 25% survey will adequately represent
the strata. The remaining survey strata (Labor Unions, Associations and Academia) will be
surveyed at 50% to provide the necessary representation of that segment of the industry
population. Based on previous experiences we believe that a 50% response rate will be
achieved and therefore we conservatively estimate a minimum response from 45 organizations
for this survey.

3.

Survey Strata of
Industry Population

Frame #’s per
Strata

Class I Passenger
Class I Freight
Short Line & Regional
Labor Unions
Associations
Academia
Total

5
10
250
15
20
18
318

Derived Survey
Invitations per
Strata
5
10
50
7
10
9
91

Estimated Survey
Responses per
Strata
3
5
25
3
5
4
45

Description of methods to maximize response rate and to deal with non-response
issues.

This qualitative survey data collection effort will be carried out in collaboration with the
leadership of the relevant stakeholder groups. A standard introductory email will be created by
FRA. This email will alert the survey sample that they will receive an invitation to participate in
the survey. All potential participants will receive the same information.
The survey invitations have been designed to enhance response rates. Specifically, introductory
emails sent to respondents will come from official email addresses. The introductory emails will
explicitly state the purpose of the study and the estimated time to complete the survey. The
emails will also stress that no special preparation is needed to participate. The survey has also
been extensively reviewed to insure that the questions use simple and straightforward language.
Estimated time to complete the 30 question survey is 20 minutes.
5

Participation will be encouraged by FRA leadership, indicating that the importance of the survey
data in structuring workforce strategies and identifying future workforce development activities.
By completing the survey, they will be able to influence how these interventions are designed
and implemented. The opportunity to provide their input should provide adequate motivation for
participation.
Prior to the issuing of survey invitations, a pre-notification of the survey will be released. This
pre-notification will be available on the FRA’s website. This pre-notification will explain the
purpose of the survey including: when data will be collected, how data will be collected, the type
of information to be collected, and why this information is being requested. Pre-notification and
survey invitations will contain contact information for FRA personnel who can be contacted for
additional information about the survey.
The survey will be a web-based (Survey Monkey) data collection effort. Respondents will be
invited to respond via an e-mail containing a hyperlink to an online survey instrument. FRA is
asking for response by survey respondents within a few days of receipt of the information as emails tend to get lost and respondents may forget about the survey invitation. To alleviate
forgetfulness, reminder notices will be sent, one week, two weeks, three weeks, one month, and
45 days after the initial invitation is sent. In an effort to increase response rates for this survey
data collection effort, reminder email subjects will be modified. Respondents will be given the
flexibility to respond at any time up to 60 days after the initial survey invitation is sent. These
reminders should increase data collection for those groups who didn’t initially respond,
increasing the number of respondents.
If after 45 days FRA finds a low response rate, time to respond will be extended out to 90 days
and FRA will encourage those who have received an invitation to complete the survey within the
new extended timeframe.
Individuals may be more comfortable completing a paper version of the survey. The email
invitation to participate will also include information regarding how to request and complete a
paper survey for those who are uncomfortable or unwilling to complete the survey online. A
printable electronic copy of the survey will be provided. Those who receive an email invitation
to complete the survey but would prefer to do so via paper will receive instructions on how to
obtain a paper copy and where to send it upon completion.
Each question in the survey will also have a “prefer not to answer” option for those questions
where the decision by the respondent not to respond was intentional. Analysis of non-responses
will not be pursued.
4.

Describe any test procedures for procedures or methods to be undertaken.

Pre-testing of the survey questions and survey tool functionality was conducted prior to ensure
questions are clear and that the tool is designed to present questions as required.
6

The pre-testing process proceeded as follows:
•

Key personnel at the FRA were consulted to elicit their input on what data collection
is critical for the survey.

•

Analysis of the most recent Modal Profile was conducted to inform and validate data
collection goals.

•

Survey questions were developed and modified to improve clarity and align to survey
items and objectives.

•

Knowledge derived from the above was augmented by workforce development
experts to validate and confirm approach and target goals.

•

The survey was developed and revised by a team consisting of contractor and FRA
personnel.

•

A draft final survey will be created in the survey tool (Survey Monkey) and tested by
FRA personnel to insure proper functionality.

After the initial survey administration, FRA plans to review the survey with relevant
stakeholders prior to subsequent deployment. FRA will discuss any changes to survey questions,
sampling, or administration strategy as required with OMB and resubmit the survey information
to OMB for approval as needed.
5.

Provide name and phone number of individuals consulted on statistical aspects of
study design and other persons who will collect/analyze information for agency.

Ramesh Rajagolapan, Ph.D. (Consultant)
Dennis Lasley, M.Ed., Lean Six Sigma Black Belt (Consultant)
Tosan Tutse-Tonwe (Consultant)
Eric Saa (Consultant)
Monique Stewart, FRA (Executive Champion)

7