2016_LEMAS_Part_B_FINAL

Part B. Collection of Information Employing Statistical Methods

1. Universe and Respondent Selection

The target population for the 2016 Law Enforcement Management and Administrative Statistics (LEMAS) survey is all local and county police, sheriff departments, and primary state police agencies that employ the equivalent of at least one full-time sworn officer (FTE).¹ The 2016 LEMAS will use the 2014 Census of State and Local Law Enforcement Agencies (CSLLEA) to identify the universe of eligible agencies. The sample of law enforcement agencies (LEAs) to be surveyed for the 2016 LEMAS will be drawn from this list based upon the sampling design described below.

Sampling Frame

The CSLLEA is the most systematic source of national data on the number of sworn and non-sworn personnel employed by LEAs nationwide. The CSLLEA is a census of all general and special purpose law enforcement agencies in the U.S. It collects information on LEA functions, facilities, personnel, and budget. The CSLLEA frame was created by using the 2008 CSLLEA and updating this content by obtaining each states’ most current Police Officer Standards and Training (POST) agency list. Information from the 2008 CSLLEA and POST lists were merged and vetted, resulting in the most comprehensive list of law enforcement agencies known.

LEMAS Sampling Designs and Response Rates

The LEMAS will use a sampling design based on the protocol used to develop the sample for past LEMAS surveys. Specifically, LEMAS uses a stratified simple random sample design in which LEAs are stratified by agency type and agency size. Agency type has three categories: (1) local police, (2) sheriff’s offices, and (3) state police. To obtain a representative sample of all agency sizes, the sample is stratified by agency size. Agency size is split into seven categories: (1) 1 FTE, (2) 2 – 4 FTEs (3) 5 – 9 FTEs, (4) 10 – 24 FTEs, (5) 25 – 49 FTEs, (6) 50 – 99 FTEs, and (7) 100 or more FTEs. In the LEMAS, LEAs with 100 or more FTEs are sampled with certainty making these agencies self-representing (SR). For agencies with less than 100 FTEs (non-self-representing), the LEMAS has employed various allocation methods depending on the analytic goals of the particular iteration.

The LEMAS has traditionally experienced a high response rate. For example, the 2013 LEMAS had an overall response rate of 86.3 percent. However, as seen in Table 1, the response rate varied by agency type and agency size. Additionally, many of these agencies, particularly those in the SR strata, will have received multiple surveys within a short time frame – a body worn camera survey from the Police Executive Research Forum (PERF) and the 2016 LEMAS Body-Worn Camera Survey Supplement. As a result, we will assume a response rate that differs by agency type and size with lower response rates for smaller agencies than in previous LEMAS waves.

Table 1: 2013 LEMAS survey response rates, by agency type and size

Agency Type	Agency Sizea	Sample Size	Response Rate
Local Police	100+	632	91.1%
	50-99	311	91.3%
	25-49	366	89.0%
	10-24	457	87.9%
	5-9	347	84.0%
	2-4	188	87.8%
	1	52	69.8%
Sheriff’s Office	100+	370	81.9%
	50-99	109	74.8%
	25-49	135	88.1%
	10-24	162	77.0%
	5-9	109	74.5%
	2-4	42	85.4%
	1	6	83.3%
State	All	50	92.0%

^a Number of full-time sworn officers.

Past experiences on the LEMAS suggest that response rates will be lower among smaller agencies and non-local agencies. Lower response rates and differential response rates are recognized as potential limitations to the 2016 LEMAS, and steps will be taken to identify and ameliorate any issues caused by such an occurrence. First, tests for potential bias in response will be performed by comparing agencies on known characteristics, including agency size, type, and geographic region. Second, the likelihood of potential bias will be reduced through nonresponse adjustments, which are discussed in a later section. The goal of this survey is to have unbiased estimates that can be obtained with lower response rates with the proper adjustments. Table 2 presents the assumed response rates for the 2016 LEMAS.

Table 2: Assumed response rates for the 2016 LEMAS, by agency type and self-representation status

Agency Type	Self-representation Status	Response Rate
Local Police	Self-representing	95%
	Non-self-representinga	85%
Sheriff’s Office	Self-representing	85%
	Non-self-representinga	75%
State Police	Self-representing	90%
All agencies		86%

^a Non-self-representing agencies are comprised of agencies with less than 100 FTEs.

Sample Allocation and Sample Size

As was done in prior LEMAS surveys, the 2016 LEMAS will select all large law enforcement agencies with 100 or more full-time equivalent sworn officers; these SR agencies are found in cells numbered 7, 14 and 15 in Table 3. This ensures our ability to measure change overtime within a consistent set of large agencies. The 2016 LEMAS will sample within the other cells in Table 3; these are the non-self-representing (NSR) agencies with less than 100 FTEs.

Table 3: Sampling Strata for the 2016 LEMAS

Agency Size Categories	Local Police	Sheriff Offices	State Police
One FTE Sworn Personnel	1	8
2 to 4 FTE	2	9
5 to 9 FTE	3	10
10 to 24 FTE	4	11
25 to 49 FTE	5	12
49 to 99 FTE	6	13
100 or More FTE	7	14	15

The NSR agencies have traditionally been subdivided into six strata based on the number of full time equivalent officers (FTE). These groupings have been used historically by BJS for reporting key characteristics of officers and agencies. They also serve as a stratification factor since independent samples would be drawn from each agency-type-by-size stratum. Such stratification will increase the statistical precision of estimates at both officer and agency levels. The size cut points have been used in prior LEMAS sampling designs for police departments and we retained them in this design for consistency between the police and law enforcement strata.

The 2016 LEMAS survey will address a wide array of topics. To optimize the sampling allocation, multiple objectives of estimation have been considered. We considered multiple allocation methods including proportional to number of agencies, proportional to number of FTE, and proportional to square root of FTE. We compared the allocations by evaluating the precision of estimates from each of the allocations. Six statistics have been identified as appropriate to consider in the allocation:

• three agency-level statistics—average annual operating budget, percent of agencies using body-worn cameras (BWC), and percent of agencies with community policing in their mission statement

• three personnel-level statistics—number of full-time sworn officers (FTS), number of full-time non-sworn (FTNS) officers, and percent of sworn officers that are female

Historic data from the 2013 LEMAS will be used to estimate the mean, variance, and standard deviation of the six statistics in each stratum (see Table 4 and Table 5). Using the 2014 CSLLEA, frame counts for each stratum are included in Table 6 as well as the distribution of agencies, officers, and square root of officers. For sample planning, a preliminary 2014 CSLLEA file is used. The final 2014 CSLLEA file is still being cleaned and processed.

Table 4: Estimates of means/percentages of selected statistics, by stratum, 2013 LEMAS

Agency Type	Agency Sizea	Budget	BWC Usage	Community Policing	Number FTS Officers	Number FTNS Staff	Female Officers
Local Police	100+	$65,644,976	22%	88%	456.6	131.5	15%
	50-99	9,738,990	23%	86%	66.3	18.4	9%
	25-49	4,329,273	24%	83%	34.8	8.8	8%
	10-24	1,789,268	33%	79%	15.5	3.3	7%
	5-9	596,791	40%	64%	6.6	1.1	6%
	2-4	268,367	29%	52%	2.9	0.2	6%
	1	100,667	36%	41%	1.1	0.1	5%
Sheriff Office	100+	76,150,598	22%	72%	346.9	296.6	15%
	50-99	12,140,765	25%	59%	71.7	65.3	12%
	25-49	5,127,476	34%	61%	38.2	33.1	11%
	10-24	2,131,020	29%	49%	18.3	15.1	11%
	5-9	933,274	41%	31%	8.3	7.9	9%
	2-4	401,919	17%	35%	4.1	3.1	6%
	1	131,076	40%	40%	1.6	0.2	5%b
State Police	All	255,956,091	23%	50%	1,168.4	601.5	6%

^a Number of full-time sworn officers.

^b For estimation purposes, this percent is set to 5% to have a valid standard deviation though the estimate was 0%.

Table 5: Estimates of standard deviations of selected statistics, by stratum, 2013 LEMAS

Agency Type	Agency Sizea	Budget	BWC Usage	Community Policing	Number FTS Officers	Number FTNS Staff	Female Officers
Local Police	100+	$222,396,327	2%	1%	1,662.6	641.4	0.1%
	50-99	4,320,058	1%	1%	15.7	10.8	0.2%
	25-49	1,807,782	1%	1%	7.7	6.5	0.3%
	10-24	860,957	1%	1%	4.6	2.8	0.3%
	5-9	267,601	1%	1%	1.9	1.5	0.6%
	2-4	260,990	1%	1%	1.2	0.6	1.1%
	1	64,142	1%	1%	0.9	0.4	3.7%
Sheriff’s Office	100+	201,958,784	2%	2%	653.0	637.6	0.1%
	50-99	7,533,508	2%	3%	14.8	48.9	0.4%
	25-49	4,558,408	2%	2%	11.1	31.4	0.5%
	10-24	1,595,649	1%	2%	6.9	15.2	0.6%
	5-9	478,142	2%	2%	2.8	7.2	1.2%
	2-4	335,159	3%	3%	1.3	5.8	2.0%
	1	93,117	10%	10%	0.9	0.4	7.7%
State Police	All	342,672,015	6%	7%	1,305.3	822.2	0.1%

^a Number of full-time sworn officers.

^b For the 2013 LEMAS, there was only respondent for entry-level salary for sheriff’s offices with an equivalent of 1 full-time sworn officer and thus no valid variance estimate, so a standard deviation of 7,000 is used for optimization.

Table 6: Distribution of agencies by stratum based on sample allocation, 2014 CSLLEA

Agency Type	Agency Sizea	Number of Agencies	Percent Agencies	Percent FTE	Percent Square Root FTE
Local Police	100+	677	4.1%	40.9%	21.1%
	50-99	887	5.4%	7.8%	9.3%
	25-49	1,728	10.5%	7.6%	9.1%
	10-24	3,452	20.9%	6.8%	8.6%
	5-9	3,096	18.8%	2.7%	5.4%
	2-4	2,472	15.0%	0.9%	3.1%
	1	1,086	6.6%	0.1%	1.2%
Sheriff’s Office	100+	401	2.4%	17.1%	13.7%
	50-99	366	2.2%	3.3%	6.0%
	25-49	611	3.7%	2.7%	5.4%
	10-24	921	5.6%	1.9%	4.5%
	5-9	513	3.1%	0.5%	2.2%
	2-4	210	1.3%	0.1%	0.9%
	1	26	0.2%	0.0%	0.2%
State Police	All	50	0.3%	7.7%	9.2%

^a Number of full-time equivalent officers.

The sample allocation for the three methods is calculated under the constraint that each stratum must have a respondent size of at least 2 agencies and is only applied to the NSR strata. Table 7 shows the sample allocation assuming 3,000 agencies respond with the response rates from Table 2.

Table 7: Estimated final response size by stratum based on allocation method, 2014 CSLLEA

Agency Type	Agency Sizea	Percent Agencies	Percent FTE	Percent Square Root FTE
Local Police	100+	643	643	643
	50-99	114	450	326
	25-49	221	439	321
	10-24	443	389	303
	5-9	397	152	190
	2-4	317	51	110
	1	140	8	43
Sheriff’s Office	100+	341	341	341
	50-99	46	187	210
	25-49	79	155	191
	10-24	118	107	159
	5-9	66	26	78
	2-4	27	5	34
	1	3	2	6
State Police	All	45	45	45

^a Number of full-time equivalent officers.

To compare the allocations, the relative standard error (RSE) is computed for each estimate, where the RSE is the ratio of a measure and its standard error (Table 8). RSE is a standardized measure of precision regardless of estimate value. While none of the allocations is best for all measures, the allocation proportional to the number of agencies is chosen, as it is the best when there are the largest differences – namely for percent female officers and body worn camera usage.

Table 8: RSE of estimates for each sample allocation, 2016 LEMAS

Allocation	Budget	BWC Usage	Community Policing	Number FTS Officers	Number FTNS Officers	Female Officers
Agencies	2.0%	2.9%	1.3%	1.5%	2.2%	6.5%
FTE	1.9%	5.6%	2.8%	1.5%	2.1%	11.2%
Square Root FTE	1.9%	3.9%	1.8%	1.5%	2.1%	8.1%

The sample size allocation obtained through the proportional number of agencies is presented in Table 9. Based on the response rate assumptions, the design calls for a sample size of 3,499 with 3,000 complete questionnaires expected.

Table 9: Sample size allocation based on the proportion to number of agencies by stratum, 2016 LEMAS

Agency Type	Agency Sizea	Sample Size	Expected Respondents
Local Police	100+	677	643
	50-99	134	114
	25-49	260	221
	10-24	521	443
	5-9	467	397
	2-4	373	317
	1	165	140
Sheriff’s Office	100+	401	341
	50-99	61	46
	25-49	105	79
	10-24	157	118
	5-9	88	66
	2-4	36	27
	1	4	3
State Police	All	50	45
Total		3,499	3,000

^a Number of full-time equivalent officers.

Sampling Error

Although the allocation of sample size will be made with consideration for the overall national estimates, it is not the only domain of interest. Other domains of interest include:

• Local police departments – all sizes

• Sheriff’s offices – all sizes

• State police departments – all sizes

• Local police departments – non-self-representing

• Sheriff’s offices – non-self-representing

The estimates RSEs for each domain are presented in Table 10. The domain with the highest RSEs is the state agencies domain. Since all agencies in this group are sampled with certainty, no allocation could improve the RSEs for this domain. Because the prevalence of female officers is lower than the other proportions of interest, the associated RSE is higher than those for the other selected statistics.

Table 10: RSEs, by selected statistic and domain, 2016 LEMAS

Domain	Budget	BWC Usage	Community Policing	Number of FTS Officers	Number FTNS Staff	Female Officers
National	1.95%	2.89%	1.35%	1.53%	2.24%	6.51%
Local Police	2.01%	3.20%	1.41%	1.98%	2.96%	7.69%
Sheriff’s Office	4.35%	6.74%	4.17%	2.67%	3.51%	12.25%
State Police	6.38%	8.78%	4.76%	5.32%	6.51%	18.15%
Non-Self-Representing	1.85%	3.03%	1.47%	0.75%	3.33%	7.45%
Self-Representing	2.63%	1.77%	0.48%	2.33%	2.89%	2.27%
Local Police – Non-Self-Representing	1.66%	3.32%	1.51%	0.83%	2.57%	8.59%
Sheriff’s Office – Non-Self-Representing	4.92%	7.46%	5.11%	1.65%	5.48%	15.01%

Final Sampling Design

The final design for the 2016 LEMAS will mirror the methods used in the previous LEMAS administrations: (1) strata will be based on agency type and number of full-time equivalent sworn officer as shown in Table 2 and (2) sample size allocation will be based on the proportional number of agencies within in each strata. The LEAR will serve as the sampling frame with and estimated sample size of 3,499 agencies for the 2016 LEMAS. Sample allocation by strata for the 2016 LEMAS are shown in Table 9.

2. Procedures for Collecting Information

Data Collection Procedures. The 2016 LEMAS will involve a series of mailings and non-response follow-up activities. Data collection will begin with a survey invitation letter (mailed via USPS) and an email to the point of contact (POC) for each LEA to inform him or her about the survey. This letter will be signed by the Director of BJS and explain the purpose and significance of the survey. It will include the survey web address and agency-specific log-in credentials (Attachment 7). The survey invitation letter also will provide a toll-free telephone number and project-specific e-mail address for the survey Help Desk should the POC have any questions. Instructions for changing the POC via the survey website, fax, or telephone will be included in the event the LEA needs to change the POC to a more appropriate person. Included with the survey invitation letter will be an informational flyer (Attachment 8). The flyer will explain the new design of the LEMAS—the supplement and core approach—and the importance of agency participation in each survey. The flyer will also explain to SR agencies and any overlapping NSR agencies why they are receiving the 2016 LEMAS core shortly after they have received the 2016 Body-Worn Camera Supplement. Accompanying this lead letter will be BJS Confidentiality Assurances (Attachment 9), a letter of support from the Police Executive Research Forum (PERF) (Attachment 10), a POC Update Form (Attachment 11) that the recipient can use to fax contact information for a newly designated POC, and a paper copy of the 2016 LEMAS instrument to inform respondents about the types of information they will need to collect prior to completing the survey (Attachment 1). Approximately 1 week after sending the survey invitation letter, RTI will send an e-mail message that is identical to the survey invitation letter to those recipients for whom an email address is available to confirm receipt of the study materials. Within three weeks of receiving a survey, the respondent will receive a thank you email or letter depending on the completion mode (Attachment 12). The thank you will formally acknowledge receipt of the survey and state that the agency may be contacted for clarification once their survey responses are processed.

Three weeks after sending the survey invitation letters, a reminder email will be sent to all POCs, including those who are newly identified (Attachment 13). These emails, signed by the BJS Project Manager, will express the importance of the LEMAS to the LEA community and encourage response via the online survey (or paper copy, if preferred). The reminder email will also include a copy of the paper questionnaire, so that respondents can gather the appropriate resources before beginning the survey. Three weeks after the reminder emails are sent, RTI will mail a second reminder letter to POCs (Attachment 14) that will include the website information and instructions on how to download another paper copy of the questionnaire if needed. Three weeks after sending the second reminder, we will send a third reminder via email (Attachment 15) and begin telephone follow-up with all non-responding LEAs. The procedures for this activity and the final two nonresponse follow-up contacts are discussed in Section 3.

Upon receipt of a completed survey (web or paper copy), data will be reviewed and edited, and if needed, the respondent will be contacted to clarify answers or provide missing information. RTI will enter data from all paper copy questionnaires into the web survey application. This will ensure that the same data quality review procedures are applied to all survey data, regardless of response mode. Prior to contacting the respondent, RTI staff will aim to address data inconsistencies via BJS-approved editing specifications. RTI also will ensure that responses fall within the proper coding schemes specified by BJS. The following is a summary of the data quality assurance steps that RTI will observe during the data collection and processing period:

Data Editing. RTI will attempt to reconcile missing or erroneous data through automated and manual edits of each questionnaire. In collaboration with BJS, RTI will develop a list of edits that can be completed by referring to other data provided by the respondent on the survey instrument. Through this process, RTI can quickly identify which paper copy cases require follow-up and indicate the items that need clarification or retrieval from the respondent.

Data Retrieval. When it is determined that additional data retrieval is needed, an Agency Liaison (AL) will contact the respondent for clarification. Throughout the data retrieval process, RTI will document the questions needing retrieval (e.g. missing or inconsistent data elements), request clarification on the provided information, obtain values for missing data elements, and examine any other issues related to the respondent’s submission.

Data Quality Review. To confirm that editing rules are being followed, RTI will review frequencies for the entered data after the first 10 percent of cases are received. Any issues will be investigated and resolved. Throughout the remainder of the data collection period, RTI staff will conduct regular data frequency reviews to evaluate the quality and completeness of data captured in both the web and paper copy modes.

3. Methods to Maximize Response Rates

The previous waves of the LEMAS survey have achieved high rates of survey response, typically meeting or exceeding 90 percent. BJS and RTI will undertake various procedures to ensure that response rates for the LEMAS are as high as possible.

BJS will use a web-based instrument supported by several online help functions to maximize response rates. For convenience respondents will receive the survey link in an email invitation and a mailed paper copy invitation. A Help Desk will be available to provide both substantive and technical assistance. In addition, the web survey interface is user-friendly, which encourages response and ensures more accurate responses. Because online submission is such an important response method, close attention will be paid to the formatting of the web survey instrument. The online application will be flexible so it can adapt to meet the needs of multiple device types (e.g., desktop computer and tablet), browser types (e.g., Internet Explorer and Google Chrome), and screen sizes. Other features in the instrument will include the following:

• Respondents’ answers will be saved automatically, and they will have the option to leave the survey partway through and return later to finish.

• The online instrument will be programmed with data consistency checks and automatic prompts to ensure inter-item consistency and reduce the likelihood of “don’t know” and out-of-range responses, thereby reducing the need for follow-up with the respondent after survey submission.

• Respondents will encounter a main menu when they enter the web survey that allows them to complete the survey module-by-module. This will be useful so that respondents can delegate specific modules to other individuals within their organization.

• Upon submission, respondents will receive a message that confirms receipt of their survey.

• LEAs may also download and print a paper version of the survey from the website.

In order to obtain higher response rates and to ensure unbiased estimates, multi-stage survey administration and follow-up procedures have been incorporated into BJS’s response plans. Ensuring adequate response (not just department response rates, but also item responses) begins with introducing agencies to the survey. This will be accomplished initially through the initial invitation letter and accompanying documents (Attachments 1, 7-11). Resources available to help the respondent complete the survey (e.g. telephone- or email-based Help Desk support) will be described in detail. We will provide LEAs with online and fax methods to identify respondents and change the POC assignment if needed. POCs will also be able to delegate specific modules of the survey to others within their organization. The survey instrument will capture the name of the individual who completed a module so that it can be used for potential quality control follow-up later. We will also provide a paper copy of the 2016 LEMAS instrument to inform respondents about the types of information they will need to collect prior to completing the survey

Approximately 1 week after sending the survey invitation letter, RTI will send an e-mail message that is identical to the survey invitation letter to those recipients for whom an email address is available to confirm receipt of the study materials. Within three weeks of receiving a survey, the respondent will receive a thank you email or letter depending on the completion mode (Attachment 12). The thank you will formally acknowledge receipt of the survey and state that the agency may be contacted for clarification once their survey responses are processed.

The data collection schedule is designed to include several follow-up communications to allow the LEA to complete the survey at a time most convenient for them. Three weeks after the initial invitation, we will send an e-mail to all LEAs. This e-mail communication will serve as a reminder for those agencies who have not yet submitted their information (Attachment 13). Following this reminder message, the second nonresponse message will be sent via USPS to any to-date nonrespondents once again asking them to complete the web survey (Attachment 14). If no survey response is received after the second nonresponse message, a third nonresponse message will be sent via email to to-date nonrespondents. Like previous communications, it will provide information on how to complete the web survey, including the URL and the LEA’s unique survey access code (Attachment 15). The nonresponse email will be distributed coinciding with the launch of telephone calling.

In addition to these written communications, telephone calls will begin with the to-date nonrespondents (see Attachment 16 for sample scripts). In preparation for this outreach, Agency Liaisons (ALs) will be trained on the study protocol and procedures for contacting nonresponding agencies. Most notably, ALs will receive training on how to ask agencies to complete the web survey, administering the web survey by telephone, arranging for LEAs to receive paper copy questionnaires, and tracking cases (including contact attempts). After nonresponse telephone calls, ALs will make targeted attempts with nonresponding agencies to capture critical items.

The largest 7 percent of state and local agencies comprise almost two-thirds of the sworn personnel, nationally. It is critical that we obtain responses from these agencies. Recognizing the benefits of their existing rapport with the LEAs, PERF will take the lead for follow-up with these state and local agencies.

Finally, two more nonresponse messages will be sent to any to-date nonrespondents. First, 2-3 weeks after telephone follow-up begins, we will send a reminder email to LEAs (Attachment 17). Second, 4 weeks before data collection ends, a final letter will be sent to the POC which will include an end-of-study message. This message will go to any to-date nonrespondents to announce the forthcoming closure of the study and make a final appeal to participate. We will also include a paper survey and a business reply envelope to facilitate completion via mail in this communication (Attachment 18).

Nonresponse Adjustments

Unit nonresponse. With any survey, it is typically the case that some of the selected subjects will not respond to the survey request (i.e., unit nonresponse) and some will not respond to particular questions (i.e., item nonresponse), despite best efforts made to collect all the data. Weighting will be used to adjust for unit nonresponse in the 2016 LEMAS, using the LEAR. To determine which factors to use in the facility nonresponse weight adjustments, a procedure available in RTI’s SUDAAN software based on the Generalized Exponential Model (GEM) will be used to model the response propensity based on information from the sampling frame (e.g., agency characteristics such as geography, operating budget, whether officers arrest people, etc.) within sampling strata (Folsom, 2000). Ideally, only variables highly correlated with the outcomes of interest will be included in the model in order to reduce the potential for bias. As described above, given the expected differential response rates by agency type and size, the weighting adjustment procedures will attempt to minimize the bias in the estimates within these domains.

Nonresponse bias analysis. As previously stated and based on the 2013 LEMAS responding, an overall response rate of approximately 86 percent is expected (Table 2). In order to ensure those agencies that do not participate in the study are not fundamentally different than those that do, a nonresponse bias analysis will be conducted if the agency-level response rate obtained in the 2016 LEMAS falls below 80 percent. The following administrative data on agency characteristics will be used in the nonresponse bias analysis:

• Agency type,

• Agency size, and

• Census region or division.

For each agency characteristic, BJS will compare the distribution of the respondents to the nonrespondents. A Cohn’s Effect Size statistic will be calculated for each characteristic. If any characteristic has an effect size that falls into the “medium” or “high” category, as defined by Cohn, then there is a potential for bias in the estimates. Each estimate will be included in a nonresponse model to adjust weights to minimize the potential for bias in the estimates.

In addition to estimating effect sizes, an examination of early and late responders will be conducted. If late responders (i.e., those that take more contact attempts before responding) are significantly different on the key outcomes of interest, that is an indication of potential bias. This comparison will be made within each strata to determine if the potential for bias varies by strata.

4. Final Testing of Procedures

The 2016 LEMAS survey instrument is built upon previous waves of the LEMAS survey to ensure data comparability. As a result, improvements to the questionnaire were done with caution. BJS shared a copy of the draft 2016 LEMAS survey instrument with research scholars with a known interest in law enforcement issues and law enforcement professionals. The expert reviewers were given an electronic draft of the survey instrument and asked to comment on question wording, response categories, as well as overall structure and layout. Responses were primarily received as written annotations within the document. Further information on the results of the expert panel review are provided in Part A. Only five new items were added to the 51-item instrument. For three other items, additional response categories were added. These adjustments were added to address recommendations from the President’s Task Force on 21^st Century Policing (see Attachment 3). We received expert feedback on these new items and where possible, question wording was consistent with other previously tested items. Additionally, we will conduct post-processing assessment of these new items.

5. Contacts for Statistical Aspects and Data Collection

1. BJS contacts include:

Shelley Hyland, PhD

202-305-5552

Shelley.Hyland@usdoj.gov

2. Persons consulted on statistical methodology:

Stephanie Zimmer, PhD

RTI International

3. Persons consulted on data collection and analysis:

Tim Smith

RTI International

Travis Taniguchi, PhD

RTI International

References

Barrick, K.,  Hickman, M. J., & Strom, K. J. (2014). Representative policing and violence towards the police. Policing.

Bies, K. J.,  Deporto, I. M.,  Long, D. G.,  McKoy, M. S.,  Mukamal, D. A. &,  Sklansky, D. A. (2015). Stuck in the '70s: The demographics of California prosecutors. Stanford, CA: Stanford Law School, Stanford Criminal Justice Center

Bromley, M. & Reaves, B. (1998). Comparing campus and city police operational practices. Journal of Security Administration, 21(2), 41-54.

Cave, B., Telep, C. W., & Grieco, J. (2015). Rigorous evaluation research among US police departments: Special cases or a representative sample? Police Practice and Research, 16(3), 254-15. doi:10.1080/15614263.2014.888348

Chalfin, A., & McCrary, J. (2013). The effect of police on crime: New evidence from U.S. cities, 1960-2010. NBER working paper series. 18815, Cambridge, MA: National Bureau of Economic Research.

DeCarlo, J., & Jenkins, M. J. (2015). Labor unions, management innovation and organizational change in police departments. Springer Briefs in Criminology. Switzerland: Springer International Publishing doi:10.1007/978-3-319-21584-6

Farrell, A. (2014). Environmental and institutional influences on police agency responses to human trafficking. Police Quarterly. 17(1), 3-29.

Folsom, R.E., & Singh, A.C. (2000). The generalized model for sampling weight calibration for extreme values, nonresponse, and poststratification. In Proceedings of the American Statistical Association’s Survey Research Methods Section, 598-603.

Groves, R. & Cork, D. (2009). Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics. Washington, D.C. National Academies Press.

Gustafson, J. (2013). Diversity in municipal police agencies: A national examination of minority hiring and promotion. Policing. 36(4), 719-736.

Hickman, M. & Piquero, A. (2009). Organizational, administrative, and environmental correlates of complaints about police use of force: Does minority representation matter? Crime and Delinquency, 15(1), 3-27.

Hur, Y. (2013). Racial diversity, is it a blessing to an organization? Examining its organizational consequences in municipal police departments. International review of administrative sciences. 79(1), 149-164.

Langton, L. (2010). Women in Law Enforcement, 1987 – 2008. Washington, D.C.: Bureau of Justice Statistics.

Matusiak, M. C., Campbell, B. A., & King, W. R. (2014). The legacy of LEMAS: Effects on police scholarship of a federally administered, multi-wave establishment survey. Policing. 37(3), 630-648.

Melekian, B. K. (2012). Policing in the new economy: A new report on the emerging trends from the Office of Community Oriented Policing Services. Police Chief, 79, 16–19.

Perez, N. M., & Bromley, M. (2015). Comparing campus and city police human resource and select community outreach policies and practices: An update. Policing: An International Journal of Police Strategies & Management, 38(4), 664-674.

President’s Task Force on 21st Century Policing. (2015). Final Report of the President’s Task Force on 21st Century Policing. Washington, DC: Office of Community Oriented Policing Services.

Randol, Blake M. (2013). An exploratory analysis of terrorism prevention and response preparedness efforts in municipal police departments in the United States: Which agencies participate in terrorism prevention and why? Police Journal. 86(2), 158-181.

Reaves, B. (2011). Census of State and Local Law Enforcement Agencies, 2008. Washington, D.C., Bureau of Justice Statistics.

Reaves, B. (2015). Local police departments, 2013: Equipment and technology. Washington, D.C.: Bureau of Justice Statistics.

Roberts, A. & Roberts, Jr., J. M. (2015). Crime and temporal variation in police investigative workload: Evidence from National Incident-Based Reporting System (NIBRS) data. Journal of Quantitative Criminology, 1-24. doi:10.1007/s10940-015-9270-9

Schuck, A. M. (2014). Female representation in law enforcement: The influence of screening, unions, incentives, community policing, CALEA, and size. Police Quarterly. 17(1), 54-78.

Schuck, A. M., & Rabe-Hemp, C. (2014). Citizen complaints and gender diversity in police organisations. Policing and Society.

Sharp, E. B. (2014). Minority representation and order maintenance policing: Toward a contingent view. Social Science Quarterly. 95 (4), 1155-1171.

Shjarback, J. A. (2015). Emerging early intervention systems: An agency-specific pre-post comparison of formal citizen complaints of use of force. Policing. doi:10.1093/police/pav006

Shjarback, J. A., & White, M. D. (2015). Departmental professionalism and its impact on indicators of violence in police-citizen encounters. Police Quarterly. doi:10.1177/1098611115604449

Smith, B. W.,  Wareham, J., & Lambert, E. G. (2014). Community and organizational influences on voluntary turnover in law enforcement. Journal of Criminal Justice. 37(3), 377-398.

Walfield, S. M. (2015). When a cleared rape is not cleared: A multilevel study of arrest and exceptional clearance. Journal of Interpersonal Violence. doi:10.1177/0886260515569062

Wareham, J.,  Smith, B.W.,  & Lambert, E. G. (2015). Rates and patterns of law enforcement turnover: A research note. Criminal Justice Policy Review. 26(4), 345-370. doi:10.1177/0887403413514439

Willits, D. W. (2014). The organisational structure of police departments and assaults on police officers. International Journal of Police Science and Management. 16(2), 140-154.

1 Two part-time officers are equivalent to one full-time officer.

15