Updated Stats Methods for Level 1 Evaluation_052610

COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

1. Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection.

• Estimation procedure.

• Degree of accuracy needed for the purpose described in the justification.

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The FBI National Academy offers a 10-week training program on a quarterly basis. Approximately 1,020 law enforcement professionals attend the training program each year. These law professionals represent state and local police and sheriff’s departments, military police organizations and federal law enforcement agencies from the United States and over 150 foreign nations. The FBI National Academy invites all students to complete two questionnaires; the Student Course Questionnaire and the General Remarks Questionnaire. Two week before the end of the training program an invitation is sent to all students attending the program via an e-mail message which includes the link directing all students to the questionnaires. Each questionnaire includes a description of its aim and the purposes to why the data is being collected. The questionnaires will also explain to respondents that their participation is completely anonymous.

After completing the questionnaires, the respondents will be instructed to select “submit” in order to record their responses. Because all FBI National Academy students will be offered the opportunity to participate in the questionnaires, the questionnaires do not involve any sampling, methods for stratification, imputations or similar statistical methods.

2. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

To ensure a high response rate, several proactive measures have been put into place. For numerical data collection an electronic application, Key Survey, will be used to calculate simple totals, percentages, and cumulative percentages. Key Survey will also capture any student comments entered in response to open-ended questions. Next the questionnaires will focus strictly on those issues of great interest in order to keep it as short as possible while still accomplishing the goals of the questionnaires. In addition, by offering the questionnaires online, we reduce the level of effort needed to complete it, as well as the steps involved in creating the database for analysis. Response rates will also be maximized by e-mail reminders which are sent out automatically through Key Survey; five days after the questionnaires have been sent. Students are also reminded about the questionnaires throughout the two–week evaluation period during their group meetings while at the FBI National Academy. This face-to-face reminder is being done by FBI National Academy personnel. Finally, by including all students we feel confident that the resulting data will reflect the experience of a wide range of respondents, including the full variety of law enforcement agencies and personnel levels. It is with the understanding of the students that by completing these evaluations they will help the FBI determine the strengths and weaknesses of the existing program.

Each student attending the FBI National Academy takes approximately six to seven courses during their ten-week training session. The students are asked to complete an evaluation for each of their courses and complete an evaluation of their overall experience while at the FBI National Academy. The students are asked to begin completing their evaluations two weeks before the training program ends to allow sufficient time. This approach has always been handled this way and has not changed to survey the students after they have left.

We email the surveys vs. passing the surveys out to the students at the end of class to save time and money.  In the past after handing out the surveys, the Evaluation Program would receive 66 envelopes with approximately 40 surveys in each envelope. Then we would compile the ratings and transcribe the student’s comments. This would take several weeks to complete.  Using the Key Survey Application allows us to collect, analyze, and compiled the data much quicker.  The application compiles the results and transcribes student’s comments all in one per survey. Less paper is being use and reports of results for all surveys are complete within one week. Students are always eager to provide their responses because they feel it’s an honor to a part of this program. The response rate has not lower since using the Key Survey application.

The students are informed on the very first day of training during orientation that they are going to receive evaluations on each of their courses. The students are told the evaluations will be sent around the 8^th week of training (2 wks before graduation). Class Counselors remind the students about the evaluations on the 7^th week of training and inform them the evaluations are going to be sent the following next week.

During orientation, personnel from FBI National Academy ask the students to provide either their work or personal email addresses. Students attending FBI National Academy are law enforcement professionals from federal, state and local, military and foreign agencies. Inasmuch as emails from their employment have high security settings some students create new email accounts while they are attending the FBI National Academy. These new email accounts have lower security settings. Students are asked to provide the email addresses that will enable them to receive the surveys without experiencing any technical difficulties.

Once the surveys are sent and the two-week evaluation period begins students are reminded seven days after. Rarely is there a needed for a second reminder but one is available if necessary.

Based on the last six FBI National Academy sessions the overall response rate has ranged from 72% - 86%, therefore the FBI believe the overall response rate for these surveys will be approximately 79%.

3. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The questionnaire forms included as part of this package were developed following extensive interviews with FBI National Academy personnel and instructors. After determining the desired content, items were developed by experienced questionnaire experts. A thorough review of the content centered on several issues, including:

• Clarity of instructions

• Ease with which information requested can be provided

• Length of the instrument, and

• Quality of the resulting data in terms of how it can be used to answer the questions of interest.

When the draft instrument was signed off by FBI National Academy personnel, programming was undertaken. Following this step, internal tests were run to ensure functionality. After a thorough check, a sample of nine students at the FBI National Academy was asked to complete the online survey. Each respondent was interviewed afterwards to determine whether they had any problems completing the questionnaire and to get their input on changes that might clarify its content. Only minor wording changes were suggested and implemented.

4. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The following individual was consulted regarding the statistical aspects of the project:

Candace Matthews

Evaluation Program Manager

Curriculum Development and Evaluation Unit

FBI Academy

Quantico, Virginia 22135

The person who will be involved in collecting and analyzing the questionnaire data:

Laleatha B. Goode

Management Program Analyst

Curriculum Development and Evaluation Unit

FBI Academy

Quantico, Virginia 22135