0704-0610_OSIE_SSB_9.12.2024_v4

NOTE: Complete Part B for Survey ICR Requests

SUPPORTING STATEMENT – PART B

B.  COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

If the collection of information employs statistical methods, the following information should be provided in this Supporting Statement:

1.  Description of the Activity

These information collections continue to support a high-visibility requirement codified in the Department of Defense Instruction (DoDI) 6400.11, Integrated Primary Prevention Policy for Prevention Workforce and Leaders. Section 5.2 directs the Under Secretary of Defense for Personnel and Readiness (USD(P&R)) to conduct On-Site Installation Evaluations (OSIEs) every other year in accordance with the March 30, 2022 Secretary of Defense Memorandum.

As of December 2024, 74 On-Site Installation Evaluations (OSIEs) have been completed around the world on military installations and ships. Biennial information collections will continue to evaluate 15-20 DoD installations or ships each year to determine where the military community is at increased or decreased risk for harmful behaviors based on the methods and metrics refined through the initial two rounds of OSIE visits. Effective prevention decreases risk factors and increases protective factors. On-site evaluations will answer: Are the installation’s prevention capabilities poised to address the risk detected on the command climate assessment? If so, what is going right that could be replicated elsewhere? If not, what concrete actions could be taken to improve the installation’s ability to address risk and enhance promise?

DoD Office of Force Resiliency (OFR) will identify 15 DoD installations to take part in each biennial OSIE. The evolving nature of these analyses is captured in the OSIE reports published on prevention.mil. Continued iterations of these methods will be used for site selectionse methods will be are provided in TAB EXA for the renewal period. Units and installations of risk and promise will be identified based on analysis of risk and protective factors for prohibited behaviors, including the command climate assessments that formed the basis of pilot site identification. Sites with extreme scores for risk or promise will be selected for site visits. Identified sites for the 2025 biennial OSIE are TBD.

At each site, in addition to assessing installation assets (e.g., prevention personnel that provide services to the entire installation), data will be collected from the 1-3 units that drive the installation’s risk and protective factor scores. There will be three data sources: 1) responding to a “request for information”; 2) participating in discussions during site visits; and 3) completing a survey. The discussions and survey constitute the information collections covered in this request.

The request for information asks a series of questions about experiences conducting prevention activities targeting a range of harmful behaviors, including suicide, alcohol misuse, , sexual assault, harassment, child abuse and neglect, retaliation, and domestic abuse. The request for information also asks what kind of prevention activities service members conduct, what their relationships are like with their colleagues at their location who also are involved in prevention, and any relevant documents about the prevention activities conducted. The request for information will be sent before the site visit.

The site visit discussions will involve staff from DoD (with support from a Booz Allen Hamilton contract team) asking questions to follow-up on data elements in the request for information, along with additional questions about prevention at each location. The duration of site visits will vary based on the number of engagements planned for each site and the number of units selected at each installation, but are estimated to last between three and ten days. Depending upon roles of each service member at the different locations, they may be asked to participate in only one day or in multiple days of OSIE discussions.

The survey asks questions about individual levels of knowledge and skills conducting prevention activities. Depending on their role, they may be asked to participate in all three activities, two activities, or just one. Participation will be determined in consultation with contract and OFR staff. The survey will be collected on paper or electronically via QR code at the site visit. Surveys will be completed in on-site meetings and then uploaded via secure platform, Qualtrics, to be sent back to the agency for data entry.

The request for information and site visit discussions will have identifiers, including the name of the respondent, email addresses, and phone numbers. Surveys will not have any individual identifiers. None of the information is sensitive, but mostly describes activities that while not well known, are public (existence of prevention programs at an installation). Some of the information will be about topics that are not publicly known—e.g., the individual prevention competencies of an individual respondent.

The respondents will be a cross section of personnel at the selected installations that fall into two general categories. First, are personnel that are specific to prevention and intervention activities relating to a variety of negative behaviors (e.g., alcohol use, suicide, sexual harassment). These include: Integrated Primary Prevention Workforce (IPPW), Sexual assault response coordinators, victim advocates, Family Advocacy Program staff, MEO/EEO staff, Mental Health Professionals, Enlisted medical personnel (e.g., medics, corpsmen), Inspectors general and misdemeanor and felony-level law enforcement representatives, Chaplains, MWR and community/support services staff, total force fitness practitioners, and physical health professionals. We are collecting data from these individuals because they have first-hand knowledge of how prevention activities are carried out at the installation. In most cases, given the size of the identified sites, samples of participants will represent a census of the personnel at these locations.

The second category are general personnel that will be important to talk to for their perceptions of how prevention is prioritized and experienced at the installation and will include (the target sample is in parentheses): Installation commander/Senior Mission commander and command leadership team (12), E1-E4 Men (12), E1-E4 Women (12), O1-O3 Men (12), O1-O3 Women (12), E5-E6 Men (12), E5-E6 Women (12), O4-O5 Men (12), O4-O5 Women (12), E7-E9 Men (12), E7-E9 Women (12). These individuals will be selected from the specific units that drive command climate risk or promise at the site. We will use personnel data to determine how representative these respondents are of the total base population. Additionally, family members of these Service members will be invited to focus groups to better understand the impact of the mission on the Service member and their support systems

Respondents’ participation in these information collections is voluntary.

2.  Procedures for the Collection of Information

There will be three data sources: 1) responding to a “request for information”; 2) participating in discussions during a site visit; and 3) completing a survey. Data sources 1 and 2 will be similar questions. The request for information will be sent ahead via an emailed Word document to a point of contact (POC) at each installation. The POC will arrange to have the relevant individual(s) answer the questions in the Word document and send back to the government representative leading the OSIE team by email. In our experience, these types of requests for information are often returned incomplete, unclear, or inaccurate. Thus, at the site visits, site visit staff will follow up to confirm the information and fill in any missing data. During the site visit, various groups of individuals specified above will be organized into a series of discussion groups in one hour and fifteen minute slots (see Table 1). Although general guidance will be provided, each installation will determine the best individuals to participate.

Table 1. Hypothetical schedule of data collection at each installation *The site visit schedule will be tailored to reflect the available demographics of the specific units of interest (i.e., a unit may not have 12 E7-E9 females).
	Tuesday		Wednesday		Thursday		Friday
	Team 1	Team 2	Team 1	Team 2	Team 1	Team 2	Team 1	Team 2
0900-1015	Installation commander/ command team in-brief and interviews	Tour of installation housing and work areas	E1-E4 Men (12)1	E1-E4 Women (12)	MWR and community/ support services staff (12)	Chaplains (5-8)	Farewell meeting with installation commander, if s/he would like	Family members/spouses
1030-1145	O1-O3 Men (12)	O1-O3 Women (12)	E5-E6 Men (12)	E5-E6 Women (12)	ASC/FMRC Family Readiness (12)	Sexual assault response coordinators, victim advocates, Family Advocacy Program staff (12)
1200-1300 Lunch on the installation
1300-1415	O4-O5 Men (12)	O4-O5 Women (12)	E7-E9 Men (12)	E7-E9 Women (12)	MEO/EEO staff (5-8)	Mental Health Professionals* (12)
1430-1545	Integrated Primary Prevention Workforce (5-8)	Prevention Collaboration Forum (e.g., CAT/CAB, C2RC) (12)	Installation Commander/ Senior Mission Commanders and command team (12)	Enlisted medical personnel (e.g., medics, corpsmen) (12)	Inspectors general AND both misdemeanor and felony-level law enforcement representatives (12)	Physical health professionals** (12)
1 Parentheses indicate project sample size *Mental health professional group should include at least two of each of the following: (a) psychiatrists, (b) psychologists or social workers, and (c) MFLC counselors (Military and Family Life Counselors) AND at least one staff member from the substance/drug and alcohol abuse program **Physical health professionals should include at least two each of the following: (a) primary care physicians (b) emergency room nurses (c) physician assistants FRG = Family Readiness Group (Navy), SFRG = Soldier and Family Readiness Group (Army), FRP = Family Readiness Program (Marine Corps), Key Spouse program is Air Force; MWR = morale, welfare, recreation programs; MEO = Military Equal Opportunity program for military personnel, EEO = Equal Employment Opportunity program for civilian personnel

The questions for 1 and 2 will be open-ended questions about the prevention activities. The discussions will NOT be recorded, but a notetaker will be part of the site visit team and will take detailed notes. Those notes will be uploaded via DoDSAFE to be uploaded to DoD SharePoint for analysis. Data source 3 will be a survey of individual competencies to carry out prevention activities. The individuals slated for this survey are those personnel that are specific to prevention and leaders who could possibly oversee those individuals. This could be a mix of enlisted and officers depending how prevention tasks are assigned at a site. These individuals will complete the survey at their respective discussion group slot during the site visit. The survey will be offered electronically via QR code or by paper and pencil and collected by site visit staff who will hand carry the completed the paper forms back to the agency for data entry and analysis; electronic forms are completed through a secure platform, Qualtrics, as managed by Miami University

3.  Maximization of Response Rates, Non-response, and Reliability

High response rates are expected given this data collection has been ordered by the Secretary of Defense. The request for information (RFI) was the most challenging to have completed during the pilot OSIEs. In order to ease the administrative burden and increase response rate and time, the RFI has been streamlined significantly. Regardless of RFI success, we specifically are employing the site visit mechanism to ensure that all questions will have complete responses. Regarding the survey, we will administer it at the end of each discussion group time slot and immediately collect paper surveys back; surveys completed via QR code will be encouraged to be completed before exiting the session. Thus, we expect the response rates to be around 85% based on previous site visits.

We will use multiple procedures to ensure high reliability. Each site visit will comprise five or six government individuals (team lead and policy office representatives to lead the discussions), and three contract staff to support notetaking. There will be four to eight individuals on each OSIE with a total project team projected to comprise seven to twelve total personnel. All staff (DoD and contractors) will be trained to use the site visit protocol using hypothetical data. For data analysis teams, training will continue until at least an interrater reliability score of .8 is achieved across all raters. After each visit, the raters from each site visit team will independently make their ratings. Interrater reliability will be calculated using these separate scores. Then, the raters will discuss their ratings and come to a concensus when they disagree. It is this consensus score that will be used in analysis.

The data collection is not intended to be generalizable to a larger universe. That is because the data collection is intended to be an installation-level evaluation of that installation’s activities. The data will be used to help each installation improve as well as provide an overview of progress to DoD leadership.

4.  Tests of Procedures

Prior to site visits for the pilot OSIEs, RAND convened an expert panel of five subject matter experts who reviewed and rated measures for validity and importance. Also, RAND administered the discussion questions to other DoD personnel who were not involved with the project otherwise as a further check on wording and feasibility (the data was not retained). In addition, RAND asked each respondent at the end of each group discussion slot about their perceptions of the process (e.g., what worked well; what should be improved). Feedback from group discussion during pilot OSIEs were used to refine the data collection for future use.

5.  Statistical Consultation and Information Analysis

a. Provide names and telephone number of individual(s) consulted on statistical aspects of the design.

Paul D. Flaspohler, Ph.D., Miami University, 90 N. Patterson Ave #212, Oxford, OH 45056, 513-529-2469,

b. Provide name and organization of person(s) who will actually collect and analyze the collected information.

Team Members	Organization/Role
Dr. Andra Tharp	DoD/Team Lead
Mr. Travis Bartholomew	DoD/Team Lead
Dr. Andrew Moon	DoD/Team Lead
Dr. Elizabeth Gray	DoD/Team Lead
Mr. Cody Matthews	DoD/Team Lead
Dr. Rachel Clare	DoD/Lead Analyst
Dr. Paul Flaspohler	Miami University of Ohio/Lead Analyst

Contract support for biennial OSIEs will be performed by Booz Allen Hamilton. Names of DoD key personnel who support the OSIEs are listed below.

2