Pilot Protocol

Attachment 4

Pilot Protocol, Schedule, and Documents

The purpose of the pilot-test of the survey was to ensure that all components of the survey function as intended before implementation as a full-scale survey. The process supported measurement error control by conducting a pretest of the survey components among a subset of respondents from the identified respondent pool.

The pilot-test protocol was implemented in consultation with the Substance Abuse and Mental Health Services Administration government project officers who work closely with the grantees. The BHITs project team the grantee government project officers identified potential participants from the universe of respondents. From this group, 9 (nine) individuals were recruited for participation in the pilot. Conditions for this participation included the role of the participant in the grant-funded program – either as the Project Director or as the Program Manager; and the individual’s willingness and ability to set aside one hour for testing the survey instrument in realistic conditions, then participating in a teleconference call to debrief the BHITs project team members.

Once the pool of potential participants was identified by the government project officers, BHITs project team members contacted each individual directly by email. The message asked the individual to confirm their interest in participating and to include a date and a one-hour time slot of their choosing for participation. Respondents who declined received a message thanking them for considering the request. Respondents who accepted received a message thanking them for their willingness to assist; a brief summary of the pilot project that included general information about the survey itself; the plan for data collection; and how the pilot survey data would be collected and used. It also provided information on next steps.

The survey pilot participants who responded with a time and date were asked to expect a message and a calendar invitation with three attachments. The attachments included:

1. “The Pilot Survey - What to Expect”.

2. “Day of the Survey Instructions”

3. “What Do You Think?”

The participant was asked to read the “Pilot Survey - What to Expect” when they received the calendar invitation. This provided high-level information on the participant’s role and the associated expectations, in a single page. The guidance asked the participant to print the “Day of the Survey Instructions” and the “”What Do You Think?” documents before the start of the survey. It also reminded participants that the hour allotted included time for the post-survey interview, and it provided the name and contact information for the interview. It explained that the participant would be expected to call immediately after completing the survey. It also assured the participant that the interviewer would also be on “standby” while the participant was taking the survey, to respond to any issues that might arise, especially technical difficulties. The document included information about the survey itself: the reasons for the survey; the plan for data collection; and how the information from the survey would be used.

The “Day of the Survey Instructions” was also a one page document, but more brief and printed in large text. It included: a reminder to use the “What Do You Think?” outline or some other means to take notes; the link to the survey; the name and number of the person to call for the post-survey interview; and an indication that this was also the name and number to use to report technical difficulties encountered during the survey.

“What Do You Think” provided a loosely organized paper-based environment for the participants to note their observations, including their start time for each section. It was provided for the participants’ convenience, as a reference tool for use in the post-survey interview. It was separate from the online instrument.

Copies of these documents are included in this appendix.

Immediately after the survey, the BHITs project team conducted a fifteen-minute post-survey interview, via teleconference. The purpose of the interview was to review the participant’s survey experience. The BHITs team member conducting the interview engaged the participant in a discussion of each section by asking open-ended questions and encouraging the participant to speak frankly about any issues or concerns.

The survey pilot and the post survey interview provided the BHITs project team with the qualitative data necessary to assess and evaluate the participants’ individual experiences. These data were collected, interpreted and summarized, then communicated to the Task Leader and the federal agency’s Contract Officer Representative as a summary report with recommendations. The summary reported participant observations on:

1. Questionnaire design, including the implementation of the “skip logic” pattern.

2. Variations in patterns to understanding and response among the different types of grant-funded programs, identifying implications for deployment of the instrument over the web. The survey assumed at least some level of knowledge about components of health information technology, and the pilot will helped to confirm this assumption.

3. Item feasibility, including the degree of complexity of the question as it related to the individual’s ability to easily interpret the meaning and then select an answer they feel accurately reflects their experience.

4. Potential obstacles to collecting the data in a web-based format, including questions that did not lend themselves to an electronic environment, and the individual’s ability to navigate the instrument without confusion.

5. Identification of survey editing recommendations.

6. Confirmation of the estimate of the timeframe for completing the survey, including the shortest amount of time estimated, and the longest amount of time estimated.

Responses to the summary recommendations from the Contract Officer Representative and the Task Leader were fully incorporated into the final survey instrument, prior to its deployment among the pool of respondents.

% complete	2 COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHOD	START	END
	2. 4 Tests and procedures	1/15/2015	1/31/2015
	2.4.1 Pilot test approved draft
100	2.4.1.1 Develop and submit Pilot Protocol	1-Jan	15-Jan
100	2.4.1.2 Prepare and submit timeframe	16-Jan	20-Jan
100	2.4.1.3 Deploy draft survey on web	16-Jan	20-Jan
100	2.4.1.4 In consultation with GPOs, identify pool of pilot survey respondents	16-Jan	20-Jan
0%	2.4.1.5 Contact participants individually by phone to confirm willingness to participate	20-Jan	22-Jan
25%	2.4.1.6 Develop and submit draft logistics letter with range of select times/dates for responses	20-Jan	21-Jan
0%	2.4.1.7 Deploy approved invitation letter with times and dates for selection	22-Jan	22-Jan
0%	2.4.1.8 Collect responses and schedule participation	22-Jan	23-Jan
0%	2.4.1.9 Confirm individual appointments, communicate survey procedure transmit pilot workbook	26-Jan	27-Jan
0%	2.4.1.10 Conduct survey pilots with interviews	27-Jan	30-Jan
0%	2.4.1.11 Prepare report with recommendations	27-Jan	30-Jan
0%	2.4.1.12 Obtain FEi and SAMHSA feedback	Jan-30	31-Jan
0%	2.4.1.14 Incorporate changes and deploy on Web	Jan-31	31-Jan
0%	2.4.1.14 Final printed versions incorporated into OMB package	Jan-31	30-Jan

The Pilot Survey - What to Expect

First, thank you again for agreeing to participate in the pilot of this survey. The survey will collect standardized data on the state of adoption of health information technology by behavioral health providers within selected Substance Abuse and Mental Health Agency grant. The data gathered will help to inform the agency’s strategies and approaches to foster the effective use of information technology by the behavioral healthcare community.

The purpose of the pilot-test of the survey is to ensure that all components of the survey function as intended before it is implemented as a full-scale survey. The process supports measurement error control by conducting a pretest of the survey components among a subset of respondents from the identified respondent pool.

You will be interviewed about the survey immediately after you have completed it. We will ask you to initiate contact with the individual identified in the “Day of the Survey Instructions,” who is also on standby during the survey to assist with any technical difficulties that may arise.

To get the most out of this interview, we have attached a document called “What Do You Think?” It lists the survey questions with space for your comments and observations. We recommend printing this document and the attached “Day of the Survey Instructions” in advance so that you are ready to take notes and then call the person listed in the “Day of the Survey Instructions” for the post-survey interview. We will be waiting for your call!

The survey itself is in six sections, but it uses “Skip Logic.” Your answers to questions at the beginning of each section may bring you through all or part of the questions in that section, or you may even skip the section entirely. If you are not sure what section you are in, just check the top of the survey screen. It will always indicate the survey section.

We are very grateful to you for taking the time to assist us with this project, and we are also looking forward to getting your feedback!

Day-of-the-Survey Instructions

• Print the “What Do You Think?” doc (attached to the calendar invitation) for taking notes, or else use a method that works best for you.

• Call the number below when you have completed the survey, for the post-survey interview. This contact is also posted online at the beginning and end of the survey

• The interview will take 15 minutes and we only have one hour. If you are not able to complete the survey within 45 minutes, STOP work and initiate the call.

To begin the survey, click the link below, or else copy and paste into the browser address bar.

https://www.surveymonkey.com/r/ZTGWYD3

When you finish the survey (or after 45 minutes) contact

Colleen O’Donnell

202-684-3747

for a brief post-survey interview. You may also use this contact to for assistance during the survey

What Do You Think?

This is offered as an optional, paper-based tool. There is a space to note the time that you start each section of the survey, and also any observations about questions in that section you want to remember for the post-survey interview. It can be printed out before the survey, for use while you are taking the survey.

If it does not seem helpful, you don’t have to use it - but if you can, please take a moment to note the time that you start each section of the survey.

Section 1: Grantee Program Information and Characteristics “I started this section at around ___:____”
Question #	Comments
Section 2: Information Technology Infrastructure “I started this section at around ___:____”
Question #	Comments

Section 3: Certified Electronic Health Record Information Technology “I started this section at around ___:____”
Question #	Comments

Section 4: Use of Telehealth Technology “I started this section at around ___:____”
Question #	Comments

Section 5: Use of Mobile Technology “I started this section at around ___:____”
Question #	Comments

Section 6: Use of Social Media “I started this section at around ___:____”
Question #	Comments

Pilot Protocol, Schedule, and Documents, Page 11