The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Behavioral Health Statistics and Quality (CBHSQ), is requesting approval for conducting cognitive testing on the use of electronic health records (EHRs) by substance abuse and mental health treatment facilities in the United States. The final goal of this cognitive testing is to incorporate questions on electronic health records to SAMHSA’s National Survey of Substance Abuse Treatment Services (N-SSATS) and the National Mental Health Services Survey (N-MHSS).
In the last decade, state-of-the art techniques have been increasingly instituted by Federal agencies, and are now routinely used to improve the quality and timeliness of survey data and analyses, while simultaneously reducing respondents’ cognitive workload and burden. As a federal statistical unit, CBHSQ anticipates the benefit of increased response rates through improved survey design; a goal tied directly to improving response rates and reducing non-response bias.
Cognitive interviews are considered a qualitative method that refers to a set of tools employed to study and identify errors that are introduced during the survey process. This method is generally conducted one-on-one with respondents. Cognitive interviews are generally used to clarify the question-response process. Interviews may be conducted with respondents providing concurrent verbal protocols as they think aloud while answering survey questions or with retrospective protocols with information provided afterwards via response-elicitation techniques. These include follow-up probing, memory cue tasks, paraphrasing, confidence rating, response latency measurements, free and dimensional sort classification tasks, and vignette classifications. The objective of all of these techniques is to aid in the development of surveys that work with respondents’ thought processes, thus reducing response error and burden. These techniques have also proven useful for studying and revising pre-existing questionnaires.
Currently, there is a lack of national level data that exists on behavioral health care providers’ progress toward interoperability. The National Council for Behavioral Health in 2011/2012 conducted a survey to determine health information technology (IT) readiness. This data focused only on the membership of the National Council for Behavioral Health and does not provide national baseline data on the four domains of interoperability that are outlined in the Interoperability Roadmap (finding, sending, receiving and integrating data into EHRs) for behavioral health care providers. Currently, these providers are not eligible to participate in interoperability driving efforts such as the Medicare Access and CHIP Reauthorization Act of 2015 (MACRA) initiative. However, some behavioral health providers may be eligible in the future to participate in value-based payment initiatives such as the Merit-Based Incentive Payment System (MIPS). Measuring and reporting the state of interoperability will help to determine the type of support these providers need and their readiness to participate in delivery system reform efforts in the future.
Collaboration between the Office of the National Coordinator for Health Information Technology (ONC) and SAMHSA on this data collection effort will provide an efficient manner to track trends in health IT adoption, use, and interoperability among behavioral health care providers. In addition, this collaboration will contribute to the development of strategic efforts to leverage health IT in behavioral health care settings to provide cost effective, high quality and patient-centered care. Results from this testing will allow ONC and SAMHSA to work together to quantitatively assess health IT adoption and interoperability among behavioral health care providers using SAMHSA’s current national surveys, the National Survey of Substance Abuse Treatment Services (N-SSATS) and the National Mental Health Services Survey (N-MHSS).
ONC and SAMHSA have collaborated in the past on survey efforts, though on a smaller scale. In 2017, ONC and SAMHSA collaborated to conduct a survey among SAMHSA grantees on health IT adoption and interoperability. Given the grantees are not representative of behavioral health settings nationally, the findings are not broadly generalizable though were still useful for programmatic purposes. The online survey examined the adoption and use of health IT and engagement in the four domains of interoperability across community behavioral health care settings. The measures for the grantee survey were developed to be consistent with other ONC measures of health IT adoption and interoperability that have been used across other settings such as hospitals and office-based physicians. The survey items on the grantee survey informed the development of the items for inclusion in the N-SSATS and N-MHSS.
N-SSATS is an annual census of substance abuse treatment facilities which collects data on the location, scope, services provided, and operational characteristics of all known substance abuse treatment facilities in the United States and its jurisdictions, and on utilization of services by means of a single-day count of clients in treatment. It is conducted under the authority of Section 505 of the Public Health Service Act (42 U.S.C. 290aa-4) to meet the specific mandates for annual information about public and private sub- stance abuse treatment providers and the clients they serve.
N-MHSS collects information about mental health treatment service providers across the nation and the services they provide to persons with mental illness. It provides national and state-level data on the number and characteristics of mental health treatment facilities in the United States. This data collection is authorized by Section 505(b) [42 USC 290aa—4] of the Public Health Service Act which mandates the collection of data on the number and variety of public and private nonprofit mental health programs and persons who receive care from them.
The information obtained from these efforts will be used to develop a new set of questions on the use and implementation of EHRs in behavioral health facilities for the N-SSATS and the N-MHSS surveys. Specifically, the information from the testing will be used to reduce respondent burden while simultaneously improving the quality of the data collected in these surveys. These objectives are met when respondents are presented with plain, coherent and unambiguous questionnaires that ask for data compatible with respondents’ memory and/or current reporting and record keeping practices.
Improved surveys will inform policy decisions on behavioral health, as well as contributing to increased agency efficiency and reduced survey costs. In addition, methodological findings have broader implications for survey study and may be presented in technical papers at conferences or published in the proceedings of conferences or in journals.
Results of this test will not be disseminated or used to inform policy, program, or budget decisions. Findings will be shared between ONC and SAMHSA staff to decide how the tested questions will be incorporated in the surveys.
For this cognitive testing, no automated methods will be used for data collection.
This study does not duplicate any other questionnaire design work being done by SAMHSA or other Federal agencies. No other federal agency or private organization collects information about the types of public and nongovernmental facilities that comprise the behavioral health care service delivery system on a state and national level. The information on mental health facilities already available from other data collection efforts cannot be used because the scope of coverage is limited or available data typically are outdated and not standardized across types of facilities.
The study may also involve joint efforts with staff from ONC. All efforts would be collaborative in nature, and no duplication in this area is anticipated.
The cognitive test will involve some contact with small entities. This study has been designed as relatively small-scale data collection efforts. This will minimize the amount of burden required to improve questionnaires and procedures, test new ideas, and refine or improve upon positive or unclear results from other tests.
If the requested information is not collected, there is a risk to data quality as new questionnaire items will be implemented without adequate testing and refinement. Without adequate testing, data collected may be of poor quality, resulting in additional resources required to process data or negative impacts on survey estimates. Not collecting this data will also affect the deeper understanding of how behavioral health treatment facilities answer surveys and how SAMHSA can better serve them. Lastly, SAMHSA’s ability to develop timely new well-designed surveys will be diminished.
This information collection fully complies with 5 CFR 1320.5(d)(2).
A Federal Register notice was published on July 27, 2018 (83 FR 35669). No public comments were received.
It is critical to the development or improvement process for any survey that SAMHSA ensures that the proposed survey questions can be answered by the target population, and that the questions are asked such that they provide for the most uniform comprehension possible. Respondent involvement in questionnaire development serves to ensure that respondents understand and can answer the survey questions, thus reducing overall respondent burden and improving data quality.
No payment or gifts are provided to respondents for participation in this cognitive testing.
All respondents who participate in this study under this clearance will be informed that the information they provide is confidential and that their participation is voluntary. For personal visit and telephone interviews, this information will be conveyed verbally by the interviewer. For personal visit interviews, respondents will also be notified in writing so they will have something they can keep and read. For telephone interviews, a notification in writing will be provided to the respondent via e-mail prior to the interview. All participants in cognitive testing will be required to sign written notification concerning the voluntary and confidential nature of their participation. SAMHSA will also inform respondents in writing of the need to have an OMB number.
All forms in this test will include a Pledge to Respondents stating that the information provided will be protected to the fullest extent allowable under Section 501(n) of the Public Health Service Act (42 USC 290aa(n)) – Limitation on the Use of Certain Information. This law permits the public release of identifiable information about an establishment only with the consent of that establishment and limits the use of information to the purposes for which it was supplied. All of data collected will be according to the protection of Title 42, U.S.C., Section 242m. ONC is entering into this agreement pursuant to the authority of 31 U.S.C. 1535, The Economy Act.
The series of questions to be tested (Attachment A) do not include questions of a sensitive nature.
The total estimated burden for this study is 39.2 hours for the period from September through December 2018. These hours will be distributed as follows:
Survey |
No.
of |
Responses per respondent |
Total number of responses |
Hours per response |
Total burden hours |
Hourly |
Total hour cost |
Interviews |
80 |
1 |
80 |
.50 |
40 |
$47.29 |
$1,891.60 |
Basis for Burden Hour Estimate:
A pre-testing of these questions was conducted with 9 mental health and substance abuse treatment facilities during November and December of 2017. The average time for the responses was 30 minutes (0.50 hours), which was used to calculate the burden hour estimates. The expected number of interviews will be 80, therefore the total number of burden hours of 40 hours.
Basis for Hourly Wage Rate Estimate:
The estimated hourly wage rate is based on the median hourly pay of $47.29 for medical and health service managers as reported in the Bureau of Labor Statistics, U.S. Department of Labor Occupational Outlook Handbook, 2017-18 Edition, Medical and Health Service Managers, at http://www.bls.gov/ooh/management/medical-and-health-services-managers.htm. Based on the hourly pay and the total burden hours, the total hourly cost is $1,891.60.
The cost to the Government for this cognitive testing is estimated to be $112,000. The project-related activities included in this estimate are: developing and coordinating review of materials among SAMHSA, ONC, and contractor; developing questionnaire, consent form, and other required documents; tracking and monitoring participation; recruiting cases for testing; analyzing responses to the questions; and writing summary report with recommendations. These activities will be shared between SAMHSA and the Contractor for the project.
Contract Monitoring. The cost for monitoring the contract during the testing period and carrying out related work including the salary and travel for contractor site visits for one FTE totals approximately $70,000.
Time Schedule
The cycle of activities for this as follows:
Activity |
Completion Date |
Cognitive Testing of EHR Questions for N-SSATS & N-MHSS
|
|
Finalization of questions to be tested
|
April 2018
|
Testing: Phase I (includes recruiting and interviewing)
|
January-February 2019
|
Testing: Phase II (includes recruiting and interviewing)
|
March-April 2019
|
Analysis & Report Writing
|
May-July 2019 |
b. Analyses and Publications
Contractor will be responsible for summarizing findings from both phases of the testing in a report that must include the following: issues with questions, summary of comments and responses from participants, and recommendations for implementation. Contractor will share final report with SAMHSA. ONC and SAMHSA will discuss the findings and recommendation from the report and will decide the next plan of action for implementation.
The expiration date will be displayed.
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | National Mental Health Services Survey (N-MHSS) Revision |
| Author | laura milazzo-sayre |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |