CMS-10754 - Fast Track Supporting Statement

Request for Approval under the “Generic Clearance for the Collection of Routine Customer Feedback” (OMB Control Number: 0938-1185)

TITLE OF INFORMATION COLLECTION: User Acceptance Testing for the Medicare Ground Ambulance Data Collection System (GADCS) (CMS-10754)

PURPOSE:

The Centers for Medicare & Medicaid Services (CMS) will conduct user testing and gather feedback on the Medicare Ground Ambulance Data Collection System (GADCS). CMS is required by Section 1834 (l)(17)(A) of the Social Security Act “to develop a data collection system to collect cost, revenue, utilization, and other information determined from providers and suppliers of ground ambulance services (“ground ambulance organizations”).” This section also requires that providers and supplies of ground ambulance services selected who are required to submit sufficient data and subject to a payment reduction if they do not submit sufficient data. To satisfy this requirement, CMS created a web-based data collection system, hosted on the CMS Enterprise Portal, for ground ambulance organizations to report this information. CMS has received several comments from the public stating that testing is a critical step in the development of its data collection system. Thus, CMS will invite a group of volunteers to test the data collection system and provide feedback on usability, technical issues, and other aspects of the system. CMS will consider these suggestions and make modifications, if feasible, to the design of the GADCS as well as supplemental documentation and educational resources. After user testing, CMS hopes to streamline the data collection and ultimately minimize the reporting burden for selected ground ambulance organizations.

In 2021, CMS conducted its initial user acceptance testing. Based on feedback from testers, CMS went back to the GADCS and added several enhancements. For example, CMS improved the autosave feature and included an import function for certain data fields. CMS believes these enhancements will vastly improve the data reporting experience in GADCS. We hope to apply feedback from our user testing by early summer of 2022 and finalize any changes to our system by the end of fall. Therefore, CMS requests permission to conduct another round of user acceptance testing before the production launch of our system.

DESCRIPTION OF RESPONDENTS:

CMS will invite ground ambulance organizations to test the web-based GADCS. We expect that different types of personnel at these ground ambulance organizations will receive the invitations to participate in testing, respond to the invitations, and actually test the system. These personnel types include clerical and administrative staff, emergency medical technicians, fire chiefs, and other personnel. The system requires each ground ambulance organization to have staff registered in a “submitter” and “certifier” role.

CMS will conduct two types of testing. One group (N=51) will proceed through the testing experience unobserved and self-paced. This group will be directed to GADCS landing page and provided with the user guide. CMS believes it is important to assess whether users can complete the GADCS based on the user guide and other instructions such as pop-up messages and other built-in features. This group will then be asked to complete a short survey about their experience with the system and the user guide.

Another group (N=9) will proceed through the testing experience observed so that CMS can assess how an organization reads, reacts, and responds to the questions within GADCS in real time. CMS contractors will observe how participants use and navigate through the GADCS. Observers will document utilization of tools such as pop-up instructions and the user guide and probe on comprehension. They will document how users navigate through the instrument sections. Throughout, observers will ask questions about how participants account for complex cost situations like cost and revenue allocation, depreciation expenses, and shared resources. Furthermore, participants may also ask questions and provide feedback regarding the platform and/or instrument questions in real time. This virtual observed format will allow CMS to identify instructions or features of the GADCS that might need modification and provide further opportunity to identify items that need clarification via education sessions and FAQs. Observers will follow a guide for each session (attached protocol) with semi-scripted probes and space to document observations related to use of instructions, navigation, and user questions and volunteered feedback.

TYPE OF COLLECTION: (Check one)

[ ] Customer Comment Card/Complaint Form [ ] Customer Satisfaction Survey

[X] Usability Testing (e.g., Website or Software) [ ] Small Discussion Group

[ ] Focus Group [ ] Other: ______________________

CERTIFICATION:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low-burden for respondents and low-cost for the Federal Government.

3. The collection is non-controversial and does not raise issues of concern to other federal agencies.

4. The results are not intended to be disseminated to the public.

5. Information gathered will not be used for the purpose of substantially informing influential policy decisions.

6. The collection is targeted to the solicitation of opinions from respondents who have experience with the program or may have experience with the program in the future.

Name: Steve Chu

To assist review, please provide answers to the following question:

Personally Identifiable Information:

1. Is personally identifiable information (PII) collected? [ ] Yes [X] No

2. If Yes, will any information that is collected be included in records that are subject to the Privacy Act of 1974? [ ] Yes [ ] No

3. If Yes, has an up-to-date System of Records Notice (SORN) been published? [ ] Yes [ ] No

Gifts or Payments:

Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to participants? [ ] Yes [X] No

BURDEN HOURS

Category of Respondent	No. of Respondents	Participation Time	Burden
Private Sector	30	120	60
State, local or tribal governments	30	120	60
Totals	60		120

*Note: If CMS could not determine an organizations category, CMS placed the organization in the “private sector” category.

The data collection instrument includes screening questions and skip patterns that directs respondents to only view and respond to their specific type of organization. For example, a private sector organization, such as a for-profit non-government ground ambulance organization, may not use volunteer labor or share operational costs such as building space or personnel (such as fire-fighters or EMTs) with another public safety organization or receive municipal taxes. Responses to those questions will not be required as they are not applicable.

As previously stated, the GADCS is hosted on the CMS Enterprise Portal. Thus, users must enter the GADCS through the enterprise portal, which requires registering it to as an individual. Each user must then link to an organization through by entering a National Provider Identifier (NPI). CMS will use the contact information to send login information as well as reminder notifications and a confirmation email upon successful completion of the GADCS.

CMS estimates it will take on average two hours to complete the testing experience. In other words, CMS rolled the observed and unobserved testing participants together in its estimate. Although CMS believes it would take longer for the observed group to go through testing and shorter for the unobserved group, because there are so many more unobserved participants than observed participants, it weighs the number closer to the 2-hours.

First, respondents will take 15 minutes to read the invitation email and agree to participate. Second, respondents will take another 15 minutes to setup an account in the GADCS. Third, respondents will take another 15 minutes to read the instrument instructions. Fourth, the respondents will an hour to complete the entire instrument. CMS does not require nor even expect respondents will use actual cost and revenue data during the user testing. Instead, organizations will be encouraged simply to enter any response to practice using the GADCS. Fifth, respondents will take up to 15 minutes to complete a short feedback form. Finally, CMS will wipe all data collected during observed and unobserved testing.

Burden Hours	=	[(Unobserved group x participation time) + (observed group x participation time)] / 60
Burden Hours	=	[(50 participants x 110 minutes) + (9 participants x 180 minutes)] / 60
Burden Hours	=	[5500 minutes + 1620 minutes] / 60
Burden Hours	=	[7120 minutes] / 60
Burden Hours	=	118.6667

CMS then rounded up and thus arrived at 120 Burden Hours.

FEDERAL COST: The estimated annual cost to the Federal government involves contractors and employee staff time. CMS contractors will email invitations for both unobserved and observed testing and will monitor responses. Furthermore, CMS contractors will setup accounts for the respondents and collect feedback on the user experience. These tasks are included in the overall cost of the system development and are not easily quantifiable. CMS contractors will then provide the results to CMS staff to receive and analyze the results from user testing during the course of normal Federal duties.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents

1. Do you have a customer list or something similar that defines the universe of potential respondents and do you have a sampling plan for selecting from this universe? [X] Yes [ ] No

If the answer is yes, please provide a description of both below (or attach the sampling plan)? If the answer is no, please provide a description of how you plan to identify your potential group of respondents and how you will select them?

Previously, CMS selected 500 of these organizations broadly representative of the universe of Medicare ground ambulance organizations in terms of their provider versus supplier status, service area, population density, ownership, and annual transport volume. Within this group, CMS only received an initial response rate of roughly 6%. Even with additional reminder and outreach efforts, CMS never received more than 45 organizations willing to participate in the user testing experience. Furthermore, several organizations that agreed to participate never actually participated in user testing.

With these lessons in mind, for this round of testing CMS will recruit organizations to participate with a more targeted approach. CMS will contact trade associations and specialty groups, and past participants of various CMS public presentations for volunteers. CMS believes this approach will reduce the overall burden to the public and instead only include the most motivated and engaged user population. Thus, CMS anticipates it will receive up to 60 organizations who will participate in the user testing.¹

Administration of the Instrument

1. How will you collect the information? (Check all that apply)

[X] Web-based or other forms of Social Media

[ ] Telephone

[ ] In-person

[ ] Mail

[ ] Other, Explain

2. Will interviewers or facilitators be used? [X] Yes* [X] No

   1. Only for the observed group of 9 participants.

Please make sure that all instruments, instructions, and scripts are submitted with the request.

Instructions for completing Request for Approval under the “Generic Clearance for the Collection of Routine Customer Feedback”

TITLE OF INFORMATION COLLECTION: Provide the name of the collection that is the subject of the request. (e.g. Comment card for soliciting feedback on xxxx)

PURPOSE: Provide a brief description of the purpose of this collection and how it will be used. If this is part of a larger study or effort, please include this in your explanation.

DESCRIPTION OF RESPONDENTS: Provide a brief description of the targeted group or groups for this collection of information. These groups must have experience with the program.

TYPE OF COLLECTION: Check one box. If you are requesting approval of other instruments under the generic, you must complete a form for each instrument.

CERTIFICATION: Please read the certification carefully. If you incorrectly certify, the collection will be returned as improperly submitted or it will be disapproved.

Personally Identifiable Information: Provide answers to the questions. Note: Agencies should only collect PII to the extent necessary, and they should only retain PII for the period of time that is necessary to achieve a specific objective.

Gifts or Payments: If you answer yes to the question, please describe the incentive and provide a justification for the amount.

BURDEN HOURS:

Category of Respondents: Identify who you expect the respondents to be in terms of the following categories: (1) Individuals or Households; (2) Private Sector; (3) State, local, or tribal governments; or (4) Federal Government. Only one type of respondent can be selected per row.

No. of Respondents: Provide an estimate of the Number of Respondents.

Participation Time: Provide an estimate of the amount of time (in minutes) required for a respondent to participate (e.g. fill out a survey or participate in a focus group)

Burden: Provide the Annual burden hours: Multiply the Number of Respondents and the Participation Time then divide by 60.

FEDERAL COST: Provide an estimate of the annual cost to the Federal government.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents. Please provide a description of how you plan to identify your potential group of respondents and how you will select them. If the answer is yes, to the first question, you may provide the sampling plan in an attachment.

Administration of the Instrument: Identify how the information will be collected. More than one box may be checked. Indicate whether there will be interviewers (e.g. for surveys) or facilitators (e.g., for focus groups) used.

Submit all instruments, instructions, and scripts with the request.

• Survey Instrument

  • Link

• Interview Protocol

  • (attached)

• User Guide

  • (attached)

• Sample Feedback Form

  • (attached)

• Recruitment Email

  • (attached)

PRA Disclosure Statement will be added as a link to the bottom of the survey. (OMB control number and expiration date will be added when survey is approved).

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number.  The valid OMB control number for this voluntary information collection is 0938-1185.   The expiration date is (XX/XX/XXXX). The purpose of this voluntary information collection request is to collect feedback about immediate technical issues users may experience on accessing and using the Medicare Ground Ambulance Data Collection System (GADCS).  The end goal of this effort is to test the data collection system and provide feedback on usability, technical issues, and other aspects of the system. The time required to complete this voluntary information collection is estimated to average 2 hours per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection  If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports Clearance Officer, Mail Stop C4-26-05, Baltimore, Maryland 21244-1850.

****CMS Disclosure****  Please do not send applications, claims, payments, medical records or any documents containing sensitive information to the PRA Reports Clearance Office.  Please note that any correspondence not pertaining to the information collection burden approved under the associated OMB control number listed on this form will not be reviewed, forwarded, or retained. If you have questions about the survey please contact [Steve Chu, 410-786-1489, steve.chu@cms.hhs.gov].

1CMS used the upper range to calculate the burden hours

8