SUPPORTING STATEMENT - PART A
JOINT OUTPATIENT EXPERIENCE SURVEY (JOES) SURVEY SUITE – 0720-JOES
Need for the Information Collection
The Joint Outpatient Experience Survey (JOES) is an ongoing, unified outpatient survey system that combines and standardizes long-standing methods used by the Army, Navy, Air Force and National Capital Region to learn about beneficiary health care experiences with the goal of making them better. The JOES suite of surveys includes six patient experience surveys for various outpatient environments: JOES, JOES-CAHPS, JOES-Dental, JOES-Ambulatory, JOES Walk-in Contraceptive Clinic, and JOES-Emergency Department, hereafter referred to simply as the JOES Survey Suite (all six surveys). Details of each survey are below. The JOES Survey Suite elicit feedback from Military Health System (MHS) users (beneficiaries) regarding their outpatient care experiences. The MHS provides care to approximately 9.6 million beneficiaries composed of uniformed service members, military retirees, and family members.
The core objective of the JOES Survey Suite is to systematically assess MHS beneficiaries' perceptions of their outpatient care quality when received directly at Military Treatment Facilities (MTFs) and through civilian network providers reimbursed by the MHS (private sector care). Data insights guide efforts to optimize care delivery across the MHS. The JOES Survey Suite leverages industry best practices to regularly measure patient satisfaction across the facets of outpatient care. Data-driven insights from this comprehensive assessment inform systematic quality improvement efforts for healthcare delivery optimization across the Military Health System.
The JOES Survey Suite focuses primarily on eliciting information on the following:
Access to care,
Doctor or provider communication and courtesy,
Effectiveness of the clerks/receptionists at the doctor’s office,
Perceptions of the MHS,
Overall satisfaction with the provider and healthcare.
Regulatory authorities and directives mandating JOES data collection include:
2021 Executive Order 13985 Executive Order on Transforming Federal Customer Experience and Service Delivery to Rebuild Trust in Government
2011 Executive Order 13571 Streamlining Service Delivery and Improving Customer Service
FY 2016 National Defense Authorization Act (NDAA) Section 713 Expansion of Evaluation of Effectiveness of the TRICARE Program to Include Information on Patient Safety, Quality of Care, and Access to Care at Military Medical Treatment Facilities (Public Law 114-92)
FY 2013 NDAA Section 724 Annual Healthcare Survey (Public Law 102-484)
Use of the Information
The six JOES surveys are:
JOES: JOES reports on the experiences of outpatient beneficiaries receiving care from MTFs. It is the successor of Service-specific outpatient satisfaction surveys. It is administered every business day via email and text message. Results are analyzed and provided at the Provider-level, MTF-level and the overall MHS.
JOES-CAHPS: JOES-CAHPS reports on the experiences of outpatient beneficiaries receiving care from MHS’s direct care via military treatment facilities (MTFs) and through its civilian providers, private sector care. The survey is administered monthly via email and mail, adhering to industry protocols outlined by the Clinician & Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS) developed by the Agency for Healthcare Research and Quality (AHRQ) for measuring patient satisfaction. More information about CG-CAHPS can be found at CAHPS Clinician & Group Survey | Agency for Healthcare Research and Quality (ahrq.gov). Results are analyzed and provided at the MTF-level and are benchmarked against CG-CAHPS national civilian scores.
JOES-Dental: JOES-Dental reports on the experiences of outpatient beneficiaries receiving dental care from MTFs. The survey is designed to be a companion to the JOES. The survey is administered every business day via email and text message. JOES-Dental results are analyzed and provided at the Provider-level, MTF-level, and overall MHS.
JOES-Ambulatory: The JOES-Ambulatory survey reports on the experiences of outpatient beneficiaries receiving care from ambulatory surgery centers within the MHS. The survey is administered every business day via email and text message. JOES-Ambulatory results are analyzed and provided at the Provider-level, MTF-level, and overall MHS. Data are used for reporting to Leapfrog.
JOES Walk-in Contraceptive Clinic: The JOES Walk-in Contraceptive Clinic survey reports on the experiences of outpatient beneficiaries receiving care from walk-in contraceptive clinics throughout the MHS. The survey is administered via a QR code at the point of service. The survey is fielded continuously based upon the schedule of the walk-in contraceptive clinics.
JOES-Emergency Department: The JOES-Emergency Department survey reports on the experiences of outpatient beneficiaries receiving care from emergency departments within the MHS. The survey is currently administered only at the Walter Reed National Military Medical Center Emergency Department but may be expanded to other emergency departments in the future. The survey is administered every business day via text message. JOES-Emergency Department results are analyzed and provided at the Provider-level and MTF-level.
Aside from JOES-C that is modeled from CG-CAHPS, the other JOES surveys may rotate or update questions as DoD priorities change with regard to patient experience with their health care visit. Examples of rotating questions include questions regarding telehealth during the COVID-19 pandemic and questions about pharmacy experiences. Additional rotations include health IT questions and emergency department questions. When questions are changed, we ensure the survey burden on the respondents is not impacted (i.e. we will replace a question with an existing one if new information is being sought). The primary goal when rotating questions is to ensure we elicit actionable information while keeping respondent burden to a minimum. Succinct and engaging surveys often lead to higher beneficiary response rates and thus the JOES surveys often rotate questions to maximize response rates.
The JOES Survey Suite collects feedback from military personnel, military family members, and retirees regarding their outpatient care experiences at MTFs (direct care) and civilian network providers (private sector care). These surveys gather insights into beneficiaries' perspectives on the care they received during outpatient visits in order to identify opportunities to improve the care provided by MHS providers and systems.
Participation is voluntary and confidential; respondents can skip questions and are assured their information is protected. Participation is encouraged to help the DHA enhance care quality. The JOES studies use stratified random sampling at the facility level and collect data via mail, email, SMS, and point-of-service QR codes (details per survey below). This mixed-mode approach allows for comprehensive feedback.
Format of each of the JOES survey instruments:
JOES: Administered via email and text message
JOES-CAHPS: Administered via email and mail (following CG-CAHPS protocol)
JOES-Dental: Administered via email and text message
JOES-Ambulatory: Administered via email and test message
JOES Walk-in Contraceptive Clinic: Administered via QR code at the point of service
JOES-Emergency Department: Administered via text message
JOES surveys can be completed by mail and online/text message. Currently, the JOES-CAHPS survey is the only survey where mail is a mode available for survey completion. A survey that is mailed to beneficiaries and completed by mail is received and scanned through an optical scanning machine whereby a digital image of the questionnaire is created. The imaging software evaluates the typical mark for each respondent using an algorithm and assigns a confidence level to each mark in the capture box. The capture boxes are established during programming and include the box and a portion immediately surrounding the box. The algorithm uses the density of the mark to make its selection. When the respondent makes a mark in the capture box but this mark falls below the confidence level, the mark is sent to an operator for evaluation. Operators have procedures including utilization of a full-page view for when there are either scratch outs or a lot of writing on a survey so as not to send false positive marks.
JOES, JOES-Dental, JOES-Emergency Department, JOES-Ambulatory, and JOES Walk-in Contraceptive Clinic are administered electronically whereby beneficiaries can provide feedback online. The online survey platform requires beneficiaries to enter the unique ID and password found in the survey invitations. The online survey contains data verification programs that prevents respondents from entering invalid values. The online survey system allows participants to exit an in-progress survey and return to it later for completion. Respondents have 24/7 access to the online survey. Once the online survey is completed and submitted, the respondent is not permitted to access the system again. Technical assistance for the online survey is available via a 24/7 toll-free helpdesk number and an email-based online helpdesk which is available during regular business hours.
Both the mail and online data are loaded into a response-based database table with their raw responses. In this database table, there is no personally identifiable information (PII). The raw survey data go through validation, editing, consistency checks and formatting for general data cleaning purposes. Only the cleaned records are used for analysis and reporting.
JOES and JOES-CAHPS results are reported weekly on an interactive, CAC-enabled website. JOES-Dental results are reported monthly on the same CAC-enable website. The JOES website is used by MTF patient advocates, MTF staff (providers, nurses, clerks and receptionist, and support staff), TRICARE Health Plan analysts, the DHA Survey Working Group, and DHA leaders for monitoring JOES scores over time. The other JOES surveys are reported monthly via reports sent to appropriate DHA program offices. The JOES Walk-in Contraceptive Clinic survey reports in near real-time because it uses QR codes. These timely results enable data-driven quality improvements across the MHS.
Data from the various JOES surveys have been reported in the Annual TRICARE Evaluation Report to Congress, on various DHA/MHS performance dashboards such as the Measures Library, internal MTF reports, and as part of the Leapfrog Hospital Survey. Quarterly and annual reports that synthesize key insights are sent to DHA and DoD stakeholders to inform policy and resourcing decisions.
The Dimensions online data collection platform will be used initially. We have been using this platform for fielding of the JOES. In the near future, we anticipate using the Medallia online data collection platform.
3. Use of Information Technology
The JOES Survey Suite employ a multifaceted data collection strategy, with approximately 70% of responses presently gathered electronically. To further augment efficiency, plans aim to boost electronic collection rates by curtailing paper surveys and increasing email and SMS modes. As CG-CAHPS protocols do not require paper surveys, phasing out this mode will align with industry best practices.
Participants will be contacted via mail, email, or SMS after an eligible medical or dental visit. The survey will be sent to patient home addresses and email addresses obtained from Defense Eligibility Enrollment Reporting System (DEERS). Patient SMS information will be obtained from a SMS consent database.
In addition to leveraging email and SMS, an innovative point-of-service modality shows promise for rapid real time feedback. Early pilot successes will pave the way for scaled implementation across a broader spectrum of clinics and facilities.
Data collection modernization efforts for the JOES surveys center on transitioning away from paper dependency toward integrated digital modes. This progression not only improves collection efficiency, but also provides more nimble analytics to drive quality improvement in a data-driven manner.
4. Non-duplication
The information obtained through this collection is unique and is not already available for use or adaptation from another cleared source.
5. Burden on Small Businesses
This information collection does not impose a significant economic impact on a substantial number of small businesses or entities.
6. Less Frequent Collection
The JOES Survey Suite are administered as needed. It employs sophisticated data infrastructure to enable continuous knowledge generation and data collection. Beneficiary encounter and demographic data are drawn from the Military Data Repository (MDR), DEERS, and HealtheAnalytics. The data are then integrated into the JOES database on a daily basis alongside survey response data. Given the daily update of the MDR and DEERS and the time required to synthesize collected survey data, the most infrequent collection of results for the majority of the JOES surveys would be one business day.
However, the Walk-in Contraceptive Clinic survey uniquely collects near real time feedback at the point of service, unlocking near real time insights. In the future, the point of service modality will be expanded to obtain actionable near real time feedback from a broader spectrum of patients.
There are approximately 45 million visits to MTFs per year. We are surveying approximately 4 million of these visits for a rate of less than 10% of all visits.
7. Paperwork Reduction Act Guidelines
This collection of information does not require collection to be conducted in a manner inconsistent with the guidelines delineated in 5 CFR 1320.5(d)(2).
8. Consultation and Public Comments
Part A: PUBLIC NOTICE
A 60-Day Federal Register Notice (FRN) for the collection published on Tuesday, July 9, 2024. The 60-Day FRN citation is 89 FRN 56347.
Two comments were received during the 60-Day Comment Period. They are included below in the order they were received, as well as our Agency’s response to the comment.
Comment #1:
“Gaza: Number of children killed higher than from four years of world conflict. You are funding this genocide. Stop. “This war is a war on children. It is a war on their childhood and their future,” said UNRWA Commissioner-General Philippe Lazzarini, who described as “staggering” the latest Gaza health authority data indicating that at least 12,300 youngsters have died in the enclave in the last four months, compared with 12,193 globally between 2019 and 2022. Writing on X, formerly Twitter, late on Tuesday, the UNRWA chief reiterated repeated international calls for an immediate ceasefire in the enclave, where intense Israeli bombardment in response to Hamas-led terror attacks in Israel on 7 October has levelled entire neighbourhoods. To date, more than 31,184 Palestinians have been killed and 72,889 injured, according to the local health authorities. As of 12 March, 247 Israeli soldiers have been killed in Gaza with 1,475 injured since the start of the ground operation, Israeli army data shows.On Wednesday the agency reported that at least one staffer had been killed and another 22 injured when Israeli Forces hit a food distribution centre in the eastern part of Rafah in the far south of the Strip. “Today’s attack on one of the very few remaining UNRWA distribution centres in the Gaza Strip comes as food supplies are running out, hunger is widespread and, in some areas, turning into famine”, Mr. Lazzarini said. “Every day, we share the coordinates of all our facilities across the Gaza Strip with parties to the conflict. The Israeli army received the coordinates including of this facility yesterday,” said the UNRWA chief Since the war began five months ago, UNRWA has recorded an unprecedented number of violations against its staff and facilities. At least 165 UNRWA team members have been killed including while in the line of duty, with more than 150 facilities hit, among them many schools. Israel is nondiscriminately attacking hospitals, refugee camps and schools. The development came as UN humanitarians repeated dire warnings about the catastrophic situation in Gaza, where one in four is close to famine – at least 576,000 people. Some 25 individuals have now died from severe acute malnutrition and dehydration in northern Gaza according to UN aid coordination office, OCHA, 21 of them reportedly children. Youngsters are among those least able to cope with hunger and disease, the UN Children’s Fund, UNICEF, has warned, with one million youngsters already uprooted from their homes by the war and some 17,000 unaccompanied or separated children – one per cent of the 1.7 million Gazans displaced.”
Comment #2:
“See attached file(s) All authority of government will be held accountable for their failure to comply”
DHA RESPONSE: Comment #1 is not relevant to this information collection. Comment #2 contained the attachments “FDIC Deposit Insurance Misrepresentation Form” and State of Illinois Application for a Permit to Organize a State Bank. This is also not relevant to this information collection.
A 30-Day Federal Register Notice for the collection published on Tuesday, November 19, 2024 The 30-Day FRN citation is 89 FRN 91373
Part B: CONSULTATION
No additional consultation apart from soliciting public comments through the Federal Register was conducted for this submission.
9. Gifts or Payment
No payments or gifts are being offered to respondents as an incentive to participate in the collection.
10. Confidentiality
A Privacy Act Statement is provided on each survey instrument. The email script contains Privacy Advisory as well.
The applicable System of Records Notice (SORN) is: EDHA 07, Military Health Information System (June 15, 2020; 85 FR 36190) https://dpcld.defense.gov/Portals/49/Documents/Privacy/SORNs/DHA/EDHA-07.pdf
A Privacy Impact Assessment (PIA) is not required for this collection because PII is not being collected electronically.
11. Sensitive Questions
Some JOES instruments include questions related to self-reported mental health status. Mental health status among Active Duty Service Members is a top priority for the DoD and MHS. The JOES Survey Suite is an invaluable source of this information and for tracking changes in self-reported mental health status overtime. Additionally, we have established a process for handling JOES Critical Alerts; survey comments from beneficiaries, including Active Duty Service Members, indicating intent to harm self or others. As soon as a Critical Alert comment is received, DHA contacts the MTF where the patient was seen and the clinic/hospital staff make contact with the patient. We receive approximately 2-3 critical alerts per week from the JOES.
12. Respondent Burden and its Labor Costs
Part A: ESTIMATION OF RESPONDENT BURDEN
Collection Instrument(s)
Joint Outpatient Experience Survey (JOES)
Number of Respondents: 250,000
Number of Responses Per Respondent: 1
Number of Total Annual Responses: 250,000
Response Time: 5 minutes
Respondent Burden Hours: 20,833 hours
JOES-CAHPS
Number of Respondents: 40,000
Number of Responses Per Respondent: 1
Number of Total Annual Responses: 40,000
Response Time: 7 minutes
Respondent Burden Hours: 4,667 hours
JOES-Dental
Number of Respondents: 10,000
Number of Responses Per Respondent: 1
Number of Total Annual Responses: 10,000
Response Time: 3 minutes
Respondent Burden Hours: 500 hours
JOES-Ambulatory
Number of Respondents: 1,500
Number of Responses Per Respondent: 1
Number of Total Annual Responses: 1,500
Response Time: 3 minutes
Respondent Burden Hours: 75 hours
JOES-Walk-in Contraceptive Clinic
Number of Respondents: 7,500
Number of Responses Per Respondent: 1
Number of Total Annual Responses: 7,500
Response Time: 3 minutes
Respondent Burden Hours: 375 hours
JOES-Emergency Department
Number of Respondents: 1,000
Number of Responses Per Respondent: 1
Number of Total Annual Responses: 1,000
Response Time: 5 minutes
Respondent Burden Hours: 83 hours
Total Submission Burden (Summation or average based on collection)
Total Number of Respondents: 310,000
Total Number of Annual Responses 310,000
Total Respondent Burden Hours: 26,533 hours
Part B: LABOR COST OF RESPONDENT BURDEN
Collection Instrument(s)
Joint Outpatient Experience Survey (JOES)
Number of Total Annual Responses: 250,000
Response Time: 5 minutes
Respondent Hourly Wage: $31.48
Labor Burden per Response: $2.62
Total Labor Burden: $655,00
JOES-CAHPS
Number of Total Annual Responses: 40,000
Response Time: 7 minutes
Respondent Hourly Wage: $31.48
Labor Burden per Response: $3.67
Total Labor Burden: $146,800
JOES-Dental
Number of Total Annual Responses: 10,000
Response Time: 3 minutes
Respondent Hourly Wage: $31.48
Labor Burden per Response: $1.57
Total Labor Burden: $15,700
JOES-Ambulatory
Number of Total Annual Responses: 1,500
Response Time: 3 minutes
Respondent Hourly Wage: $31.48
Labor Burden per Response: $1.57
Total Labor Burden: $2,355
JOES Walk-in Contraceptive Clinic
Number of Total Annual Responses: 7,500
Response Time: 3 minutes
Respondent Hourly Wage: $31.48
Labor Burden per Response: $1.57
Total Labor Burden: $11,775
JOES-Emergency Department
Number of Total Annual Responses: 1,000
Response Time: 5 minutes
Respondent Hourly Wage: $31.48
Labor Burden per Response: $2.62
Total Labor Burden: $2,620
Overall Labor Burden
Total Number of Annual Responses: 310,000
Total Labor Burden: $834,250
The Respondent hourly wage was determined by using the Department of Labor Wage Website ([https://www.bls.gov/oes/current/oes_nat.htm#00-0000]).
13. Respondent Costs Other Than Burden Hour Costs
There are no annualized costs to respondents other than the labor burden costs addressed in Section 12 of this document to complete this collection.
14. Cost to the Federal Government
Part A: LABOR COST TO THE FEDERAL GOVERNMENT
Collection Instrument(s)
JOES Survey Suite
Number of Total Annual Responses: 310,000
Processing Time per Response: 0 hours
Hourly Wage of Worker(s) Processing Responses: $0
Cost to Process Each Response: $0
Total Cost to Process Response: $0
Overall Labor Burden to the Federal Government
Total Number of Annual Responses: 310,000
Total Labor Burden: $0
Labor cost to the government calculated based on government oversight of the contract who will be conducting this survey as $45,505.00. All processing is performed by the contractor and are captured in the “other” contract cost in Part B below.
Part B: OPERATIONAL AND MAINTENANCE COSTS
Cost Categories
Equipment: $
Printing: $
Postage: $
Software Purchases: $
Licensing Costs: $
Other: The total cost of the contract supporting this survey is $3,199,389.
Total Operational and Maintenance Cost: $3,199,389.
Part C: TOTAL COST TO THE FEDERAL GOVERNMENT
Total Labor Cost to the Federal Government: $45,505
Total Operational and Maintenance Costs: $3,199,389
Total Cost to the Federal Government: $3,164,894
15. Reasons for Change in Burden
This is a new collection with a new associated burden.
16. Publication of Results
Results from the JOES Survey Suite will be published as part of an internal (DoD) annual report released ahead of the close of each fiscal year to facilitate patient experience improvement efforts. Results are also published in the Annual TRICARE Evaluation Report to Congress. Additionally, the results may be considered for external publication and may be published by Leapfrog. However, no opportunities for external publication have been identified at this time.
17. Non-Display of OMB Expiration Date
We are not seeking approval to omit the display of the expiration date of the OMB approval on the collection instrument.
18. Exceptions to “Certification for Paperwork Reduction Submissions”
We are not requesting any exemptions to the provisions stated in 5 CFR 1320.9.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Kaitlin Chiarelli |
| File Modified | 0000-00-00 |
| File Created | 2024-11-20 |