VHA OCC Mobile App Field Testing Instrument

Request for Approval under the “Generic Clearance for the Collection of Routine Customer Feedback” (OMB Control Number: 2900-0770)

TITLE OF INFORMATION COLLECTION: VHA OCC Mobile App Field Testing

PURPOSE:

The Veterans Health Administration (VHA) Office of Connected Care (OCC) is developing mobile applications (apps) to support patient care provided to Veterans. The purpose of VA Mobile App Field Testing is to collect data to assist the agency in improving existing or future deliveries of products or communication materials, which are important to support the implementation of these apps. Field Testing occurs after completion of development and technical testing and is focused on understanding, preferences, and experiences (including usability and workflow) of the app. Field Testing allows participants to use the app in the real world for a predetermined period of time (usually 2-4 weeks) and provide feedback on their experience. Participants in Field Testing are provided with an introduction to the app and the ability to provide feedback on their user experience and satisfaction. If applicable, training on the use of the app is provided however, part of what Field Testing is meant to determine is if users need training before they can successfully use the app. Participation in VA Mobile App Field Testing is voluntary and participants may decline participation at any time. Feedback will be aggregated such that no individuals’ comments are attributed to them, nor will any personally identifiable information be included. Feedback is then summarized for the development team to enable prioritization of enhancements and correction of defects before wider release of the app. Participants will be instructed that participation is voluntary and feedback will have no impact on their VA benefits.

Upcoming apps expected to require participants for VA Mobile App Field Testing include:

• CareHub

• CHAMPVA Pay (Civilian Health and Medical Program of the Dept. of Veterans Affairs)

• GDx (Genetic Diagnostic Testing)

• Dementia Care

• Mental Health Checkup

• MobileKidney

• MyVA Health Journal

• MyVA Images

• Pain Coach

• Rx Refill iOS (Prescription Refill)

• Summary of Care

• Sync My Health Data

• VA Health Chat

• VA Fit Heart

DESCRIPTION OF RESPONDENTS:

Field Testing covers a myriad of apps, as listed above. Because functionality and target audiences for each app varies, inclusion and exclusion criteria for participation in VA Mobile App Field Tests may vary. For example, participants in the Pain Coach Field Test, may need to be experiencing pain that is being managed in some way.

However, across all apps, respondents will:

• Be Veterans or family Caregivers

• Receive at least some care at the VA, or provide care for a Veteran who does

• Have credentials, as needed, to access the apps that relate directly to clinical care

• Be recruited based on app specific criteria

TYPE OF COLLECTION: (Check one)

[ ] Customer Comment Card/Complaint Form [ ] Customer Satisfaction Survey

[ X ] Usability Testing (e.g., Website or Software) [ ] Small Discussion Group

[ ] Focus Group [ ] Other: ______________________

CERTIFICATION:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low-burden for respondents and low-cost for the Federal Government.

3. The collection is non-controversial and does not raise issues of concern to other federal agencies.

4. The results are not intended to be disseminated to the public.

5. Information gathered will not be used for the purpose of substantially informing influential policy decisions.

6. The collection is targeted to the solicitation of opinions from respondents who have experience with the program or may have experience with the program in the future.

Name, Position Title and Credentials:

Nancy R. Wilck; Director, Connected Health Implementation Strategies

To assist review, please provide answers to the following question:

Personally Identifiable Information:

1. Is personally identifiable information (PII) collected? [ ] Yes [X] No

2. If Yes, will any information that is collected be included in records that are subject to the Privacy Act of 1974? [ ] Yes [ ] No

3. If Yes, has an up-to-date System of Records Notice (SORN) been published? [ ] Yes [ ] No

Gifts or Payments:

Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to participants? [ X ] Yes [ ] No

Further details are provided below, under the Cost calculation.

BURDEN HOURS

Category of Respondent	No. of Respondents	Participation Time	Burden
Individuals or Households	950*	60 minutes*	950 hours
Totals			950 hours

* Projected

FEDERAL COST: The estimated annual cost to the Federal government is: $177,286.**

** Cost to the government has two components:

1. That of one FTEE analyst (government or contractor) who plans, conducts, and analyzes data from the Field Tests. For the calculation, a GS 13-5 salary was assumed: $98,886.

2. Incentives to participants: Based on past payments for similar work done in VA, the average cost per response is approximately $392. This includes reimbursement to test participants (usually $50 per test, with increased amounts for longer duration) as well as cost for contractor support (recruiting, screening, and support of testing as needed).

It is estimated that 20 Field Tests will require use of incentives, and that each of these Field Tests will enroll 10 Participants: 20 * 10 * $392 = $78,400.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents

1. Do you have a customer list or something similar that defines the universe of potential respondents and do you have a sampling plan for selecting from this universe? [ ] Yes [ X ] No

If the answer is yes, please provide a description of both below (or attach the sampling plan)? If the answer is no, please provide a description of how you plan to identify your potential group of respondents and how you will select them?

Participants will be selected on a Field Test by Field Test basis, with specific criteria derived in collaboration with the app’s business owner. Sample criteria might include something such as “Received primary care at a Community Based Outpatient Clinic in the past year.” Participants will be invited to participate in one of two ways:

1. Use of a contract. For apps with broad criteria or without a need to interact with certain VA providers, OCC has paid for part of a recruitment contract administered by Human Factors Engineering (also within the Office of Health Informatics).

2. Use of VA staff/providers. For apps in which criteria dictate a specific patient-provider relationship or type of relationship with VA (e.g., care by a specialist who is willing to participate as well), Veterans will be recruited via the volunteer staff. In all cases, participants will be instructed that their VA benefits will not be affected by their participation or lack of participation.

Administration of the Instrument

1. How will you collect the information? (Check all that apply)

[X ] Web-based using survey or VA App Store Feedback tab

[X ] Telephone

[X ] In-person

[ ] Mail

[ ] Other, Explain

Each app has a Feedback form on its respective page in the VA App Store for participants to complete at the end of their testing. Participants will also be offered the chance to provide verbal comments via phone or in person. Some studies will also involve the use of a short (approximately 5-10 question) survey.

2. Will interviewers or facilitators be used? [ ] Yes [ X ] No

We don’t expect interviews per se, but if participants want to share any experience via the phone, we will encourage and accommodate that feedback.

Please make sure that all instruments, instructions, and scripts are submitted with the request.

Instructions for completing Request for Approval under the “Generic Clearance for the Collection of Routine Customer Feedback”

TITLE OF INFORMATION COLLECTION: Provide the name of the collection that is the subject of the request. (e.g. Comment card for soliciting feedback on xxxx)

PURPOSE: Provide a brief description of the purpose of this collection and how it will be used. If this is part of a larger study or effort, please include this in your explanation.

DESCRIPTION OF RESPONDENTS: Provide a brief description of the targeted group or groups for this collection of information. These groups must have experience with the program.

TYPE OF COLLECTION: Check one box. If you are requesting approval of other instruments under the generic, you must complete a form for each instrument.

CERTIFICATION: Please read the certification carefully. If you incorrectly certify, the collection will be returned as improperly submitted or it will be disapproved.

Personally Identifiable Information: Provide answers to the questions. Note: Agencies should only collect PII to the extent necessary, and they should only retain PII for the period of time that is necessary to achieve a specific objective.

Gifts or Payments: If you answer yes to the question, please describe the incentive and provide a justification for the amount.

BURDEN HOURS:

Category of Respondents: Identify who you expect the respondents to be in terms of the following categories: (1) Individuals or Households;(2) Private Sector; (3) State, local, or tribal governments; or (4) Federal Government. Only one type of respondent can be selected per row.

No. of Respondents: Provide an estimate of the Number of respondents.

Participation Time: Provide an estimate of the amount of time required for a respondent to participate (e.g. fill out a survey or participate in a focus group)

Burden: Provide the Annual burden hours: Multiply the Number of responses and the participation time and divide by 60.

FEDERAL COST: Provide an estimate of the annual cost to the Federal government.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents. Please provide a description of how you plan to identify your potential group of respondents and how you will select them. If the answer is yes, to the first question, you may provide the sampling plan in an attachment.

Administration of the Instrument: Identify how the information will be collected. More than one box may be checked. Indicate whether there will be interviewers (e.g. for surveys) or facilitators (e.g., for focus groups) used.

Submit all instruments, instructions, and scripts are submitted with the request.

8