3_Generic Application FEMA App Beta Testing Process

Request for Approval under the “Generic Clearance for Improving Customer Experience: OMB Circular A-11, Section 280 Implementation”

(OMB Control Number: 1601-0029)

TITLE OF INFORMATION COLLECTION: FEMA MOBILE APP: BETA TESTING PROGRAM

PURPOSE OF COLLECTION:

What are you hoping to learn / improve? How do you plan to use what you learn? Are there artifacts (user personas, journey maps, digital roadmaps, summary of customer insights to inform service improvements, performance dashboards) the data from this collection will feed?

XXX

TYPE OF ACTIVITY: (Check one)

[ ] Customer Research (Interview, Focus Groups, Surveys)

[ ] Customer Feedback Survey

[ x ] Usability Testing of Products or Services

ACTIVITY DETAILS

1. If this is a survey, will the results of this survey be reported to Touchpoints as part of quarterly reporting obligations specified in OMB Circular A-11 Section 280?

[ ] Yes

[ ] No

[x ] Not a survey

2. How will you collect the information? (Check all that apply)

[ ] Web-based or other forms of Social Media

[ ] Telephone

[ ] In-person

[ ] Mail

[x ] Other, Explain

Beta testers will submit their feedback via Apple and TestFlight (iOS devices) and Google Play (Android devices.

3. Who will you collect the information from?

Explain who will be interviewed and why the group is appropriate for the Federal program / service to connect with. Please provide a description of how you plan to identify your potential group of respondents and if only a sample will be solicited for feedback, how you will select them(e.g., anyone who provided an email address to a call center rep, a representative sample of Veterans who received outpatient services in May 2019, do you have a list of customers to reach out to (e.g., a CRM database that has the contact information, intercept interviews at a particular field office?)

The FEMA app team will market the beta testing initiative to:

• People in Online Professional Groups on LinkedIn. In particular, technologists and quality assurance professionals.

• English and Spanish language visitors on FEMA.gov

• FEMA social media users on Facebook and Instagram.

• FEMA ODIC stakeholders and nonprofits.

• FEMA app power users. Power users are people who have emailed us at least once.

• Disaster Prep bloggers and influencers.

Beta testers will need to meet the minimum device operating system requirements in order to be eligible for testing the app. Beta testers will be selected to join on a first come, first serve basis and up to approximately 300 testers.

4. How will you ask a respondent to provide this information?

(e.g., after an application is submitted online, the final screen will present the opportunity to provide feedback by presenting a link to a feedback form / an actual feedback form)

iOS Beta testers will be asked to:

1. Open the TestFlight app on their device.

2. From the Apps list, tap the FEMA app.

3. Tap Send Beta Feedback.

4. In the share dialog, tap Include Screenshot to choose a screenshot. If the tester doesn’t want to send an attachment, tap Don't Include Screenshot.

5. Beta tester will add their comments (up to 4,000 characters), and optionally enter their email address if they were invited with a public link.

6. Tap Submit.

Android Beta testers will be asked to submit their feedback directly to fema-app@fema.dhs.gov.

5. What will the activity look like?

Describe the information collection activity – e.g. what happens when a person agrees to participate? Will facilitators or interviewers be used? What’s the format of the interview/focus group? If a survey, describe the overall survey layout/length/other details? If User Testing, what actions will you observe / how will you have respondents interact with a product you need feedback on?

Beta testers will be given an objective and a set of instructions with every beta app update. The instructions may include specific areas of the app to test and how they should approach the testing process.

6. Please provide your question list.

Paste here the questions or prompts presented to participants in your activity. If you have an interview / facilitator guide, that can be attached to the submission and referenced here.

Please make sure that all instruments, instructions, and scripts are submitted with the request.

Beta test questions are dependent on the new features, enhancements and/or bugs in the app update. Here is a set of instructions and questions that we used as part of a recent user acceptance test:

NEW “Allow Push Notifications” feature during Onboarding & confirm selection in Settings

Test Instructions:

1. Delete the beta test app by holding down on the app icon and selecting “Remove app” or “Uninstall.”

2. Go to Google Play or the Test Flight app.

3. Confirm the app version is the most recent version and then reinstall the test app.

4. Launch the updated beta test app.

5. After the splash screen loads (FEMA seal on white background), you should see a pop-up: “FEMA” Would Like to Send You Notifications.

6. Select Allow Notifications

7. Proceed through Onboarding pages, filling out your Name and Location, until you arrive at the Home screen.

8. Go to Settings (gear icon) > Notifications

Expected Results

• During Step 5, you should have seen a pop-up for push notifications.

  • Did you see the pop-up? If not, please describe what you experienced.

• During Step 6, after selecting “Allow” in the pop-up you should be able to proceed through Onboarding without any issues.

  • Were you able to select “Allow” and proceed through Onboarding? If not, please describe what you experienced.

• During Step 7, once you arrive on the Home screen, confirm the Name and Location that you entered during Onboarding is reflected on the Home screen.

  • Did you see your name and location on the Home screen, above the main graphic?

• During Step 8, on the Settings page, the Allow Notification option should be turned ON and Silent Notifications set to Never.

  • Did you achieve the expected result? If not, please describe your experience.

7. When will the activity happen?

Describe the time frame or number of events that will occur (e.g., We will conduct focus groups on May 13,14,15, We plan to conduct customer intercept interviews over the course of the Summer at the field offices identified in response to #2 based on scheduling logistics concluding by Sept. 10^th, or “This survey will remain on our website in alignment with the timing of the overall clearance.”)

Updated beta versions of the FEMA app will be made available as new features are developed and internally tested. The app team operates on an 8-10 week sprint cycle. In one year, there is time for about 5 sprints and each sprint will have at least one round of beta testing.

8. Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to participants?

[ ] Yes [ x ] No

If Yes, describe:

BURDEN HOURS

Category of Respondent	No. of Respondents	Participation Time	Burden Hours
Individuals or Households	300	30 minutes	150
Totals	300	30 minutes	150

CERTIFICATION:

I certify the following to be true:

1. The collections are voluntary;

2. The collections are low-burden for respondents (based on considerations of total burden hours or burden-hours per respondent) and are low-cost for both the respondents and the Federal Government;

3. The collections are non-controversial;

4. Any collection is targeted to the solicitation of opinions from respondents who have experience with the program or may have experience with the program in the near future;

5. Personally identifiable information (PII) is collected only to the extent necessary and is not retained;

6. Information gathered is intended to be used for general service improvement and program management purposes

7. The agency will follow the procedures specified in OMB Circular A-11 Section 280 for the required quarterly reporting to OMB of trust data and experience driver data from surveys.

8. Outside of the quarterly reporting mentioned in the bullet immediately above, if the agency intends to release journey maps, user personas, reports, or other data-related summaries stemming from this collection, the agency must include appropriate caveats around those summaries, noting that conclusions should not be generalized beyond the sample, considering the sample size and response rates. The agency must submit the data summary itself (e.g., the report) and the caveat language mentioned above to OMB before it releases them outside the agency. OMB will engage in a passback process with the agency.

Name and email address of person who developed this survey/focus group/interview:

Name: Olivia Delfert________________

Email address: olivia.delfert@fema.dhs.gov_______

All instruments used to collect information must include:

OMB Control No. 1601-0029

Expiration Date: 12/31/2026

HELP SHEET

(OMB Control Number: XXXX-XXXX)

TITLE OF INFORMATION COLLECTION: Provide the name of the collection that is the subject of the request. (e.g. Comment card for soliciting feedback on xxxx)

PURPOSE: Provide a brief description of the purpose of this collection and how it will be used. If this is part of a larger study or effort, please include this in your explanation.

TYPE OF COLLECTION: Check one box. If you are requesting approval of other instruments under the generic, you must complete a form for each instrument.

CERTIFICATION: Please read the certification carefully. If you incorrectly certify, the collection will be returned as improperly submitted or it will be disapproved.

Personally Identifiable Information: Agencies should only collect PII to the extent necessary, and they should only retain PII for the period of time that is necessary to achieve a specific objective.

BURDEN HOURS:

Category of Respondents: Identify who you expect the respondents to be in terms of the following categories: (1) Individuals or Households;(2) Private Sector; (3) State, local, or tribal governments; or (4) Federal Government. Only one type of respondent can be selected per row.

No. of Respondents: Provide an estimate of the Number of respondents.

Participation Time: Provide an estimate of the amount of time required for a respondent to participate (e.g. fill out a survey or participate in a focus group)

Burden: Provide the Annual burden hours: Multiply the Number of responses and the participation time and divide by 60.

8