Home Visiting Evidence of Effectiveness – Website Information Architecture Testing

Home Visiting Evidence of Effectiveness – Website Information Architecture Testing instrument, instructions, and scripts

Document Overview

This document describes our plan for testing the Home Visiting Evidence of Effectiveness (HomVEE) website’s Information Architecture to discover issues impacting the usability of the site.

Participants

The test participants will include members of the website’s intended audience, recruited from two primary listservs: (1) subscribers to the HomVEE email distribution list, and (2) subscribers to OPRE’s newsletter. These individuals include researchers and experts in the field of child welfare and related fields, home visiting implementation agencies and providers, and state policy makers and federal staff.

Procedure

We will conduct three types of tests which will be packaged together as part of a single, 20-minute, self-directed online activity for which participants will receive a link.

Tree testing will be used to test HomVEE’s navigation and menu structure. Participants will be shown a hierarchical list of the site’s main sections and asked to find specific types of information (see Exhibit 1). To minimize order bias, the order of options will be randomized and different testers will be given different versions of the navigation structure and labels to enable comparison testing.

E


xhibit 1. Examples of tree testing interface

Card sorting tests how site users would organize and label site content. Results from card sorting tests are used to improve a site’s information architecture and terminology. Testing will focus on the structure and terms used on the model report and model search pages. Participants will be presented with an unsorted list of model content items and model search terms and asked to sort them into groups and then label the groups (see Exhibit 2). The order of items is randomized to minimize order bias.

Exhibit 2. Example of card sorting interface

First click tests are used to test web page designs to ensure important page elements are easy to locate and interpret. Participants will be given a task and shown a clickable image of a web page and asked what they would click on first to complete the task (see Exhibit 3). This part of the test will focus on the HomVEE home and model search pages.

Exhibit 3. Example of first click testing interface

Announcement and solicitation of testers

Testers will be invited to participate through an announcement that will be incorporated in materials sent to two potential audiences: (1) an “eblast” email announcement to current subscribers to the HomVEE email distribution list, and (2) the announcement being included in an upcoming issue of OPRE’s newsletter. The invitation will briefly explain the purpose and the reasons for the test and a link to complete the test online. Testers complete the test independently.

Text of announcement:

Help inform HomVEE’s website redesign! Over the next year, ACF will be redesigning the HomVEE website with a new look and new features. Use the “What’s New!” section of the home page to stay informed about changes. As part of a site redesign, we are seeking users familiar with HomVEE or the early childhood home visiting field and willing to participate in a brief activity to help us uncover challenges with navigating the website. To participate in this 20-minute, self-directed online activity, click this link.

PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN: Through this information collection, ACF is gathering information to inform the re-design of a public website containing information about evidence-based home visiting programs. The purpose of this information collection is to inform the website’s content and design. Public reporting burden for this collection of information is estimated to average 20 minutes per respondent, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. This is a voluntary collection of information. agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB # is 0970-0401 and the expiration date is 05/31/2027. If you have any comments on this collection of information, please contact your test facilitator.

Test instrument (begins after users click link in announcement)

Introduction

HomVEE is a comprehensive review of the evidence base for early childhood home visiting models for families with pregnant people and children from birth to kindergarten entry. The review is conducted by Mathematica and sponsored by the Office of Planning, Research, and Evaluation (OPRE) of the Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services (HHS), in partnership with the Health Resources and Services Administration (HRSA). For more information, visit the project’s website: https://homvee.acf.hhs.gov.

Over the next year, ACF will be redesigning the HomVEE website with a new look and new features. As part of a site redesign, we are seeking users familiar with HomVEE or the early childhood home visiting field and willing to participate in a brief activity to help us uncover challenges with navigating the website.

Instructions

Your participation is completely voluntary, and we are grateful for your help. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0401 and the expiration date is 05/31/2027.

This test has 4 parts:

1. Three brief, pre-test questions.

2. Next, to test the site’s navigation and menu structure, we will show you a hierarchical list of the site’s main sections and ask you to find specific types of information.

3. Then, to help us better organize and label HomVEE content, you will be presented with an unsorted list of items and asked to sort them into groups and label your groups.

4. Finally, to test page designs, we give you a task, present you with an image of a website page, and ask you to click on the element on the page that will enable you to complete the task.

All parts of this test should take approximately 20 minutes to complete. Please remember, this is not a test of your ability, there are no right or wrong answers.

That's it, let's get started!

Pre-Test Questions

1. Which describes you best. Select all that apply.

   • I am a researcher.

   • I am a policy maker.

   • I design and/or implement early childhood home visiting models.

   • I am a home visiting service provider.

   • None of these labels applies to me.

2. How often do you use websites to find research?

   • Never

   • Rarely

   • Sometimes

   • Often

3. Select other evidence review websites you have visited. Select all that apply.

• The What Works Clearinghouse (WWC)

• The Clearinghouse for Labor Evaluation and Research (CLEAR)

• The Pathways to Work Evidence Clearinghouse

• The Employment Strategies for Low-Income Adults Evidence Review (ESER)

• The Teen Pregnancy Prevention Clearinghouse (TPPER)

• The Prevention Services Clearinghouse

Example Tasks

Note: The specific tasks tested with each tester will be tailored based on the status of site redesign, but will include a subset of the types of sample tasks below.

Tree test tasks:

1. Where would you find information on how the HomVEE review is conducted?

2. Where would you find models eligible for Maternal, Infant, and Early Childhood Home Visiting funding?

3. Where would you find the criteria HomVEE uses for rating the quality of research?

4. Where would you find the criteria HomVEE uses for determining if a home visiting model is evidence based?

5. Where would you find a list of home visiting models that are evidence based according to the Health & Human Services criteria?

6. Where would you find out about recent updates to the HomVEE website?

7. Where would you find information on Tribal Home Visiting?

8. Where would you find information about the intended population of the Child First program?

9. Where would you find a summary of HomVEE’s review of findings from a study authored by Barlow (as first author) in 2015?

10. Where would you find information on implementing the Family Connects program?

11. Where would you find research summaries about early childhood home visiting’s impacts on maternal health?

12. Where would you find the age range of children served by the Family Spirit® program?

Card sorting tasks:

1. Present the tester with a randomized list of model search filters and ask them to group the filters and label their groups.

2. Present the tester with a randomized list of model report sections and ask them to group the sections and label their groups.

First click testing tasks:

Present user with a clickable image of the HomVEE home page and ask them to:

1. Find information on HomVEE’s purpose, directives, and key staff.

2. Find models that were evaluated with tribal populations.

3. Find HomVEE’s definition of an evidence based model.

4. Find individual studies reviewed by HomVEE.

Present user with a clickable image of the HomVEE model search page and ask them to:

1. Find models with Spanish language support.

2. Find models eligible for MIECHV funding.

3. Find models with favorable impacts on child health.

4. Clear all applied search filters.

Testing metrics

This section describes the usability metrics our team will use to assess the test outcomes.

• Time to complete (all test types). How long does it take users to complete a task?

• Success rate (tree and first click testing). Can testers complete tasks? Did testers select the expected location? What were the most common unexpected selections?

• Efficiency (tree testing). How often do testers use the most direct route to the resource?

• Option comparisons (tree testing). Which organization/labeling options scored highest on efficiency and success rate metrics?

• Common categorization and labeling (card sorting): Identify common patterns with how items are grouped and labeled.

4