SSA - MIHOPE-K Pretesting GenIC

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Mother and Infant Home Visiting

Program Evaluation:

Kindergarten Follow-Up (MIHOPE-K)

Pre-testing of Evaluation Data Collection Activities

0970 - 0355

Supporting Statement

Part A

May 2021

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Nancy Geyelin Margie

Laura Nerenberg

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Pre-testing of Evaluation Data Collection Activities (0970-0355).

• Description of Request:

The goal of this generic information collection (GenIC) request is to develop and test remote administration of direct assessments of children, direct assessments of caregivers, and videotaped caregiver-child interactions. Intended use of the resulting data is to evaluate and improve the quality of the data gathered through ACF/OPRE’s Mother and Infant Home Visiting Program Evaluation kindergarten follow-up (MIHOPE-K, approved under OMB #0970-0402). The MIHOPE-K data collection began in 2018 and its in-home assessment elements (direct assessments of children, direct assessments of caregivers, and videotaped caregiver-child interactions) need to be adapted for remote administration so that data collection can safely continue amid the COVID-19 pandemic. We do not intend for this information to be used as the principal basis for public policy decisions.

• Time Sensitivity:

These pretesting and piloting efforts must conclude in time to submit a nonsubstantive change incorporating what is learned in the pilot into the full MIHOPE-K package (OMB #0970-0402), program alterations to instruments, and resume kindergarten follow-up data collection at the start of January 2022. The OMB # for the MIHOPE-K data collection expires in November 2021 and we need to finalize these changes prior to submitting the full request to continue data collection beyond November. MIHOPE-K data collection has already been postponed one year due to the COVID-19 pandemic and cannot begin later than January 2022 due to the critical period in children’s development that this round of data collection aims to capture. We request a response as soon as possible.

A1. Necessity for Collection

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval to conduct pretesting for remote administration of a long-term follow-up of the families participating in the Mother and Infant Home Visiting Program Evaluation (MIHOPE). Due to the COVID-19 pandemic, the MIHOPE study may need to remotely administer elements of its kindergarten follow-up data collection that had previously been conducted in families’ homes. These three data collection elements are direct child assessments, direct caregiver assessments, and videotaped caregiver-child interactions. Remote administration of direct child assessments, direct caregiver assessments, and videotaped caregiver-child interactions has never been conducted with a sample of this size. Given the lack of precedent, successfully conducting remote data collection requires pretesting to ensure the new mode of data collection goes smoothly and results in high quality data.

Study Background

MIHOPE is providing information about the effectiveness of the Maternal, Infant, and Early Childhood Home Visiting program (MIECHV) in its first few years of operation and providing information to help states and others develop and strengthen home visiting programs in the future. The goals of the study are:

1. to understand the effects of home visiting programs on parent and child outcomes, both overall and for key subgroups of families,

2. to understand how home visiting programs were implemented and how implementation varied across programs, and

3. to understand which features of local home visiting programs are associated with larger or smaller program impacts.

Data collection has been ongoing since 2012 (see OMB #0970-0402¹ for previously approved MIHOPE data collection activities). Most recently, in November 2018, OMB approved the collection of follow-up data when the children in the study were in kindergarten. The instruments to be pretested through this generic information collection (GenIC) request are specific to this kindergarten follow-up data collection.

Legal or Administrative Requirements that Necessitate the Collection

There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

This proposed information collection meets the primary goals of ACF’s generic clearance for pre-testing (0970-0355): to develop and test information collection instruments and procedures. The purpose of the information collection is to research how the data collection elements previously conducted in families’ homes need to be modified to support virtual/remote administration. The proposed pretest will consist of remote administration of the data collection elements previously conducted in families’ homes followed by debrief interviews of families, assessors, and support staff. During the debrief interviews, families will report on the usability of the equipment, experience with the equipment drop off and pickup, and any logistical issues they experienced. During their debrief interviews, assessors and support staff will report on their experiences administering the data collection elements remotely, child and caregiver engagement during the assessment, and any challenges they faced administering the assessments or families faced in completing them. Debrief interviews with assessors and support staff are not included in burden estimates as these staff are study team members. The study team will use the data collected through the proposed pretesting to determine what modifications to data collection instruments are needed to improve child and caregiver engagement, understanding of the instructions, and fidelity to the instruments. The final revised instruments will be incorporated into the MIHOPE kindergarten follow-up data collection, approved under OMB #0970-0402.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions or Tests

The research question the MIHOPE-K pretest will try to answer is:

1. What modifications to the MIHOPE-K elements that had been conducted in families’ homes are necessary to support virtual/remote administration?

Study Design

The design of this two-phase pretest consists of (1) implementing the remote protocol for the MIHOPE-K elements that had been conducted in families’ homes, (2) gathering feedback from caregivers, assessors, and field staff, (3) revising the instrument and procedures based on that feedback, and (4) repeating steps one through three a second time with the revised remote protocol.

For an explanation of why the study design is appropriate for the intended use of the resulting information, see GenIC Supporting Statement B.1. Appropriateness of Study Design and Methods for Planned Uses.

For information about whether the results of this pretest are designed to be generalizable to a given subpopulation, see GenIC Supporting Statement B.1. Generalizablility of Results.

For information about the limitations of this study design for its purpose and intended use, see GenIC Supporting Statement B.1 Appropriateness of Study Design and Methods for Planned Uses and GenIC Supporting Statement B.2 Target Population.

Data Collection Activity	Instruments	Respondent, Content, Purpose of Collection	Mode and Duration
Pretesting of virtual administration of MIHOPE-K elements that had been conducted in families’ homes	Instrument 1: Recruitment Screener Instrument 2: Virtual visit (Direct assessments of children, Direct assessments of caregivers, Videotaped caregiver-child interactions, Debrief)	Respondents: The sample of 55 families participating in the proposed pretest includes both English and Spanish speakers, child ages and characteristics comparable to the focal children in the MIHOPE sample, and caregivers and children with a variety of levels of tech savviness and home distractions. We anticipate 70 participants will need to be screened in order to reach this sample of 55. Content: Direct child assessments such as Hearts & Flowers and the Woodcock Johnson IV Picture Vocabulary and Oral Comprehension subtests will be administered to assess the child’s receptive language skills, early numeracy, working memory, inhibitory control, and cognitive flexibility. Assessors will also observe and rate parental warmth and the child’s emotions, attention, and behavior. Direct caregiver assessments (i.e. the Digit Span) will be administered to assess maternal self-regulation.  Observations of caregiver-child interactions will be conducted using a videotaped interaction. Children’s behaviors towards the caregiver will be gathered in the context of caregiver-child interaction, including engagement with the caregiver and negativity toward the caregiver. Caregivers’ parenting behaviors, including supportiveness and respect for child’s autonomy, will also be assessed, as well as features of the caregiver-child dyad (e.g., affective mutuality). Purpose: The purpose of this collection is to develop and test virtual administration of data collection instruments.	Mode: Virtual Duration: 6.5 minutes for the recruitment screener and up to 2 hours for the virtual visit

Other Data Sources and Uses of Information

This GenIC will inform data collection for the MIHOPE Kindergarten Follow-up study that had previously been designed to occur within a respondent’s home. The results of these pre-testing activities will be incorporated into the materials approved under OMB #0970-0402. The study team will compare some aspects of the data (e.g., rate of missingness) to data from prior rounds of MIHOPE-K data collection.

A3. Use of Information Technology to Reduce Burden

This study will use information technology, when possible, to minimize respondent burden and to collect data efficiently. The proposed collection of information involves the use of technology and electronic submission of responses so that data collection can occur remotely during the ongoing COVID-19 pandemic. Direct child assessments, direct caregiver assessments, and videotaped caregiver-child interactions will occur through electronic devices connecting participating families with a remote assessor virtually and recording responses electronically. The use of this technology will ensure that assessors can guide children and caregivers through the assessments efficiently, thereby reducing respondent burden. 

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

This research does not duplicate any other data collection design work being conducted. The purpose of this GenIC is to better inform and improve the quality of virtual/remote data collection conducted as a part of the MIHOPE-K study. We are not aware of any data collection efforts that have conducted virtual/remote assessments of the scale envisioned here.

A5. Impact on Small Businesses

No small businesses will be involved with this information collection.

A6. Consequences of Less Frequent Collection

This is a one-time data collection.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for pre-testing activities. This notice was published on October 20, 2017; Volume 82, Number 202, page 48820, and provided a sixty-day period for public comment. During the notice and comment period, no substantive comments were received.

Consultation with Experts Outside of the Study

The following experts have provided guidance on the MIHOPE-K remote assessments:

• Data Recognition Corporation (DRC) regarding allowable/recommended modifications to the preLAS Simon Says subtest

• Riverside Insights regarding allowable/recommended modifications to the Woodcock-Johnson® IV Picture Vocabulary and Woodcock Johnson® III Applied Problems.

A9. Tokens of Appreciation

We propose a token of appreciation of $75 and a small book or toy for the child for completing the pretest, which is estimated to take up to 2 hours and include a post-visit debrief that is estimated to take up to 15 minutes. In comparison, the standard administration of the in-home assessment is estimated to take up to about 1 hour and 40 minutes and the OMB approved token of appreciation for completion was $50 and a small book or toy for the child. The token of appreciation is higher for pretesting because of the debrief that follows the instrument, and we do not anticipate incorporating changes to the token of appreciation into the full request. Combined with other administrative efforts intended to communicate the study’s relevance and salience to participants, tokens of appreciation are an important means for improving participant engagement throughout the pretest; securing an adequate response rate to answer research questions; and reducing differential attrition of specific subgroups of interest. The token of appreciation is intended to reduce nonresponse bias, differential attrition, and overall attrition to ensure that the pretest has a sufficiently representative sample to answer its key research question. A high response rate makes it more likely that pretest respondents are representative of the variation in the MIHOPE-K sample, which is crucial when determining whether the virtual protocol will be usable for the MIHOPE-K sample. Tokens of appreciation are also intended to reduce the number of contact attempts needed to complete cases, which would both reduce the costs of data collection and improve data comparability.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

The collection requests the names and contact information of participants. This PII will be used to contact caregivers and deliver equipment participants need to participate in the pretest. The child and caregiver assessments will be audio-recorded so the study team understands needed adjustments to the protocol and instruments. The caregiver-child interactions will be videotaped to determine whether the videos collected through a remote administration are codable and worth collecting full-scale from the MIHOPE sample.

Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier. Personally identifiable information will be removed from study files, which will contain a linking identification number that can be used to match records from one data file to another.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.

Assurances of privacy are included in the introductory script of the virtual visit (see Instrument 2) and the caregiver consent form (see Appendix A or Instrument 2). When participants begin the virtual visit, they will be reminded of the study goals, time required, and the nature of the virtual visit activities. Caregivers will be assured that their responses will be shared only with researchers, will be reported only in the aggregate as part of statistical analyses, and that participating in the study is completely their choice. They will also be told that all data collection activities are voluntary, and they can refuse to participate in any and all activities (including being video recorded) without penalty. Via the consent form, caregivers will be assured that the research team follows strict rules to keep their information private, videos will be stored in a secure location, and videos will only be used to help the study team learn about the quality of videos collected virtually, how we can improve our procedures, and for study staff training purposes.

The study team is committed to protecting the privacy of participants and maintaining the privacy of the data that are entrusted to us and is experienced in implementing stringent security procedures. Every MDRC and Mathematica employee, including field staff employed for data collection, is required to sign a pledge to assure participants of nondisclosure of private information. Field staff will also be trained in maintaining respondent privacy and data security. 

Data Security and Monitoring

MDRC shall use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. MDRC shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. MDRC shall: ensure that this standard is incorporated into the MDRC’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, MDRC shall minimize the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information that ensures secure storage and limits on access.   

Mathematica’s Sample Management System (SMS), which has been used for all previous rounds of MIHOPE data collection (see OMB #0970-0402), will be the central clearinghouse for all contact information on pretest participants. Personally identifiable information will be removed from study files, which will contain a linking identification number that can be used to match records from one data file to another. Finally, data will be available only to staff associated with the project through password protection and encryption keys. 

A11. Sensitive Information ²

The proposed data collection does not collect any sensitive information.

A12. Burden

Estimated Annualized Burden and Cost to Respondents

For collecting data from families, an hourly wage of $16.80 was assumed for caregivers, which is the median hourly wage for workers 25 years old or older with a high school diploma.³

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total/ Annual Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Instrument 1: Recruitment Screener	70	1	0.11	7.7	$16.80	$129.36
Instrument 2: Virtual Visit (Direct assessments of children, Direct assessments of caregivers, Videotaped caregiver-child interactions, Debrief)	110 (55 caregivers and 55 children)	1	1.92	211.2	$16.80	$3,548.16
Total	125			218.90		$3,677.52

Note: The total number of respondents participating in the virtual visit (125) was calculated by taking the sum of the number of virtual visit respondents, 110 (55 caregivers and 55 children), and the number of additional respondents who will need to be screened, 15, to achieve a sample size of 55 families.

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

Cost Category	Estimated Costs
Instrument Development and OMB Clearance	$30,000
Field Work	$225,000
Data Analysis	$35,000
Total/Annual costs over the request period	$290,000

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella generic clearance for pre-testing (0970-0355).

A16. Timeline

We propose to field the pretest approximately one month after OMB approval. The fielding of the pretest is expected to take approximately two months.

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

• Instrument 1_MIHOPE-K_Pilot_Recruitment Script

• Instrument 2_MIHOPE-K_Pilot_Virtual Visit Specifications

• Appendix A_MIHOPE-K_Pilot_Caregiver Contact Materials

1 https://www.reginfo.gov/public/do/PRAOMBHistory?ombControlNumber=0970-0402

2 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

3 https://data.bls.gov/PDQWeb/le

10