OMB HtE KS-MO_36M Supp Statement-Part B

Contract No.: HHS 233-01-0012

Contract Amount: $23.78 million

SUPPORTING STATEMENT

FOR OMB CLEARANCE

PART B

DHHS/ACF/ASPE/DOL

ENHANCED SERVICES FOR THE HARD-TO-EMPLOY (HtE)

DEMONSTRATION AND EVALUATION PROJECT

KANSAS-MISSOURI 36-MONTH DATA COLLECTION INSTRUMENTS

June 21, 2007

Prepared for:
U.S. Department of Health and Human Services
	Office of Assistant Secretary for Planning
Administration for Children and Families	and Evaluation
370 L’Enfant Promenade, SW	200 Independence Avenue, SW
Washington, DC 20447	Washington, DC 20201
Phone: 202-401-5070	Phone: 202-260-0384
Project Officer: Girley A. Wright	Project Officer: Flavio Menasce
U.S. Department of Labor	Prepared by:
Employment and Training Administration	MDRC
200 Constitution Avenue, NW	16 East 34th Street, 19th Floor
Washington, DC 20210	New York, NY 10016
Phone: 202-693-3654	Phone: 212-532-3200
Project Officer: Roxie Nicholson	Project Directors: David Butler/Barbara Goldman

TABLE OF CONTENTS

B. COLLECTION OF INFORMATION USING STATISTICAL METHODS

B1. Sampling 1

B1.1 Minimum Detectable Effects for Key Outcomes in Effect Size Units 1

B2. Procedures for Collection of Information 2

B2.1 Procedures for the Administration of the Survey and Direct Child Assessments 2

B3. Maximizing Response Rates 3

B4. Pre-testing 4

B5. Consultants on Statistical Aspects of the Design 4

LIST OF APPENDICES

A.1: Parent Consent Form

A.2: Consent Form for Videotaping Direct Child Assessments

B: 36-Month Follow-Up Parent-Reported Survey

C: 36-Month Follow-Up Protocol for Conducting Direct Child Assessments

D.1: Federal Register Published 60-Day Notice

D.2: Federal Register Draft 30-Day Notice

E: Statute/Regulation Authorizing Evaluation and Data Collection: Social Security Act, Section 1110

F: References

B. COLLECTION OF INFORMATION USING STATISTICAL METHODS

B1. Sampling

The follow-up sample will consist of 488 parents and 488 children. The follow-up sample estimate is based on the assumption that 80 percent of the research sample will be successfully interviewed.

The evaluation literature often discusses the appropriateness of the sample size for a study by focusing on the smallest program impacts that are likely to be detected with a specified level of confidence, assuming a sample of a given size and characteristics. These are usually called the program’s “minimum detectable effects” (MDEs). Analysis of MDEs is also referred to as “power analysis,” as it estimates the study's power to measure the effects it was designed to find.

As a guide to determining appropriate sample sizes, the following table projects the statistical power of sampling plans for a two-group impact estimate using results for parents approximately 28 months after random assignment and children aged three years old in the Early Head Start Research and Evaluation Project.¹ This table reports MDEs, the minimum program impact that a sample has an acceptable chance of detecting (with a .10 significance level, and .80 power) for two parental outcomes and two child outcomes, and for samples of different sizes.

B1.1. Minimum Detectable Effects for Key Outcomes²

Size of Program and Control Group	Percent of Parents Ever Employed	Percent of Parents with Incomes Above the Poverty Level	Children’s Early Literacy Skills3	Children’s Aggressive Behaviors4
75/75	11.9	20.1	6.3	2.6
100/100	10.3	17.4	5.5	2.3
150/150	8.4	14.2	4.5	1.9
200/200	7.3	12.3	3.9	1.6
300/300	6.0	10.1	3.2	1.3

SOURCE: Unpublished calculations using data from the Early Head Start Research and Evaluation Project (U.S. Department of Health and Human Services, 2002).

B2. Procedures for Collection of Information

The 36-month follow-up survey data will be collected primarily by CATI, although a mixture of telephone and in-person outreach and interviewing strategies will be used to maximize response rates. The direct child assessments will be conducted in the home by interviewers. MDRC will work with HumRRO to develop strategies that ensure 80 percent response rates for both of these instruments.

All completed interviews and direct child assessments will be reviewed to ensure all applicable fields are correctly completed and that all relevant interviewer notes are included in the data set. Any open ended and “other, please specify” items will be coded based on codes developed by HumRRO and approved by MDRC. Preliminary data files will be created and shared – with documentation – with MDRC on an agreed-upon schedule.

B2.1 Procedures for the Administration of the Survey and Direct Child Assessments

Interviewer selection. MDRC will work with HumRRO to ensure that the interviewers administering these follow-up surveys are professional interviewers, many of whom have worked on social science research projects. Prefer­ence will be given to those who are multilingual, depending on the languages spoken by the research sample. Familiarity with the special requirements of interviewing low-income populations will be desirable. New personnel will be trained along with the seasoned interviewers.

Interviewer training. MDRC will work with HumRRO to ensure sufficient interviewer training. In the past, this has typically involved two training sessions, each of which lasts about three to four days. Personnel who are new to interviewing were trained in general interviewing techniques and approaches in the first day of the session. Interviewers will then be trained on the administration of the direct child assessments for the remaining three days of the training session. In the next 3-day training session, interviewers will be trained on the administration of the survey and will receive a refresher on the administration of the direct child assessments. Some pre-training exercises are likely to be required, and the actual training will include an item-by-item or task-by-task review of the survey instrument and direct child assessments, practice interviews and administrations, and critiques of those interviews and assessments.

Training will take place close to the time when the first cohorts of research subjects reach the 36-month anniversary of their random assignment date.

All interviewers will sign a confidentiality pledge during training. They will be instructed on the importance of maintaining confidentiality and told that breaches of confidentiality will lead to dismissal.

MDRC will also work with HumRRO to monitor early interviews for each interviewer and periodically monitor interviews and administrations over the course of fielding the data collection instruments (e.g., listening in on telephone interviews, reviewing videotapes of administrations of direct child assessments). Feedback will be provided to the interviewers based on these monitoring efforts.

All interviewers will also undergo a certification process before administering any direct child assessments in the field to ensure that interviewers are qualified to work with children and that a high quality of data is collected (see Section A16.1a).

Conducting interviews and direct child assessments. In all cases, the interviewers will explain the purpose of the interview, and inform respondents that they will receive a small incentive for completing the survey and their children will receive a small gift for attempting to complete the direct child assessments. Each interviewer will be prepared to answer any questions about the study that sample members might have.

Interviewer Supervision. Interviewing staff will be supervised directly by staff from HumRRO.

B3. Maximizing Response Rates

The goal will be to achieve an 80 percent response rate for the parent survey and the direct child assessments. Procedures for obtaining the maximum degree of cooperation include:

• Conveying the purposes of the survey and direct child assessments to respondents so they will thoroughly understand the purposes of the survey and perceive that cooperating is worthwhile;

• Providing a toll-free number for respondents to use to ask questions about the survey, direct child assessments, and the survey firm’s staff;

• Training site staff to be encouraging and supportive, and to provide assistance to parents and children as needed;

• Hiring interviewers who have necessary skills for encouraging cooperation from parents and children;

• Training interviewers to maintain one-on-one personal rapport with parents and children; and

• Offering appropriate payments to parents and a small gift to children for attempting the assessments.

For the most part, direct child assessments will be conducted in person because the direct child assessments will be conducted in the home. However, the follow-up survey is designed to be administered in the home or by telephone. If the interview cannot be administered in person, or the parent chooses not to have the child participate in direct child assessments, the parent can be reached by telephone to complete the interview. Once contacted, they will be asked to complete the interview using their own phone or a cell phone provided by the representative of the survey firm.

Interviewers will also be trained to distinguish "soft" refusals from "hard" ones. Soft refusals often occur when the sample member has been reached at an inopportune time. In these cases, it is important to back off gracefully and to establish a convenient time to call or come back rather than to persist at the moment. Hard refusals do occur and must also be accepted gracefully by the interviewer.

These strategies were used at the 15-month follow-up as well. Thus far our response rate is fairly high, which bodes well for the data collection strategies being proposed for the 36-month follow-up.

B4. Pre-testing

Most of the questions proposed for this survey are either identical to questions used in prior MDRC evaluations or are similar, if not identical, to questions used in previous national surveys or major evaluations. The measures of the direct child assessments have also been drawn from prior research. Consequently, many of the items and measures have been thoroughly tested on larger samples.

The follow-up survey has already undergone a number of revisions, following critiques by internal staff, by project consultants, and by staff at HHS. Revisions were also made on the basis of our small-scale consultations. A similar set of reviews will be conducted with the protocols for the direct child assessments.

MDRC will also work closely with HumRRO’s senior staff to conduct formal pre-tests of the follow-up survey and protocol for the direct child assessments with a convenience sample recruited from an EHS program in Kansas or Missouri that is not participating in the HtE evaluation. These pre-tests will provide more definitive estimates about the length of each survey and direct child assessments – and their various components – as well as lead to improvements in questions, introduction scripts, wording and document formatting. Following each of the pre-tests, respondents will be debriefed about the clarity of the questions and any potential problems with the instruments. Interviewers will also be debriefed concerning any problems they encountered in survey and direct child assessment administration – and they will recommend improvements. Translated versions of the survey and protocol for direct child assessments will be developed once English versions are finalized.

B5. Consultants on Statistical Aspects of the Design

There were no consultants on the statistical aspects of the design.

1 U.S. Department of Health and Human Services, 2002.

2 Results in Exhibit B1-1 reflect unadjusted E-C group differences. Minimum detectable effects are for two-tailed tests at 0.10 significance with 80 percent power. Accounting for differences in baseline characteristics across individuals will result in smaller MDEs.

3 Calculations are based on the Peabody Picture Vocabulary Test standard scores.

4 Calculations are based on aggressive behaviors assessed with the Child Behavior Checklist.