H2E KS-MO 36M_Responses to OMB Questions_10-26-07 (2)

to: U.S. Department of Health and Human Services

from: JoAnn Hsueh, Pamela Morris, and David Butler

date: October 26, 2007

subject: Responses to Comments on OMB Submission for 36-Month Data Collection Instruments in Kansas and Missouri Sites of the Hard-to-Employ Project

We appreciate the thoughtfulness of the comments on the submission requesting approval of the 36-month data collection instruments being proposed for use in the Kansas and Missouri sites of the Enhanced Services for the Hard-to-Employ (HtE) Demonstration and Evaluation Project. In the following pages, we have made an effort to address the issues and provide clarification to the questions that were raised. In order to ensure that each of the points has been addressed, we have separated out and responded to each concern individually.

Question/Comment #1

Will individual-level data be available from administrative records on parental employment?

Response

Yes, individual-level data will be available from administrative records on parental employment.

Question/Comment #2

Where will administrative data be “supplemented” and what is the nature of this supplementation? Is the administrative data not reliable? Some employment off-the-books or informal employment, which could occur in low-income populations, may not be reflected in administrative data sources.

Response

You are correct about the limitations of administrative data for capturing employment and earnings. However, the advantage of administrative records is that they are available for all study participants and will provide the most accurate record of reported earnings and employment for the full study sample. Self-reported survey data, in contrast, is only available for a sub-sample of the study population and is subject to the limitations of response rates. However, the advantage of survey data is that it will capture at least some off-the-books and informal employment and sources of income. Moreover, the survey includes other key information about work and employment status that are not readily available from administrative data such as wages, benefits, work hours, work schedules, occupational complexity, and job mobility. The impact study will use both of these complementary sources to evaluate the effects of the program.

Question/Comment #3

Will the cost-benefit analysis conform with principles in OMB circular A4 (attached)? To what extent will the analytical methods diverge from A4? Precisely what costs and what benefits will be included in the calculation, how will the benefits (particularly any health benefits) be calculated, and will the CBA be undertaken from the point of view of society as a whole or only from the point of view of one (group of) stakeholders?

Response

Although we originally proposed a cost-benefit analysis as part of the HtE Kansas and Missouri sites, this analysis is no longer planned.

Question/Comment #4

Information about pre-test and pilot tests is included in the OMB supporting statement. Are the pre-test and/or pilot tests distinct efforts? What is the design of the pre-test and/or pilot test? Have this burden for the pre-test and/or pilot tests been included in the statement?  How many people will be involved in the pre-test and/or pilot test? Are any of them actual study participants? Will separate OMB submissions be submitted for approval of the pre-test and/or pilot test?

Response

We apologize for the confusion regarding the reference to the pre-test and pilot tests in the OMB supporting statement. The terms pre-test and pilot tests represent a single effort to further streamline these data collection components and ensure that all components of the survey instrument and protocol for the direct child assessments will be implemented as intended in the full-scale follow-up effort. Drafts of the survey will be tested prior to fielding the actual instruments with up to nine parents and the draft of the protocol for the administration of the direct child assessments will be tested with up to nine children. The pre-test sample will be selected to be similar in demographic characteristics to the survey sample. None of the pre-test respondents will be part of the actual study sample. The design of the pretest is intended to reflect the realistic conditions that are expected during the actual full-scale fielding of the data collection instruments. By doing so, the research team aims to mimic those conditions that are likely to pose difficulties during fielding. As such, the survey will be administered to pre-test respondents in person and/or over the phone. The direct child assessments will be conducted in person. Because the sample for the pre-test will only include nine study participants, our understanding is that this effort does not require a separate OMB review and approval process, and these hours are not included in our burden estimates.

Question/Comment #5

The OMB submission states that the research team will be cutting the questionnaire down if administration goes over 45 minutes.  Have you identified the information that would be dropped?  How might this impact the results?

Response

Should the survey instrument take longer than 45 minutes to administer, the research team has identified several items in the following areas of inquiry as possible candidates for streamlining and cutting the survey: parental psychological well-being, stress, parenting style, family routines, father involvement, parental engagement, child care, and services receipt. As a result, the research team will likely propose measuring these constructs with fewer items in the final version of the survey instrument. While using a larger number of items increases the reliability of the information gathered, we believe that in some cases, fewer measures will also provide adequate reliability for tapping the constructs of interest and we do not expect these cuts to the survey to have a substantial impact on the quality and the reliability of the data being collected. Notably, we are not intending to cut any key constructs from the survey to meet length requirements, only components of those constructs.

Question/Comment #6

How will you assure the accuracy of off-the-books income?  Wouldn't participants tend to withhold it because it might lead to loss of EHS services, other services, or even tax auditing, etc.

Response

We agree that participants may be hesitant to report off-the-books income. The accuracy of the information being collected will largely be dependent upon the interviewers’ abilities to develop rapport with the respondents, a strategy which has been used successfully in ethnographic studies collecting information about low-income mothers’ off-the-books employment and other informal sources of income (e.g., Edin & Lein, 1997). Furthermore, even though some study participants may withhold information about informal sources of income, we do not expect any bias to differ across the program and control groups and, thus any impacts of the intervention will likely not be biased.

Question/Comment #7

Why wouldn't you use the same data source to collect information on program participation for experimental and control groups?

Response

The survey will be used to collect information about program participation for respondents in both the program and control groups.

 

Question/Comment #8

Why does a 3-6 year old need a $10 toy?  We think $5 is sufficient for this age group.  In addition, with children this age, the incentive for participation comes form the rapport that the interviewer builds with the parent and child.  As with previous studies (e.g. Head Start Impact Study), would ACF be amenable to providing a non-monetary incentive not to exceed $5 for young children?

Response

We do agree that the incentive for the participation in the direct child assessments for children of this age comes from the rapport that the interviewer builds with the parent and the child. If OMB would prefer a smaller incentive, we are willing to offer it.

Question/Comment #9

What were the previous response rates for the Urban Change Project conducted in 2001? In that study, ACF provided $20 for a 90 minute interview. Why is ACF proposing to provide $35 for a 45 minute interview this time?

Response

 

In the Urban Change Project, response rates were relatively high, at about 80%. However, these families were living in very high poverty neighborhoods. The sample in the HtE sites in Kansas and Missouri is somewhat more advantaged than that in the Urban Change study. As such, a financial incentive of $20 may not yield the same response rates for the population in the current study.

In HtE, we have received OMB approval to give financial incentives of up to $25 for participation in the 15-month follow-up survey, which takes 45 minutes to complete in the Kansas and Missouri sites (OMB Control Number: 0970-0276). For the 36-month follow-up effort, we propose a slightly higher financial incentive ($35) in order to achieve the desired response rates due to the relatively long follow-up period here and difficulties encouraging respondents to participate in the survey as a result.

Question/Comment #10

What will you be doing to account for multiple comparisons - when using so many different configurations to approximate an outcome, something is bound to show up positive.  How will you go back to test whether the effect is likely to be 'real'?

Response

Since we will analyze multiple outcomes, we will explore the possibility of adjusting estimates to account for this fact, for example, by using a Bonferroni correction (Darlington, 1990) or other omnibus test (such as those discussed in Cooper & Hedges, 1994). We will also be examining the pattern of impacts across multiple outcomes to determine whether hypotheses regarding the expected impacts of the intervention are supported across multiple outcomes.

Question/Comment #11

If ACF has the statutory authority to provide assurances of confidentiality, please cite the statute on this consent form for videotaping. If not, the word “confidential” should not be used on this form. Also, in general, while the language in the supporting statement and agreement to participate in the study is “technically” correct, I’m not sure it is written clearly enough to inform the respondents that there are situations like FOIA where ACF might need to disclose respondent information.

Response

Thank you for raising this point. It is our understanding that ACF does not have the statutory authority to provide assurances of confidentiality. We have revised the language in the consent form accordingly to reflect situations in which respondent information might be disclosed. The language in the consent form for videotaping has been amended to read as follows (a revised copy of the consent form accompanies this memo as Appendix A):

“Privacy

The videotaped information, and all the other information collected, is confidential. In other words, no one other than the researchers will see the videotaped information that we are getting from you today. All the videotapes will be kept in locked cabinets. The videotapes will not be edited or copied. All the videotapes will be destroyed upon completion of the study. Your child’s image, voice, and first name will be on the videotape, but no identifying information, such as full name and address, will be released to anyone, except as required by law. The research team will combine your information with information from all of the other participants to create group statistics. Although every effort will be made to keep research records private, there may be times when federal or state law requires the disclosure of such records, including personal information. This is very unlikely, but if disclosure is ever required, the research team will take all steps allowable by law to protect the privacy of personal information.

Exceptions to Privacy

There are some exceptions to our promise to keep this information private. If you or your child tells us that you or your child are in immediate danger, or that you or your child intend to harm yourself or someone else, we may need to inform the appropriate authorities according to state and local law.”

5