Response to OMB 10-4-07

IDENTIFYING PROMISING TANF DIVERSION PRACTICES STUDY

RESPONSES TO OMB QUESTIONS

October 4, 2007

In general, I think this study is fine but I am concerned about the study limitation pointed out in the supporting statement. It is unclear to me how you can really identify the most “promising” diversion strategies without looking at outcomes. It’s also not clear how you can implement a purposive sampling plan based on strategies states would most want to replicate without considering the effectiveness of the strategies themselves. Presumably, the strategies states will want to replicate most are those that are most effective. As such, it seems to me that it is this criterion--i.e., effectiveness—which should drive the purposive sampling (e.g. site visits to those states that have had the best outcomes), rather than the criterion that has been proposed (i.e. ease of replicability). Why would states want to implement strategies solely because they are easy to replicate? Even if they would want to do that, why should we encourage this?

The overall purpose of the Identifying Promising TANF Diversion Practices study is to identify and describe the practices that states and local offices have undertaken to divert clients from their TANF caseloads. We have used the term “promising practices” instead of “best practices” precisely because we do not expect to be able to measure the effectiveness of these practices. This study aims to identify strategies that are promising based on the strength of their logic models, their successful implementation of the planned activities, and analysis of available data on their outcomes.

Staff at Mathematica Policy Research, Inc. (MPR) will identify these promising practices through three study activities. First, the study’s literature review will identify diversion programs studied and found to be effective in other evaluations. Second, during telephone interviews with state and local staff, MPR interviewers will ask respondents about the goals of their programs and the extent to which they believe these goals are being met. Third, interviewers will request and analyze existing data to look for initial evidence about whether programs have been fully implemented and appear to be achieving their intended goals. Since, at most, the study is likely to collect data on the number of diverted cases, the study will not be able to ascertain the number of cases that have been diverted appropriately, or how many diverted cases eventually return to TANF.

Based on the data collected from all of these sources, we will be able to report on states’ practices and indicate which appear to be most successful at achieving the goal of appropriately diverting customers from the TANF caseloads. MPR also will use all data—including information collected through the literature review, telephone interviews, and analysis of outcomes data—to identify states for site visits. The data from these sources will help us determine which practices are promising, but since the study is not an impact evaluation of diversion programs, we will not be able to determine their effectiveness.

I see from the interview schedules that ACF does plan to ask interviewees for data on outcomes (although data is not requested for certain modules for some reason). Presumably, those states that have effective diversion strategies will be more than happy to share their data with ACF. Why not analyze those outcomes (after, of course, screening them to make sure that the outcomes reported, and the methodology used to get them, are indeed valid and reliable)? And based on those outcomes, why not figure out which states seem to have the most effective diversion strategies and base the purposive sampling on that? For those modules where data are currently not being requested, I would strongly suggest that ACF request the data. States can always decline to provide it: it doesn't hurt to ask.

MPR modified the interview protocols to ensure that data are requested as part of each module. While project resources may not permit a thorough review of data to confirm their validity and reliability, the study will examine the reported data. Indeed, one purpose of the study is to assess the availability of state data on these activities. MPR proposes to use the state data as one source of information for the purposive sampling of states for site visits. Other relevant criteria include whether additional information about the diversion activity can be obtained through site visits and whether the activity has been in operation long enough to have resolved initial implementation issues.

It also seems important to ensure that a good mix of different strategies are represented in the purposive sampling. For example, it is unclear why “states selected for site visits will have work-related application requirements and/or a pre-TANF job search program” (page 6). Is there some prior research indicating that these strategies are the best strategies? From a research point of view, the solely state-funded cash assistance programs seem the most novel (and appear to be a direct response to the DRA requirements). As such, isn't it important to explore states that use this strategy in depth?

The sampling strategy will ensure that a mix of different diversion practices—such as work-related application requirements, pre-TANF job search programs, and other innovative approaches—identified through the literature review and interviews will be included in the site visits. However, the information currently available on the solely state-funded programs indicates that many are in the very early stages of implementation and most represent a change in funding source rather than the creation of a new program that differs substantially from the state’s TANF program. Consequently, we expect there would be little to learn about these programs on visits to local sites. The telephone interviews are likely to provide most of the pertinent information about their implementation. Questions about how local sites implement work-related application requirements or job-search programs may be more appropriate for site visits.

And for all of these programs, isn't it important to ask States what the opportunity costs were in funding these programs? (i.e. in order to fund these programs and at least give the appearance that welfare rolls were being reduced, what state programs that were previously funded and could no longer be funded?) This will enable the study to provide at least a rough estimate of the impact of these DRA requirements.

MPR revised the state-level telephone protocol to include a question about the state programs no longer funded as a result of the implementation of solely state-funded cash assistance programs.

Finally, it is unclear whether the Privacy Act applies to this information since the respondents are directors of TANF programs. Please check with your GC and ensure that the Privacy Act applies. If it doesn't (and ACF has no other statutory authority for providing assurances of confidentiality which will stand up to, say, a FOIA request), then ACF should not use the term "confidential" in any of the materials provided to respondents. The extent that could be said in this case would be that ACF would "keep responses private to the extent permitted by law.” The response to A10 would also need to be revised.

MPR revised language in the supporting justification and data collection instrument to state that the study will inform respondents that their responses will be “kept private to the extent permitted by law.” We also will stress to respondents that study reports will aggregate information across sites and states, so that it will not be possible to identify individuals.

Also, the state TANF director questionnaire is missing from the ICR submission. It sounds like the questionnaire is short, so I am willing to overlook this omission. But please send me a copy via email and be careful in the future, as this is not the first time incomplete submissions have been sent to OMB.

The questionnaire is attached.

Response to OMB 3 02/05/21