Ticket to Work Program Evaluation Survey (National Beneficiary Survey)
OMB No. 0960-0666
Addendum to the Supporting Statement
This addendum will discuss the CATI/CAPI experiment, including its purpose and how we will use the information. Note that we will not need to collect additional data for this study; rather, we will be assigning the data we have to different groups and cross-referencing and comparing these groups. See Part B of the Supporting Statement for further details.
Overview/Background of NBS Round 4 Survey
The 1999 Ticket to Work and Work Incentives Improvement Act (“Ticket Act”) (Public Law 106-170) established the Ticket to Work program (TTW) to create additional access to services for Social Security beneficiaries through a new system of public and private Employment Network (EN) providers. Along with establishing the TTW program, the legislation mandated an evaluation of the program. The Social Security Administration (SSA) designed an evaluation consisting of several components to assess the impact of TTW relative to the previous system. The evaluation includes a process analysis, as well as an impact and a participation analysis (approved under the original OMB submission). Primary among these evaluation activities was a multi-wave survey of Social Security Disability Insurance (SSDI) and SSI (Supplemental Security Income) beneficiaries, the National Beneficiary Survey (NBS).
The NBS collects data needed for the TTW evaluation that are not available from SSA administrative data or other sources. This survey has five key objectives:
To provide critical data on the work-related activities of SSI and SSDI beneficiaries, particularly as these activities relate to TTW implementation;
To collect data on the characteristics and program experiences of beneficiaries who use their Ticket;
To gather information about beneficiaries who do not use their Ticket, and the reasons for this choice;
To collect data that will allow us to evaluate the employment outcomes of Ticket users and other SSI and SSDI beneficiaries; and
To collect data on service use, barriers to work, and beneficiary perceptions about TTW and other SSA programs designed to help SSA beneficiaries with disabilities find and keep jobs.
The NBS collects data from two samples of beneficiaries: a national sample of SSI and SSDI beneficiaries (hereafter referred to as the Representative Beneficiary Sample), and a sample of TTW participants (hereafter referred to as the Ticket Participant Sample). We designed the NBS as a dual-mode survey: we collect data using computer-assisted telephone interviewing (CATI), with computer-assisted personal interviewing (CAPI) follow-ups of beneficiaries who do not respond to the CATI interview (as well as those who cannot be located, or who request an in-person interview to facilitate their participation in the survey). The survey instrument is identical in each mode. In all cases, we attempt interviews with the sample person. However, if a sample person is unable to complete either a telephone or in-person interview, we will seek a proxy respondent.
In February 2003, SSA sought and received OMB approval to conduct data collection activities necessary for the evaluation of TTW under OMB No. 0960-0666. At the time of SSA’s request for clearance, we anticipated beginning data collection for the NBS in 2003 and completing data collection in 2006. Once the evaluation contract was awarded, we revised the schedule to begin data collection in 2004 and complete it by October 2007. In February 2006, SSA requested, and received in May, an extension to the existing OMB clearance to October 2007.
In 2008, we made significant changes to the TTW program. Because of these changes, we postponed the final data collection wave until 2010. In addition, changes to the program, including the way the state vocational rehabilitation agencies (SVRAs) can provide services, necessitated revisions to the data collection instrument and sample design. SSA also deleted questions from the survey that were no longer necessary and added some questions to answer questions raised in reports based on prior rounds (such as why services were not received or were not useful).
SSA is submitting this request to make revisions to the original OMB clearance to conduct surveys to evaluate the TTW program (OMB No. 0960-0666). Specifically, we are requesting clearance to conduct round 4 of the NBS using a modified sample design and a slightly revised survey instrument.
In addition, SSA seeks clearance to conduct a methodological survey which would examine how the use of CATI compared to CAPI affects responses from subjects with disabilities. Ewe will randomly select a subset of cases as the treatment group to receive CAPI only. The remaining subjects, assigned to CATI/CAPI, will be treated as CATI only. These cases will not receive a CATI attempt, but will go directly to in-person interviewing. All other cases will receive the standard NBS treatment (CATI with CAPI follow-up). Cases that do not complete in CATI will still be sent to CAPI per the study design, but will be considered non-respondents for the purposes of the experiment..
CATI/CAPI Experiment and Analyses
1. Background
Researcher generally view in-person interviews as necessary to adequately survey a population of persons with disabilities, since sample members’ disabilities may preclude them from completing an interview by telephone or mail. Additionally, because in-person interviewing is generally more expensive than other methods, it is often necessary to collect as many responses as possible using less costly means. Utilizing a mixed-mode methodology, in which CATI attempts are made before deploying field interviewers, can result in significant cost savings over a CAPI-only method while furnishing higher response rates than would be achievable with CATI-only. However, there is a risk that different data collection modes may not produce equivalent results and that non-response bias may not necessarily be reduced even if the overall response rate is increased (Lyberg and Kasprzyk, 1991).
Measurement error can be introduced if the mode of data collection has an independent effect on the interview process and the data collected (Voogt & Saris, 2005). In the case of a dual mode telephone and face-to-face survey, such mode effects could be the result of inherent differences in the attributes of these two modes. While telephone and face-to-face interviews are similar in many ways because both involve an interviewer, they are different in the channels of communication available to the interviewer and respondent (de Leeuw, 2005).
This, combined with norms of conversation via telephone compared to face-to-face, can result in differences in ability to build rapport and engage respondents, respondents’ willingness to reveal information, the pace of the interaction, and the cognitive complexity of the task experienced by respondents. Krosnick et al. (2002) suggests that the higher the cognitive demands placed on respondents, the more likely they are to take shortcuts, or to engage in “satisficing”. Respondents with limited cognitive abilities and low motivation may be most susceptible to this phenomenon and most likely to exhibit “strong satisficing” behavior; making efforts to give a seemingly reasonable answer while putting in minimal effort. There is some evidence to suggest that telephone respondents display more acquiescence, choose more extreme categories, refuse more items, and display more evidence of recency effects than face-to-face respondents (Jordan, Marcus, & Reeder, 1980; Locander & Burton, 1976).
In a post-hoc analysis of NBS data, Sloan, Wright and Barrett (2006) found that overall there were few differences in NBS data collected by CATI versus CAPI for persons with mental and physical disabilities, though there was some evidence that data collected via in-person interviews had lower item non-response and showed less evidence of socially desirable responses and acquiescence than data collected by telephone, particularly for items that were vague or demanded more attention and cognitive processing. However, while this study attempted to control for differences in the characteristics of individuals who self-selected into each mode, due to the non-experimental nature of the comparisons, differences may be an artifact of underlying differences in respondents who responded by CATI rather than CAPI. A rigorous randomized study is the most appropriate method to determine if there are differences in data collected that are due to the data collection mode.
Additionally, given the costs associated with field operations, we are interesting in analyzing existing “paradata” (survey process details about specific interviews, such as how many attempts were required to contact a sampled unit) to determine whether bias would be incurred if field operations were scaled back or eliminated. In particular, this would inform SSA about the tradeoff between costs associated with the extensive field operations currently in place and the relatively high response rate and precision that are obtained, and the lower response rate and precision that would result if field efforts were less intense.
2. Design
The proposed experiment and non-response bias analyses would examine how the use of CATI compared to CAPI affects responses collected from persons with disabilities. There are two key issues related to fielding multi-mode surveys for persons with disabilities this experiment would answer:
1. Are data collected by CATI and CAPI comparable?
2. To what extent is non-response bias reduced by including an in-person CAPI component?
To answer the question of data comparability between the two modes, a subset of cases will be randomly selected as the treatment group to receive CAPI only. The remaining subjects, assigned to CATI/CAPI, will be treated as CATI only. These cases will not receive a CATI attempt, but will go directly to in-person interviewing. All other cases will receive the standard NBS treatment (CATI with CAPI follow-up). Cases that do not complete in CATI will still be sent to CAPI per the study design, but will be considered non-respondents for the purposes of the experiment. The additional costs associated with sending cases directly to CAPI will be covered by the StatsRRTC budget. The estimated number of sample members that are needed for a minimum detectable treatment effect of .075 is 360 in the treatment group.
As indicators of data comparability across modes, we will examine item non-response, number of options checked for check-all-that-apply items, non-differentiation among items in a series, proportion of agree/yes responses, length of responses to open-ended items, and distribution of responses or means for sensitive items. In addition, we will examine response effects by type of disability (cognitive or physical).
To determine mode effects, cross tabulations will be developed to test the hypothesis of no association between modes for categorical variables. If the expected count in one or more cell in the contingency table is less than five, Fisher’s Exact Test will be used rather than the chi-square statistic. Means will be calculated to test the hypothesis of no difference between modes for continuous variables. A t-test will be used to determine whether a significant difference exists between modes for these items.
0960-0666 Addendum
| File Type | application/msword |
| File Title | Ticket to Work Program Evaluation Survey (National Beneficiary Survey) |
| Author | 666429 |
| File Modified | 2009-12-08 |
| File Created | 2009-12-08 |