sub-study1 miniSSA test

Mini Supporting Statement A For

“Questionnaire Cognitive Interviewing and Pretesting (NCI)”

0925-0589, Expiration Date 04/30/2014

Title of Sub-Study: Reliability of Computer Adaptive Tests (CAT) Study for the NIH-SSA Collaboration to Improve Disability Determination

Daniel Hobbs, Management Analyst, Epidemiology & Biostatistics Section, Rehabilitation Medicine Department, National Institutes of Health Clinical Center, NIH

Section A

A.1 Circumstances Making the Collection of Information Necessary

The Public Health Service Act (42 USC § 282) authorizes the National Institutes of Health to collect this information. The Rehabilitation Medicine Department (RMD) at the NIH Clinical Center is currently working with the Social Security Administration (SSA) through a multiyear, inter-agency agreement (IAA). The IAA, established in 2008, provides support for the RMD to conduct research focused on improving the SSA’s disability determination process. This research includes the feasibility of comprehensively examining function through development of Computer Adaptive Tests (CAT) that could improve the SSA disability determination process.

The Epidemiology and Biostatistics section in the RMD will be collecting information through a contractor (Boston University-Health and Disability Research Institute (BU-HDR)) to assess the reliability and consistency of the CAT tool as part of the disability determination process. This study involves formative research to refine the software and confirm consistency of the CAT. Specifically, this study will test the reliability of two existing SSA CAT instruments that assess physical functioning and behavioral health functioning on the same individuals no more than 7 days apart.

This study is part of a larger series of studies that are being conducted between the SSA and NIH. Three projects have already received OMB approval¹ and another project received NIH Clinical Exemption (CE #2013-07-001). In February 2013, OMB recommended this study be submitted under Dr. Gordon’s Willis’ formative generic. This is the final formative study planned as part of the development of the physical function and behavioral health CAT instruments with the SSA. This study fits within scope of the full generic as “other questionnaire testing and development” (Supporting Statement A written January, 2011, p. 11) in which program staff will “perform testing of other questionnaires that require development over a short time-frame.”

A.2 Purpose and Use of the Information Collection

The proposed reliability study will examine the outcome consistency of the developing CAT instruments, and aims to identify and resolve potential CAT reliability issues prior to finalizing these instruments.

Specific study primary objectives include:

1. To examine the existing CAT instruments’ consistency and reliability in scoring function of individuals in a normative population sample and those with disabilities, and

2. To differentiate between the fluctuation of individual CAT scores as functional status changes occur in claimants over a period of time and limitations of the CAT instruments.

The instruments will assess the physical function and behavioral health function CAT (Attachments A, B, and C), as compared to the Veterans Rand 12-Item Health Survey (VR-12) Legacy Instrument (Attachment E). All three surveys will be administered to a normative sample and a disabled sample of individuals. Additionally, all three surveys will be administered to the same individuals twice; initially as a baseline and then again no more than seven days apart.

A.3 Use of Improved Information Technology and Burden Reduction

The use of Computer Adaptive Tests (CAT) will decrease the response burden to the claimant and eliminate the need for data entry.

The instruments will be administered to respondents electronically, through the subcontracted professional survey firm YouGov’s website.

A.4 Efforts to Identify Duplication and Use of Similar Information

This is the first time that the CAT instruments have been systematically tested to ensure reliability. There are no data regarding the consistency and reliability of CAT instruments over multiple administrations.

A.5 Impact on Small Businesses or Other Small Entities

No small businesses or other small entities will be impacted.

A.6 Consequence of Collecting the Information Less Frequently

Instrument development would continue without adequately addressing potential inconsistencies in the ability of CAT instruments to successfully assess function. This has the potential to substantially delay completion and implementation of the CAT instruments, and will increase associated development costs to the government.

A.7 Special Circumstances Relating to the Guidelines in 5 C.F.R. 1320.5

There is one exception to the guidelines of 5 C.F. R. 1320.5. This study will administer 2 tests within 7 days. This is to ensure consistent content and calculations.

A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

NIH Clinical Center is currently working with the Social Security Administration (SSA) through a multiyear, inter-agency agreement (IAA). The Epidemiology and Biostatistics section in the RMD will be collecting information through a contractor (Boston University-Health and Disability Research Institute (BU-HDR).

A.9 Explanation of Any Payment or Gift to Respondents

The study participants will be an opt-in pool of voluntary respondents, recruited through the standard practices of private survey firms, including incentivizing through a point-based system. Survey respondents will receive 1000 points for initial completion of CAT instruments, and 3000 points for completion of follow-up testing. These amounts of accrued points equate to an equivalent monetary value of $1 and $3, respectively.

A.10 Assurance of Confidentiality Provided to Respondents

Basic personally identifiable information (PII) will collected for this study so that the same respondents can be contacted twice.

IRB approval for this study was sought through Boston University Medical Center (Attachment D). NIH developed a reliance agreement with Boston University signed on 2/19/13.

A.11 Justification for Sensitive Questions

PII is collected in the form of basic demographic information including gender, ethnicity, race, education, marital status, and zip code of residence. For the disabled study population, broad nature (primarily physical, mental or both) and duration of disability will be collected.

Questions included in the CAT instruments for behavioral health are designed to assess a respondent’s functioning in interpersonal domains. This includes sensitive questions that can be regarded as “psychological problems” including questions about feelings towards others, and mood swings. These questions are essential to include, as it would be impossible to fully capture someone’s behavioral health functioning without including this information.

A.12 Estimates of Annualized Burden Hours and Costs

All three surveys will be administered to a normative sample and a work disabled sample of individuals twice; initially as a baseline and then again no more than seven days apart.

The total respondent burden for this proposed effort is estimated to be no more than 1001 hours (Table A.12-1).

Table A.12-1. Estimates of Burden Hours

Form Name	Type of Respondent	Number of Respondents*	Number of Responses per Respondent**	Average Burden Per Response (in hours)	Total Burden Hours
Consent	Normative and Work Disabled Individuals	800	1	5/60	67
VR-12 Legacy	Normative Individuals	400	2	5/60	67
	Work Disabled Individuals	400	2	5/60	67
Physical Function CAT	Normative Individuals	400	2	15/60	200
	Work Disabled Individuals	400	2	15/60	200
Behavioral Health Function CAT	Normative Individuals	400	2	15/60	200
	Work Disabled Individuals	400	2	15/60	200
Totals					1001

* The normative sample n=400 and the work disabled n=400. These 800 individuals will be taking the same three surveys.

** Respondents will respond to the survey initially, at baseline, and then again within 7 days later.

The May 2012 National Occupational Employment and Wage Estimates for the United States reports that the mean hourly wage rate for all occupations is $22.01. It is estimated that the cost to the respondents to be $22,032 over the course of this study (Table A.12-2).

Table A.12-2. Cost to Respondents

Form Name	Type of Respondent	Number of Respondents	Total Burden Hours	Hourly Wage Rate	Total Cost
Consent	Normative and Work Disabled Individuals	800	67	$22.01	$1474.67
VR-12 Legacy	Normative Individuals	400	67	$22.01	$1,474.67
	Work Disabled Individuals	400	67	$22.01	$1,474.67
Physical Function CAT	Normative Individuals	400	200	$22.01	$4,402.00
	Work Disabled Individuals	400	200	$22.01	$4,402.00
Behavioral Health Function CAT	Normative Individuals	400	200	$22.01	$4,402.00
	Work Disabled Individuals	400	200	$22.01	$4,402.00
Totals			934		$22,032.01

A.13 Estimate of Other Total Annual Cost Burden to Respondents and Record Keepers

There is no operating, maintenance, and capital costs associated with this study.

A.14 Annualized Cost to the Federal Government

$40,150 for the survey firm YouGov to conduct this study. There is an additional $15,000 in Federal Government staff time to oversee this contract and manage the project. This totals a cost of $55,150 to the Federal Government.

A.15 Explanation for Program Changes or Adjustments

This is a new collection of information.

A.16 Plans for Tabulation, Publication and Project Time Schedule

The primary objective is to examine the consistency and reliability of the CAT instruments over multiple administrations. Outcomes of this study will be used to inform future tool development, as well as inform future SSA/NIH collaboration planning.

The results of this study may be combined and published with the conclusions drawn from previously approved studies relating to the validity of the CAT instruments in a general context.

This sub-study, pending approval, is tentatively scheduled to commence in June 2014, with data collection completed within three weeks and data analysis completed by the end of the calendar year. BU-HDR will be responsible for the analysis of the data and provide a report to the NIH for review and acceptance.

A.17 Reason(s) Display OMB Expiration Date is Inappropriate

We are not requesting exemption from the display of the OMB expiration date.

A.18 Exceptions to Certification for Paperwork Reduction Act Submissions

This information collection will comply with the requirements in 5 CFR 1320.9.

List of instruments, instructions, and scripts submitted with this request:

Attachment A: All Behavioral Health CAT Items

Attachment B: All Physical Functioning CAT Items

Attachment C: CAT Instrument Initial screenshot

Attachment D: Expedited IRB Review approval

Attachment E: VR-12 Legacy Assessment

Attachment F: IRB Approved Consent

1 The three projects include:

• OMB No. 0925-0659, “The SSA-NIH Collaboration to Improve the Disability Determination Process: Validation of IRT-CAT Tools (CC)”, approved on 6/12/2012;

• OMB No. 0925-0642-33 “User Simulation Study for NIH-SSA Collaboration to Improve Disability Determination”, approved on 11/7/2013; and

• OMB No. 0925-0642-35, “Cognitive Interviewing for Item Bank Fields in a Computer Adaptive Testing Instrument,” approved on 12/31/2013.

7