Mini Supporting Statement A For
“Questionnaire Cognitive Interviewing and Pretesting (NCI)”
0925-0589, Expiration Date 04/30/2014
Title of Sub-Study: Reliability of Computer Adaptive Tests (CAT) Study for the NIH-SSA Collaboration to Improve Disability Determination
Daniel Hobbs, Management Analyst, Epidemiology & Biostatistics Section, Rehabilitation Medicine Department, National Institutes of Health Clinical Center, NIH
Section A
A.1 Circumstances Making the Collection of Information Necessary
The Public Health Service Act (42 USC § 282) authorizes the National Institutes of Health to collect this information. The Rehabilitation Medicine Department (RMD) at the NIH Clinical Center is currently working with the Social Security Administration (SSA) through a multiyear, inter-agency agreement (IAA). The IAA, established in 2008, provides support for the RMD to conduct research focused on improving the SSA’s disability determination process. This research includes the feasibility of comprehensively examining function through development of Computer Adaptive Tests (CAT) that could improve the SSA disability determination process.
The Epidemiology and Biostatistics section in the RMD will be collecting information through a contractor (Boston University-Health and Disability Research Institute (BU-HDR)) to assess the reliability and consistency of the CAT tool as part of the disability determination process. This study involves formative research to refine the software and confirm consistency of the CAT. Specifically, this study will test the reliability of two existing SSA CAT instruments that assess physical functioning and behavioral health functioning on the same individuals no more than 7 days apart.
This study is part of a larger series of studies that are being conducted between the SSA and NIH. Three projects have already received OMB approval1 and another project received NIH Clinical Exemption (CE #2013-07-001). In February 2013, OMB recommended this study be submitted under Dr. Gordon’s Willis’ formative generic. This is the final formative study planned as part of the development of the physical function and behavioral health CAT instruments with the SSA. This study fits within scope of the full generic as “other questionnaire testing and development” (Supporting Statement A written January, 2011, p. 11) in which program staff will “perform testing of other questionnaires that require development over a short time-frame.”
A.2 Purpose and Use of the Information Collection
The proposed reliability study will examine the outcome consistency of the developing CAT instruments, and aims to identify and resolve potential CAT reliability issues prior to finalizing these instruments.
Specific study primary objectives include:
To examine the existing CAT instruments’ consistency and reliability in scoring function of individuals in a normative population sample and those with disabilities, and
To differentiate between the fluctuation of individual CAT scores as functional status changes occur in claimants over a period of time and limitations of the CAT instruments.
The instruments will assess the physical function and behavioral health function CAT (Attachments A, B, and C), as compared to the Veterans Rand 12-Item Health Survey (VR-12) Legacy Instrument (Attachment E). All three surveys will be administered to a normative sample and a disabled sample of individuals. Additionally, all three surveys will be administered to the same individuals twice; initially as a baseline and then again no more than seven days apart.
A.3 Use of Improved Information Technology and Burden Reduction
The use of Computer Adaptive Tests (CAT) will decrease the response burden to the claimant and eliminate the need for data entry.
The instruments will be administered to respondents electronically, through the subcontracted professional survey firm YouGov’s website.
A.4 Efforts to Identify Duplication and Use of Similar Information
This is the first time that the CAT instruments have been systematically tested to ensure reliability. There are no data regarding the consistency and reliability of CAT instruments over multiple administrations.
A.5 Impact on Small Businesses or Other Small Entities
No small businesses or other small entities will be impacted.
A.6 Consequence of Collecting the Information Less Frequently
Instrument development would continue without adequately addressing potential inconsistencies in the ability of CAT instruments to successfully assess function. This has the potential to substantially delay completion and implementation of the CAT instruments, and will increase associated development costs to the government.
A.7 Special Circumstances Relating to the Guidelines in 5 C.F.R. 1320.5
There is one exception to the guidelines of 5 C.F. R. 1320.5. This study will administer 2 tests within 7 days. This is to ensure consistent content and calculations.
A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency
NIH Clinical Center is currently working with the Social Security Administration (SSA) through a multiyear, inter-agency agreement (IAA). The Epidemiology and Biostatistics section in the RMD will be collecting information through a contractor (Boston University-Health and Disability Research Institute (BU-HDR).
A.9 Explanation of Any Payment or Gift to Respondents
The study participants will be an opt-in pool of voluntary respondents, recruited through the standard practices of private survey firms, including incentivizing through a point-based system. Survey respondents will receive 1000 points for initial completion of CAT instruments, and 3000 points for completion of follow-up testing. These amounts of accrued points equate to an equivalent monetary value of $1 and $3, respectively.
A.10 Assurance of Confidentiality Provided to Respondents
Basic personally identifiable information (PII) will collected for this study so that the same respondents can be contacted twice.
IRB approval for this study was sought through Boston University Medical Center (Attachment D). NIH developed a reliance agreement with Boston University signed on 2/19/13.
A.11 Justification for Sensitive Questions
PII is collected in the form of basic demographic information including gender, ethnicity, race, education, marital status, and zip code of residence. For the disabled study population, broad nature (primarily physical, mental or both) and duration of disability will be collected.
Questions included in the CAT instruments for behavioral health are designed to assess a respondent’s functioning in interpersonal domains. This includes sensitive questions that can be regarded as “psychological problems” including questions about feelings towards others, and mood swings. These questions are essential to include, as it would be impossible to fully capture someone’s behavioral health functioning without including this information.
A.12 Estimates of Annualized Burden Hours and Costs
All three surveys will be administered to a normative sample and a work disabled sample of individuals twice; initially as a baseline and then again no more than seven days apart.
The total respondent burden for this proposed effort is estimated to be no more than 1001 hours (Table A.12-1).
Table A.12-1. Estimates of Burden Hours
Form Name |
Type of Respondent |
Number of Respondents* |
Number of Responses per Respondent** |
Average Burden Per Response (in hours) |
Total Burden Hours |
Consent |
Normative and Work Disabled Individuals |
800 |
1 |
5/60 |
67 |
VR-12 Legacy |
Normative Individuals |
400 |
2 |
5/60 |
67 |
Work Disabled Individuals |
400 |
2 |
5/60 |
67 |
|
Physical Function CAT |
Normative Individuals |
400 |
2 |
15/60 |
200 |
Work Disabled Individuals |
400 |
2 |
15/60 |
200 |
|
Behavioral Health Function CAT |
Normative Individuals |
400 |
2 |
15/60 |
200 |
Work Disabled Individuals |
400 |
2 |
15/60 |
200 |
|
Totals |
|
|
|
|
1001 |
* The normative sample n=400 and the work disabled n=400. These 800 individuals will be taking the same three surveys.
** Respondents will respond to the survey initially, at baseline, and then again within 7 days later.
The May 2012 National Occupational Employment and Wage Estimates for the United States reports that the mean hourly wage rate for all occupations is $22.01. It is estimated that the cost to the respondents to be $22,032 over the course of this study (Table A.12-2).
Table
A.12-2. Cost to Respondents
Form Name |
Type of Respondent |
Number of Respondents |
Total Burden Hours |
Hourly Wage Rate |
Total Cost |
Consent |
Normative and Work Disabled Individuals |
800 |
67 |
$22.01 |
$1474.67 |
VR-12 Legacy |
Normative Individuals |
400 |
67 |
$22.01 |
$1,474.67 |
Work Disabled Individuals |
400 |
67 |
$22.01 |
$1,474.67 |
|
Physical Function CAT |
Normative Individuals |
400 |
200 |
$22.01 |
$4,402.00 |
Work Disabled Individuals |
400 |
200 |
$22.01 |
$4,402.00 |
|
Behavioral Health Function CAT |
Normative Individuals |
400 |
200 |
$22.01 |
$4,402.00 |
Work Disabled Individuals |
400 |
200 |
$22.01 |
$4,402.00 |
|
Totals |
|
|
934 |
|
$22,032.01 |
A.13 Estimate of Other Total Annual Cost Burden to Respondents and Record Keepers
There is no operating, maintenance, and capital costs associated with this study.
A.14 Annualized Cost to the Federal Government
$40,150 for the survey firm YouGov to conduct this study. There is an additional $15,000 in Federal Government staff time to oversee this contract and manage the project. This totals a cost of $55,150 to the Federal Government.
A.15 Explanation for Program Changes or Adjustments
This is a new collection of information.
A.16 Plans for Tabulation, Publication and Project Time Schedule
The primary objective is to examine the consistency and reliability of the CAT instruments over multiple administrations. Outcomes of this study will be used to inform future tool development, as well as inform future SSA/NIH collaboration planning.
The results of this study may be combined and published with the conclusions drawn from previously approved studies relating to the validity of the CAT instruments in a general context.
This sub-study, pending approval, is tentatively scheduled to commence in June 2014, with data collection completed within three weeks and data analysis completed by the end of the calendar year. BU-HDR will be responsible for the analysis of the data and provide a report to the NIH for review and acceptance.
A.17 Reason(s) Display OMB Expiration Date is Inappropriate
We are not requesting exemption from the display of the OMB expiration date.
A.18 Exceptions to Certification for Paperwork Reduction Act Submissions
This information collection will comply with the requirements in 5 CFR 1320.9.
List of instruments, instructions, and scripts submitted with this request:
Attachment A: All Behavioral Health CAT Items
Attachment B: All Physical Functioning CAT Items
Attachment C: CAT Instrument Initial screenshot
Attachment D: Expedited IRB Review approval
Attachment E: VR-12 Legacy Assessment
Attachment F: IRB Approved Consent
1 The three projects include:
OMB No. 0925-0659, “The SSA-NIH Collaboration to Improve the Disability Determination Process: Validation of IRT-CAT Tools (CC)”, approved on 6/12/2012;
OMB No. 0925-0642-33 “User Simulation Study for NIH-SSA Collaboration to Improve Disability Determination”, approved on 11/7/2013; and
OMB No. 0925-0642-35, “Cognitive Interviewing for Item Bank Fields in a Computer Adaptive Testing Instrument,” approved on 12/31/2013.
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
Author | Hobbs, Daniel (NIH/CC/RMD) [C] |
File Modified | 0000-00-00 |
File Created | 2021-01-24 |