Final Report of the National Health Interview Survey Review Panel
to the NCHS Board of Scientific Counselors
Overview of the Report
The Board of Scientific Counselors (BSC) of the National Center for Health Statistics (NCHS) commissioned a panel to review the National Health Interview Survey (NHIS) as part of an on-going program review process and to report its findings to the BSC. This report summarizes the review process; provides background information on the NHIS; outlines the accomplishments and contributions of the survey to the health information of the nation; describes the current status of the survey and the challenges and opportunities it faces; and presents a series of recommendations.
The NHIS Review Panel members (attachment 1) met on June 9 and 10 at the National Center for Health Statistics, following an established agenda (attachment 2). In its deliberations, the Panel followed the “Procedures for Reviewing NCHS Programs” established by the BSC (attachment 3) which calls for the reviewers to examine the current status, scientific quality and responsiveness of each program within the context of its mission. Further, the review procedures require that the review take into account future availability of financial and staffing resources, emphasize forward thinking and future plans as well as assess current operations, and conduct an interactive review that obtains information from written materials, presentations, and discussion with program staff.
In advance, the Panel received a number of documents (attachment 4) providing background information on the NHIS. Panel members also submitted additional questions to which responses were provided prior to the meeting (attachment 5). During the two-day meeting, they heard presentations from Dr. Edward Sondik, NCHS Director; Dr. Jennifer Madans, Associate Director for Science and Acting Deputy Director, NCHS; Dr. Jane Gentleman, Director, Division of Health Interview Statistics; and Dr. Richard Nahin, National Center for Complementary and Alternative Medicine, NIH. NHIS staff (Marcie Cynamon, Chief, Survey Planning and Special Surveys Branch; Anne Stratton, DHIS Deputy Director and Acting Chief, Data Production and Systems Branch; Eve Powell-Griner, Chief, Data Analysis and Quality Assurance Branch and Acting DHIS Associate Director for Science; Brenda LaRochelle, Public Health Analyst; and Christopher Moriarity, Mathematical Statistician) were present to answer questions, provide more detailed information, and discuss points with the Panel. Panel members reviewed and revised the Panel’s draft report. Panel Chair Dr. Robert Hummer presented the report to the Board of Scientific Counselors at the September 18-19 meeting. This final report updates the earlier drafts of this report. At the September 2009 BSC meeting, NHIS will report on the progress made on the recommendations and other actions taken as a result of the Panel’s report.
The NHIS is the gold standard for U.S. health survey data. The size, scope and the quality of the NHIS data set it apart from the vast majority of other U.S. health surveys.
It is a unique survey in that it produces data on health status, health behaviors and health care utilization, along with in-depth demographic and socioeconomic characteristics of the population. The NHIS relies on the information that people are able and willing to report to the interviewer and these subjective questions are the most revealing for a number of measures, such as knowledge and attitudes about health and health behavior, limitation of activity and impact on daily activities, and experiences of pain, among others.
It is one of the few health data sets to allow for detailed subgroup analyses—by race/ethnicity and other demographic characteristics—using a nationally representative sample. The geocoding of the entire sample also permits detailed analysis of the association between factors in the environment and a variety of key health outcomes.
Perhaps most important, the NHIS has critical advantages over nearly all of the other U.S. health surveys that rely on telephone survey procedures (with the exception of the NHANES, which involves a much smaller sample). The fact that the NHIS is still using in-person household interviews with area probability sampling means that it is unparalleled in the comprehensiveness of its coverage of the population and in achieving high response rates. These facts mean that the resulting data meet higher standards for quality than any other source of nationally representative health data in the United States.
Survey Overview
The NHIS is a large-scale, household interview survey of the nation’s civilian, noninstitutionalized population designed to gather data on a wide range of health topics. The survey was established with the 1956 National Health Survey Act, was first fielded in 1957, and has been in continuous operation since then. The survey is comprised of a core questionnaire that covers health status, utilization of health care services, health insurance coverage, health related behaviors, risk factors and demographic and socio-economic characteristics of the population. In addition, supplemental questions on special topics are added to the NHIS questionnaire each year, co-sponsored by other government agencies. The sample of the survey is designed to be representative of the noninstitutionalized civilian population; to improve the precision of estimates for certain minority groups, the survey oversamples black, Hispanic and Asian persons.
The U.S. Bureau of the Census provides survey operations for the NHIS. Census and NCHS share the sample design development. Interviews for the NHIS have been conducted by Bureau of the Census since the inception of the survey. Currently, Census Bureau Field Representatives (FR’s) utilize computer-assisted personal interviewing technology to conduct in-person interviews in the home. The CAPI technology was adopted in 1997, at the same time that the NHIS questionnaire was extensively revised and shortened. The re-engineering project of 2001-2003 adopted and developed new CAPI software (Blaise) to replace the old CAPI software (CASES). Field Representatives receive refresher training each year (except in years when cost-cutting measures required elimination of the training) to prepare for new aspects of the survey or focus on problem areas.
The NHIS sample size has ranged from a low of 62,000 persons in 1986 to a high of 139,000 persons in 1966. These changes over time in part reflect the impact of budget uncertainties from year to year as well as changes in survey design and survey costs. The sample size for 2007 was 76,000, which is consistent with the overall long-term slowly declining trend in yearly sample size. Half of the interviewed households from NHIS are reserved for subsequent follow-up by the Medical Expenditure Panel Survey (MEPS) that is conducted by the Agency for Healthcare Research and Quality. MEPS collects additional data from some of the NHIS respondents about health care use, health care expenses, and health insurance coverage. NHIS microdata are periodically linked with three other microdata files: the National Death Index, Medicare Enrollment and Claims data, and the Social Security Benefit History data.
Data are collected about all family members by the Family Section of the NHIS core, from one randomly selected adult for the Sample Adult Section, and about one randomly selected child for the Sample Child Section. Questions in the NHIS core are generally stable from year to year; the last major revision of the core questionnaire was implemented in 1997. Survey planners balance the need for continuity in the analysis of trend data and the modifications necessary to reflect societal shifts, changes in health care delivery, scientific advances in diagnoses and treatment or other developments.
Supplements to the survey offer the opportunity to collect timely, topical and specialized data. Co-sponsored by an array of other government agencies on a cost-recovery basis, the supplements address the specific data needed for program planning and management, policy and research. DHIS staff work closely with co-sponsoring agencies--often many years in advance--to plan and incorporate a supplement into the survey. Frequent co-sponsors include other CDC programs, NIH Institutes, and other DHHS agencies, such as the Substance Abuse and Mental Health Service Administration and the DHHS Office of the Assistant Secretary for Planning and Evaluation. A few of the supplements fielded in recent years include:
2008 Balance and Dizziness Supplement. Co-sponsor: National Center for Deafness and Other Communicative Disorders, NIH
2008 Oral Health Supplement. Co-sponsor: National Institute of Dental and Craniofacial Research, NIH
2007 Complementary and Alternative Medicine Supplement. National Center for Complementary and Alternative Medicine, NIH. This large supplement is scheduled to be repeated in 2012.
2006 Supplementary questions from the Healthy People 2010 Program. Co-sponsors: CDC and NIH
2005 Cancer Control Supplement. Co-sponsors: National Center Institute, NIH and National Center for Chronic Disease Prevention and Health Promotion, CDC. This large supplement is scheduled to be repeated in 2010.
2004 Children’s Mental Health Supplement. Co-sponsor: National Institute of Mental Health, NIH.
NHIS data are made available to users in a variety of forms. Annually, DHIS releases NHIS microdata and documentation to the public free of charge on the NCHS Website. Other data products include a series of publications in the long-running Vital and Health Statistics Series and issues in the Advance Data Series, both available in printed and electronic formats. The Web-only Health E-Stats series features NCHS survey data, including NHIS findings, in summary form.
DHIS has made improving the timeliness of microdata release a major priority. That focus led to the development of the Early Release of Selected Estimates Based on Data from the National Health Interview Survey, a collection of key indicators on which data are released quarterly on the Website. The Early Release now includes 15 major indicators and shows the estimates by key variables, such as age, sex, and race/ethnicity. Also part of the Early Release program are reports on health insurance coverage and cell phone usage.
A wide array of stakeholders rely on NHIS data for scientific studies, public health surveillance, and policy analyses. Data users run the full gamut of researchers, policymakers, academicians, government and non-government programs, business, media and the general public. They obtain NHIS data and analytical products from the Website or by contacting NCHS or NHIS staff directly. Information services are provided by a centralized NCHS data dissemination program and by NHIS staff through a structured inquiries processing system.
The Division of Health Interview Statistics (DHIS) conducts the National Health Interview Survey. In addition to the Office of the Director, there are three branches that contribute to the execution of the survey: Survey Planning and Special Surveys Branch, Data Production and Systems Branch, and Data Analysis and Quality Assurance Branch. The Division currently has 43 full-time staff members (four of whom work exclusively on the SLAITS program and 2 who devote from 50 to 85 percent of their time to SLAITS) and 4 part-time (80 percent of full-time) employees. The FY 2008 NCHS budget for DHIS is $14.6 million, with an additional $6.6 million from reimbursable funds generated from agencies which have added supplemental questions to the NHIS. The single biggest expense for the NHIS is the payment to Census for fielding the survey. This expense was $18.5 million in FY2007, was estimated to be around $21.0 million in FY2008 (prior to any cuts in sample size but with cuts in associated activities), and is projected to increase by about 8% annually over the next several years.
The costs of conducting the NHIS are outstripping current and anticipated resources. At the time this report was prepared, another $0.6 million was estimated to be needed to conduct the survey in 2009 and this number jumps to $8.5 million by 2012. Budget uncertainty is likely to change these figures. These deficits just reflect current operations and not the resources needed to conduct the research, planning and restructuring to maintain a viable and effective NHIS in future years.
Cost savings steps have already taken place, including sample size reduction, cutting back on interviewer training, and deferring critical investments in survey methods research, all of which are needed to maintain the survey’s scientific quality and cost effectiveness.
Sample sizes have been reduced either on a weekly basis for a number of weeks or by panel cuts, achieved by cancelling interviews for two of the four equally-representative subsamples during the last quarter of the fiscal year. More recent news from the DHIS Director is that the NHIS will move to a half-sample approach in October of 2008, which will probably continue through all of 2009. This is due to the fact that current funding levels cannot support current survey operations and to budget uncertainty regarding the post fall 2008 election period. Such huge cuts in sample size have very serious costs in the use-value of the survey and for health estimates among racial/ethnic minority and other sub-populations.
To reduce costs, the NCHS cut back interviewer training, normally done on an annual basis in classrooms with Census FR’s to maintain and improve the quality of data collection.
Cost savings have also been achieved by not investing in the research component of the survey. In 2007, DHIS opted out of a joint research project with the Census Bureau to purchase and study commercially-available address files in order to develop alternative systems for obtaining lists of households to sample. Currently, Census Bureau staff travel to selected geographic areas where they list all of the households in the area and then the resulting addresses are sampled for the survey. Using commercial listings is one potential approach to finding an efficient, less costly method of obtaining those addresses.
Despite the reductions that have already been made, additional reductions will be required to meet anticipated budget levels in future years. Options considered to date include eliminating the oversampling of minority populations, and reducing questionnaire content or reducing sample size again, but these cuts don’t result in commensurate cost savings. Most notably, fixed survey costs remain, even while sample size is cut. Further, reducing questionnaire content or minority representation may result in supplement co-sponsors dropping out and the subsequent loss of those additional funds.
The Bureau of the Census conducts the fieldwork for the NHIS under an interagency agreement. There has often been a positive relationship between the Census Bureau and NHIS, and they have worked together in many productive ways, including the re-engineering of the survey a few years ago.
However, there are some important constraints in the data collection arrangements. NCHS receives limited information on performance standards, interviewing techniques, interviewing quality, and the costs associated with various aspects of the fieldwork. Most of the analysis of paradata has been done by DHIS staff. A lack of timely and comprehensive information on operational aspects of the survey limits DHIS’ ability to develop and evaluate alternatives in survey design or fieldwork procedures to either save costs or improve data quality. Census does not provide the range of information needed to consider the advantages, benefits or limitations in redesigning the survey or phases of its methodology.
The Bureau of the Census has also made little use of the paradata (data on interviewing) that are generated by the data collection instruments and has been reluctant to aggressively follow-up on DHIS recommendations. The Panel heard several examples from DHIS employees about Census interviewers performing lackluster in their duties, yet little corrective action was apparently taken. DHIS would like to take a more proactive stance, deal quickly with interviewer performance problems, and implement changes to improve interview quality. Census Regional Offices are not consistent in the way they manage interviewers and assess interviewer performance and quality standards. As a result, NHIS has recommended that re-interviews be done independently of each Regional Office and in a centralized location and standardized manner.
While changes are occurring which affect the day-to-day operations of the survey, other changes are taking place which affect all of the Centers for Disease Control and Prevention (CDC), NCHS and thereby, DHIS. For the past 20 years, NCHS has been a part of CDC and for many of those years continued to operate more or less independently. NCHS was involved in and responsive to the overall mission of CDC, collaborated with other CDC programs, and worked on important CDC-wide programs and projects. NCHS collaborated with and supported multiple DHHS agencies in its role as the Department’s key source of health statistics for the nation. Administrative services, personnel, planning, budget and other functions were coordinated but not centralized.
A recent reorganization of CDC has changed that system in two ways. First, CDC has been reorganized to focus on issues considered to be of greatest importance to public health and has established a process of creating goals and the organizational structure to support the development, management, and evaluation of those goals to ensure the greatest public health impact. DHIS, like all NCHS programs, has been called upon to provide staff to participate in the goals process. Staff provide technical and subject-matter assistance in the selection and documentation of goals and the development and analysis of the data needed to monitor the goals. While NCHS data have always been a major resource for public health planning and programs, this new system entails additional time-consuming functions which drain staff leadership resources from their primary responsibilities for planning and conducting health surveys.
Secondly, the centralized administrative services now consume a greater amount of high-level staff time than the previous support systems. Almost all of the CDC agencies are located in the Atlanta area and this, along with less understanding of NCHS’s unique and additional mission as a Federal statistical agency, has meant that the systems are not as efficient and effective for NCHS. They take more NCHS staff time but—to date—haven’t offered the level of service associated with the old systems. This restructuring of administrative services has resulted in a greater administrative burden on DHIS staff, both drawing them away from the core NHIS work for which they are uniquely qualified and making the administrative processes more cumbersome, slower, and less responsive, as well as more costly to NCHS. In a program where every staff hour is a valuable and scarce commodity, these effects of the reorganization have had real consequences for the NHIS.
DHIS staff engages in a range of analytical projects, from descriptive statistical reports to more complex analyses of survey findings. Three summary statistics reports issued each year present the basic findings from the survey on children, adults, and the overall population measures. Staff members also analyze and present findings on selected topics, ranging from health behaviors to insurance coverage to health profiles of selected population groups. However, the analytical program of the survey is severely limited due to lack of staff resources and the competing demands for data production-related activities. Staff can on occasion collaborate with external researchers but again these opportunities are limited.
There are no full-time researchers in the DHIS. Methodological studies come at the expense of other activities performed by staff. Most studies are small in scope and focused on useful, immediately applicable results. Opportunities for joint methodological work with the Bureau of the Census have also been limited. The absence of a strong, methodological research component affects not only current issues, such as the release of State estimates with appropriate confidentiality protection, but the evaluation of future options for survey design and operations. A particularly urgent need for methodological expertise and attention is the decision about who should do NHIS data collection in the future—a topic which will be discussed in detail later in the report.
No analysis of the current status of the NHIS and plans for its future would be complete without an understanding of the important contributions of the NHIS to the nation’s health information. Indeed, the NHIS has an impressive history of accomplishments, replete with landmark findings on some of the most important health issues of the past 50 years. The survey has collaborated with other government agencies to ensure that data to develop policies and establish and direct health programs are available.
Just a few of the noteworthy NHIS contributions to science and the public’s health include:
Disparities in health and health care - DHHS and other government programs to eliminate disparities in health and health care rely extensively on the NHIS data. They use these data to measure racial, ethnic and socioeconomic disparities in the prevalence of chronic conditions and health behaviors, urban/rural differences in health care access, and disparities in the use of health services between persons with health insurance and those who are uninsured. This important role of the NHIS data is enabled by oversampling in the survey design and its detailed information on race and ethnicity, education, income, and other socio-demographic data.
Smoking and other health behaviors – NHIS has been the source of data on tobacco use for more than 4 decades and continues to be the single most important source of the information to evaluate the public health campaign to prevent smoking and help direct resources to population groups that are in greatest need of education and assistance about smoking. The NHIS has been, and continues to be, a key data tool in efforts by the federal government and public health agencies throughout the United States to prevent smoking and the adverse health effects of tobacco use—one of the most successful public health campaigns in American history.
AIDS/HIV – The NHIS provided the first national data on knowledge and attitudes on HIV/AIDS. This included estimates of the population’s knowledge of factors associated with transmission, how information is obtained, assessment of personal risk behaviors and other information intended to direct the nation’s HIV prevention program. In 1987, when this survey component was fielded, there was little other reliable information and little experience with collecting information on AIDS from a general population. The NHIS had to develop the survey module, and test and field it in an extremely short turnaround. The findings guided education and prevention programs for years.
Disability – The most extensive survey of the extent of disability and characteristics of the population with disability was conducted in 1994 and 1995 as part of the NHIS, with a more in-depth follow-up with selected respondents. Developed in collaboration with the DHHS Assistant Secretary for Planning and Evaluation, the survey provided much of the information to monitor compliance with the provisions of the Americans with Disability Act.
Insurance coverage – From the second year on, the NHIS has collected data on insurance coverage, providing an insight into those who don’t have insurance and what that means in terms of their health status and their use of health services, including preventive services. The NHIS pioneered the use of several separate measures to show lack of insurance coverage at the time of interview, at any time in the past year, and for a year or more. Looking at the data by race/ethnicity and income level lets policymakers determine how best to meet the health needs of the uninsured.
Complementary and Alternative Medicine (CAM) - A 2002 supplement on CAM co-sponsored by the National Institute for Complementary and Alternative Medicine produced the first national data on this topic. With the current interest in alternative methods of dealing with the complex array of chronic conditions facing an aging population, this supplement produced particularly useful data and was repeated in 2007.
Wireless Only Households – NHIS recently documented that 1/8 of all U.S. households have only wireless phones instead of land lines, and that this percentage varies widely across U.S. family contexts and by household age composition. The collection and dissemination of this information has been used by health researchers and private companies in many areas and is important for future survey methodologists in thinking about the changing context of survey research, particularly the growing concerns about representativeness of telephone surveys that involve only samples of households with a land line.
The long-term continuous conduct of the NHIS has positioned the survey to play a vital and unique role in Federal health initiatives. For example, the NHIS provides a substantial portion of the data used to monitor the successive Healthy People (HP) initiatives. NHIS has sufficient sample size and content scope to provide the detailed information needed to monitor many of the more than 500 HP health objectives. In addition, the survey design oversampling has generated reliable results on many health indicators by race and ethnicity. The data by socioeconomic status are another strong feature of the NHIS. The detailed education categories for example, allow researchers to distinguish between the behaviors, risks, and even mortality outcomes among those who have completed 12 years of education and those who achieved a GED, and among those with a college education versus those with advanced degrees. Such detail may be very useful for the formulation of health and social policy. Urban/rural differences in health have also been well documented by researchers using the NHIS, which are also useful findings in the allocation of health resources and formulation of policy.
The accomplishments and contributions of the NHIS are directly attributed to the NHIS staff, a highly competent, strongly dedicated group of individuals who have many years of experience in conducting the survey and analyzing its results. There are many examples of the staff being generous with its time and knowledge in assisting users to understand and correctly utilize and apply the data. One of the recent achievements, most specifically, includes the speeding up of data release on a time-scale that is incredibly efficient for such a large survey. This is the result of leadership and staff commitment to build a better program despite the burden of maintaining all on-going activities and programs.
The panel proposes a number of recommendations for the future of the NHIS. Some are long-term and may take some time to put in place; others could be implemented fairly quickly and achieve a relatively quick payoff in terms of benefits. In accordance with its charge, the Panel did not recommend the infusion of vast new funding or personnel resources but did indicate that a modest increase in some areas and/or the redirection of other funds and staff should be considered. The Panel recognizes that not all recommendations could be implemented at once, though some are inherently linked. Thus, in the following sections, we first provide one over-arching recommendation, then a set of specific high prior items, and finally a set of other recommendations.
Over-arching Recommendation
The first over-arching recommendation is that the survey be continued. The Panel unanimously viewed the survey as a public good of immense importance and remarkable contributions over its 50-year history. Before any changes are made to the survey, the Panel wanted to ensure that the current capabilities of the survey would be enhanced rather than diluted in any way. NHIS is the gold standard for large, in-person household health surveys. It may be improved by considering and implementing some of the specific recommendations listed below, but changes need to be made in a scientific, empirical manner, with sufficient planning, testing and evaluation before implementation.
Specific High Priority Recommendations
The following recommendations were deemed critical for both advancing the survey and dealing with the current and future challenges in resources and operations. The development of a strategic plan for the survey is the starting point and will provide the framework for the other recommendations.
Strategic Planning Process – NHIS needs a strong, effective strategic planning process that involves stakeholders. This process should determine the core mission of NHIS with respect to population health surveillance, research and policy. What will follow is the delineation of the minimum survey design to accomplish that mission, as well as analytical and dissemination plans. The determination of the core mission also relates to funding. The core mission should be able to be maintained with the on-going NCHS appropriation and not be subject to the contributions from outside agencies which may or may not be available from year to year. A plan needs to be in place for an annual NHIS that will occur without funding from outside agencies. The Division should specify the minimum core resources needed to achieve the minimum survey design, and it should specify the design that can be supported with the existing level of core funds designated for the NHIS. In other words, first define and plan to implement the best NHIS that the current NCHS budget of $14.6 million will buy. If $14.6 million will not fund a survey that will meet some minimum set of the nation’s needs for health survey statistics, or if there are valuable statistical goals that cannot be achieved within that budget, it is incumbent upon the NHIS and NCHS leaders and staff to publicize those deficits and make every effort to secure the funding in core appropriations that will ensure that a proper NHIS will be conducted each year.
NCHS should continue to seek additional resources for the NHIS and its other data collection systems. Future changes in health policies and programs, national priorities in health and health information, the climate for collection and use of data and other factors may make those efforts successful in one or more ways. Such additional funding should be used first to support a solid, stable NHIS on a year-to-year basis and also enable the NHIS to respond to the health priorities of the nation and take advantage of the ideas and innovations of the NHIS leadership as well as the NHIS advisory body. The optimal use of additional funding should be a key component of the strategic plan.
Methodological Studies – DHIS needs to invest extensively in methodological research to support the survey’s redesign and to inform decisions made on all aspects of survey operations and data dissemination. The program does not currently employ or have access to the range of methodological skills needed to conduct the required studies. DHIS needs to draw in the external expertise needed but to link it closely with internal staff and direct the research to its very specific needs. This isn’t an opportunity to buy methodology “off the shelf.” To the contrary, the research will need to be closely attuned to the problems and challenges facing the survey and the environment in which it exists.
Redesign – DHIS needs to consider all possible modes of survey design, including merging with other survey mechanisms, dual-frame, RDD overlay, etc. The guiding philosophy should be to retain in-person interviewing until such time as an alternative method is shown to yield data that meets the standard for data quality of in-person designs, as well as being cost effective. It’s clear that NHIS cannot continue as it is with current and anticipated funding, so DHIS must carefully consider alternative designs while considering both resources and data quality. For example, the Current Population Survey conducts the vast majority of their interviews over the telephone, saving them large sums of money. DHIS should begin methodological work as soon as possible with research into various options with current overall design. Some of the experiments could pay off in cost savings now. Among the possibilities to carefully evaluate are a merger with the National Health and Nutrition Examination Survey (NHANES) with NHIS and NHANES using the same household interview in part or total as NHIS.
Add biomarkers to the NHIS. The NHIS is a well-designed interview survey conducted in households across the nation. There is the potential to add objective biomarkers to the survey to confirm and/or complement data collected in the interviewing process. Increasingly, biomarkers have been incorporated into health and social surveys over the last decade, allowing researchers and policymakers to consider objective markers of health as well as the most commonly included subjective measures. Such objective measure could also be important in better evaluating time trends and group differences in health outcomes. Subjective measure can fluctuate up or down based on social and historical context, but objective measures could potentially control for those forces and allow for more objective and precise health comparisons across social groups and across historical time. Moreover, it is clear that more and more public health and population based research is utilizing biomarker data and that the demand for such data is increasing in the user community. DHIS should contact other survey programs which have successfully integrated biomarkers in an interview setting to learn of their experiences and tap into the research already done in this area. In addition, from the strategic planning process as well as user input, DHIS will be able to evaluate level of interest in the health community and determine how privacy and confidentiality concerns will be handled. At the minimum, DHIS could consider adding the collection of blood spots and/or buccal cell swab kits, both of which yield a wide array of health information at a cost that is relatively low. Further, height and weight measures could be accurately collected by interviewers (perhaps for a sample of respondents) quite quickly and at low cost to obtain more accurate information than self-reports of weight and height. While obviously there is some additional cost for the NHIS, biomarker collection and processing costs have decreased in recent years and should be considered as a potential addition to the NHIS survey effort.
Linkage – Linkage with Medicare, Medicaid, Social Security, and mortality records greatly increases the analytical power and value of the NHIS data. Data linkage is extremely important to the scientific value of the NHIS. Clearly, the greater the accuracy and completeness of identifying information collected during the interviews, the more accurate such linkages will be and the more useful the linked data will then become. Therefore, DHIS should explore improved methods of collecting identifying information in the survey which could then improve data quality and use of the linked data. Recently, for example, the collection of the last four digits of the Social Security Number (SSN) was a good advancement in this area. Looking ahead, DHIS should continue to collaborate with the Special Projects Branch (SPB) of the Office of Analysis and Epidemiology to design questions and interviewing procedures that improve the completeness and quality of identifying information in the NHIS. DHIS should also continue to collaborate with the SPB to test new methodologies for obtaining informed consent for subsequent survey record linkage, which could also work to improve the quality and completeness of linked data resources.
State-level data – DHIS needs to find a way to produce state-level data and methodological work to make this possible can begin with current files and extend into issues related to new and innovative survey design approaches. There are ways to make state-level data available now with acceptable levels of precision for many uses. DHIS can flag unstable estimates, provide more information and guidance to users, create perturbed files for public use, and find ways to smear PSU boundaries, for example.
Data quality – Keeping in mind that scientific rigor is a hallmark of the NHIS, DHIS needs to conduct research to evaluate the quality of data. The research needs to evaluate quality by content and for subpopulation groups as well as overall assessments.
Leadership in Health Survey Statistics – Since the NHIS is the flagship survey of the nation’s center for health statistics, it is also appropriate for the NHIS staff to exert a leadership role in promoting research on how to improve health survey methods more generally. In its early years, NCHS/NHIS were at the forefront of efforts to study sources of survey error and strategies to reduce it. Not only did the NHIS benefit from such work, but so did researchers from around the world. In recent years, with very few exceptions, NCHS and NHIS have not had the staff or financial resources to play this role. There is no other federal agency whose mission includes working to improve the quality of the nation’s health statistics. It would serve the nation well if NHIS had some resources and made a commitment to reassume that leadership role.
Stakeholders’ Input/Advisory Body – The decision-making process is inextricably linked to input from stakeholders and feedback from data users. To conduct the most effective NHIS, the data must meet the current and emerging needs of current and future data users. NHIS should employ a number of mechanisms to obtain this information. NHIS may want to convene an appropriate group of stakeholders as a permanent advisory body, meeting regularly to gain familiarity not only with information needs and uses but also with the structure and constraints under which the survey operates. NHIS also needs to investigate multiple ways of obtaining information on users and user needs. The survey should systematically employ all reasonable approaches to gaining user feedback, from customer satisfaction surveys to embedding user information and feedback into dissemination systems, such as having users register and report on their use of data each time they download files.
Fieldwork Agreement – DHIS needs to enter into a process to renew the agreement or obtain field work and data collection services from a new survey organization to begin with the data collection of 2014. Therefore, DHIS should develop a request for proposals (RFP) that includes its future requirements (including, for example, the collection of specific biomarker data, paradata delivery, ability to change the instrument when needed in a relatively short time, etc…). This RFP could first be sent to the Census Bureau and, if the Bureau’s response is deemed inadequate, then it could be sent to other potential vendors. Alternatively, the RFP could be sent to both the Census Bureau and other potential vendors at the same time. DHIS may also wish to consult with outside advisors in evaluating the proposals.
For a new fieldwork agreement to be in place in time, planning and preparation need to begin immediately. In developing the standards for a data collection agreement, DHIS needs to be explicit about the requirements (e.g., specify that changes sometimes need to be quickly made, the organization must be responsive to data quality issues discovered in the paradata, etc.). Thus, the survey organization needs to be responsive to DHIS’s need for a range of information on survey operations. Only with detailed information to match survey costs with varying methodological approaches and operational aspects can DHIS make informed decisions. The survey organization needs to be responsive to changing priorities, emerging data needs, changes in data collection environments and unexpected opportunities and challenges. It needs to be nimble and able to implement changes on a timely basis, through the innovative design of its systems or through current and future technological advances. The organization should be able and willing to use the feedback from DHIS paradata or other information gained from its long survey experience to refine its field work practices and approaches and to consistently seek ways to improve the quality of data collection. The size and complexity of the survey ensure that it can only be handled by a large, experienced organization, perhaps the Census Bureau or a large-scale, private agency or organization. This agreement is critical and DHIS needs to expend its senior staff resources in ensuring that this process culminates in the best possible results.
Title 13, Title 15, or Other. The Panel carefully considered the pros and cons of the NHIS becoming a Census Bureau Title 13 survey. NHIS is now conducted under Title 15. Title 13 would have the advantage of allowing NCHS to use the Census Bureau’s Master Address File. Under Title 13, however, even though an NHIS public use microdata file would be created and disseminated, there would be limitations on access by NCHS data users to the microdata generated from the survey. However, a memorandum of agreement with the Census Bureau could potentially reduce those limitations, but the requirement that the Census Bureau conduct all follow-up surveys that re-contact respondents/households would remain. Thus, the Medical Expenditure Panel Survey, which uses the NHIS sample, for example, could not be conducted by any contractor other than the Census Bureau were NCHS to move to Title 13.
Title 15 keeps access as it is now, with NCHS regulations covering confidentiality protections and appropriate use of the data, but NCHS must finance the household listing process, a major and costly early step in survey operations. Currently, NCHS shares the infrastructure costs of household listing with certain other Federal agencies that use the Census Bureau for data collection, and each agency also pays for its proportionate share of listings for its own sampling frame. By the time of the implementation of the 2014 sample redesign, the Census Bureau anticipates that the other Federal agencies will drastically reduce or eliminate their need to use listing and will rely on the Census Bureau’s Master Address File, enhanced by data from address information resellers, to supply the listings for their sampling frames. This would leave NCHS to bear the listing infrastructure costs alone for the NHIS. Costs to NCHS to continue with listing as done today would then rise from about $1-2 million per year to an estimated $4-5 million per year in 2014. However, research is underway to determine how the costs of listing as done for the NHIS today could be reduced by using data from address information resellers.
As stated earlier, NCHS will either continue renewing its annual agreement with the Census Bureau or obtain the services of another organization for the NHIS data collection by 2014. Major private survey organizations that conduct large household surveys have their own listing infrastructures in place, and they are also researching the use of data from address information resellers, so NCHS might again be able to take advantage of sharing the listing infrastructure with other clients. New technology may also expand the options for listing and reduce costs, e.g., by deploying GPS and other advanced systems to obtain address information. Another possible strategy—although considered a very unlikely one—would be to attempt to ease the legal and other restrictions on use of the Master Address File by non-Title 13 surveys; these restrictions cause those surveys to duplicate the heavy costs of creating address files.
At present, the Panel concluded that the limitations on data use outweigh the efficiencies and potential cost savings of moving to Title 13. That is, the Panel thought that providing and expanding very wide data access is paramount for the future of the NHIS. Thus, at the moment it appears that becoming a Title 13 Survey would be an unworkable solution because it would undercut a key goal of the NHIS: providing population health data that are widely used for scientific research, public health surveillance, and policy and program development. The final answer to this question, though, most clearly rests on whichever contractor agreement would best and most cost effectively meet the needs of the survey, given future projected budget constraints.
State-level Estimates and Data – NHIS should make the development of state-level estimates a very high priority. While this is admittedly a difficult step to take (see, e.g., Appendix C of the “Responses to Questions from the NCHS BSC NHIS Review Panel,” dated May 21, 2008, included in this report as appendix 5), producing state-level estimates is critical to making the NHIS relevant to policy needs with audiences, a key mission of NCHS and the NHIS. Providing state-level estimates would enable NCHS to encourage states to fund supplemental samples for their own states, which would enhance the statistical power for such estimates. Producing state-level estimates could be facilitated by a redesign of the sample, but it is entirely feasible to generate state-level estimates with the existing design by merging data for variables that are common across consecutive survey years for small states. Confidentiality concerns and statistical-analytical issues related to NHIS’s complex sample design can be addressed by using web-based data analysis tools and user tutorials. The web-base query system developed for the California Health Interview Survey has successfully addressed both types of issues. Public use data files should also include state identifiers for most states, with remaining small states grouped sufficiently to address confidentiality concerns. An option could also be explored to periodically oversample small states on some systematic basis to make it possible for those states to get solid estimates from time to time. NHIS also should explore including Metropolitan Statistical Area (MSA) identifiers on public use data sets.
If necessary, the NHIS could consider using techniques that others have advocated for protection of privacy while still allowing some level of access to state (and MSA) identifiers. For example, DHIS could use data swapping techniques and/or produce a synthetic dataset that could be used for state level research. At a minimum, DHIS should provide guidance to researchers who visit the NCHS Research Data Center (RDC) about the publication and use of its state (and MSA) identifiers (e.g., a set of best practices). Overall, the ability to produce some state (and MSA) estimates and to make available state (and MSA) identifiers will greatly improve the NHIS for health policy research because many important initiatives (e.g., anti-smoking programs, health insurance programs) are implemented, funded, and tweaked at the state level.
Other Recommendations
The Panel recommends another series of steps be taken to improve the survey and the use and usefulness of its data. These primarily concerned the dissemination of the survey findings, documenting the policy uses of the data, and marketing of the survey and its data products. All of these steps would increase the visibility of the survey, enhance the value of the data and increase support for the survey by policymakers and the broad spectrum of data users.
Dissemination – DHIS should have a well-thought out dissemination policy and the practices in place which support that policy. The program has many elements of one now but there are gaps and areas which need additional attention. For example, DHIS could gain important information from users in the dissemination process which would help the program to plan better to meet those needs. In addition, DHIS has made impressive progress on speeding the release of data and that goal should continue to drive the program.
The Research Data Center (RDC) at NCHS will play a central roll in data access for researchers who would like to work with linked NHIS data and for researchers who would like access to state identifiers and other lower level geography. NCHS has made great strides in making the RDC more accessible and responsive to researchers needs over the last couple of years under its new leadership. While these strides are very welcome by the research community, improvement is always possible. We would recommend that NCHS and RDC hold some informal focus groups with researchers who have used the RDC facilities in attempt to see how further improvements could be made. For example, the RDC could provide on-line codebooks for the restricted data sets that are analyzed at the RDC. The RDC will be a vital access point to critical NHIS data for health researchers. Focus groups with clients could help identify key areas for the RDC to invest in over the next few years. Furthermore, this investment may come with increased costs of operation. These costs could be recovered through increased fees or some internal source. Given the potential importance of gaining access to restricted versions of the NHIS data, the RDC should be viewed by NCHS as a key investment for the future dissemination.
Data users receive NHIS data assistance through a system of revolving staff members assigned to receive and triage data requests. This often results in lack of continuity for the data users and specialists assisting them. DHIS needs to develop a system where data users have some continuity and can continue to work with one or several staff members when that would be productive. DHIS also needs to highlight the policy relevance of its data during the dissemination process, from the selection of topics for analysis; through the presentation of those findings, in terms of style and format; and finally by the use of appropriate channels for reaching key, target audiences.
DHIS needs to encourage staff to be more keenly attuned to current policy issues and to apply that perspective to the design and creation of policy-relevant products. The latter are generally less technical than most of the current products and focus more on the key findings and the impact of those findings rather than the methodological aspects of the data. NCHS has moved to address this audience with its new Data Briefs and DHIS should use that series or similar approaches to a greater extent.
DHIS should accelerate its efforts to provide a user-friendly web-based query system that will enable potential data users, including those without sophisticated analytic capacity, to gain access to NHIS data. Such a system must meet the NHIS requirements for protecting data confidentiality, but it should also make the data highly accessible and easily tailored to user needs. Such a query system would also facilitate the provision of state- and even MSA-level estimates without violating data confidentiality.
DHIS has also partnered with researchers at the Minnesota Population Center over the last few years to help disseminate NHIS microdata and documentation in a harmonized fashion from 1969 to the present (www.ihis.us). DHIS also worked very hard to release public use files for the 1963-1968 NHIS data that will soon be put into the harmonized data system. This data will provide important historical context for researchers interested in examining the full length of data through this crucial decade in public health (e.g., Medicare/Medicaid and smoking research will benefit from access to these data). This partnership should continue, because it will continue to increase the visibility/usefulness of the survey and further enhance the use value of the complete set of NHIS data sets going back in time.
Documentation of Policy Uses – NHIS data have been used for important policy applications. DHIS needs to document those policy uses for several purposes: to encourage additional use, to gain support for the survey, and to interest potential collaborators. DHIS could devote some staff time to document articles which have used NHIS data; a librarian could probably assist in this task. As mentioned, the data dissemination process should automatically capture information on data users and the data uses they plan. DHIS should consider holding a policy uses conference (perhaps alternating with the Data Users Conference) to showcase the policy relevance of the NHIS data and the uses to which the data have been applied. Poster and presentation competitions and conferences or workshops on specific policy-relevant themes are other possibilities. DHIS could partner with an external organization, e.g. American Heart Association and focus on the NHIS findings relevant to heart disease prevention and management, and conduct a workshop or piggyback on events already planned. That approach would take advantage of audience and opportunity without the setup costs.
Marketing - DHIS will need a systematic, innovative marketing strategy to elicit the support and participation of other agencies. The marketing strategy should be a part of the overall strategic planning process but it can also take advantage of the DHIS efforts to improve data dissemination documentation and the focus on policy relevant findings. Another marketing plus would be a revamped program to provide access to state-level data both from current data sets and through survey redesign opportunities. Marketing the survey requires the involvement of high-level staff to both seek out possible collaborating agencies and to make the necessary contacts to pitch the survey. To date, other partners and supporters have been other Federal agencies. DHIS could seek out non-profit or other private organizations to be survey co-sponsors but needs to be careful to maintain the survey’s reputation for objectivity. These contacts outside the public arena might be done best through the CDC foundation or other intermediary.
Conclusion
The NHIS is a valued and valuable centerpiece in the nation’s health information infrastructure, both currently and for the future. Its past contributions and current role, however, are being jeopardized by the resource and operational challenges it now faces. Continuing cuts to the NCHS and DHIS budget and draining of human resources have created a significant barrier to the NHIS being able to provide the high-quality data that DHHS and the public have had and continue to expect.
There are critical decisions the DHIS and NCHS must make to keep the survey viable and effective and there must be more and better information on which to base those decisions. Communication with its stakeholders and potential partners will provide some of the needed information as will a rigorous program of methodological research conducted over the next few years. As an added benefit, some of the improvements may be implemented on an earlier time frame to both gain experience and perhaps reduce some costs. Planning for the major restructuring in survey design and selection of an agency or organization to conduct the survey in 2014 must begin now. Indeed, above we recommend setting up the criteria for this selection in the form of an RFP to best determine which data collection agency can best meet the future needs of the survey.
Finally, while funding issues have helped to bring about the most single important problem in fielding the current NHIS, the central issue to focus on for the future is one of identity and role. That is, what is the unique function and role of the NHIS and how can it be funded and managed to carry out that role to the benefit of the nation for years to come?
| File Type | application/msword |
| File Title | July 3, 2008 |
| Author | owner |
| Last Modified By | hcr8 |
| File Modified | 2011-06-09 |
| File Created | 2011-06-09 |