Guidance for Proposing Content for the NCHS Rapid Surveys System

Guidance for Proposing Content for the NCHS Rapid Surveys System

Version: 1 May 2023

The National Center for Health Statistics (NCHS) is launching the Rapid Surveys System to provide CDC and other partners with time-sensitive data of known quality about emerging and priority health concerns, use these data collections to continue learning and evaluation of the quality of public health estimates generated from commercial online panels, and improve methods to appropriately communicate the fitness for use of public health estimates generated from commercial online panels.

In 2023, cross-sectional data from approximately 6,000 adult participants each quarter will be collected using two panels – AmeriSpeak (conducted by NORC) and KnowledgePanel (conducted by IPSOS Public Affairs). NCHS will be conducting extensive scientific and statistical evaluations of data quality, including identifying new methods for combining data from the two panels, evaluating different postsurvey weighting adjustments, calibrating the estimates to benchmarks, and assessing the fitness-for-use of the panel survey data for its intended purposes. Key estimates, online dashboards, and microdata files with documentation are expected to be available within six months after each quarter’s survey topics are finalized, though additional time may be needed initially as program operations are established.

CDC programs and other interested HHS agencies have an opportunity to partner with NCHS to propose survey content. Potential partners should be aware of the following considerations.

Considerations for adding questions

• Content proposed for the NCHS Rapid Surveys System must be consistent with NCHS authorizing legislation. The National Health Survey Act of 1956 provided for special studies to secure accurate and current statistical information on the amount, distribution, and effects of illness and disability in the United States and the services rendered for or because of such conditions. This was further codified in Section 306 of the Public Health Service Act. Examples include measures of health status, causes of morbidity and mortality, populations at risk, behavioral risk and protective factors, intermediate outcomes, access to or utilization of medical or social services, determinants of health (including attitudes to illness and treatment), and topics related to family formation, growth, and dissolution.

• Data collected under the new Rapid Surveys System are intended to complement and not replace the current household survey systems at NCHS, including the National Health Interview Survey (NHIS). The Rapid Surveys approach has a greater potential for coverage and nonresponse bias and smaller sample sizes, thus resulting in lower precision (especially for subgroups). The Rapid Surveys System is particularly well suited for the following situations.

1. Time-sensitive data needs. The survey platform is optimal for obtaining data about emerging and priority public health topics for which existing sources are not of sufficient or known quality or are otherwise unavailable within the timeframe needed.

2. Public health attitudes: Opinions, beliefs, stated preferences, and hypotheticals measure constructs that are expected to vary relatively quickly over time. If asked as part of surveys with lengthy data collection periods, the estimates could be outdated by the time they were available. These types of questions may, however, be suitable for the Rapid Surveys System. A connection should be obvious between these attitudes and either a) the extent, nature, or impact of illness and disability or b) public health programs or policy. Questions on rapidly changing behaviors (e.g., in response to public health messaging) may also be appropriate for this new program.

3. Developmental work to improve concept measurement. The Rapid Surveys System can be used to inform question design for emerging health topics, including understanding how survey respondents interpret and understand questions and concepts related to those topics. This can be accomplished with experimental design (e.g., half of the respondents receive one survey wording whereas half receive an alternative) or by embedding cognitive probes with the newly developed questions.

4. Methodological studies. The Rapid Surveys System can be used to compare, test, and develop approaches to data collection and analysis designed, for instance, to improve data quality, coverage of subpopulations, response rates, and to compare data collection modes, question order, and develop scales.

• The Rapid Surveys System, on the other hand, is generally not appropriate for topics or constructs that are well understood and expected to remain steady over time. NCHS will not include the same questions for multiple quarters unless the underlying concept is likely to change quickly over time and is important for informing public health programs or policy or the response to a public health threat that is covered within NCHS’ statutory authority. Rather, topics or constructs that are well understood and stable over time should be studied with larger and more representative surveys over longer time periods. Exceptions include:

• The Rapid Surveys System can be used to approximate nationally representative estimates for timely and preliminary programmatic decisions, and then – if the constructs are expected to remain steady over time – the questions can be added to larger and more representative surveys to identify important covariates and disparities.

• For constructs that are expected to change quickly over time (e.g., within a given year), NCHS will consider including the same questions for multiple quarters. Each round of the Rapid Surveys System is independent, and estimates produced are cross-sectional. The program is not designed for assessing intra-individual change over time.

• Questions and response options proposed for the Rapid Surveys System will be evaluated by NCHS. They should be simple and written in such a way that respondents with varying levels of cognitive ability can understand the question, recall the relevant information, and provide a straightforward response.

Topics that are not suitable

• Questions that address similar content to what is already being collected as part of the NHIS or other federal surveys of equivalent quality are not appropriate for the Rapid Surveys System, unless there is a time-sensitive need to explore the topic more deeply than done by the other surveys, or if the other surveys will be unable to provide timely estimates to meet the urgent data need.

• NCHS will not accept questions related to consumer market research, such as media habits and non-health purchasing behaviors. CDC licenses Styles data from Porter Novelli for those programs that want to guide their overall communication strategies by better understanding their target audiences’ lifestyles, attitudes, and behaviors.

• Because of the relationship between these types of variables and the likelihood of participating in online panels, the Rapid Surveys System will avoid topics that have the potential to be related to the data collection methods (e.g., internet and technology use) and willingness to participate (e.g., civic engagement, volunteering, voting). However, questions on these topics may be included by NCHS primarily for weighting, calibration, or other methodological purposes.

• Because of the relationship between income/affluence, acculturation, and the likelihood of participating in online survey panels, the Rapid Surveys System may not be the appropriate platform for understanding income or acculturation or documenting differences between subgroups defined by income or affluence. This hypothesis and potential limitation will be evaluated by NCHS using the Rapid Surveys System data and appropriate benchmarks.

• The Rapid Surveys System is not expected to be appropriate for low prevalence estimates given sample size limitations. However, when baseline estimates of health outcomes or covariates associated with an emerging trend are urgently needed, a construct with a low baseline prevalence may be considered for inclusion if the expectation is that the estimate will increase over time (e.g., the receipt of a new vaccine) and the question(s) will be included for several quarters of the Rapid Surveys System (or for several years of an established population survey) to track trends.

• The Rapid Surveys System will be of limited utility for producing estimates for hard-to-reach or low prevalence subpopulations given coverage and sample size limitations. However, depending on the subpopulation and ability to expand samples, special targeted surveys may be feasible. Such surveys may require a longer lead time for development, planning and field time.

• By statute, NCHS survey questions must be objective and survey content must not advocate for or appear to favor any specific health policy or political viewpoint. Content should be policy neutral and program/policy relevant.

Process for partnering with NCHS

• Programs are advised to start a conversation with NCHS as soon as a need is identified. Survey design experts in NCHS’s Collaborating Center for Questionnaire Design and Evaluation Research and Division of Health Interview Statistics will work with programs to refine constructs, questions, and response options for clarity, objectivity, and understandability.

• NCHS makes the final decisions regarding what content will be included in its questionnaires. Prioritization criteria for the Rapid Surveys System include alignment with the CDC and NCHS missions,¹ appropriateness of the content for the program, and the expected quality of the resulting data. Time-sensitive, mission-critical content will have top priority if space is limited.

• If there is more demand than space available, a review committee will consider the prioritization criteria and make decisions regarding each quarter’s content. The committee will meet once per quarter, shortly before the deadline for finalizing content. The Rapid Surveys System is designed to be flexible and to support urgent data needs. Content proposed close to the deadline may be given higher priority than content proposed months earlier, if the former better fulfills the prioritization criteria. Proposed content that is not included in one quarter can be considered again for the following quarter’s questionnaire.

• The average length of each Rapid Surveys questionnaire will be approximately 20 minutes. NCHS expects that at least 10 minutes (50 questions or more) will be reserved for content proposed by our partners. The remaining time is for standard variables to be used for sample weighting and calibration, and for selected portions of existing content from NCHS surveys (such as NHIS, the National Survey of Family Growth, or the National Health and Nutrition Examination Survey). The latter will be used to compare the panel estimates to these benchmarks, to assess the fitness-for-use of the panel survey data, and other methodological purposes.

• All items proposed for the Rapid Surveys questionnaires will undergo question quality evaluation using methods such as expert consultations. When estimating national prevalence is time-sensitive, then focus groups, cognitive testing, or field tests can be done in parallel with fielding the Rapid Surveys questionnaire.

• NCHS ultimately is responsible for the questions and content included on the Rapid Surveys questionnaires. NCHS is responsible for working with the NCHS Ethics Review Board (ERB) and OMB to obtain their approvals under the Common Rule, NCHS Practices and Procedures for the Protection of Human Subjects, and the Paperwork Reduction Act. Changes may be needed to any proposed content depending on ERB or OMB feedback. NCHS will work with our partners to resolve any issues.

1 https://www.cdc.gov/about/organization/mission.htm and https://www.cdc.gov/nchs/about/mission.htm