2021brfss Ssb

The Behavioral Risk Factor Surveillance System (BRFSS) is a collaborative project of the Centers for Disease Control and Prevention (CDC) and U.S. states, the District of Columbia, and U.S. territories (collectively called “states” or “jurisdictions” in this document). The BRFSS is a coordinated series of interviews that collects information about preventive health practices and behavioral risk factors that are linked to chronic diseases, injuries, and preventable infectious diseases. Respondents are adults, ages > 18 years. Information collection is conducted annually.

The BRFSS is administered through cooperative agreements with state health departments. A representative sample of respondents is drawn for each state. Each state administers a state-tailored questionnaire which consists of (1) a standard core administered by all states, and (2) optional modules selected by the states. The state-tailored questionnaires and samples are designed individually by each state with technical assistance provided by the CDC.

BRFSS is a unique collaboration between the federal government and states. It is highly responsive to diverse needs and priorities for states, the federal government, and non-governmental agencies. Although federal funding is essential to its continued operation, the BRFSS includes funding from state and local government sources as well as from nonprofit agencies in some instances. Thus, decisions about the BRFSS encompass consideration of need for all of these partners. All BRFSS collaborators recognize the need for a high level of data standardization while still being flexible to meet the particular public health surveillance needs of individual states. For example, regular training programs and contact with the data collectors and state BRFSS coordinators help to establish agreed-upon protocols for data collection.

However, it may be difficult in some cases to find balance and consensus among the CDC, other federal agencies, and state-specific needs. As a result, specific methodological challenges may arise from this unique partnership but also provide great opportunity for innovation. BRFSS provides guidance to both collaborators and users of BRFSS data as necessary to address methodological issues. BRFSS is addressing specific issues as follows:

1. Sampling

a) Strengths: Drawing samples at the state level produces datasets that are actionable for the principal use, such as, managing public health programs at the state/substate jurisdiction level. States may determine that substate regions should be drawn by health districts, counties or groups of counties as determined by state health policy implementation and service delivery.

b) Challenges: Given the diverse public health needs and focus of the BRFSS, a major challenge is to be able to produce reliable and valid estimates at differing geographic levels, including sub-state, state and national levels.

c) Comments: The BRFSS has undertaken a number of steps to make sampling more efficient and to recruit respondents to take part in the survey. These efforts include: enhancements to increase cell phone participation and response, the geographic targeting of cell phone numbers, overlapping sampling of landline and cell phone respondents, and inclusion of respondents who live in college housing. In order to ensure that reduced response rates do not result in bias, enhancements have also been made in post data collection processes. Studies are being conducted to provide guidance on both national and sub-state estimates from the state-based BRFSS estimates. In 2021 the BRFSS will be using a new protocol to sample residents of each state using cell phone screening which relies on addresses and reduces the number of out-of-state participants in each state sample. Landline phone numbers are substantially reduced as a portion of the total sample.

d) Recent pilots: Using the CDC/ATSDR Formative Research and Tool Development generic package (OMB #0920-1154), several pilots of sampling methods are underway. These include two projects on post data collection analysis methods to examine small area estimation protocols, and three pilots on alternative data collection methods using internet panels and RDD push -to-web, and address-based sampling. The BRSS is also testing the validity of self-reported measures by sampling subjects from a large electronic health record system and administering a subset of the BRFSS. Self-reports will then be compared to information within the electronic health records.

2. Telephone-based mode of survey administration

a) Strengths: Telephone surveys have been shown to be relatively agile and low-cost methods for population studies. The BRFSS remains current in terms of population data collection methods. The BRFSS follows standards of the American Association of Public Opinion Research (AAPOR) and presents results of internal research on BRFSS paradata at annual conferences. Validity checks are conducted by comparing data from the BRFSS against similar questionnaire items from other surveys (usually the NHIS, NHANES and/ or NSDUH). Updating methods also includes constant review of all field methods, questionnaire design and item analyses.

b) Challenges: In recent years, there have been declines in response rate for phone surveys, although the BRFSS performs well when compared to other telephone surveys. Assessment of the quality of telephone surveys requires assessment of alternative modes of data collection.

c) Comments: The Division of Population Health will continue to pursue assessment studies of complementary modes of data collection as well as continue to examine nonresponse bias to ensure data quality and representativeness of the population.

d) Recent pilots: Two pilots were completed in 2018-2019. These provided preliminary information on the use of sampling other than reliance on a random digit dialed (RDD) sample and also examined whether online data collection was possible using an RDD (as opposed to an address-based sample). These initial tests indicated that a change to ABS would not improve administration of the survey. The initial test to push respondents from the phone to the web indicated that about 20% of those who were asked to complete an online portion of the BRFSS did follow through. This test indicated that a text message with an embedded link was the most productive method of achieving a completed interview. As is noted above 6 tests of methods are currently underway. Data from these test will inform the Division of Population Health as to the potential to change methods of sampling and/or data collection

3. Overall complexity of the system

a) Strengths: The BRFSS platform is a source of innovation in data collection and has been a leader in understanding how population surveys can contribute to understanding of state-level health data.

b) Challenges: There is a constant need to maintain data quality standards, provide training and technical assistance, and to keep up with new personal communication technologies.

c) Comments: The DPH conducts field studies and provides training, technical assistance, and quality improvement services to the states at annual meetings and through bimonthly webinars.

d) The Division of Public Health recently completed a redesign of the core questionnaire to reduce the number of questions and streamline content (see Attachment 16). Some sections were moved to modules (Arthritis Burden) and others (Fruits and Vegetables, Detailed Physical Activity, Falls, Drinking and Driving/Seat Belt Use) will move to less frequent rotation. In some instances, core sections remained by questions were modified to improve the flow of the questionnaire or decrease respondent confusion.

The BRFSS is moving toward the use of a single, cell phone based sample of phone numbers with only supplemental landline interviews for states (such as Alaska) where cell phone coverage is problematic. These changes could simplify the design weighting process and reduce the administrative burden of incorporating two samples into a single weighting process.

1. Respondent Universe and Sampling Methods

Respondent Universe

The target population for BRFSS information collection is adults (18 years of age or older) living in private households or college housing. An eligible household is defined as a housing unit that has a separate entrance, where occupants eat separately from other persons on the property, and that is occupied by its members as their principal or secondary place of residence. The following are non-eligible households: vacation homes not occupied by household members for more than 30 days per year, group homes, institutions, and (in the landline telephone sample) households in states other than the one conducting the particular BRFSS questionnaire.

Eligible household members include all related adults (aged 18 years or older), unrelated adults, boarders/roomers, and domestic workers who consider the household their home, even though they may not be home at the time of the call. Household members do not include adult family members who are currently living elsewhere. Since 2012, adult students living in college housing have been included as eligible respondents. Persons living in college housing are treated as single adult households.

The BRFSS does not conduct proxy interviews.

State-tailored Samples

An independent sample is drawn for each state by the state health department. The size of each state sample is estimated according to the number of completed interviews for the previous year, but may be adjusted depending on state objectives and funding. States may subdivide their state by geographic region/geostrata (such as public health districts, counties or groups of counties). States may also target population groups within their sample. States with sufficient sample size may choose to “split” samples into versions order to obtain information on a broader array of topics. A state may field up to three versions of its annual questionnaire, where each version is comprised of the standard core and a different set of optional modules. To ensure an adequate number of responses for weighting purposes, the state must conduct at least 2,500 interviews for each version of the BRFSS questionnaire. The minimum number is set to allow for comparisons by sex, age and/or racial groups. The minimum sample size for the split versions was set in response to a recommendation from a working group of the American Statistical Association when versions were first used in 2004. See Attachment 8 for a summary of the U.S. adult population and estimated size of the current BRFSS sample, by state. The total number of respondents for the BRFSS varies from year to year but will not exceed 480,000 in 2022-2024. As was noted earlier, the size of the sample for each state is determined by state needs and resources and not by the CDC. Samples in any given year also vary by state, according to the resources that states have, specific information need and the need to target specific groups. CDC provides technical assistance to states in drawing their samples, but does not determine the sample design nor the size of the sample. CDC provides technical assistance to the states in sampling design as needed or requested. Thus the final data set is an aggregate of state-level samples, not a national sample.

States draw their samples based on substate regions (with a few exceptions of states which do not have substate regions and draw only a state-wide sample).

Sampling Frame

To provide rapid and flexible access to respondents and contain costs, BRFSS data collection is conducted through telephone interviews (except in very limited circumstances in which interviews must be conducted in-person). A sample record is one telephone number in the list of all telephone numbers selected for dialing. To meet the BRFSS sample design standards, sample records must be justifiable as a probability sample of all persons with telephones in the state. The BRFSS sample is randomly selected from working phone numbers within each jurisdiction. CDC uses an overlapping sample of landline and cell phone numbers. No direct method of accounting for non-telephone coverage is employed for the BRFSS. As was noted earlier, the proportion of the sample comprised of landline number is reduced each year. In 2021, states are targeting 80% of their sample to be taken from cell phone numbers. In 2019, approximately 65% of all interviews were conducted by cell phone.

The landline sample for each state is based on a disproportionate stratified sample (DSS) design in which telephone numbers are assigned to two separately sampled strata based on the presumed density of residential (non-business) telephone numbers. The high-density and medium-density strata contain telephone numbers that are expected to belong mostly to households. Whether a telephone number goes into the high-density or medium-density stratum is determined by the number of listed residential numbers in each hundred block, or set of 100 telephone numbers with the same area code, prefix, the first two digits of the suffix and all possible combinations of the last two digits. Numbers that come from hundred blocks with one or more listed household numbers (“1+ blocks,” or “banks”) are put in either the high-density stratum (“listed 1+ blocks”) or medium-density stratum (“unlisted 1 + blocks”). The sampling ratio between listed one-plus block and not-listed one-plus block household density strata in a DSS design is 1.5:1 in which the listed will be sampled at the rate of 1.5 times that of not-listed. Geographic stratification for landline telephones within a state is defined by counties, health districts, cities, zip codes, and/or census tracts.

The cellphone sample for each state is randomly selected from lists of all working cell phone numbers. Cellular telephone interviews are conducted with respondents who answer the number called and are treated as one-person households. Persons who have moved to other states and who have cell phone numbers with area codes/ prefixes from other states are eligible for interview. Data collected from persons who have moved into a state with cell phone prefixes from other states, will be transferred to the appropriate data file at the end of the calendar year. In 2021 a new process to sample cell phones not only by area code, but also by address was introduced. By mid-2021, an assessment of the efficiency of this screening will be available to understand how the new sample has improved targeting of state residents. For the BRFSS accurate state residence is of paramount importance, since optional modules vary by state. In the past out-of-state residents were interviewed, but only core questions were administered. This resulted in missing data for optional modules when data were transferred to the state of residence. The new protocols will alleviate the problem of missing optional module data, but the extent of the improvement is not known at this time.

The field test sample is a smaller sample designed to produce a minimum number of completed responses (300-500) per field test. The field test is conducted after the BRFSS annual meeting, when new questions are voted on by the states. Generally field tests take place in early summer or late spring, annually. The sample is taken from a single state and includes only cell phone numbers. The state which conducts the field test is selected from among those states which have in-house capacity to conduct surveys using their own Computer Assisted Telephone Interview (CATI) systems. States may volunteer to conduct the test on behalf of the BRFSS and report back to their BRFSS colleagues and partners. The field test sample is not designed to be representative, and is not intended for population estimation, but is used to test the questionnaire and related software prior to the launch of the annual survey.

CDC obtains the lists of phone numbers from a private vendor (currently GENESYS Sampling-Marketing Systems Group MSG) and distributes it to the states on a monthly or quarterly basis, according to the state’s preference. Within the calling centers, samples are released in a controlled fashion by replicate. Each replicate is a sub-sample containing 30 telephone numbers. The BRFSS list-assisted method (using numbers that are present in a directory of household phone numbers) has been shown to improve efficiency and reduce unproductive calling among landline numbers. Samples of landline telephone numbers purchased using this method have been prescreened to ensure that they are residential phone numbers. Dedicated FAX and computer lines, ported cell phone numbers, business numbers and other ineligible numbers have been identified prior to implementation of the survey. The list assisted design reduces the number of unproductive screening calls while maintaining sampling weights that are roughly equal.

Summary of Similarities and Differences in Sampling by State
Similarities	Differences
Targets for proportions of cell/landline phone numbers Single sampling vendor	Sampling designs/geostrata Oversampling of targeted populations Quarterly/monthly sampling Split sampling to allow for optimal number of modules in some states One territory with low telephone coverage uses a geographic sample to conduct personal interviews

The BRFSS sample is weighted to state populations and is intended for use as separate statewide datasets. However many researchers aggregate the state data to estimate prevalence at the national level. The PHSB recognizes that there may be differences in national estimates drawn from a national sample and nationwide estimates drawn from a set of state-level samples. For example, although each state’s population is appropriately weighted, the estimated percentage for Hispanics in the aggregated data set for 2015 was15.5%, while a national weighting method would reduce that proportion to 15%, a more accurate representation of national percentages. The potential demographic bias in the aggregated method, therefore, may have implications for health outcomes that may show variations across demographic groups. To control for this potential bias, the national weights could be raked at the national level using as many of the raking dimensions—among those used at the state level—as possible for convergence and stability. In addition, national raking methods could use states as an additional weighting margin to preserve state totals and to reproduce state estimates. PHSB therefore examined a series of reweighting methods using a range of raking margins. Six methods for adjustment in the aggregated dataset were tested using estimates from the National Health Interview Survey (NHIS) as a benchmark. A number of prevalence estimates from chronic conditions, health behaviors and demographic characteristics were tested using these six methods. Although differences in estimates were minor across the tested indicators, the research resulted in a methodology for national weights for the state-based BRFSS. Data users who aggregate data from all states would benefit from the use of these new national weights. However, researchers using data from only a few states would find that the weights associated with state level populations would be better suited to their analyses. Likewise an analysis that used data from BRFSS modules administered to residents in only a few states should use state-level weights rather than a national weight. This research resulted in a peer-reviewed article entitled “National Weighting of Data from the Behavioral Risk Factor Surveillance System (BRFSS)” and was published in BMC Medical Research Methodology. It is available using a link from the BRFSS website or at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5109644/.

This ICR includes a request to have some participants complete the BRFSS online. There will not be online options for participants in 2022, but in 2023-2024, the BRFSS may offer respondents the option to complete the survey online. If the BRFSS adopts this option, the respective change request for the year(s) of online administration will detail the related data collection methods.

2. Procedures for the Collection of Information

Procedures are constructed to produce a coordinated series of state-tailored surveys that are unified by a common purpose, a common reference set of questions (divided into core questions and optional modules), and common protocols for sampling and questionnaire administration. Some flexibility in the content of each state-tailored questionnaire and operations management is allowed within parameters established by the BRFSS cooperative agreement. Participants in the design and implementation of the data collection process include CDC, state health departments; and data collection contractors.

Summary of Steps, Roles, and Responsibilities

CDC and BRFSS awardees participate in an annual discussion (generally in late March or early April) to determine the exact wording of the questions in any part of the BRFSS and vote to adopt (or reject) changes or new questions. Changes to the core, other than editorial changes, are rare, but must also be approved by the states. In some instances, changes are required when questions become outdated. (For example when preventive screening questions refer to testing which is no longer recommended.) Discussions that precede voting include cognitive testing results, field testing results from previous years, question validity testing from other surveys and other validity testing if available. States may question the programs or agencies which propose new items or changes in wording based on their own experiences. In most cases, questions which are adopted for inclusion as optional modules have been used on other surveys, or as state added questions. Changes to the reference set of approved BRFSS questions may be submitted by BRFSS awardees or CDC programs for consideration. The state BRFSS coordinators may also vote to add questions on emerging issues (such as the H1N1 flu questions added in 2009). Once voting is complete, states may not change the wording of any questions in the approved reference set of core questions and optional modules. This discussion also determines the content of the next (calendar) year’s core survey.
A field test of new questions is conducted by a single state with the oversight of the CDC (generally in June or early July). Field testing is conducted in a manner that mimics the full-scale project protocol, to the degree that is feasible, but with only those parts of the questionnaire which are new, have had substantive change or which are proximal to changed/new sections. Field tests are used to identify problems with instrument documentation or instructions, problems with conditional logic (e.g., skip patterns), or other implementation and usability issues. Approximately 500 respondents will be recruited by phone for the field test. Field tests are not cognitive tests, but may identify some problems with question wording or response sets that have not been noted previously. Following the field tests, suggestions for change may be forwarded to the states for review prior to finalization of the questionnaire. Suggestions for question order, interviewer instructions, CATI programming or other protocols may also be made as a result of field testing.
CDC annually compiles the revised reference set of approved BRFSS core questions and approved changes to the optional modules. CDC also produces data processing layouts, while taking state priorities, potential funding, and other practical aspects into consideration. The new BRFSS materials for the next surveillance (calendar) year are then sent to the states, which then may add their own state-added questions that they have designed or acquired. Because states determine which of the optional modules they will include and may also add state added questions, the final questionnaire produced for each state is unique to that state, although all states must include questions from the common core.
Information collection is conducted by telephone interview. CDC provides Computer-Assisted Telephone Interviewing (CATI) programming to states for their use. States may opt to use their own CATI programming software. States may send advance letters to households/persons using address matching to inform potential respondents of their selection and increase response rates. This ICR includes the collection of the BRFSS online. This may be administered in 2023-2024. If the online option is adopted, details of methods will be included in respective years’ change requests.
CDC works with each state to determine its sample size and then communicates sample specifications to the vendor. The vendor will release lists of telephone numbers to each state on a monthly or quarterly basis, as requested. All calls for a given survey month (or quarter) should be completed within the same sample month (or quarter).
BRFSS awardees are responsible for field operations and determine how their data will be collected within BRFSS guidelines (see the current Data Collectors’ Protocol in Attachment 10).. States may collect data using in-house calling centers, hire vendors using RFP procedures, or contract with universities. Data collectors must develop and maintain procedures to ensure respondents’ privacy, assure and document the quality of the interviewing process, and supervise and monitor the interviewers. Files containing phone numbers must be maintained separately from any files containing responses.
States submit de-identified data files to CDC on a monthly basis for cleaning and weighting. CDC returns clean, weighted data files to the state of origin for its use. Through the BRFSS website, CDC also makes cleaned subsets of state data files available for public use, along with information about data quality and analysis, weighting (see Attachment 7, BRFSS Weighting). Detailed information about selected steps in the process is provided below and in Attachment 11, BRFSS Questionnaire Development Process.

Content and Construction of the Annual BRFSS Questionnaire(s)

The BRFSS questionnaire is comprised of an annual standard core which includes questions asked of respondents each year, a rotating core which include questions asked only in even or odd numbered years/three-or four-year cycles, optional modules which includes standardized questions adopted verbatim by the states, and state-added questions which can be individually customized by states. Attachment 4 provides questions for all optional modules and Attachment 3 provides questions for core sections including the periodicity of administration. Attachment 13 provides and example of an annual questionnaire. All questions included in the BRFSS core and optional modules, with the exception of state-added questions, are cognitively tested prior to inclusion in the questionnaire.

Fixed Core Questions: The portion of the questionnaire that is included each year and must be asked by all states. Each year, the core includes questions about emerging or “late-breaking” health issues. After one year, these questions are either discontinued or incorporated into the standard core, rotating core, or optional modules. (See Attachments 3a and 3b.)
Rotating Core Questions: The portion of the questionnaire asked by all states on a two-, three- or four- year basis. In 2021 the number of questions which are cycled in three years increased following a redesign of the core questionnaire (see Attachment 16 for a summary of the BRFSS Core Questionnaire Resign).
Optional Modules: Sets of standardized questions on various topics that each state may select and include in its questionnaire. Once selected, a module must be used in its entirety and asked of all eligible respondents. If an optional module is modified in any way (e.g., if a question is omitted), then the questions will be treated as state-added questions (see below). (See Attachment 4.)
State-added Questions: States may choose to gather data on additional topics related to their specific health priorities through the use of extra questions they choose to add to their questionnaire. The CDC neither reviews, makes suggestions for nor processes data related to state added questions.

All versions of the BRFSS questionnaire include the fixed core. An updated version of each year’s questionnaire will be uploaded to Reginfo.gov. Attachment 13 provides a final version of the 2021 BRFSS Questionnaire including rotating year questions and optional modules available in 2021. Not all modules are used in every year. A list of optional modules used, by state, is provided in Attachment 15. Each year, the previous year’s approved list of optional modules will be used for planning purposes, and the list will be updated as plans are finalized. This listing also illustrates how the states split samples in order to include a wider range of topics on optional modules.

CDC takes care to ensure that all new questions are adequately cognitively tested. The Division of Population Health (DPH) recognizes the need to enhance cognitive testing methods. In 2020, the DPH was approved to conduct cognitive testing under OMB review. This will increase the number of cognitive testing subjects and increase evaluative capacity on all new questions and modifications of questions for use on the BRFSS. In addition, the DPH works with other CDC programs and federal agencies to harmonize question format, wording and response sets whenever possible. In cases where new topics are under development (such as is the case with opioid use surveillance) the DPH is an active partner in cross- agency committees and workgroups to work on the development of new questions.

Call/Interview Guidelines

Data collection follows a suggested BRFSS interviewing schedule. The protocol suggests up to 15 calling attempts for each landline phone number and up to 8 for each cell phone number in the sample, depending on state regulations for calling and outcomes of previous calling attempts. In 2021 the number of attempts was adjusted to allow for efficiencies when phone numbers do not reach potential participants. This results in a cost savings when the history of calling outcomes indicates that it is very unlikely the number will result in a completed interview (see Attachment 10). Some states make calling attempts over the totals suggested by the BRFSS protocol. Although states may have some flexibility in distribution of calling times, in general, surveys are conducted using the following calling occasions:

Conduct 20% of the landline interviews on weekdays (prior to 5:00 pm)
Conduct 80% of the landline interviews on weeknights (after 5:00 pm) and weekends
Conduct cell phone interviews during all three calling occasions (weekday, weeknight and weekend) approximately 30% of cell phone calls on weekend calling occasions.
Change schedules to accommodate holidays and special events
Make weeknight calls just after 5:00 pm
Make callbacks during hours that are not scheduled for other interviews, generally on weekdays
With the exception of verbally abusive respondents, eligible persons who initially refuse to be interviewed may be contacted at least one additional time and given the opportunity to be interviewed. Preferably, this second contact will be made by a supervisor or a different interviewer. Some states have regulations on whether refusals should be called again.
Adhere to respondents’ requests for specific callback times whenever possible

Since response rates are calculated from optimizing calling protocols, there is an incentive to work the sample as efficiently as possible over the course of the month/quarter that it is allocated. Poor use of the sample would be noted by the CDC in the YTD quarterly reports of sample use which are provided on the upload site to the states throughout the year and in the annual summary data quality report on the public website.

Calling Dispositions

States are required to give a final disposition for every number in the sample, usually within the same month of the sample. Each telephone number in the CDC-provided sample must be assigned a final disposition code to indicate a particular result of calling the number:

A completed or partially completed interview or
A determination that:
- A household was eligible to be included but an interview was not completed or
- A telephone number was ineligible or could not have its eligibility determined.

An interview is considered to be a partial complete if respondents are asked questions which are used in weighting (approximately half-way through the core BRFSS questionnaire). These variables include, race, ethnicity, sex, marital status, education, home ownership, type of phone ownership (i.e. landline only, cell phone only or dual user), and geographic/ (sub) state region. If values on weighting variables are not entered due to respondent refusal, imputed values will be generated and used only to assign weights.

The final disposition codes are then used to calculate response, cooperation and refusal rates. The distribution of individual disposition codes and the rates of cooperation, refusal, and response are published annually in the Summary Data Quality Reports. The BRFSS uses standards set by the American Association of Public Opinion Research (AAPOR) to determine disposition codes and response rates. All BRFSS disposition codes and rules for assigning disposition codes are provided in Attachment 10: Data Collectors’ Protocol. Data collectors must adhere to the rules for assigning disposition codes and train and monitor interviewers in the use of specific dispositions.

Procedures to Promote Data Quality and Comparability

In order to maintain consistency across states and allow for state-to-state comparisons, the BRFSS sets standard protocols for data collection which all states are encouraged to adopt with technical assistance provided by CDC. The following items are included in the BRFSS survey protocol:

All states must ask the core questions without modification. States may choose to add any, all, or none of the optional modules and state-added questions after the core component. Interviewers may not offer information to respondents on the meaning of questions, words or phrases beyond the interviewer instructions provided by CDC and/or the state BRFSS coordinators.
Interviewers should be trained specifically for the BRFSS and retrained each year.
Systematic, unobtrusive electronic monitoring is a routine and integral part of monthly survey procedures for all interviewers. States may also use callback verification procedures to ensure data quality. Unless electronic monitoring of 10% of all interviews is being routinely conducted, a 5% random sample of each month’s interviews must be called back to verify selected responses for quality assurance.

General calling rules, listed below, are established by the BRFSS and states are encouraged to adhere to them whenever possible. It is understood that the calling rules are not universally applicable to each state.

All cellular telephone numbers must be hand-dialed.
If possible, calls made to non-English speaking households are assigned the interim disposition code of 5330 (household language barrier), should be attempted again with an interviewer who is fluent in the household language (e.g. Spanish).
States should maximize calling attempts as outlined in Attachment 10. The maximum number of attempts (15 for landline telephone and 8 for cellular telephone) may be exceeded if formal appointments are made with potential respondents.
Calling attempts should allow for a minimum of 6 rings and up to 10 rings if not answered or diverted to answering devices.

The BRFSS produces a Summary Data Quality Report which is published annually on its website. The report includes information on calling attempts, sample quality, rates of completion and response. The 2019 Summary Data Quality Report is provided in Attachment 9.

3. Methods to Maximize Response Rates and Deal with Nonresponse

The vendor prescreens landline phone numbers to determine whether they are assigned to businesses or are nonworking.

The sampled phone numbers are called by each state or their data collector and screened to determine whether the numbers dialed are residential numbers and, for those that are, eligible individuals are identified.

As noted in the preceding section, the BRFSS uses a number of techniques to deal with nonresponse. These include providing the interview in languages other than English, creating a number of call back protocols designed to convert refusals, and alternating times and days of calling attempts. In addition, the BRFSS advises states to make use of caller ID to inform potential respondents that state health departments are making the calls. States may also use advance letters to inform respondents that their household or cell phone number has been selected to participate in the BRFSS for landline interviews. Potential respondents are informed of the purpose of the call and the importance of their response early in the screening script. Experienced interviewers are used for callbacks when respondents initially refuse to take part in the survey. Hard refusals (where potential respondents state that they are not interested in completing the interview) are not called back.

States must maintain training for all interviewers involved in the BRFSS. Issues related to response rates are discussed in large annual meetings of the data collectors. Data collectors also participate in bimonthly conference calls organized by the CDC to discuss best practices, and share experiences.

The CDC and the states have conducted a number of pilot studies in recent years to identify methods that might improve response rates and alleviate potential nonresponse bias. Current pilots include two projects on post data collection analysis methods to examine small area estimation protocols, and three pilots on alternative data collection methods using internet panels and RDD push -to-web, and address-based sampling. The BRSS is also testing the validity of self-reported measures by sampling subjects from a large electronic health record system and administering a subset of the BRFSS. Self-reports will then be compared to information within the electronic health records.

In 2018 and 2019, a two feasibility tests were conducted. One of these tested cell phone respondents who are sampled by one state but who live in another could be diverted to the web in order to complete state-specific modules which otherwise would have resulted in missing data. (Currently these respondents are interviewed but only core questions are asked). Another tested the use of address-based sample in a single state. Also in 2029, the BRFSS conducted research to examine the impact of moving to an all cell phone sample. This research included analysis of the previous three years of data after removing the landline portion of the sample, reweighting cell phone interviews and comparing prevalence estimates with the landline/cell phone sample. In addition, in 2020-2021, the DPH completed a redesign of the core questionnaire. The purpose of the redesign was to establish standards for core questions, reduce the length of the core and update question formats. The final report on the BRFSS Redesign is provided in Attachment 16.

Although response rates overall for telephone surveys are declining, the BRFSS maintains a relatively high level of response when compared to other surveys. Response rates, cooperation rates, and refusal rates for BRFSS are calculated and published annually using standards set by the American Association of Public Opinion Research (AAPOR) [1]. The BRFSS calculates response rates using AAPOR Response Rate #4, which is in keeping with rates provided by BRFSS prior to 2011 using rates from the Council of American Survey Research Organizations (CASRO) [2]. Attachment 9 provides the most recent state by state information on response rates for landline and cell phone samples of the BRFSS.

Based on the guidelines of AAPOR, response rate calculations include assumptions of eligibility among potential respondents/households that are not interviewed. The BRFSS calculates “likely eligible” phone numbers using the proportions of eligible households among all phone numbers where eligibility has been determined. This “eligibility factor” appears in calculations of response-, cooperation-, resolution-, and refusal rates.

The calculations of calling outcome rates are based on final disposition codes that are assigned after all calling attempts have been exhausted. The BRFSS may make up to 15 attempts to reach respondents prior to assigning a final disposition code. The BRFSS uses a single set of disposition codes for both landline and cell phones, adapted from standardized AAPOR disposition codes for telephone surveys. A few disposition codes apply only to landline telephone or cellular telephone sample numbers. For example, answering-device messages may confirm household eligibility for landline telephone numbers but are not used to determine eligibility of cellular telephone numbers. Disposition codes reflect whether interviewers have completed or partially completed an interview (1000 level codes), determined that the household was eligible without completing an interview (2000 level codes), determined that a household or respondent was ineligible (4000 level codes), or was unable to determine the eligibility of a household and/or respondent (3000 level codes). The BRFSS uses an overlapping sample in that persons who are reached on cell phones are eligible even if they also have landline phones and vice versa. If a number in the landline sample reaches a person on a cellphone that person is not within the landline sample and therefore not eligible. If the same number appears in the cell phone sample then the person would be eligible to be interviewed. Attachment 10 provides disposition codes used by the BRFSS and it notes the instances where codes are used only for landline telephone or cellular telephone sample numbers. Assignment of disposition codes may vary slightly from one state to another, but all numbers must be assigned a final (not an interim) disposition code prior to data submission. Factors affecting the distribution of disposition codes by state include differences in telephone systems, sample designs, surveyed populations, and data collection processes.

The table below illustrates the categories of disposition codes which are used to calculate outcome rates. The totals for each category are then used in the formulae below to determine rates of completion, cooperation, refusal, and response according to AAPOR standards and guidelines. Response rates are provided by landline and cell phone. An overall response rate is also calculated for each state, weighted by the proportions of the sample which are landline and cell phone. Details for calculations of these rates are published annually for each state on the BRFSS website at http://www.cdc.gov/brfss/annual_data/annual_data.htm .

Table 2 Landline and Cellular Telephone BRFSS Disposition Codes
Category	Disposition Code Definitions	Formulae Abbreviation
Completed interviews	1100+1200	COIN
Eligible	1100+1200+2111+2112+2120+2210+2220+2320+2330	ELIG
Contacted eligible	1100+1200+2111+2112+2120+2210+2320+2330	CONELIG
Terminations and refusals	2111+2112+2120	TERE
Ineligible phone numbers	All 4000 level disposition codes	INELIG
Unknown whether eligible	All 3000 level disposition codes	UNKELIG
Eligibility factor	ELIG/(ELIG + INELIG)	E

Eligibility Factor

E = ELIG/ (ELIG + INELIG)

The Eligibility Factor is the proportion of eligible phone numbers from among all sample numbers for which eligibility has been determined. The eligibility factor, therefore, provides a measure of eligibility that can be applied to sample numbers with unknown eligibility. The purpose of the eligibility factor is to estimate the proportion of the sample that is likely to be eligible. The eligibility factor is used in the calculations of refusal and response rates. Separate eligibility factors are calculated for landline telephones and cellular telephone samples for each state and territory.

Resolution Rate

((ELIG + INELIG) / (ELIG+INELIG+UNKELIG))*100

The Resolution Rate is the percentage of numbers in the total sample for which eligibility has been determined. The total number of eligible and ineligible sample phone numbers is divided by the total number of phone numbers in the entire sample. The result is multiplied by 100 to calculate the percentage of the sample for which eligibility is determined. Separate resolution rates are calculated for landline telephone and cellular telephone samples for each state and territory.

Interview Completion Rate

(COIN / (COIN + TERE)) * 100

The Interview Completion Rate is the rate of completed interviews among all respondents who have been determined to be eligible and selected for interviewing. The numerator is the number of complete and partially completed interviews. This number is divided by the number of completed interviews, partially completed interviews, and all break offs, refusals, and terminations. The result is multiplied by 100 to provide the percentage of completed interviews among eligible respondents who are contacted by interviewers. Separate interview completion rates are calculated for landline telephone and cellular telephone samples for each state and territory.

Cooperation Rate

(COIN / CONELIG) *100

The AAPOR Cooperation Rate is the number of complete and partial complete interviews divided by the number of contacted and eligible respondents. The BRFSS Cooperation Rate follows the guidelines of AAPOR Cooperation Rate #2. Separate cooperation rates are calculated for landline telephone and cellular telephone samples for each state and territory.

Refusal Rate

(TERE / (ELIG + (E * UNKELIG))) * 100

The BRFSS Refusal Rate is the proportion of all eligible respondents who refused to complete an interview or terminated an interview prior to the threshold required to be considered a partial interview. Refusals and terminations (TERE) are in the numerator, and the denominator includes all eligible numbers and a proportion of the numbers with unknown eligibility. The proportion of numbers with unknown eligibility is determined by the eligibility factor (E; described above). The result is then multiplied by 100 to provide a percentage of refusals among all eligible and likely to be eligible numbers in the sample. Separate refusal rates are calculated for landline telephone and cellular telephone samples for each state and territory.

Response Rate

(COIN / ((ELIG + (E * UNKELIG))) * 100

A Response Rate is an outcome rate with the number of complete and partial interviews in the numerator and an estimate of the number of eligible units in the sample in the denominator. The BRFSS Response Rate calculation assumes that the unresolved numbers contain the same percentage of eligible households or eligible personal cell phones as the records whose eligibility or ineligibility are determined. The BRFSS Response Rate follows the guidelines for AAPOR Response Rate #4. It also is similar to the BRFSS CASRO Rates reported prior to 2011. Separate eligibility factors are calculated for landline telephone and cellular telephone samples for each state and territory and a combined Response Rate for landline telephone and cellular telephone also is calculated. The combined landline telephone and cellular telephone response rate is generated by weighting to the respective size of the two samples. The total sample equals the landline telephone sample plus cellular telephone sample. The proportion of each sample is calculated using the total sample as the denominator. The formulae for the proportions of the sample are found below:

P₁ = TOTAL LANDLINE SAMPLE /
(TOTAL LANDLINE SAMPLE + TOTAL CELL PHONE SAMPLE);

P₂ = TOTAL CELL PHONE SAMPLE /
(TOTAL LANDLINE SAMPLE + TOTAL CELL PHONE SAMPLE);

The formula for the Combined Landline Telephone and Cellular Telephone Weighted Response Rate, therefore, is described below:

COMBINED RESPONSE RATE=
(P₁ * LANDLINE RESPONSE RATE) + (P₂ * CELL PHONE RESPONSE RATE).

4. Tests of Procedures or Methods to be Undertaken

BRFSS protocols have been adapted over time to meet the needs of the data collection process and maximize response rates while minimizing respondent burden. The BRFSS continually assesses its methods and procedures through comparisons with industry standards, consultation with BRFSS coordinators and other experts in the field, and real-world experience and feedback from the BRFSS data collectors. The PHSB convenes an Expert Panel meeting approximately every three years. The last Expert Panel meeting was held in 2017. An Expert Panel meeting scheduled for 2020 was postponed due to the COVID pandemic. All questions which are included in the BRFSS are cognitively tested, with the exception of state-added questions. Field tests are also conducted each year to ensure that the questionnaire is ready for fielding. New questions are tested each year before adoption, mostly as changes to existing or additional optional modules. The methods by which questions are adopted are provided in Attachment 11. As this document indicates, SMEs from CDC and other federal agencies, state health department representatives and survey experts are involved in the process of question development. Many of the questions which are included in the BRFSS appear on other surveys including the National Health Interview Survey (NHIS), the National Adult Tobacco Survey (NATS), the National Health and Nutrition Examination Survey (NHANES), the National Immunization Survey (NIS) and others. The use of identical or similar questions is advantageous in that it allows researchers to make comparisons across different samples, different geographic areas or over time. The BRFSS website maintains a listing of validity studies conducted by CDC staff (http://www.cdc.gov/brfss/publications/ methodology/data_qvr.htm) and other researchers (http://www.cdc.gov/brfss/publications/ methodology/mvr.html) on validity and reliability tests of the BRFSS, its methods and questions.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

CDC personnel are responsible for all statistical aspects of the BRFSS including data analyses and reporting. The following staff members are primarily responsible for BRFSS data reporting.

Name	Title	Phone	E-mail
Machell Town	Population Health Surveillance Branch Chief	770-488-2533	mpt2@cdc.gov
William Garvin	Team Lead, survey operations	770-488-2459	wsg1@cdc.gov
Carol Pierannunzi	Senior Survey Methodologist	770-488-4609	ivk7@cdc.gov

In addition staff members attend the annual meetings of the American Association of Public Opinion Research (AAPOR) and the Joint Statistical Meetings (JSM) at which experts provide guidance, comments and suggestions on staff methodological research presentations.

References

1. The American Association for Public Opinion Research. 2011. Standard Definitions: Final dispositions of case codes and outcome rates for surveys. 7^th edition.
http://www.aapor.org/AM/Template.cfm?Section=Standard_Definitions2&Template=/CM/ContentDisplay.cfm&ContentID=3156

2. The Council of American Survey Research Organizations. 2013. Code of standards and ethics for market, opinion, and social research http://c.ymcdn.com/sites/ www.casro.org/resource/resmgr/code/september_2013_revised_code.pdf?hhSearchTerms=%22casro+and+response+and+rate%22

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	cww6
File Modified	0000-00-00
File Created	2021-10-08