Best Practices

Government & Academic Research

Best Practices for ICR Supporting Documentation for KnowledgePanel® Surveys
Prepared by J. M. Dennis
December 1, 2010
Federal agencies develop Information Collection Requests (ICRs) that are submitted to the Office of
Management and Budget (OMB) for new surveys that will impose a reporting burden on the U.S.
public as defined under the Paperwork Reduction Act of 1974 and as further required under the socalled Data Quality Act.
This document is an attempt to provide textual support to Federal agencies proposing to use
KnowledgePanel sample for studies about which an ICR will be submitted for OMB review.
Because OMB’s stated objection to KnowledgePanel is the cumulative response rate (~10%), the
information below emphasizes nonresponse bias measurement. KN has led the survey research
industry in providing standards for response reporting for web-based surveys and has disclosed its
response rates in Public Opinion Quarterly.1
Best Practices for ICRs involve descriptions of these tasks:
 Sampling
 Data Collection Procedures
 Nonresponse Bias Measurement
Statistical weighting is not covered in this document as KN’s standard procedures have been
considered sufficient by OMB, in past ICR reviews.
Summary
Key Justification Points for using KnowledgePanel sample in Federal information collections:
 Uses probability-based sampling consistent with traditional sampling theory;
 Provides single mode of data collection (web based), obviating potential for data collection
mode effects;
 Includes sample coverage of non-Internet households (via computer device and ISP provision)
and Spanish-language households;
1

See Callegaro, Mario & Disogra, Charles (2008). Computing Response Metrics for Online Panels. Public Opinion
Quarterly. 72(5) pp. 1008-1031.
Knowledge Networks, Inc.
phone: (650) 289-

-









Supports multimedia surveys/stated-preference methodology through the web mode of data
collection;
Supports targeted sampling for studies of subpopulations;
Supports longitudinal information collections with high follow-up cooperation rates;
Reduces reporting burden on the U.S. public by re-using previously consented sample and by
eliminating the re-asking of previously asked survey questions;
Benefits from informed consent having already been acquired from research subjects;
Supports cost-effective measurement of nonresponse bias;
Achieves within-panel survey cooperation rates of 70% and higher, minimizing the potential
of nonresponse bias from self-selection into a specific study.

Best Practices for ICRs, to reduce it to the essentials, are as follows:
 Survey-Specific Sampling
o Draw the KnowledgePanel sample exclusively from the 2008 and later cohorts sourced
from Address-Based Sampling (ABS);
o Include KnowledgePanel Latino;
o Use modified stratified sampling with completion propensity adjustment, a sampling
selection procedure that takes into account between-group differences in survey completion
rates to KnowledgePanel online surveys.


Data Collection Procedures to Maximize Within-Panel Survey Cooperation Rate
o Send pre-notification email to sampled respondents 2-3 days before sending the actual
survey invitation;
o Field Survey for two to three weeks;
o Include cash-equivalent incentives of $5 to $10 for longer surveys (25 minutes or longer);
o Use cash-equivalent incentives selectively to target nonresponders late in the field period;
o Use email reminders and telephone-based reminder calls with nonresponders.



Nonresponse Bias Measurement
o To identify possible self-selection effects at the panel recruitment and retention stages,
statistically compare demographic and household characteristics of (i) the sample invited to
the KN panel and (ii) the subset of actual survey participants (i.e., the estimating sample);
o To identify possible within-panel self-selection effects, statistically compare demographic
and household characteristics of (i) the sample invited to a specific online panel survey and (ii)
the panelists participating in the survey on which the estimates are based.
o Benchmark KN panel survey estimates by placing benchmarking survey questions on the
KN panel survey instrument, and comparing the surveys estimates to benchmarks from goldstandard surveys (e.g., NHIS, GSS, SIPP, etc); the selection of the survey questions should be
informed by a theory that the survey measures are related to the study topics of interest (e.g.,
political ideology measure in a study on attitudes towards an government regulation of an
environmental good);
o (Optional, but not essential) Measure nonresponse bias directly through a technique
sometimes called “double sampling” or a “nonresponse follow-up survey.” The procedure is to
randomly subsample households that initially were selected to join KnowledgePanel but
refused to do so, as well as subsample households that agreed to join the panel but

Knowledge Networks, Inc.
phone: (650) 289-

-

subsequently refused to participate. In this approach, a subset of items from the main survey
questionnaire is administered to the selected samples by either a mail survey or web survey.
In the next two sections, more information is provided on the Sampling and Nonresponse Bias
Measurement Tasks.
Sampling
1. Restrict Panel Samples to ABS-Sourced Respondents
As of December, 2010, approximately 40% of the active KnowledgePanel households are sourced
from a sample frame called “Address-Based Sampling,” while the remainder is sourced from randomdigit dialing (RDD). For the information collections requiring OMB review, we recommend that the
KnowledgePanel sample be restricted to the ABS-sourced sample in order to provide the most
representative sample possible. ABS-sourced sample is advantaged by providing improved
representation of certain segments, particularly young adults, cell-phone-only households, and
nonwhites. In addition, by restricting the sample to ABS, valuable ancillary person-level and
household-level characteristic data are available for the ABS sample units, making possible a
descriptive comparison of the characteristics of the entire invited sample and the subset of survey
participants.
Between 1999 and April 2009, KnowledgePanel’s probability-based recruitment had been based
exclusively on a national RDD frame. In April 2009, Knowledge Networks added the ABS frame (to
supplement the RDD frame) in response to the growing number of cell-phone-only households
(CPOHHs) that are outside of the RDD frame. In January 2010, Knowledge Networks transitioned
completely to ABS-sourced panel recruitment and ceased recruitment using RDD and telephone
methods, with the exception of some Spanish-language telephone-based recruitment to support
KnowledgePanel Latino.
ABS involves probability-based sampling of addresses from the U.S. Postal Service’s Delivery
Sequence File (DSF). Post office boxes and rural route addresses are included.
Business and institutional addresses (i.e., dormitories, nursing homes, group homes, jails, etc.) are
removed from the frame, as is military housing. Also removed are those multi-dwelling residential
structures that have only a single address (called a drop point address) and for which there is no unitlevel identifying information (mail is internally distributed).
Randomly sampled addresses are invited to join KnowledgePanel through a series of mailings.
Telephone follow-up calls are made to nonresponders when a telephone number can be matched to the
sampled address. Invited households can join the panel by one of several means:
 Completing and mailing back an acceptance form in a postage-paid envelope;
 Calling a toll-free hotline staffed by bilingual recruitment agents; or
 Going to a dedicated KN recruitment website and completing the recruitment information on
line by using a unique PIN provided in the advance letter.

Knowledge Networks, Inc.
phone: (650) 289-

-

After initially accepting the invitation to join the panel, respondents are then “profiled” online,
answering key demographic questions about themselves. This profile is maintained using the same
procedures established for the RDD-recruited research subjects. Respondents not having an Internet
connection are provided a laptop computer and free Internet service. Respondents sampled from the
ABS frame, like those from the RDD frame, are provided the same privacy terms and protections to
the extent permitted by law that we have developed over the years and that have been reviewed by
dozens of Institutional Review Boards.
The key advantage of the ABS sample frame is that it allows sampling of almost all U.S. households.
An estimated 97% of households are “covered” in sampling nomenclature. Regardless of household
telephone status, they can be reached and contacted via the mail. Second, ABS pilot project has other
advantages beyond the expected improvement in recruiting young adults from CPOHHs, such as
improved sample representativeness for minority racial and ethnic groups and improved inclusion of
lower educated and low-income households.
2. Include KnowledgePanel Latino Sample in the Panel Sample Draws
To achieve improved sample coverage, inclusion of the Spanish-language dominant households can be
important for certain studies. Approximately 4% of the U.S. adult population is not covered for a
general population study when the sample rule excludes Spanish-speaking adults who are
insufficiently literate in English for self-administered English-language surveys.
3. Use Modified Stratified Sampling with Completion Propensity Adjustment
Certain demographic segments have survey cooperation rates that are predictably lower or higher than
average. If these groups are sampled for a panel survey in proportion to their share of the U.S.
population, then the unweighted share of the interviews from the low-cooperation-rate groups will be
less than their share of the U.S population, while the unweighted share of the high-cooperation rate
groups will be higher than their share of the U.S. population.
For studies requiring an ICR, Knowledge Networks employs a refinement to the KN standard protocol
for drawing samples from the panel (see U.S. Patent No. 7,269,570). The modified approach is
designed to improve further the demographic similarities between the completed panel interviews and
the U.S. Census population benchmarks by factoring estimated survey completion rates for key
demographic groups into the sample draw selection probabilities. Knowledge Networks has ample
experiential data upon which to calculate reliable completion propensities for specific demographic
groups. Essentially, by oversampling groups that have consistently lower completion rates and
undersampling groups that tend to have higher rates, the valid completed interviews can mirror the
Census demographic benchmarks more closely. This approach can be employed when it is essential to
minimize the range of a study’s post-stratification weights and the resultant design effect. This
modified sampling approach is executed by first constructing 576 cells using the following six
variables and then adjusting the fielded sample size for each cell by the response propensity for each
cell: Age (18-24, 25-34, 35-44, 45-54, 55-64, 65+); Education (Less than high school, High school,
Some college, College degree +); Hispanic (Hispanic, Non-Hispanic); Race (White, Black, Other);
Gender (Male, Female); Household income (Less than $75K, $75K+).

Knowledge Networks, Inc.
phone: (650) 289-

-

Nonresponse Bias Measurement
This section will describe basic statistical tests of nonresponse bias measurement, and one direct
measurement technique. A summary of past research on nonresponse bias measurement in the context
of Knowledge Networks surveys is available.2
1. Statistical Comparison of Demographic and Household Characteristics of the Sample Frame
versus the Subset of Actual Survey Participants
This approach attempts to measure self-selection bias among the estimating sample making up the
completed interviews. The approach is possible only for general population studies of U.S. adults
where the interview sample size requirement is 5,000 interviews or less (subject to increase as the
ABS-sourced sample increases over time). The approach works best when limiting the KN panel
sample draw to ABS-sourced panelists.3 ABS sourcing is important because a specific benefit of
address-based sampling: the ability to append to the sample frame many person-level and householdlevel ancillary data associated with an address. Commercial databases (e.g., Experian, infoUSA, and
Acxiom) are used to append to the sample frame observed and modeled information at various levels
of aggregation. These same ancillary data are also used to analyze nonresponse bias by comparing the
ancillary data available for the entire sample invited to join the KnowledgePanel and the small subset
of recruited study participants that participate in any given study. If the study requires a general
population adult sample, the expectation is that the estimating sample of completed interviews will
have marginal distributions on person-level and household-level characteristics that are statistically
similar to the distributions of the entire invited sample.
Statistical comparisons for specific studies can be made between the total invited sample for the panel
recruitment and the estimating sample for these variables:
Household level
Number of adults in the household
Presence of children (yes, no)
Home ownership (own, rent)
Household income (12 levels recoded to <$25K. $25-$49K, $50-$74K, $75K+)
Person level
Marital status (married, single)
Education of head of household (less than high school, high school, some college, BA, higher)
Age of householders
Race/Ethnicity (White, African American, Hispanic, Other)

2

See Dennis, J. Michael. 2010. KnowledgePanel®: Processes & Procedures Contributing to Sample
Representativeness & Tests for Self-Selection Bias. The paper may be downloaded from
http://www.knowledgenetworks.com/ganp/reviewer-info.html.
3
The approach is technically possible when using the RDD-sourced portion of KnowledgePanel; however, the
ancillary data attached to the sample frame will have more unit-level missing data.
Knowledge Networks, Inc.
phone: (650) 289-

-

The approach is explained in more detail in an article by DiSogra, Dennis, and Fahimi in the 2010
Proceedings of the Joint Statistical Meetings.4 An aggregate error rate can be calculated as the sum of
the differences in the distributions between the expected values from the total invited sample
compared to the actual values (from the estimating sample of completed interviews).
2. Statistical Comparison of Demographic and Household Characteristics of the Survey
Participants versus the Non-Responders
For studies requiring ICRs, the within-panel survey cooperation rate will be less than 100%. There is
the potential for self-selection bias at the stage of inviting KN panelists to participate in an actual
survey. This technique attempts to identify nonresponse resulting from a survey cooperation or return
rate of 70% to 80%. It involves a simple comparison using the approximately 20 person-level and
household-level characteristics (available on all KN panelists). The sample invited to participate is
compared to the sample that does participate. If the survey topic is preponderantly more attractive to
some groups rather than others, this technique will identify such patterns.
3. Benchmarking KN Panel Survey Estimates
Benchmarking KN panel survey estimates by placing benchmarking survey questions on the KN panel
survey instrument, and comparing the surveys estimates to benchmarks from gold-standard surveys
(NHIS, GSS, SIPP, etc); the selection of the survey questions should be informed by a theory that the
survey measures are related to the study topics of interest (e.g., political ideology measure in a study
on attitudes towards an government regulation of an environmental good). For more examples of
benchmarking studies, see Dennis, J. Michael. 2010, “KnowledgePanel®: Processes & Procedures
Contributing to Sample Representativeness & Tests for Self-Selection Bias,”
http://www.knowledgenetworks.com/ganp/reviewer-info.html.
A limitation of this approach is that usually the benchmarking data were not collected by the online
mode of data collections but instead by in-person interviewing. As a result, data differences observed
in the KnowledgePanel estimates and those from the benchmarking survey could the result of the
mode differences (presence versus absence of an interviewer). Differences in mode is a hypothesized
to be factor accounting for KnowledgePanel estimates between different on some items from the
General Social Survey.5
4. Direct Measurement of Nonresponse Bias
Direct measurement of nonresponse bias is infrequently undertaken because of its cost and also
because of a concern that it will introduce an additional source of error.
The technique is sometimes called “double sampling” or a “nonresponse follow-up survey.” The
procedure is to randomly subsample households that initially were selected to join KnowledgePanel
but refused to do so, as well as subsample households that agreed to join the panel but subsequently
4

The article may be downloaded from http://www.knowledgenetworks.com/ganp/reviewer-info.html. The citation is
DiSogra, Charles, J. Michael Dennis, and Mansour Fahimi. 2010. On the Quality of Ancillary Data Available for
Address-Based Sampling. Conference Proceedings of the 2010 Joint Statistical Meetings.
5
See Smith, Tom W., and J. M. Dennis. 2005. Online Versus In-Person: Experiments with Mode, Format, and
Question Wordings. Public Opinion Pros. December issue. Available under "Past Issues" at
http://www.publicopinionpros.norc.org/index.asp.
Knowledge Networks, Inc.
phone: (650) 289-

-

refused to participate. In this approach, a subset of items from the main survey questionnaire is
administered to the selected samples by either a mail survey or web survey.
Additional error can be introduced by this approach as a result of the need to use more than one mode
of data collection in order to achieve a satisfactory refusal conversion rate. Because the doublesampling approach is premised on the need to interview those who already refused to participate or
else constitute non-contacted households, it is common to supplement the web mode of data collection
with telephone-based and mail-based interviews. As a consequence, the supplemental interviews
obtained in the nonresponse follow-up interviews are from different modes, introducing measurement
differences that may be entirely attributable to the mode of data collection. The result is an inability to
isolate the cause of estimation differences resulting from sample source (KN panel recruits, KN panel
recruitment nonresponders) versus mode of data collection (online, mail, telephone, and in-person).
For more discussion and examples, see Dennis, J. Michael. 2010, “KnowledgePanel ®: Processes &
Procedures Contributing to Sample Representativeness & Tests for Self-Selection Bias,”
http://www.knowledgenetworks.com/ganp/reviewer-info.html.

Knowledge Networks, Inc.
phone: (650) 289-

-