Download:
pdf |
pdfSurvey Response Rate and Bias Results
from a Trial of Pre-notification Letters :
A Report to the Office of Management
and Budget on the National Survey on
Recreation and the Environment (NSRE)
The Office of Management and Budget has expressed concern that low response rates for the
NSRE and other public surveys are a primary cause for non- response bias. To increase response
rates, OMB required a test of prenotification letters, increased numbers of call backs to persons
not reached, and a 2-question quick survey of refusing individuals to gain information for use in
designing future NSRE surveys. This report provides in-depth analysis of the results of testing
prenotification, more call backs and refuser questions. Response rates and apparent parameter
estimate bias are the primary criteria for evaluating results.
Submitted by the US Forest Service, National Oceanic and Atmospheric Administration,
University of Georgia, and University of Tennessee.
December 6, 2006
Survey Response Rate and Bias Results
from a Trial of Pre-notification Letters :
A Report to the Office of Management
and Budget on the National Survey on
Recreation and the Environment (NSRE)
By:
Vernon Leeworthy1
Stanley Zarnoch
H. Ken Cordell
Gary T. Green
J. Mark Fly
Rebecca Stephens
1
Dr. Vernon Leeworthy, Chief Economist, NOAA National Ocean Service, Silver Spring, MD; Dr. Stanley
Zarnoch, Research Scientist/Mathematical Statistician, USDA Forest Service, Asheville, NC; Dr. H. Ken
Cordell, Pioneering Scientist, USDA Forest Service, Athens, GA; Dr. Gary T. Green, Assistant Professor,
University of Georgia, Warnell School of Forestry, Athens, GA; Dr. J. Mark Fly, Professor, and Ms.
Rebecca Stephens, Senior Research Assistant, University of Tennessee at Knoxville, Human Dimensions
Research Lab, Knoxville, TN.
Table of Contents
Page
I. Introduction: Overview of the Assessment ............................................................ 1
II. Assessment of Pre-notification Letters: Response Rates, Sample
Representativeness and Non Response Bias .................................................... 4
1. Did pre-notification letters increase response rates? ...................................... 4
2. Is there a relationship between response rates and sample
representativeness ......................................................................................... 6
3. Does improving response rates with pre-notification letters improve
representativeness of samples compared with “standard RDD”
sampling? ..................................................................................................... 12
4. Is there a relationship between response rates, sample
representativeness and non response bias? .............................................. 12
5. Is the non response bias significant? ............................................................... 18
6. Can sample weighting be used to adjust for non response bias? ................. 23
7. What is the impact of pre-notification letters on non response bias? ......... 23
8. Are the benefits of pre-notification letters worth the added costs? ............. 24
III. Assess Refusals and Non Response Bias ...............................................................
1. Did the refusal letters increase response rates (refusal conversions)? ........
2. Is there a difference in sample representativeness as a result of
increasing response rates (refusal conversions)? ....................................
3. Is there a relationship between socioeconomic/demographic
factors and activity participation? ............................................................
4. Is there a significant difference between estimates of activity
participation rates for the No Letter and Letter sample groups? .........
25
26
26
31
32
5. Do the letters to refusals decrease non response bias? .................................. 38
6. Are the benefits of the letters to refusals worth the added costs? ................ 38
IV. Overall Conclusions ................................................................................................ 45
References ...................................................................................................................... 46
i
List of Tables
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
Page
Comparative profiles: Census versus samples and response rates
by sample groups .......................................................................................... 4
Differences (Sample-Census) between Census and sample
profiles and response rates ........................................................................... 7
Differences between sample profiles. ................................................................ 8
Activity participation rates by sample group: Unweighted samples ........... 13
Activity participation rates by sample group: Weighted samples ............... 13
Definitions of variables included in Logit equations ..................................... 14
Estimated participation functions by activity: Logit equations ................... 16
Tests (P-values) on the main effects in the Logit participation
models based on the Wald Chi-Square test .............................................. 17
Differences in unweighted and weighted estimates of activity
participation rates: Full sample ................................................................. 19
Differences in unweighted and weighted estimates of activity
participation rates: Pre-notification letters .............................................. 19
Differences in unweighted and weighted estimates of activity
participation rates: Standard RDD – No pre-notification letters ........... 20
Differences in weighted estimates of activity participation rates:
Comparison pre-notification with standard RDD – No
pre-notification letters ................................................................................ 22
Comparison on mean activity participation rates between
pre-notification letter sample and the standard RDD sample:
Weighted data – difference approach ....................................................... 22
Comparison of activity participation rate estimates NSRE
2000-2001 vs. 2005 ....................................................................................... 23
Comparative profiles for refusal conversions: Census versus
samples and response rates by sample groups ......................................... 27
Refusal conversion differences (sample-Census) between Census
and sample profiles and response rates .................................................... 29
Refusal conversion differences between sample profiles ............................... 30
Refusal conversion activity participation rates by sample group:
Unweighted samples .................................................................................... 31
Refusal conversion activity participation rates by sample group:
Weighted samples. ...................................................................................... 31
Refusal conversion estimated participation functions by activity:
Logit equations ............................................................................................ 33
Tests (P-values) on the main effects in the refusal conversion logit
participation models based on the Wald Chi-Square test ....................... 35
Differences in refusal conversion unweighted and weighted estimates
of activity participation rates: All refusal conversions ........................... 35
Differences in refusal conversion unweighted and weighted estimates
of activity participation rates: No letters .................................................. 36
Differences in refusal conversion unweighted and weighted estimates
of activity participation rates: Letters ...................................................... 36
ii
List of Tables (continued)
25.
26.
27.
28.
29.
30.
31.
Page
Differences in weighted estimates of activity participation rates:
Comparison of refusal conversions with and without letters .................
Comparison of mean activity participation rates between
refusal conversions with and without refusal letters:
Weighted data – difference approach .......................................................
Refusal estimated participation functions for walking: logit question ........
Comparative profiles for refusals for Census division ..................................
Refusal differences (sample-Census) between Census and sample
profiles for Census division ........................................................................
Differences between refusal sample Census division profiles .......................
Reasons given by refusals for not participating in survey ............................
iii
37
37
39
40
41
44
45
List of Figures
Page
1.
Assessment efforts to address non response bias in NSRE 2005 .................... 3
2.
NSRE 2005 sample groups ................................................................................. 3
3.
Sample groups for refusal letter assessment .................................................. 26
4.
Sample groups for non respondent/refusal analysis ...................................... 40
iv
I. INTRODUCTION: Overview of the Assessment
In the review/approval process under the Paperwork Reduction Act (PRA), the U.S. Office
of Management and Budget (OMB) required several additional procedures for sampling and
conducting the NSRE survey (National Survey on Recreation and the Environment). The
current OMB Approval Number is 0596-0127; Expiration Date: 8/31/2007. NSRE is a
national random digit dialing (RDD) telephone survey of U.S. households. Response rates
for telephone surveys have been declining since the early 1990s. In a previous NSRE (199495), the response rate was 82 percent, while in NSRE 1999-2000 the response rate dropped
to 20 percent. The Office of Management and Budget has expressed concern that low
response rates increases the potential for non response bias. To increase response rates,
OMB imposed the following conditions:
1. An experiment be conducted on the first three versions of the survey (each version
includes approximately 5,000 completed interviews) in which RDD telephone numbers
are matched to addresses and pre-notification letters be sent to 50% of those with listed
numbers (i.e., those with matched mailing addresses). Survey Sampling, Inc., the firm
which supplies the University of Tennessee Survey Research Center with RDD
telephone numbers, takes all listed telephone numbers with addresses and does a reverse
append to match addresses to RDD telephone numbers. On average, only 40% of RDD
telephone numbers are listed with addresses. Note that each version of the NSRE
consists of a core set of questions that take, on average, about 8 minutes and is asked of
all respondents. The core questions include recreation activity participation and
socioeconomic/demographic information. The total interview time per respondent is
restricted to an average of 14 minutes. This leaves about six minutes, on average, for
additional modules of questions. A module is a set of questions designed to achieve a
specific objective or set of objectives. Different versions of the survey have different
modules.
2. Increase efforts to convert refusals by increasing call-backs from 8 to 15 before dropping
a number and sending letters to 50% of those refusals with addresses before calling one
more time.
3. For all refusals to the full survey, conduct a two-question survey to assess non response
bias from refusals. The two questions include one demographic variable related to a
selected recreation activity and if the person participated in that selected activity during
the past 12 month. The demographic variable was age and the selected activity was
“walking for exercise or pleasure.” In addition, gender of the respondent was recorded,
even though not asked.
4. After completing versions one thru three, deliver a report to OMB assessing the
costs/benefits of efforts to increase response rates.
The outcome of this assessment will determine whether the NSRE will be required to continue
the more costly procedures of sending pre-notification letters, continuing with greater efforts to
convert refusals, and administering the two-question survey to assess non-response bias due to
high refusals. The following criteria were specified in the supporting statement approved by
OMB:
“In addition to the marginal cost comparison, computed estimates for recreation
participation rates by activity for both the sample with and without the pre-notification letter
1
will be examined. If there is a statistically significant difference in estimated participation
rates between letter recipients and non recipients and if the average cost per additional
completed response (marginal cost) with an advance letter is no greater than 5% more than
the average cost per completed interview among those not receiving the advance letter (the
outer limit of the budget), then the advance letter procedure will be adopted for the duration
of the NSRE 2005.”
NSRE has been slowed due to many factors, including the hurricanes in 2005.
Consequently, only versions one and two have been completed for a total of 10,001
interviews.
To address the first three of the above four requirements, we have broken the assessment into
three main assessment tasks:
(1) Assess Pre-notification Letters: Response Rates, Sample Representativeness and NonResponse Bias (Section II)
(2) Assess Letters to Refusals (Section III, Part 1)
(3) Assess Refusals and Non Response Bias (Section III, Part 2).
See Figure 1 for a summary of tasks and related sample groups used for each assessment
task.
Note that in all the assessments, we have limited information about non respondents.
Generally, we must extrapolate from what we learn about respondents to non respondents.
The one exception is the analysis of a class of nonrespondents we label “Hard Refusals.”
These are eligible respondents who refused to answer even the two-question survey. For this
group, we test for differences in distributions across demographic characteristics by Census
Division of Residency.
2
3
II. Assessment of Pre-notification Letters: Response Rates, Sample Representativeness
and Non Response Bias
In assessing the benefits of pre-notification letters, we address eight questions in a sequential
analysis. In conducting the assessment, the sample was divided into three groups: 1)
Respondents with No Matching Addresses with RDD telephone numbers, 2) Respondents
with Matching Addresses, but did not receive the pre-notification letter, and 3) Respondents
with Matching addresses that received the pre-notification letter (See Figure 2).
Groups one and two were also combined as they represent the sample that would have been
obtained using standard RDD telephone sampling, i.e., the sample that did not receive prenotification letters. We refer to this group as the Standard RDD Sample Group. This is an
important group because it serves as the standard of comparison for the assessment of prenotification letters.
1. Did pre-notification letters increase response rates?
The simple answer to the above question is yes. The pre-notification letters increased
response rates from 14.08% for the Standard RDD Sample Group to 28.10% for the Prenotification Letter Sample Group, an increase of 14 percentage points. This is a much
larger increase than we were led to expect by project consultants. The net increase was
about 3.5 percentage points for an overall response rate of 17.62%. Again, a limitation
here is that we only have information from respondents in the data we have currently
received. This limits our ability to analyze the factors related to non response. See
Table 1 for response rates by sample groups.
Table 1. Comparative Profiles: Census versus Samples and Response Rates by Sample Groups.
Factors
Age (years)
16-24
25-34
35-44
45-54
55-64
65 and older
N
Chi-Square
P-value
Gender
Male
Female
N
Chi-Square
P-value
Race/Ethnicity
White (not Hispanic)
Black (not Hispanic)
Census
No
Address
No Letter
Address
No
Letter
16.4
17.5
19.3
18.2
12.7
15.9
11.6
16.3
20.6
22.5
16.7
12.2
8.1
14.4
18.0
21.1
18.9
19.5
3119
146.8
<0.0001
48.7
51.3
44.6
55.4
45.6
54.4
3166
21.6
<0.0001
70.6
11.7
2454
224.7
<0.0001
79.4
8.6
2496
9.4
0.0022
86.0
6.6
4
Address
Letter
5.3
10.1
17.1
22.0
21.5
23.9
4276
930.6
<0.0001
48.6
51.4
4332
0.0
0.9350
90.2
4.3
Standard
RDD
Total
Sample
10.1
15.5
19.5
21.9
17.7
15.4
8.0
13.2
18.5
21.9
19.3
19.1
9849
1013.2
<0.0001
5573
300.5
<0.0001
45.0
55.0
5662
30.4
<0.0001
82.3
7.7
46.6
53.4
9994
17.7
<0.0001
85.7
6.2
Factors
Census
Native Am./Pac.Is. (not
0.7
Hispanic)
Asian (not Hispanic)
4.4
Hispanic
12.6
N
Chi-Square
P-value
Education Attainment
Less than High School
19.6
High School or Equivalent
28.6
Some College or College
42.9
Degree
Masters, Prof. Degree, or
8.9
Doctorate
N
Chi-Square
P-value
Household Income ($)
0-24,999
20.8
25,000-49,999
29.1
50,000-99,999
34.8
100,000 and above
15.3
N
Chi-Square
P-value
Urban/Rural Residency
Urban
82.8
Rural
17.2
N
Chi-Square
P-value
Census Division of Residency
New England
5.0
Middle Atlantic
13.9
South Atlantic
18.9
East South Central
6.0
West South Central
11.1
East North Central
15.7
West North Central
6.8
Mountain
6.6
Pacific
16.0
N
Chi-Square
P-value
Response Rate (%)
N/A
No
Address
No Letter
1.9
Address
No
Letter
0.9
Address
Letter
0.9
Standard
RDD
1.5
Total
Sample
1.2
2.3
7.9
1.4
5.2
1.1
3.4
1.9
6.7
1.6
5.3
3108
206.7
<0.0001
2442
298.2
<0.0001
4252
818.5
<0.0001
5550
464.1
<0.0001
9802
1202.6
<0.0001
8.9
23.8
51.2
7.7
25.8
50.3
6.3
26.9
50.2
8.3
24.6
50.8
7.5
25.6
50.5
16.2
16.2
16.5
16.2
16.3
3128
448.9
<0.0001
21.7
25.8
32.4
20.2
2442
361.1
<0.0001
18.2
28.3
36.4
17.1
2380
51.6
<0.0001
81.6
18.4
1861
11.6
0.0090
76.6
23.4
3169
3.0
0.0821
4.5
12.5
17.5
5.9
10.3
13.2
5.3
8.5
22.3
2498
67.0
<0.0001
5.3
12.6
18.8
7.1
11.8
14.7
7.0
7.3
15.4
3169
129.5
<0.0001
12.67
2498
14.2
0.0778
16.33
5
4255
714.7
<0.0001
18.1
27.0
36.7
18.2
3307
38.3
<0.0001
78.4
21.6
4334
57.6
<0.0001
5.3
12.6
19.2
8.1
9.7
17.9
10.3
6.9
10.1
4334
230.6
<0.0001
28.10
5570
806.9
<0.0001
20.2
26.9
34.1
18.8
4241
43.3
<0.0001
79.4
20.6
5667
45.3
<0.0001
4.9
12.6
18.1
6.4
11.0
13.8
6.1
8.0
19.3
5667
81.5
<0.0001
14.08
9825
1511.9
<0.0001
19.3
26.9
35.3
18.6
7548
73.8
<0.0001
79.0
21.0
10001
101.3
<0.0001
5.0
12.6
18.5
7.1
10.4
15.6
7.9
7.5
15.3
10001
73.0
<0.0001
17.62
2. Is there a relationship between response rates and sample Representativeness?
We answer this question in two steps.
Step 1. Here we compared the distribution of each sample group with the Census of the
noninstitutionalized population age 16 years old or older for socioeconomic/demographic
factors (e.g. age, gender, race/ethnicity, educational attainment, household income,
urban/rural residency and residency by Census Division). According to the U.S. Bureau
of the Census, this population is the most appropriate population for extrapolating results
from telephone surveys and provides the “standard of comparison” for judging
representativeness of each sample group.
For assessing sample representativeness, we conducted several statistical tests on
differences between the Census distributions and the distributions for our different
sample groups for each of the socioeconomic/demographic factors. The first tests
compared the total sample distribution of each factor for each sample group against the
known Census distribution. This comparison was also conducted for the “Total Sample”.
We used the Chi-square test using the SAS Software, Inc. and PROC FREQ with the
option TESTP, which specifies the known Census population distribution. The sample
sizes, Chi-square values, and p-values for each of these tests (statistical significance of
the difference) are reported in Table 1.
Results of Test 1:
•
•
•
•
All sample groups and the Total Sample were different from the Census for Gender,
except the pre-notification letter sample group.
All sample groups and the Total Sample were different from the Census for the
following demographic factors: Age, Race/Ethnicity, Educational Attainment, and
Household Income.
All sample groups and the Total Sample, with the exception of the sample group that
had no matching addresses and received no letter, were different for Urban/Rural
Residency.
All sample groups and the Total Sample were different from the Census for Census
Division, except the Address and No Letter sample group.
The second test we conducted was to identify where specific differences existed in a
distribution that was shown earlier to be different from Census. Since a distribution for a
factor had several categories, many tests could be performed to identify specific
differences. To protect from finding false differences, we used a conservative approach
(Bonferroni adjustment), which uses an experimentwise alpha to control for the error for
the set of all tests within a distribution. In this approach, instead of using the 0.05 level
on each proportion test for a factor, we divided 0.05 by the number of category
proportions for a factor and used this value as the error rate for each test. Example:
Race/Ethnicity has five categories: 1) White, not Hispanic, 2) Black, not Hispanic, 3)
Native American, Pacific Islander, not Hispanic, 4) Asian, not Hispanic, and 5) Hispanic.
We divide 0.05 by 5, which equals an error rate of 0.01 for each category comparison
(Census versus sample), but 0.05 across all categories of the factor or the experimentwise
error. Results are summarized in Table 2.
6
Results of Test 2:
Results for test 2 confirm exactly the results of test 1. That is, there was at least one
category found significant in test 2 if the distribution was found significant from test 1.
Similarly, if the distribution was not found significant with test 1, there were no
categories found significant with test 2. This consistency was expected, especially when
using the Bonferroni approach. The value of these tests is to determine what categories
are under or over represented in a sample. For instance, consider age which had a
significantly different distribution from Census for all samples. Table 2 shows that the
younger ages (16-24 and 25-34) were significantly under represented as compared to
Census for all samples. However, the older ages (45-54 and 55-64) were significantly
over represented. The middle age (35-44) appears to match Census, except for the
Address Letter group. This is interesting information that may be useful when analyzing
the participation rates for response bias.
Step 2. We tested for differences in the distributions between sample groups for each
socioeconomic/demographic factor. In this situation we are comparing two sample
distributions as opposed to a sample distribution compared to a known distribution, as
was previously done. We conducted three comparisons: 1) The “Address & Letter”
sample groups versus the “Standard RDD” sample group, 2) The “No Address & No
Letter sample group versus the “Address & No Letter” sample group and 3) “Address &
Letter sample group versus the “Address & No Letter” sample group.
Two-way contingency tables were used to test for distributional differences using the
Chi-Square test. Subsequent specific tests between proportions of the distribution were
conducted by estimating the difference D = p1-p2 and the associated standard error and
then constructing a confidence interval. Here again, we use the Bonferroni adjustment
approach to control for experimentwise error. Results are summarized in Table 3.
Table 2. Differences (Sample-Census) between Census and Sample Profiles and Response Rates.
An * indicates significance at the experimentwise 0.05 level.
Factors
Age (years)
16-24
25-34
35-44
45-54
55-64
65 and older
Gender
Male
Female
Race/Ethnicity
White (not Hispanic)
Black (not Hispanic)
Native Am./Pac.Is. (not
Hispanic)
Census
No
Address
No Letter
Address
No
Letter
Address
Letter
Standard Total
RDD
Sample
16.4
17.5
19.3
18.2
12.7
15.9
-4.8*
-1.2
1.3
4.3*
4.0*
-3.7*
-8.3*
-3.1*
-1.3
2.9*
6.2*
3.6*
-11.1*
-7.4*
-2.2*
3.8*
8.8*
8.0*
-6.3*
-2.0*
0.2
3.7*
5.0*
-0.5
-8.4*
-4.3*
-0.8
3.7*
6.6*
3.2*
48.7
51.3
-4.1*
4.1*
-3.1*
3.1*
-0.1
0.1
-3.7*
3.7*
-2.1*
2.1*
70.6
11.7
0.7
8.8*
-3.1*
1.2*
15.4*
-5.1*
0.2
19.6*
-7.4*
0.2
11.7*
-4.0*
0.8*
15.1*
-5.5*
0.5*
7
Factors
Asian (not Hispanic)
Hispanic
Education Attainment
Less than High School
High School or Equivalent
Some College or College Degree
Masters, Prof. Degree or
Doctorate
Household Income ($)
0-24,999
25,000-49,999
50,000-99,999
100,000 and above
Urban/Rural Residency
Urban
Rural
Census Division of Residency
New England
Middle Atlantic
South Atlantic
East South Central
West South Central
East North Central
West North Central
Mountain
Pacific
Response Rate (%)
Census
4.4
12.6
No
Address
No Letter
-2.1*
-4.7*
Address
No
Letter
-3.0*
-7.4*
Address
Letter
-3.3*
-9.2*
Standard
RDD
-2.5*
-5.9*
Total
Sample
-2.8*
-7.3*
19.6
28.6
42.9
8.9
-10.7*
-4.8*
8.3*
7.3*
-11.9*
-2.8*
7.4*
7.3*
-13.3*
-1.7*
7.3*
7.6*
-11.3*
-4.0*
7.9*
7.3*
-12.1*
-3.0*
7.6*
7.4*
20.8
29.1
34.8
15.3
0.9
-3.3*
-2.4*
4.9*
-2.6*
-0.8
1.6
1.8
-2.7*
-2.1*
1.9
2.9*
-0.6
-2.2*
-0.7
3.5*
-1.5*
-2.2*
0.5
3.3*
82.8
17.2
-1.2
1.2
-6.2*
6.2*
-4.4*
4.4*
-3.4*
3.4*
-3.8*
3.8*
5.0
13.9
18.9
6.0
11.1
15.7
6.8
6.6
16.0
-0.5
-1.4
-1.4
-0.1
-0.8
-2.5*
-1.5*
1.9*
6.3*
0.3
-1.3
-0.1
1.1
0.7
-1.0
0.2
0.7
-0.6
0.3
-1.3
0.3
2.1*
-1.4*
2.2*
3.5*
0.3
-5.9*
-0.1
-1.3*
-0.8
0.4
-0.1
-1.9*
-0.7
1.4*
3.3*
0.0
-1.3*
-0.4
1.1*
-0.7
-0.1
1.1*
0.9*
-0.7
N/A
12.67
16.33
28.10
14.08
17.62
Table 3. Differences between sample profiles. An * indicates significance at the
experimentwise 0.05 level.
Factors
Age (years)
16-24
25-34
35-44
45-54
55-64
65 and older
P-value
Gender
Male
Female
P-value
Race/Ethnicity
White (not Hispanic)
Black (not Hispanic)
Address & Letter
versus Standard
RDD
-4.7*
-5.4*
-2.3*
0.1
3.8*
8.5*
No Address & No
Letter versus Address
& No Letter
3.5*
1.9
2.7
1.5
-2.3
-7.3*
<0.0001
3.6*
-3.6*
-2.8*
-4.3*
-0.8
0.9
2.6
4.4*
<0.0001
-1.1
1.1
0.0003
8.0*
-3.4*
<0.0001
3.0*
-3.0*
0.4236
-6.6*
2.1*
8
Address & Letter
Versus Address &
No Letter
0.0166
4.2*
-2.2*
Factors
Native Am./Pac.Is. (not
Hispanic)
Asian (not Hispanic)
Hispanic
P-value
Education Attainment
Less than High School
High School or Equivalent
Some College or College
Degree
Masters, Prof. Degree or
Doctorate
P-value
Household Income ($)
0-24,999
25,000-49,999
50,000-99,999
100,000 and above
P-value
Urban/Rural Residency
Urban
Rural
P-value
Census Division of Residency
New England
Middle Atlantic
South Atlantic
East South Central
West South Central
East North Central
West North Central
Mountain
Pacific
P-value
Address & Letter
versus Standard
RDD
-0.6*
No Address & No
Letter versus Address
& No Letter
0.9*
Address & Letter
Versus Address &
No Letter
-0.1
-0.7*
-3.2*
0.9*
2.7*
-0.2
-1.7*
<0.0001
<0.0001
<0.0001
-2.0*
2.3*
-0.5
1.2
-2.0
0.9
-1.4
1.1
-0.1
0.3
0.0
0.3
0.0004
-2.0
0.1
2.6
-0.6
0.1928
3.5*
-2.6
-4.0*
3.1*
0.0462
-1.0
1.0
-0.1
-1.3
0.3
1.1
0.0003
5.0*
-5.0*
0.2354
0.4
0.0
1.1
1.7*
-1.3
4.0*
4.2*
-1.0
-9.1*
0.6457
1.8
-1.8
<0.0001
-0.8
-0.1
-1.3
-1.2
-1.5
-1.5
-1.7
1.1
6.9*
<0.0001
0.1730
0.0800
0.0
-0.1
0.4
1.0
-2.1
3.2*
3.2*
-0.4
-5.2*
<0.0001
<0.0001
Results of Sample Group Comparisons:
Address & Letter versus Standard RDD:
•
•
•
There were significant differences in the overall distributions for all demographic
factors, except Urban/rural Residency.
Age. The Address & Letter sample group is older than the Standard RDD sample
group, with significantly higher concentrations of respondents in the above 55 age
groups and lower concentrations in the below age 44 groups. There was not a
significant difference in the age group 45-54.
Gender. The Address & Letter sample group has a significantly higher proportion of
males than the Standard RDD sample group.
9
•
•
•
•
•
Race/Ethnicity. The Address & Letter sample group has a significantly higher
proportion of respondents classified as White, Not Hispanic and significantly lower
proportions of respondents in all other classifications than the Standard RDD sample
group.
Educational Attainment. The Address & Letter sample group has a slightly higher
level of educational attainment than the Standard RDD sample group. The Address &
Letter sample group had a significantly lower proportion of respondents in the “less
than high school” level of education and a significantly higher proportion of
respondents in the “high school or equivalent” level of education. There were no
significant differences at the higher levels of education.
Household Income. There were no significant differences for any income
categories, even though the overall distribution was slightly significant at p = 0.0462.
This occasionally occurs because the Bonferroni adjustment is only approximate and
was not able to find where the minor difference may have existed.
Urban/Rural Residency. There was not a significant difference for urban/rural
residency.
Census Division of Residency. The Address & Letter sample group had
significantly higher proportions of respondents in the East South Central, East North
Central, and West North Central Census divisions of residency than the Standard
RDD sample group. And, the Address & Letter sample group had a significantly
lower proportion of respondents in the Pacific Census division of residency.
No Address & No Letter versus Address & No Letter:
These two sample group together compose the Standard RDD group. There are
significant differences between those who have listed telephone numbers that could be
matched and those who don’t. Of course, this fact has long been known, and is the
reason for doing RDD telephone sampling versus sampling from listings in telephone
directories. The results here simply confirm this fact.
•
•
•
•
•
•
There were significant differences in the overall sample distributions for Age,
Race/Ethnicity, Household Income, Urban/rural Residency and Census Division of
Residency.
There were no significant differences in the overall sample distributions for Gender
and Educational Attainment.
Age. Respondents to the No Address & No Letter sample group were younger than
the Address & No Letter sample group. The No Address & No Letter sample group
had a significantly higher proportion of respondents in the 16-24 age group and a
significantly lower proportion of respondents in the 65 and older age group
compared to the Address and No Letter sample group. There were no significant
differences for any other age groups.
Gender. There was no significant difference between the two sample groups.
Race/Ethnicity. The No Address & No Letter sample group had a significantly
lower proportion of respondents classified as White, Not Hispanic and a significantly
higher proportion of respondents classified in all the other race/ethnicity categories.
Educational Attainment. There were no significant differences in level of
education.
10
•
•
•
Household Income. The No Address & No Letter sample group had significantly
higher proportions of respondents in both the lowest income category ($0 - $24,999)
and the highest income category ($100,000 and above) when compared with the
Address & No Letter sample group. The No Address & No Letter sample group had
a significantly lower proportion of respondents in the $50,000 - $99,999 income
category than the Address & No Letter sample group. The No Address & No Letter
sample group had a lower proportion of respondents in the $25,000 - $49,999 income
category than the Address & No Letter sample group, but this difference was not
significant.
Urban/Rural Residency. The No Address & No Letter sample group had a
significantly higher proportion of respondents who live in urban areas than the
Address & No Letter sample group.
Census Division of Residency. The No Address & No Letter sample group had a
significantly higher proportion of respondents that live in the Pacific Census division
than the Address & No Letter sample group. There were no other significant
differences for Census division of residency.
Address & Letter versus Address & No Letter:
This comparison controls for the treatment of samples that both have listed numbers with
matching addresses and tests for differences in sample responses due to the use of the
pre-notification letter. Use of the pre-notification letter does result in some significant
differences in the demographic profiles of who responded to the survey.
•
•
•
•
•
•
•
•
•
There were significant differences in the overall sample distributions for Age,
Gender, Race/Ethnicity, and Census Division of Residency.
There were no significant differences in the overall sample distributions for
Educational Attainment, Household Income, or Urban/rural Residency.
Age. The Address & Letter sample group had a significantly lower proportion of
respondents in the two age groups under 34 years old and a significantly higher
proportion of respondents in the 65 and older age group than the Address & No
Letter sample group. There were no significant differences for the middle age
groups (35-64).
Gender. The Address & Letter sample group had a significantly higher proportion of
respondents that are male compared to the Address & No Letter sample group.
Race/Ethnicity. The Address & Letter sample group had a significantly higher
proportion of respondents classified as White, Not Hispanic and significantly lower
proportions of respondents for those who were classified as Black or African
American, Not Hispanic and those who were classified as Hispanic.
Educational Attainment. There were no significant differences for level of
education.
Household Income. There were no significant differences for household income.
Urban/Rural Residency. There was no significant difference for urban/rural
residency.
Census Division of Residency. The Address & Letter sample group had
significantly higher proportions of respondents in the East North Central and West
11
North Central Census divisions and, a significantly lower proportion of respondents
in the Pacific Census division versus the Address & No Letter sample group.
3. Does improving response rates with pre-notification letters improve representativeness of
samples compared with “standard RDD” sampling?
In the analyses presented in Tables 1-3, we found significant differences between the
Census distributions for socioeconomic/demographic factors and the distributions for the
socioeconomic/demographic factors for all our sample groups. We also found that there
were significant differences between the distributions for socioeconomic/demographic
factors in comparisons of the Standard RDD sample group and the Pre-notification Letter
sample group. Although for Gender the sample distribution of the Pre-notification Letter
sample group was closer to the Census distribution, generally we conclude that the
effect of the pre-notification letters yielded a total sample less representative than if we
had just done the Standard RDD. Below are some specific findings:
•
•
•
The Standard RDD sample group distributions were closer to the Census
distributions for Age, Race/Ethnicity, Urban/rural Residency and Census Division of
Residency based on the Chi-Square statistics.
The distributions for Education and Income were only very slightly closer to the
Census for the Pre-notification Letter sample as compared to the Standard RDD.
The Pre-notification Letter sample group distributions were closer to the Census
distribution for Gender. So pre-notification letters seem to correct for gender bias
noted in telephone surveys, but may introduce bias related to other factors.
4. Is there a relationship between response rates, sample representativeness and non
response bias?
In answering questions 1-3, we have established that there is no relationship between
response rates and sample representativeness, and that the net effect of pre-notification
letters was to reduce sample representativeness. But, we also found that all our samples
are significantly different from the Census and so our sample is not (without sample
weighting) representative of the population. However, sample representativeness is only
a necessary, not sufficient condition for establishing the existence of non response bias.
To understand non response bias, we need to establish whether there is a relationship
between any of the socioeconomic/demographic factors, for which there is over or under
representativeness, and activity participation rates (i.e. the measures we are seeking to
estimate by use of the survey).
For activity participation rates, we limited our analysis to nine activities: walking for
exercise or pleasure (walk), bird watching (bird), hunting (hunt), fishing (fish), motor
boating (mboat), swimming in natural water bodies (swim_nat), family gatherings (fam),
day hiking (hike), and mountain biking (mtnbike). These activities were chosen because
they capture the activities of greatest importance to the managing agencies, they span a
range of activities in terms of participation rates (relatively high and low), and from past
research are likely to span the range of differences in which socioeconomic/demographic
variables are important for explaining participation rates.
12
Tables 4 and 5 show the estimated activity participation rates for each of the nine
recreation activities for the total sample and each sample group for the unweighted and
weighted sample data.
Table 4. Activity Participation Rates by Sample Group: Unweighted Samples
Samples (Participation Rates)
No
Address &
Address &
Standard
Total
Activity
Address
No Letter
Letter
RDD1
Sample
Walking for exercise or pleasure
0.8672
0.8763
0.8738
0.8712
0.8723
Bird Watching
0.3932
0.4231
0.4384
0.4064
0.4202
Hunting
0.1123
0.1393
0.1271
0.1242
0.1255
Fishing
0.3376
0.3395
0.3459
0.3385
0.3417
Motor Boating
0.2872
0.2734
0.2970
0.2811
0.2880
Swimming in Natural Waterbodies
0.4771
0.4383
0.4442
0.4600
0.4531
Family Gatherings
0.7272
0.7103
0.7292
0.7193
0.7237
Day Hiking
0.3327
0.3731
0.3442
0.3520
0.3486
Mountain biking
0.2154
0.1951
0.1861
0.2057
0.1972
1. Standard RDD sample is equal to sample with no address listings plus sample with address
listings and no letter.
Table 5. Activity Participation Rates by Sample Group: Weighted Samples
Samples (Participation Rates)
No
Address &
Address &
Standard
Total
Activity
Address
No Letter
Letter
RDD1
Sample
Walking for exercise or pleasure
0.8517
0.8492
0.8523
0.8507
0.8513
Bird Watching
0.3124
0.3403
0.3777
0.3240
0.3450
Hunting
0.1039
0.1287
0.1267
0.1142
0.1191
Fishing
0.3377
0.3155
0.3527
0.3286
0.3380
Motor Boating
0.2290
0.2212
0.2640
0.2257
0.2407
Swimming in Natural Waterbodies
0.4343
0.3769
0.3923
0.4106
0.4034
Family Gatherings
0.7256
0.7061
0.7233
0.7173
0.7197
Day Hiking
0.2832
0.3142
0.3019
0.2966
0.2987
Mountain biking
0.2083
0.1896
0.1751
0.2002
0.1902
1. Standard RDD sample is equal to sample with no address listings plus sample with address listings
and no letter.
To estimate the relationship between socio-demographic factors and activity
participation we chose to use logit equations with the unweighted data. We chose the
dummy variable approach based on previous research using NSRE 1999-2000 data for
projecting participation in marine recreation (see
http://marineeconomics.noaa.gov/NSRE/NSREForecast.pdf).
For socioeconomic/demographic variables, we used those variables included in the
comparative profiles (See Tables 1, 2, and 3). We use the dummy variable approach for
explanatory variables. For variables, age, educational attainment and household income,
we chose the lowest category as the base since these variables increase in value with
each category. For all other variables, we chose the base category based on previous
work on activity participation in Leeworthy et al. (2005). See Table 6 for definitions of
the variables in the equations.
13
To test for differences by sampling method, we constructed dummy variables for two
sample treatments. The first sample treatment was the use of pre-notification letters
versus the Standard RDD. The variable created is “stndrdd” and is equal to one when the
sample group is the “Standard RDD” sample group. This will allow testing the effect of
sending pre-notification letters. The second treatment was for those who received refusal
letters or extra effort on refusal conversion (extra effort beyond what would have been
done without the extra OMB requirements). This latter treatment will receive extensive
analysis in Section III of this report. The variable for this second treatment is “Rfconv”
and is equal to one if a refusal conversion.
We also deal with a problem in unit (item) non response for a certain socioeconomic
factor (household income). Around 24.5 percent of our sample did not provide a
response to household income. We didn’t want to lose these respondents in our analyses,
so in our dummy variable approach, we include those that did not provide a response as a
special group.
Table 6. Definitions of Variables included in Logit Equations
Variable
Age16_24
Age25_34
Age35_44
Age45_54
Age55_64
Age65p
Male
White
Black
Asian
Native
Hispan
Educ11
Educhs
Educcoll
Educgrad
Educoth
Inc25
Inc50
Inc100
Inc100p
Incmiss
Urban
Cendiv1
Cendiv2
Cendiv3
Cendiv4
Description
Dummy variable for age those 16 to 24. Value 1=yes 0=no. Reference in constant in
initial full model estimation.
Dummy variable for age those 25 to 34. Value 1=yes 0=no.
Dummy variable for age those 35 to 44. Value 1=yes 0=no.
Dummy variable for age those 45 to 54. Value 1=yes 0=no.
Dummy variable for age those 55 to 64. Value 1=yes 0=no.
Dummy variable for age those 65 and over. Value 1-yes 0=no.
Dummy variable for gender. Value 1=male 0=female.
Dummy variable for Race/Ethnicity, those White-Not Hispanic. Value 1=yes 0=no.
Dummy variable for Race/Ethnicity, those Black-Not Hispanic. Value 1=yes 0=no.
Dummy variable for Race/Ethnicity, those Asian-Not Hispanic. Value 1=yes 0=no.
Dummy variable for Race/Ethnicity, those Native American, Native Hawaiian, or Pacific
Islander & Not Hispanic. Value 1=yes 0=no. Reference in constant in initial full model
estimation.
Dummy variable for Race/Ethnicity, those who are Hispanic. Value 1=yes 0=no.
Dummy variable for Education, less than High School. Value 1=yes 0=no.
Reference in constant in initial full model estimation.
Dummy variable for Education, High School or Equivalent. Value 1=yes 0=no.
Dummy Variable for Education, Some College/College Grad. Value 1=yes 0=no.
Dummy variable for Education, Graduate/Professional Degree. Value 1=yes 0=no.
Dummy variable for Education, Other not specified. Value 1=yes 0=no.
Dummy variable for Household Income, less than $25,000. Value 1=yes 0=no.
Reference in constant in initial full model estimation.
Dummy variable for Household Income, $25,000 - $49,999. Value 1=yes 0=no.
Dummy variable for Household Income, $50,000 - $99,999. Value 1=yes 0=no.
Dummy variable for Household Income, $100,000 & over. Value 1=yes 0=no.
Dummy variable for Household Income, those who did not answer. Value 1=yes 0=no.
Dummy variable for Residence, Value 1=urban 0=rural.
Dummy variable for Census Division of Residence, Northeast. Value 1=yes 0=no.
Dummy variable for Census Division of Residence, Mid Atlantic. Value 1=yes 0=no.
Dummy variable for Census Division of Residence, S. Atlantic. Value 1=yes 0=no.
Dummy variable for Census Division of Residence, E S Central. Value 1=yes 0=no.
14
Variable
Cendiv5
Cendiv6
Cendiv7
Cendiv8
Cendiv9
Stndrdd
Rfconv
Walk
Bird
Hunt
Fish
Mboat
Swim_nat
Fam
Hike
Mtnbike
Description
Dummy variable for Census Division of Residence, W S Central. Value 1=yes 0=no.
Dummy variable for Census Division of Residence, E N Central. Value 1=yes 0=no.
Dummy variable for Census Division of Residence, W N Central. Value 1=yes 0=no.
Dummy variable for Census Division of Residence, Mountain. Value 1=yes 0=no.
Reference in constant in initial full model estimation.
Dummy variable for Census Division of Residence, Pacific, Value 1=yes 0=no.
Dummy variable for sample treatment. Value 1=Standard RDD 0=Pre-notification
letter.
Dummy variable for sample treatment. Value 1=Refusal conversion
0=not a refusal conversion.
Dummy variable for Activity Participation: Walking for Exercise or Pleasure.
Value 1=yes 0=no.
Dummy variable for Activity Participation: Bird Watching. Value 1=yes 0=no.
Dummy variable for Activity Participation: Hunting. Value 1=yes 0=no.
Dummy variable for Activity Participation: Fishing. Value 1=yes 0=no.
Dummy variable for Activity Participation: Motor boating. Value 1=yes 0=no.
Dummy variable for Activity Participation: Swimming in Natural Water bodies.
Value 1-yes 0=no.
Dummy variable for Activity Participation: Family Gatherings. Value 1=yes 0=no.
Dummy variable for Activity Participation: Day Hiking. Value 1=yes 0=no.
Dummy variable for Activity Participation: Mountain Biking. Value 1=yes 0=no.
We estimated the logit equations using both the SAS 9.0 software and LIMDEP 7.0.
With the use of SAS we were able to test the “main effect” for each socio-demographic
factor and perform pairwise comparisons. This is analogous to what is usually done in
an analysis of variance. The full results are not included in this report since they are not
central to the task here. However, the results are available on request.
The results of the logit equations are summarized in Tables 7 and 8. Note that we have
included all the dummy variables corresponding to each category for each factor, except
for Gender and Urban/rural residency. These two factors are binary variables taking on
values of zero or one. A person is either male or female or lives in an urban or rural
area. For all other variables (factors) we include all the category dummy variables in the
table of results. A blank in the table indicates that the category is in the constant.
Results of the Logit Equations:
•
•
•
•
•
Age and Household Income are significant factors in all nine (9) activities tested.
Gender and Census Division of Residency are significant factors in 8 of 9 activities
tested.
Race/Ethnicity and Educational Attainment are significant factors in 7 of 9 activities
tested. The “main effects” test indicates that race/ethnicity and education attainment
were significant in 8 of the 9 activities tested. The additional activity was walking
for exercise or pleasure. The significance of the “main effects” is shown in Table 8.
Urban/rural Residency was a significant factor in 3 of 9 activities tested.
Sample treatments of pre-notification letters and refusal conversions were not
significant factors for any of the nine (9) activities tested.
15
Conclusion: There is evidence of non response bias. The sample contains over and
under representation for all socioeconomic/demographic factors and these factors are
significant factors in explaining activity participation.
Table 7. Estimated Participation Functions by Activity: Logit Equations
Activities (Participation Function Coefficients) 1
Bird
Hunt
Fish
Mboat
-1.7295 *
-2.4595 *
-0.1996
-2.2920 *
Factor
Walk
Swim_nat
Constant
2.0135 *
-0.3633
Age16_24
Age25_34
-0.4028 *
0.2950 *
0.1484
-0.1381
-0.2859 *
-0.8554 *
Age35_44
-0.4196 *
0.6511 *
0.01750
-0.0399
-0.2073
-0.8776 *
Age45_54
-0.4822 *
0.9523 *
-0.3326 *
-0.3739 *
-0.4490 *
-1.2848 *
Age55_64
-0.6961 *
1.0253 *
-0.5556 *
-0.7318 *
-0.5743 *
-1.7689 *
Age65p
-0.8667 *
0.8795 *
-1.2766 *
-1.1430 *
-1.0778 *
-2.6188 *
Male
-0.4790 *
-0.3234 *
1.9424 *
0.8941 *
0.2605 *
-0.0838
White
0.2184
0.3193
0.3034
-0.2343
0.5561 *
0.01489
Black
0.0136
-0.5101 *
-1.1572 *
-1.0233*
-1.0292 *
-1.5114 *
Asian
-0.5876
-0.3081
-2.1376 *
-0.6259 *
-0.5952
-1.0633 *
Native
Hispan
0.3984
-0.0717
-0.2655
-0.4910 *
-0.0695
-0.2902
Educ11
Educhs
0.0919
0.2228 *
0.0642
0.1732
0.3356 *
0.2535 *
Educcoll
0.6621 *
0.5283 *
-0.3799 *
-0.1052
0.5563 *
0.7895 *
Educgrad
1.1007 *
0.7098 *
-0.8481 *
-0.3349 *
0.5567 *
1.0293 *
Educoth
0.5216
0.6037 *
0.3276
0.0247
0.4603
0.5477 *
Inc25
Inc50
0.4378 *
0.1548 *
0.5207 *
0.2822 *
0.5972 *
0.4102 *
Inc100
0.5850 *
0.1602 *
0.7252 *
0.4127 *
0.9219 *
0.7177 *
Inc100p
0.8502 *
0.2737 *
0.4584 *
0.3627 *
1.1969 *
1.0508 *
Incmiss
0.1983 *
-0.04255
0.3506 *
0.1057
0.6480 *
0.3695 *
Urban
0.0056
-0.1079 *
-0.9219 *
-0.3769 *
-0.05972
0.0962
Cendiv1
-0.1376
0.4145 *
-0.8985 *
-0.1772
0.0721
1.1323 *
Cendiv2
-0.3649 *
0.0564
-0.4078 *
-0.3788 *
-0.0681
0.8177 *
Cendiv3
-0.3902 *
0.2478 *
-0.3567 *
0.2291 *
0.2390 *
0.8348 *
Cendiv4
-0.4160 *
-0.07608
-0.0617
0.1944
0.1345
0.1971
Cendiv5
-0.3848 *
-0.07841
0.4196 *
0.2287 *
0.1533
0.1306
Cendiv6
-0.3010 *
0.1329
-0.1956
-0.0907
0.3018 *
0.3401 *
Cendiv7
-0.3305 *
0.1265
0.3819 *
0.3008 *
0.6234 *
0.1505
Cendiv8
Cendiv9
-0.0444
0.1893 *
-0.6487 *
-0.3201 *
0.0646
0.5304 *
Standrdd
-0.1161
-0.0342
0.0496
-0.0295
-0.0431
-0.0823
Rfconv
-0.0120
-0.05902
0.0952
0.0469
0.0105
0.0268
1. *=significance at .05 or less and blank means dummy category in constant.
Table 7 (Continued). Estimated Participation Functions by Activity: Logit Equations
Factor
Constant
Age16_24
Age25_34
Age35_44
Activities (Participation Function Coefficients) 1
Fam
Hike
Mtnbike
1.4254 *
-0.1291
-0.4047
-0.5042
-0.3066
-0.0015
0.0409
16
-0.2590
-0.5090 *
Activities (Participation Function Coefficients) 1
Fam
Hike
Mtnbike
-0.8422 *
-0.0358
-0.8405 *
-1.0638 *
-0.2807
-1.3171 *
-1.0568 *
-0.8734 *
-2.2554 *
-0.1919 *
0.2922 *
0.5045 *
-0.0603
-0.1265
-0.4092
0.5756
-1.5053 *
-0.7284 *
-0.6144
-0.7431
-1.1048 *
Factor
Age45_54
Age55_64
Age65p
Male
White
Black
Asian
Native
Hispan
-0.0736
-0.3153
-0.6593
Educ11
Educhs
0.0311
0.0347
-0.2303
Educcoll
0.3132
0.2549
0.0179
Educgrad
0.2864
0.6239 *
0.2875
Educoth
0.5934
0.0713
0.0161
Inc25
Inc50
0.2352
0.2692 *
0.1371
Inc100
0.5907 *
0.4858 *
0.1431
Inc100p
0.5509 *
0.5160 *
0.5205 *
Incmiss
0.1868
0.0671
-0.1510
Urban
-0.2239
-0.0160
-0.0016
Cendiv1
0.3516
-0.6420 *
0.0255
Cendiv2
0.1869
-0.7786 *
-0.0032
Cendiv3
0.0070
-0.9184 *
-0.0227
Cendiv4
0.1612
-1.0940 *
-0.3645
Cendiv5
0.2237
-1.2887 *
-0.6838 *
Cendiv6
0.1661
-0.8698 *
0.1349
Cendiv7
0.1893
-0.8318 *
0.0324
Cendiv8
Cendiv9
0.4111
-0.3019 *
-0.0192
Standrdd
-0.1696
-0.0157
-0.0127
Rfconv
-0.1206
-0.0244
-0.0793
1. *=significance at .05 or less and blank means dummy category in constant.
Table 8. Tests (P-values) on the Main Effects in the Logit Participation Models Based on the
Wald Chi-Square Test.
Factor
Age
Gender
Ethrace
Educ
Income
Urban
Cendiv
Standrdd
Rfcon
Walk
<0.0001
<0.0001
0.0019
<0.0001
<0.0001
0.9509
0.0189
0.0818
0.8733
Bird
<0.0001
<0.0001
<0.0001
<0.0001
<0.0002
0.0535
<0.0001
0.4416
0.2180
Hunt
<0.0001
<0.0001
<0.0001
<0.0001
<0.0001
<0.0001
<0.0001
0.4996
0.2235
Fish
<0.0001
<0.0001
<0.0001
0.0006
<0.0001
<0.0001
<0.0001
0.5189
0.3818
17
Mboat Swim_Nat Fam
Hike
<0.0001 <0.0001 <0.0001 <0.0001
<0.0001 0.0691
0.0424 <0.0001
<0.0001 <0.0001 0.0768 <0.0001
<0.0001 <0.0001 0.1182 <0.0001
<0.0001 <0.0001 0.0012 <0.0001
0.3061
0.1067
0.0624 0.8248
<0.0001 <0.0001 0.4671 <0.0001
0.3851
0.0777
0.0805 0.7888
0.8603
0.6294
0.2559 0.7377
Mtnbike
<0.0001
<0.0001
0.0305
0.0026
<0.0001
0.9839
0.0003
0.8585
0.3900
5. Is the non response bias significant?
To answer this question, we tested whether there is a difference between unweighted and
weighted data estimates of the nine activity participation rates.
Multivariate weights were constructed for age, gender and race/ethnicity using the 2004
Census data for the noninstitutionalized population 16 years old and older and our
sample data. As with the NSRE 2000, we also applied multiplicative weights for
educational attainment and urban/rural residency.
The multivariate weights included 60 cells (age=6 categories, gender=2 categories and
race/ethnicity=5 categories). Sample sizes will not support extending multivariate
weighting to educational attainment and urban/rural residency. Extending to educational
attainment, which includes five categories, would result in a 300 cell matrix. Sample cell
densities in a 300 cell matrix would most likely not support effective sample weighting.
By effective, we mean that sample sizes would not be large enough to have
representative samples in each cell. Effective weighting, equalizing sample to
population, requires representative samples in each cell.
For statistical tests, we used four approaches. In the first approach, we constructed 95
percent confidence intervals for the estimated activity participation rates for both the
unweighted and weighted sample data. Statistically significant differences are indicated
by non overlapping confidence intervals. Statistically significant differences are
interpreted as indicating the existence of significant non response bias. Using this
approach we conducted comparisons of unweighted and weighted estimates of activity
participation for the “Full Sample” (Table 9), the “Pre-notification Letter” sample group
(Table 10) and the “Standard RDD” sample group (Table 11). We also compared the
weighted estimates of activity participation for the “Pre-notification Letter” sample
group to the “Standard RDD” sample group.
Results from Comparisons Using the Overlapping Confidence Interval Approach:
•
•
•
Full Sample. There were significant differences between the unweighted and
weighted estimates for 5 of the 9 activities tested (walk, bird, mboat, swim_nat and
hike). Unweighted estimates were always higher than non weighted estimates
indicating a general upward bias (Table 9).
Pre-notification Letter Sample. There were significant differences between the
unweighted and weighted estimates for 5 of the 9 activities tested (walk, bird, mboat,
swim_nat and hike). All the unweighted estimates were higher, except for fishing,
which was lower but not significant (Table 10).
Standard RDD Sample. There were significant differences between the unweighted
and weighted estimates for 5 of the 9 activities tested (walk, bird, mboat, swim_nat
and hike). All the unweighted estimates were higher than the weighted estimates
(Table 11).
18
Table 9. Differences in Unweighted and Weighted Estimates of Activity Participation Rates:
Full Sample
Unweighted
95% C.I.1
Weighted
95% C.I.2
Statistically Significant
Difference3
Walk
0.8723
(0.8658, 0.8788)
0.8513
(0.8442, 0.8584)
Yes, +
Bird
0.4203
(0.4107, 0.4299)
0.3450
(0.3358, 0.3542)
Yes, +
0.1255
(0.1190, 0.1320)
0.1191
(0.1128, 0.1254)
No,+
0.3417
(0.3325, 0.3509)
0.3380
(0.3288, 0.3472)
No, +
0.2880
(0.2792, 0.2968)
0.2407
(0.2323, 0.2491)
Yes, +
0.4532
(0.4434, 0.4630)
0.4034
(0.3938, 0.4130)
Yes, +
0.7237
(0.7059, 0.7415)
0.7197
(0.7019, 0.7375)
No, +
0.3486
(0.3355, 0.3617)
0.2987
(0.2860, 0.3114)
Yes, +
Sample Group/Activity
Hunt
Fish
Mboat
Swim_nat
Fam
Hike
0.1972
0.1902
(0.1862, 0.2082)
(0.1794, 0.2010)
No, +
1. 95 percent confidence interval on estimated activity participation rates using unweighted
data.
2. 95 percent confidence interval on estimated activity participation rates using weighted
data.
Mtnbike
3. Yes or No for statistically significant difference between unweighted and weighted
estimates of activity participation rates. + or - indicating unweighted estimate of
activity participation rate is greater (+) or less (-) than the weighted estimate of activity
participation rate.
Table 10. Differences in Unweighted and Weighted Estimates of Activity Participation Rates:
Pre-notification Letters
Unweighted
95% C.I.1
Weighted
95% C.I.2
Statistically Significant
Difference3
Walk
0.8738
(0.8640, 0.8836)
0.8523
(0.8417, 0.8629)
Yes, +
Bird
0.4384
(0.4237, 0.4531)
0.3777
(0.3632, 0.3922)
Yes, +
Hunt
0.1271
(0.1171, 0.1371)
0.1267
(0.1169, 0.1365)
No, +
Fish
0.3459
(0.3318, 0.3600)
0.3527
(0.3384, 0.3670)
No, -
Mboat
0.2970
(0.2835, 0.3105)
0.2640
(0.2509, 0.2771)
Yes, +
Activity
19
Unweighted
95% C.I.1
Weighted
95% C.I.2
Statistically Significant
Difference3
Swim_nat
0.4442
(0.4295, 0.4589)
0.3923
(0.3778, 0.4068)
Yes, +
Fam
0.7292
(0.7025, 0.7559)
0.7233
(0.6964, 0.7502)
No, +
0.3442
(0.3242, 0.3642)
0.3019
(0.2827, 0.3211)
Yes, +
Activity
Hike
0.1861
0.1751
(0.1698, 0.2024)
(0.1592, 0.1910)
No, +
1. 95 percent confidence interval on estimated activity participation rates using unweighted data.
2. 95 percent confidence interval on estimated activity participation rates using weighted data.
3. Yes or No for statistically significant difference between unweighted and weighted estimates of
activity participation rates. + or - indicating unweighted estimate of activity participation rate is greater
(+) or less (-) than the weighted estimate of activity participation rate.
Mtnbike
Table 11. Differences in Unweighted and Weighted Estimates of Activity Participation Rates:
Standard RDD - No Pre-notification Letters
Unweighted
Weighted
Statistically Significant
1
2
Activity
95% C.I.
95% C.I.
Difference3
0.8712
0.8507
Walk
(0.8626, 0.8798)
(0.8415, 0.8599)
Yes, +
0.4064
0.3240
Bird
(0.3937, 0.4191)
(0.3118, 0.3362)
Yes, +
0.1242
0.1141
Hunt
(0.1156, 0.1328)
(0.1059, 0.1223)
No, +
0.3384
0.3286
Fish
(0.3261, 0.3507)
(0.3164, 0.3408)
No, +
0.2811
0.2257
Mboat
(0.2693, 0.2929)
(0.2149, 0.2365)
Yes, +
0.4600
0.4106
Swim_nat
(0.4471, 0.4729)
(0.3979, 0.4233)
Yes, +
0.7193
0.7173
Fam
(0.6956, 0.7430)
(06934, 0.7412)
No, +
0.3520
0.2966
Hike
(0.3344, 0.3696)
(0.2797, 0.3135)
Yes, +
0.2057
0.2002
Mtnbike
(0.1908, 0.2206)
(0.1855, 0.2149)
No, +
1. 95 percent confidence interval on estimated activity participation rates using unweighted
data.
2. 95 percent confidence interval on estimated activity participation rates using weighted
data.
3. Yes or No for statistically significant difference between unweighted and weighted
estimates of activity participation rates. + or - indicating unweighted estimate of
activity participation rate is greater (+) or less (-) than the weighted estimate of activity
participation rate.
20
Difference Approach:
An alternative approach and the one preferred here, is to estimate the difference defined
as D = U – W, where U is the unweighted estimate of the participation rate and W is the
weighted estimate of the participation rate. Confidence intervals on the difference are
constructed as 1.96 times the square root of the variance of D (the 95 percent confidence
interval) and serve as the test criterion by comparing the interval to zero. This is a more
powerful test than the above over lapping paired confidence interval test.
We implemented this approach using a difference test: We considered using an analysis
of variance approach using PROC MIXED in SAS, but rejected this approach because of
the assumption of equal variances. Instead, we implemented the difference test by
calculating unweighted and weighted means and standard errors using PROC MEANS in
SAS and constructing 95 percent confidence intervals. This latter test relaxes the
assumption of homogeneous or equal variances. We don’t include the details of the
results since they yielded the same results as the simple overlapping confidence interval
results as above.
Comparison of Pre-notification Letter and Standard RDD
Here we applied both the overlapping confidence intervals and difference approach for
weighted estimates of activity participation rates for the Pre-notification sample group
versus the Standard RDD sample group. The differences D = P – S, where P is the
weighted estimate of the participation rate for the Pre-notification sample group and S is
the weighted estimate of the participation rate for the Standard RDD sample group.
Results of the Differences in Pre-notification and Standard RDD Estimates of
Weighted Activity Participation Rates:
•
•
Using the overlapping confidence interval approach, we found significant differences
for only two (2) of the nine (9) activities tested (bird and mboat). In both cases the
Pre-notification letter sample group estimates were higher than the Standard RDD
sample group estimates (Table 12).
Using the difference approach, we found significant differences for four (4) of the
nine (9) activities tested (bird, fish, mboat and mtnbike). Of the four significant
differences, the Pre-notification Letter sample group had higher estimates of activity
participation rates than the Standard RDD sample group (bird, fish and mboat). For
mountain biking (mtnbike), the Pre-notification Letter sample group estimates were
lower than the Standard RDD sample group. See Table 13 for a summary of the
results.
Conclusions: There is significant non response bias in some estimates of activity
participation rates. Even after applying sample weighting there are significant
differences between estimates of activity participation from the Pre-notification and
Standard RDD sample groups.
21
Table 12. Differences in Weighted Estimates of Activity Participation Rates: Comparison Prenotification with Standard RDD - No Pre-notification Letters
Pre-notification
Standard RDD
Statistically Significant
1
2
Activity
95% C.I.
95% C.I.
Difference3
0.8523
(0.8417,
0.8507
(0.8415,
Walk
0.8629)
0.8599)
No, +
0.3777
(0.3632,
0.3240
(0.3118,
Bird
0.3922)
0.3362)
Yes, +
0.1267
(0.1169,
0.1141
(0.1059,
Hunt
0.1365)
0.1223)
No, +
0.3527
(0.3383,
0.3286
(0.3164,
Fish
0.3669)
0.3408)
No, +
0.2640
(0.2509,
0.2257
(0.2149,
Mboat
0.2771)
0.2365)
Yes, +
0.3923
(0.3778,
0.4106
(0.3979,
Swim_nat
0.4068)
0.4233)
No, 0.7173
(06934,
0.7233
(0.6964,
Fam
0.7502)
0.7412)
No, +
0.3019
(0.2827,
0.2966
(0.2797,
Hike
0.3211)
0.3135)
No, +
0.1751
(0.1592,
0.2002
(0.1855,
Mtnbike
0.1910)
0.2149)
No, 1. 95 percent confidence interval on estimated activity participation rates using weighted data for those
in the pre-notification letter sample.
2. 95 percent confidence interval on estimated activity participation rates using weighted data for those
in the sample that did not receive pre-notification letters or Standard RDD.
3. Yes or No for statistically significant difference between pre-notification and Standard RDD sample
group estimates of activity participation rates. + or - indicating pre-notification sample group estimate of
activity participation rate is greater (+) or less (-) than the estimate of activity participation rate for the
Standard RDD sample group.
Table 13. Comparison of Mean Activity Participation Rates between Pre-notification Letter
Sample and the Standard RDD Sample: Weighted Data - Difference Approach1
Activity
Statistically Significant Difference2,3
Walk
No, +
Bird
Yes, +
Hunt
No, +
Fish
Yes, +
Mboat
Yes, +
Swim_nat
No, Fam
No, +
Hike
No, +
Mtnbike
Yes, 1. Difference approach compares differences in weighted means for activity participation rates for
two different sample groups.
2. Yes indicates a statistically significant difference at the 0.05 significance level and + indicates the
mean for the pre-notification sample is greater than the mean for the Standard RDD sample. No
indicates the difference is not statistically significant at the 0.05 level of significance and - indicates
that the mean for the pre-notification letter sample was less than the mean for the Standard RDD
sample.
3. Equal variance assumption relaxed in test.
22
6. Can sample weighting be used to adjust for non response bias?
We don’t know the “true” activity participation rates to make definitive judgments about
which estimates are better estimates. However, we can compare estimates from the
NSRE 2000-2001 with our current estimates using the entire sample. Generally, our
estimates for NSRE 2005 are not greatly different from those estimated in NSRE 20002001 (Table 14). For most of the nine activities analyzed in this assessment, the
weighted estimates from NSRE 2005 are closer to the estimates from NSRE 2000-2001.
In Leeworthy et al. 2005 (see
http://marineeconomics.noaa.gov/NSRE/NSREForecast.pdf), activity participation rates
for marine recreation were projected to decline from 2000 to 2005 and from 2000 to
2010 based on projected changes in the same socioeconomic/demographic factors
analyzed in this assessment. One of the most important factors driving the projected
declines was Race/Ethnicity. The projected decreases in the proportion of the population
that is White, Not Hispanic and increases in the proportions of the population that are
Black or African American, Not Hispanic and Hispanic were the major drivers of the
projected declines in activity participation rates. The reason for this is that those who are
White, Not Hispanic had generally higher activity participation rates and those who were
Black or African Americans, Not Hispanic and Hispanics had lower activity participation
rates.
Given the expected declines in 2005 activity participation rates, we might expect that
NSRE 2005 estimates would be slightly lower than NSRE 2000-2001 estimates. This
was true for six (6) of the nine (9) activities tested here using the weighted sample
estimates.
Conclusions: The sample weighting can adjust for non response bias. However, the
pre-notification letters are yielding more unrepresentative samples on key variables
such as race/ethnicity, which are imparting higher non response bias even with
sample weighting.
Table 14. Comparison of Activity Participation Rate Estimates NSRE 2000-2001 vs. 2005
Activity
Walking for exercise or pleasure
Bird Watching
Hunting
Fishing
Motor boating
Swimming in Natural Water bodies
Family Gatherings
Day Hiking
Mountain biking
NSRE 2000-2001
Weighted
83.0
32.4
11.3
34.1
24.4
41.7
73.5
33.3
21.4
NSRE 2005
Weighted Unweighted
85.1
87.2
34.5
42.0
11.9
12.5
33.8
34.2
24.1
28.8
40.3
45.3
71.2
72.4
29.9
34.9
19.0
19.7
7. What is the impact of pre-notification letters on non response bias?
The rationale behind the imposition of doing pre-notification letters is that they would
increase response rates and thereby reduce non response bias. However, the basis of
23
RDD telephone sampling is that there is a difference between simple random sampling
from listed numbers and RDD samples, with RDD samples being more representative of
the population. Thus, there is a possibility that increasing the proportion of the sample
with listed telephone numbers versus those with unlisted numbers could introduce bias
into estimates of activity participation rates. We find support for this hypothesis.
We don’t know the “true” activity participation rates, but in a Florida study where
boaters used the reefs in Southeast Florida the “true” distribution of boats by size class
was known (Jones et al., 2003). In this study, a stratified random sample was selected
from the boat registration file, which contains the names and addresses of the boat
owners, along with characteristics of the boat. Telephone numbers were not included in
the boat registration files. Florida State University researchers wanted to use a
computer-aided telephone instrument (CATI) system to conduct the survey. Telephone
listings were used to match addresses with the boat registration files to get telephone
numbers (the reverse of our problem). The result was that the sample with telephone
numbers was a biased sample. The owners of boats greater than 25 foot in length
occurred disproportionally among those with unlisted numbers. Given that the reefs
were generally four to six miles offshore, larger boats would have a higher probability of
being used to access the reefs. The telephone survey approach was abandoned and a
mail survey was used.
Conclusion: Pre-notification letters are introducing bias. The letters are generally
not correcting biases from using the Standard RDD sampling methods. Because the
letters are going only to people with listed telephone numbers that can be matched to
addresses and these people are different from those with unlisted numbers, increasing
the proportion of these types of people is making our sample more unrepresentative
and increasing bias in our estimates of activity participation.
8. Are the benefits of pre-notification letters worth the added costs?
No. After including the costs of matching telephone numbers to addresses;
stuffing/labeling letters; the printing of letters; and the postage to mail the letters, the
average cost per completed interview increased 9.8%. The offsetting cost reduction of
higher response rates was insignificant since the average completed interview required
3.5 calls for those who received pre-notification letters, while taking 4.0 calls for those
who did not receive the letter. In addition, as demonstrated above, pre-notification
letters resulted in more unrepresentative samples and biased estimates of activity
participation. Therefore, according to the criteria set out at the beginning of this
assessment, the increase in average costs per interview from the use of pre-notification
letters exceeds the 5% threshold, while also introducing significant bias to the estimation
of activity participation rates, the pre-notification letter requirement should be dropped
from all further versions of NSRE 2005.
Overall Conclusions on the use of Pre-notification Letters:
•
There is evidence of non response bias. However, there are no apparent
improvements by utilization of pre-notification letters even though pre-notification
letters increased response rates 14 percentage points (14.08% to 28.10%).
24
•
There is no relationship between response rates and non response bias. Non
response bias comes from the mix of people responding. Pre-notification letters
result in even more unrepresentative samples than simple Standard RDD samples.
•
Although there is evidence of non response bias in Standard RDD sampling,
sample weighting seems adequate to adjust for this bias.
•
Given the added cost of pre-notification letters with no corresponding benefit, prenotification letters do not pass the benefit-cost test.
III. Assess Refusals and Non Response Bias
Currently we are making 15 calls to a telephone number before dropping the number. We
will continue with this throughout NSRE 2005 and will not assess the benefits and costs of
this requirement.
For those who are contacted and refuse, at the end of each week’s surveying a special session
was set-up to call back refusals. Before these call-backs were begun, letters were sent out to
50% of the refusers with listed telephone numbers and addresses.
In call-backs to refusers, if they again refused, they were asked if they would answer two
questions. If they agreed, they were asked their age and if they had participated in walking
for exercise or pleasure during the past 12 months. Gender is also recorded, but not asked.
We were also able to construct the variable for Census Division of Residency for each
eligible telephone number.
As with pre-notification letters, we assessed whether the added efforts versus Standard RDD
sampling increased response rates, whether more representative samples were obtained,
whether non response bias exists, and if non response bias exists, is it significant. In
addition, we addressed whether sample weighting or some other correction method be used
to correct for biases. Also, do the refusal letters introduce bias? In accordance with the prenotification letter assessment we proceed in a sequential analysis.
Part 1: Assess Letters to Refusals
To support the analysis of the refusal letter, we have two sample groups; 1) those who
received a letter and 2) those who did not receive a letter (See Figure 3).
25
1. Did the refusal letters increase response rates (refusal conversions)?
Yes. Those who did not have listed telephone numbers with matching addresses and
thus did not receive a refusal letter had a response rate of 7.2%. Similarly, those who
had matched telephone numbers with addresses, but did not receive a refusal letter had a
response rate of 7.6%. On net, all refusals that did not receive a refusal letter had a
response rate of 7.4%. Those who received a refusal letter had a response rate of 14.4%.
the overall response rate for refusals was 9.4% (Tables 15 and 16).
2. Is there a difference in sample representativeness as a result of increasing response rates
(refusal conversions)?
In accordance with the pre-notification letter assessment, we first compared the two
sample groups and the total refusal sample to the Census distributions for all
socioeconomic/demographic variables using two tests. Test 1 uses SAS PROC FREQ
with the TESTP option, which specifies the known Census Distribution. A Chi-square
test is conducted for differences in distributions for each factor at the 0.05 significance
level. The second test uses the more conservative Bonferroni adjustment to control for
experiment wise error, as described earlier.
Results of Test 1:
•
Both refusal sample groups and the total sample of refusals were different from the
Census for all socioeconomic/demographic factors, with only one exception. The
exception was Gender for the refusals that received the letter. So just as with the prenotification letter, the refusal letter seems to eliminate gender bias (See Table 15).
Results of Test 2:
•
The results confirm the results from test 1 as expected. The real purpose of test 2 is
to find out where in the distributions the differences exist (See Table 16).
26
Results of Sample Group Comparisons:
Here we addressed the differences between the samples obtained in the refusal
conversion process by the use of the “refusal letters” by comparing the
socioeconomic/demographic profiles of those who received the letters versus those who
did not.
No Letter versus Letter (Table 17):
•
•
•
•
•
•
•
•
•
There were significant differences for Age, Race/Ethnicity, Educational Attainment,
and Census Division of Residency.
There were not significant differences for Gender, Household Income, and
Urban/rural Residency.
Age. The No Letter sample group was younger than the Letter sample group. The
No Letter sample group had a significantly higher proportion of respondents in the
16-24 and 24-34 age groups and a significantly lower proportion in the 65 and older
age group.
Gender. There was a lower proportion of males in the No Letter sample group, but
the difference was not significant.
Race/Ethnicity. The No Letter sample group had a significantly lower proportion of
respondents classified as White, Not Hispanic and a significantly higher proportion
of respondents classified as Asian, Not Hispanic and Hispanic than the Letter sample
group.
Educational Attainment. The No Letter sample group had a significantly lower
proportion of respondents with a High School or Equivalent level of educational
attainment than the Letter sample group.
Household Income. There were no significant differences.
Urban/Rural Residency. There wasn’t a significant difference.
Census Division of Residency. The No Letter sample group had a significantly
lower proportion of respondents in the East North Central and West North Central
Census Division and a significantly higher proportion of respondents from the
Pacific Census Division than the Letter sample group.
Conclusions: The No Letter Sample group distributions are generally closer to the
Census distributions, except for Gender and Educational Attainment. Thus, the refusal
letter, on balance seems to yield a more unrepresentative sample.
Table 15. Comparative Profiles for Refusal Conversions: Census versus Samples and Response
Rates by Sample Groups.
Factors
Age (years)
16-24
25-34
35-44
45-54
55-64
65 and older
n
Census
No Letter
Letter
Total Sample
16.4
17.5
19.3
18.2
12.7
15.9
10.7
13.9
17.2
23.6
18.1
16.5
5.3
7.7
17.1
21.2
20.2
28.5
8.5
11.4
17.1
22.6
19.0
21.4
1454
27
1008
2462
Factors
Chi-Square
P-value
Gender
Male
Female
n
Chi-Square
P-value
Race/Ethnicity
White (not Hispanic)
Black (not Hispanic)
Native Am./Pac.Is. (not Hispanic)
Asian (not Hispanic)
Hispanic
n
Chi-Square
P-value
Education Attainment
Less than High School
High School or Equivalent
Some College or College Degree
Masters, Prof. Degree, or Doctorate
n
Chi-Square
P-value
Household Income ($)
0-24,999
25,000-49,999
50,000-99,999
100,000 and above
n
Chi-Square
P-value
Urban/Rural Residency
Urban
Rural
n
Chi-Square
P-value
Census Division of Residency
New England
Middle Atlantic
South Atlantic
East South Central
West South Central
East North Central
West North Central
Mountain
Pacific
n
Chi-Square
Census
No Letter
99.4
<0.0001
Letter
284.2
<0.0001
Total Sample
302.3
<0.0001
48.7
51.3
44.9
55.1
48.2
51.8
46.2
53.8
1482
8.7
0.0032
70.6
11.7
0.7
4.4
12.6
82.4
6.7
1.7
2.7
6.6
89.4
5.4
0.9
0.9
3.4
1443
129.9
<0.0001
19.6
28.6
42.9
8.9
8.6
26.0
48.2
17.2
20.0
26.2
32.8
21.1
82.8
17.2
78.2
21.8
4.6
13.1
15.4
7.5
9.5
14.2
6.2
8.0
21.6
28
1003
123.7
<0.0001
2452
328.5
<0.0001
18.8
26.0
34.7
20.5
749
18.1
0.0004
1831
41.6
<0.0001
78.0
22.0
1018
18.6
<0.0001
5.5
11.2
18.3
8.8
10.8
20.8
9.6
5.9
9.0
1483
55.7
2444
290.8
<0.0001
7.9
27.9
48.1
16.1
77.7
22.3
1483
22.5
<0.0001
5.0
13.9
18.9
6.0
11.1
15.7
6.8
6.6
16.0
1001
179.9
<0.0001
17.2
25.8
37.4
19.6
1082
28.4
<0.0001
2500
6.1
0.0139
85.3
6.1
1.4
2.0
5.3
7.0
30.6
48.0
14.5
1449
214.9
<0.0001
20.8
29.1
34.8
15.3
1018
0.1
0.7651
2501
41.0
<0.0001
5.0
12.4
16.6
8.0
10.0
16.9
7.6
7.1
16.5
1018
80.4
2501
37.3
Factors
P-value
Census
No Letter
<0.0001
Letter
<0.0001
Total Sample
<0.0001
Response Rate (%)
N/A
7.4%
14.4%
9.4%
Table 16. Refusal Conversion Differences (Sample-Census) between Census and Sample Profiles
and Response Rates. An * indicates significance at the experimentwise 0.05 level.
Factors
Age (years)
16-24
25-34
35-44
45-54
55-64
65 and older
Gender
Male
Female
Race/Ethnicity
White (not Hispanic)
Black (not Hispanic)
Native Am./Pac.Is. (not
Hispanic)
Asian (not Hispanic)
Hispanic
Education Attainment
Less than High School
High School or Equivalent
Some College or College
Degree
Masters, Prof. Degree or
Doctorate
Household Income ($)
0-24,999
25,000-49,999
50,000-99,999
100,000 and above
Urban/Rural Residency
Urban
Rural
Census Division of Residency
New England
Middle Atlantic
South Atlantic
East South Central
West South Central
East North Central
West North Central
Mountain
Pacific
Response Rate (%)
Census
No
Letter
Letter
Total
Sample
16.4
17.5
19.3
18.2
12.7
15.9
-5.7*
-3.6*
-2.1
5.4*
5.4*
0.6
-11.1*
-9.8*
-2.2
3.0
7.5*
12.6*
-7.9*
-6.1*
-2.2*
4.4*
6.3*
5.5*
48.7
51.3
-3.8*
3.8*
-0.5
0.5
-2.5*
2.5*
70.6
11.7
0.7
11.8*
-5.0*
1.0*
18.8*
-6.3*
0.2
14.7*
-5.6*
0.7*
4.4
12.6
-1.7*
-6.0*
-3.5*
-9.2*
-2.4*
-7.3*
19.6
28.6
42.9
-11.0*
-2.6
5.3*
-12.6*
2.0
5.1*
-11.7*
-0.7
5.2*
8.9
8.3*
5.6*
7.2*
20.8
29.1
34.8
15.3
-0.8
-2.9
-2.0
5.8*
-3.6*
-3.3
2.6
4.3*
-2.0
-3.1*
-0.1
5.2*
82.8
17.2
-4.6*
4.6*
-5.1*
5.1*
-4.8*
4.8*
5.0
13.9
18.9
6.0
11.1
15.7
6.8
6.6
16.0
N/A
-0.4
-0.8
-3.5*
1.5
-1.6
-1.5
-0.6
1.4
5.6*
7.4%
0.5
-2.7
-0.6
2.8*
-0.3
5.1*
2.8*
-0.7
-7.0*
14.4%
0.0
-1.5
-2.3*
2.0*
-1.1
1.2
0.8
0.5
0.5
9.4%
29
Table 17. Refusal Conversion Differences between sample profiles. An * indicates significance
at the experimentwise 0.05 level.
Factors
Age (years)
16-24
25-34
35-44
45-54
55-64
65 and older
P-value
Gender
Male
Female
P-value
Race/Ethnicity
White (not Hispanic)
Black (not Hispanic)
Native Am./Pac.Is. (not Hispanic)
Asian (not Hispanic)
Hispanic
P-value
Education Attainment
Less than High School
High School or Equivalent
Some College or College Degree
Masters, Prof. Degree or Doctorate
P-value
Household Income ($)
0-24,999
25,000-49,999
50,000-99,999
100,000 and above
P-value
Urban/Rural Residency
Urban
Rural
P-value
Census Division of Residency
New England
Middle Atlantic
South Atlantic
East South Central
West South Central
East North Central
West North Central
Mountain
Pacific
P-value
No Letter versus Letter
5.5*
6.2*
0.1
2.4
-2.2
-12.0*
<0.0001
-3.4
3.4
0.0978
-7.0*
1.3
0.8
1.8*
3.2*
<0.0001
1.6
-4.6*
0.3
2.7
0.0280
2.7
0.4
-4.6
1.4
0.1785
0.5
-0.5
0.7892
-0.9
2.0
-2.9
-1.4
-1.3
-6.7*
-3.4*
2.1
12.5*
<0.0001
30
3. Is there a relationship between socioeconomic/demographic factors and activity
participation?
Even though in our assessment of pre-notification we already estimated logit equations
for nine selected activities, we repeat this estimation for the refusal conversion subsample. A finding of statistically significant relationships combined with over or under
representation of significant factors demonstrates non response bias. Tables 18 and 19
show the unweighted and weighted estimates of activity participation for the refusal
conversion sub-sample by treatment (letter versus no letter).
Table 18. Refusal Conversion Activity Participation Rates by Sample Group: Unweighted
Samples
Samples (Participation Rates)
No
Letter
Total
Activity
Letter
Sample
Walking for exercise or pleasure
0.8597
0.8821
0.8689
Bird Watching
0.4039
0.4136
0.4078
Hunting
0.1214
0.1415
0.1295
Fishing
0.3385
0.3428
0.3403
Motor Boating
0.2906
0.2790
0.2859
Swimming in Natural Waterbodies
0.4666
0.4086
0.4430
Family Gatherings
0.6835
0.7148
0.6980
Day Hiking
0.3445
0.3455
0.3450
Mountain biking
0.1996
0.1859
0.1917
1. Total sample is equal to sample with no letter plus sample with letter.
Table 19. Refusal Conversion Activity Participation Rates by Sample Group: Weighted
Samples
Samples (Participation Rates)
No
Letter
Total
Activity
Letter
Sample
Walking for exercise or pleasure
0.8508
0.8693
0.8577
Bird Watching
0.3070
0.3682
0.3299
Hunting
0.1167
0.1454
0.1274
Fishing
0.3159
0.3450
0.3268
Motor Boating
0.2343
0.2466
0.2389
Swimming in Natural Waterbodies
0.4278
0.3590
0.4022
Family Gatherings
0.6642
0.6888
0.6740
Day Hiking
0.2773
0.3236
0.2961
Mountain biking
0.1790
0.1891
0.1831
1. Total sample is equal to sample with no letter plus sample with letter.
As with the pre-notification letter analysis, we found significant relationships between
socioeconomic/demographic factors and activity participation. Thus, non response bias
is indicated. The results are summarized in Tables 20 and 21.
31
4. Is there a significant difference between estimates of activity participation rates for the
No Letter and Letter sample groups?
As with the pre-notification assessment, we want to determine what the effect of letters
were on our estimates of activity participation. First, we used the logit equations and
tested for the treatment effect by including the dummy variable for receiving a refusal
letter, with 1=received letter and 0=did not receive the letter (See Table 20). This test
found only one significant difference. That was for the activity walking. In the second
test, we constructed 95% confidence intervals and used the overlapping confidence
intervals by comparing the unweighted and weighted estimates of participation rates for
“all refusal conversions” (See Table 22), for refusal conversions that received no letters
(See Table 23) and for “refusal conversions that received the letter” (See Table 24). The
second test is a within sample group test for differences to test the efficacy of sample
weighting, whereas the logit equation approach was an across sample group test of the
effect of letters. In a third test we did a comparison using weighted estimates of activity
participation to test differences between the “No Letter” and “Letter” sample groups (See
Table 25). A statistically significant difference here indicates that sample weighting may
not fully adjust for non response bias. And as in the pre-notification letter assessment,
we did a fourth test, the difference test on mean activity participation rates using
weighted data (See Table 26). The fourth test is slightly more rigorous and relaxes the
assumption of equal variances used in a standard analysis of variance.
Results:
• Logit equation found that the refusal letter resulted in only one significant difference
in activity participation, holding other factors constant. This was for activity walk
(See Table 20).
• Using the confidence interval approach across all refusal conversions, unweighted
estimates of activity participation were generally higher than weighted estimates of
activity participation. The differences were statistically significant for four (4) of the
nine (9) activities tested (Bird, Mboat, Swim_nat and Hike) (See Table 22).
• Using the confidence interval approach for those who did not receive a refusal letter,
again estimates of unweighted activity participation rates were higher than estimates
of weighted activity participation rates. Statistically significant differences were
found for only two (2) of the nine (9) recreation activities tested (Bird and Mboat)
(See Table 23).
• Using the confidence interval approach for those who received the refusal letter,
unweighted estimates of activity participation were higher than weighted estimates of
activity participation for six (6) of the nine (9) activities tested (Walk, Bird, Mboat,
Swim_nat, Fam, and Hike). However, there were no statistically significant
differences(See Table 24).
• Using the confidence interval approach and weighted data, the estimates of activity
participation were higher for the “Letter” sample group than the “No Letter” sample
group for eight (8) of the nine (9) activities tested and lower for one activity
(Swim_nat). However, only two of the differences were statistically significant (Bird
and Swim_nat) (See Table 25).
• Using the difference approach on weighted data yielded the same results as the
confidence interval approach for all activities, except Hunt. The group receiving the
32
letter had a higher and statistically significant participation rate for Hunt than the “no
Letter” sample group (See Table 26).
Table 20. Refusal Conversion Estimated Participation Functions by Activity: Logit Equations.
Activity (Participation Function Coefficients) 1
Walk
Bird
Hunt
Fish
Mboat
Swim_nat
Constant
2.1376*
-2.0601*
-1.7470*
-0.0889
-2.9940*
-0.6133
Age16_2
4
Age25_3
4
-0.5535
0.6437*
0.0315
0.0238
-0.1801
-0.8706*
Age35_4
4
-0.6295
1.0025*
-0.3007
-0.1151
-0.1333
-0.9682*
Age45_5
4
-0.7177*
1.3049*
-0.6850*
-0.4530*
-0.4930*
-1.4775*
Age55_6
4
-0.9469*
1.3387*
-0.6170*
-0.4885*
-0.4297*
-1.9329*
Age65p
-1.4765*
1.2981*
-1.7358*
-1.5056*
-1.1652*
-2.7368*
Male
-0.4124*
-0.4352*
2.0994*
0.9835*
0.2371*
0.0921
White
-0.3227
0.2830
-0.3625
-0.6509
1.0784*
0.0706
Black
-0.6383
-0.3670
-2.0579*
-1.4872*
-0.4385
-1.3191*
Asian
-0.5074
-0.1795
-2.7374*
-1.1213*
0.2126
-1.2703*
Native
Hispanic
0.1159
-0.2143
-1.0869
-0.9786*
0.2051
-0.3752
Educ11
Educhs
0.1618
0.0248
0.0991
0.1616
0.2972
0.1128
Educcoll
0.7312*
0.3938
-0.4074
0.1604
0.5456*
0.6532*
Educgrad
1.0684*
0.7884*
-0.9096*
-0.1923
0.3561
0.9067*
Educoth
2.3689*
0.4049
-0.1092
0.1105
-0.2604
0.1296
Inc25
Inc50
0.3279
0.2083
0.6801*
0.2379
0.6990*
0.4125*
Inc100
0.4904*
0.1902
0.7302*
0.4073*
1.1261*
0.7200*
Inc100p
1.0068*
0.3941*
0.3886
0.3448
1.4715*
1.2243*
Incmiss
0.2388
0.1375
0.3622
0.2266
0.8368*
0.4439*
Urban
0.2783
-0.2061
-0.9686*
-0.2941*
-0.1809
0.1237
Cendiv1
-0.1192
0.6813*
-1.0268*
-0.3381
0.0312
1.5163*
Cendiv2
-0.0772
0.2185
-0.0914
-0.5388*
0.1076
1.1139*
Cendiv3
-0.1842
0.2352
-0.3562
0.1730
0.3024
1.1131*
Cendiv4
-0.1577
0.0662
0.0662
0.2868
0.3512
0.3977
Cendiv5
0.1258
0.1166
0.6271*
0.2602
0.3214
0.2871
Cendiv6
-0.0655
0.2644
-0.1421
0.0968
0.4948*
0.6501*
Cendiv7
0.0674
0.4225
0.5014
0.2989
0.9442*
0.5073*
Cendiv8
Cendiv9
-0.1071
0.1944
-0.5716
-0.3308
0.2915
0.9344*
Rfltr
0.3848*
-0.0543
0.2078
0.0464
-0.1021
-0.0268
1.
*=Significance at .05 or less and blank means dummy category in constant.
33
Table 20 (continued). Refusal Conversion Estimated Participation Functions by Activity: Logit
Equations.
Activity (Participation Function Coefficients) 1
Fam
Hike
Mtnbike
Constant
0.9950
-0.5455
-0.9547
Age16_24
Age25_34
0.3078
-0.0115
0.0334
Age35_44
0.0863
-0.0998
-0.1304
Age45_54
-0.5369
-0.0049
-0.5452
Age55_64
-0.4995
-0.4101
-1.1721*
Age65p
-1.0286*
-1.168*
-2.9934*
Male
-0.0548
0.3471*
0.6253*
White
-0.0558
-0.0226
-0.8710
Black
0.0278
-1.5900*
-2.0287*
Asian
-0.8769
0.4741
-0.7631
Native
Hispanic
-0.0577
0.0253
-1.2529
Educ11
Educhs
-0.1763
0.1651
-0.3171
Educcoll
0.3519
0.0401
-0.0748
Educgrad
0.1170
0.5108
0.3557
Educoth
0.4730
15.5335
2.9908*
Inc25
Inc50
-0.2952
0.3574
-0.2051
Inc100
0.4127
0.5377*
-0.5538
Inc100p
-0.0107
0.6329*
0.0965
Incmiss
0.0024
-0.1116
-0.5754
Urban
-0.1906
0.2583
0.6136*
Cendiv1
1.0715
-0.5218
0.8767
Cendiv2
0.0826
-0.8561*
0.6015
Cendiv3
0.3401
-0.7494*
0.3076
Cendiv4
0.3308
-0.7127*
0.1418
Cendiv5
0.0984
-1.4151*
-0.3831
Cendiv6
0.3012
-0.8694*
0.8212*
Cendiv7
0.1288
-0.4442
0.8235
Cendiv8
Cendiv9
0.2009
-0.3097
0.4429
Rfltr
0.2368
0.1454
0.2296
1.
*=Significance at .05 or less and blank means dummy category in constant.
34
Table 21. Tests (P-values) on the Main Effects in the Refusal Conversion Logit Participation
Models Based on the Wald Chi-Square Test.
Age
Gender
Race
Educ
Income
Urban
Cendiv
Rfltr
Walk
<0.0001
0.0014
0.4180
<0.0001
0.0100
0.0535
0.9742
0.0042
Bird
<0.0001
<0.0001
0.0026
<0.0001
0.2656
0.0539
0.2141
0.5501
Hunt
<0.0001
<0.0001
0.0007
0.0008
0.0235
<0.0001
<0.0001
0.1433
Fish
<0.0001
<0.0001
0.0003
0.1270
0.1798
0.0091
<0.0001
0.6334
Mboat
<0.0001
0.0139
<0.0001
0.0351
<0.0001
0.1236
0.0020
0.3090
Swim_
Nat
<0.0001
0.3260
<0.0001
<0.0001
<0.0001
0.2858
<0.0001
0.7841
Fam
0.0046
0.7889
0.7893
0.3087
0.2349
0.4721
0.8525
0.2535
Hike
<0.0001
0.0150
0.0088
0.2130
0.0140
0.1764
0.0026
0.3212
Mtn
bike
<0.0001
0.0004
0.0663
0.0067
0.0369
0.0182
0.0497
0.2000
Table 22. Differences in Refusal Conversion Unweighted and Weighted Estimates of Activity
Participation Rates: All Refusal Conversions
Unweighted
Weighted
Statistically Significant
Activity
95% C. I.1
95% C.I.2
Difference3
Walk
0.8689
0.8577
No, +
(0.8621,
(0.8507,
0.8756)
0.8647)
Bird
0.4078
0.3299
Yes, +
(0.3980,
(0.3205,
0.4177)
0.3393)
Hunt
0.1295
0.1274
No, +
(0.1228,
(0.1207,
0.1363)
0.1341)
Fish
0.3403
0.3268
No, +
(0.3308,
(0.3174,
0.3497)
0.3362)
Mboat
0.2859
0.2389
Yes, +
(0.2768,
(0.2304,
0.2949)
0.2474)
Yes, +
0.4022
Swim_na
0.4430
(0.3923,
t
(0.4331,
0.4530)
0.4120)
No, +
Fam
0.6980
0.6740
(0.6785,
(0.6541,
0.7176)
0.6940)
Hike
0.3450
0.2961
Yes, +
(0.3305,
(0.2823,
0.3594)
0.3100)
Mtnbike
0.1917
0.1831
No, +
(0.1714,
(0.1798,
0.1949)
0.2037)
1.
95 percent confidence interval on estimated activity participation rates using unweighted data.
95 percent confidence interval on estimated activity participation rates using weighted data.
3.
Yes or No for statistically significant difference between unweighted and weighted estimates of activity
participation rates. + or – indicating unweighted estimate of activity participation rate is greater (+) or less (-) than
the weighted estimate of activity participation rate.
2.
35
Table 23. Differences in Refusal Conversion Unweighted and Weighted Estimates of Activity
Participation Rates: No Letters.
Weighted
Statistically Significant
Unweighted
1
2
95%C.I.
Difference3
Activity
95% C.I.
0.8597
0.8508
Walk
No, +
(0.8420, 0.8774)
(0.8326, 0.8689)
0.4039
0.3070
Yes, +
(0.3789, 0.4289)
(0.2835, 0.3305)
0.1214
0.1167
Hunt
No, +
(0.1047, 0.1380)
(0.1003, 0.1330)
0.3385
0.3159
Fish
No, +
(0.3144, 0.3626)
(0.2922, 0.3396)
0.2906
0.2343
Mboat
Yes, +
(0.2675, 0.3138)
(0.2127, 0.2559)
0.4666
0.4278
Swim_nat
No, +
(0.4412, 0.4920)
(0.4026, 0.4531)
0.6835
0.6642
No, +
Fam
(0.6303, 0.7367)
(0.6102, 0.7183)
0.3445
0.2773
Hike
No, +
(0.3062, 0.3828)
(0.2412, 0.3134)
0.1996
0.1790
Mtnbike
No, +
(0.1646, 0.2287)
(0.1482, 0.2099)
1.
95 percent confidence interval on estimated activity participation rates using unweighted data.
2.
95 percent confidence interval on estimated activity participation rates using weighted data.
3.
Yes or No for statistically significant difference between unweighted and weighted estimates of activity
participation rates. + or – indicating unweighted estimate of activity participation rate is greater (+) or less (-) than
the weighted estimate of activity participation rate.
Bird
Table 24. Differences in Refusal Conversion Unweighted and Weighted Estimates of Activity
Participation Rates: Letters.
Unweighted
Weighted
Statistically Significant
1
2
Activity
95% C.I.
95%C.I.
Difference3
0.8821
0.8693
Walk
No, +
(0.8623, 0.9020)
(0.8486, 0.8901)
0.4136
0.3682
No, +
Bird
(0.3833, 0.4439)
(0.3385, 0.3979)
0.1415
0.1454
Hunt
No, (0.1200, 0.1629)
(0.1237, 0.1671)
0.3428
0.3450
Fish
No, (0.3136, 0.3720)
(0.3157, 0.3742)
0.2790
0.2466
Mboat
No, +
(0.2514, 0.3066)
(0.2200, 0.2731)
0.4086
0.3590
No, +
Swim_nat
(0.3784, 0.4389)
(0.3295, 0.3885)
0.7148
0.6888
Fam
No, +
(0.6592, 0.7705)
(0.6317, 0.7459)
0.3455
0.3236
Hike
No, +
(0.3034, 0.3875)
(0.2822, 0.3649)
0.1859
0.1891
Mtnbike
No, (0.1515, 0.2202)
(0.1545, 0.2237)
1.
95 percent confidence interval on estimated activity participation rates using unweighted data.
2.
95 percent confidence interval on estimated activity participation rates using weighted data.
3.
Yes or No for statistically significant difference between unweighted and weighted estimates of activity
participation rates. + or – indicating unweighted estimate of activity participation rate is greater (+) or less (-) than
the weighted estimate of activity participation rate.
36
Table 25. Differences in Weighted Estimates of Activity Participation Rates: Comparison of
Refusal Conversions With and Without Letters
Without Letters
Statistically Significant
With Letters
1
2
95% C.I.
Difference3
Activity
95% C. I.
Walk
0.8693
0.8508
No, +
(0.8486, 0.8901)
(0.8326, 0.8689)
Bird
0.3682
0.3070
Yes, +
(0.3385, 0.3979)
(0.2835, 0.3305)
Hunt
0.1454
0.1167
No, +
(0.1237, 0.1671)
(0.1003, 0.1330)
Fish
0.3450
0.3159
No, +
(0.3157, 0.3742)
(0.3174, 0.3362)
Mboat
0.2466
0.2343
No, +
(0.2200, 0.2731)
(0.2127, 0.2559)
Swim_na
0.3590
0.4278
Yes, t
(0.3295, 0.3885)
(0.4026, 0.4531)
Fam
Hike
Mtnbike
0.6888
(0.6317, 0.7459)
0.3236
(0.2822, 0.3649)
0.1891
(0.1545, 0.2237)
0.6642
(0.6102, 0.7183)
0.2773
(0.2412, 0.3134)
0.1790
(0.1482, 0.2099)
No, +
No, +
No, +
1.
95 percent confidence interval on estimated activity participation rates using weighted data for refusal
conversions that received refusal letter.
2.
95 percent confidence interval on estimated activity participation rates using weighted data for refusal
conversions that did not receive refusal letter.
3.
Yes or No for statistically significant difference between weighted estimates of activity participation rates for
those who received refusal letter and those who did not receive the refusal letter. + indicating estimate of activity
participation rate for those who did receive the refusal letter is greater (+) or less (-) than the estimate of activity
participation rate for those who did not receive the letter.
Table 26. Comparison of Mean Activity Participation Rates Between Refusal Conversions With
and Without Refusal Letters: Weighted Data – Difference Approach 1
Activity
Statistically Significant Difference 2
Walk
No, +
Bird
Yes, +
Hunt
Yes, +
Fish
No, +
Mboat
No, +
Swim_nat
Yes, Fam
No, +
Hike
No, +
Mtnbike
No, +
1. Difference approach compares differences in weighted means for activity participation rates for two different
sample groups. Refusal conversions for those who received a refusal letter versus those who did not receive a
refusal letter.
2. Yes indicates a statistically significant difference at the 0.05 level and + indicates the mean for the group that
received the letter was greater than the mean for the group that did not receive the letter. No indicates the difference
is not significantly different at the 0.05 level of significance and – indicates that the mean for the group that received
37
the letter was less than the mean of the group that did not receive the letter. This test relaxes the assumption of equal
variances.
5. Do the letters to refusals decrease non response bias?
The tests conducted above indicate that the refusal letters are yielding more
unrepresentative samples and this is having an affect on estimated participation rates.
This makes non response bias worse than without use of the letters. Sample weighting
doesn’t appear to be successful in eliminating all the differences.
6. Are the benefits of the letters to refusals worth the added costs?
No. Given that use of the refusal letters increases cost without any benefits, the
refusal letters are not worth the costs. The letters are yielding more biased samples
with effects on sample estimates of activity participation which reflect more bias than
without the letters.
Part 2: Assess Refusals and Non Response Bias
Assess Refusals: Two-question Survey. A special experiment was done by asking
refusals if they would answer just two questions. Those who agreed were asked their age
and if they participated in walking for exercise or pleasure over the past 12 months.
Gender was recorded, but not asked. We were also able to create a third variable, Census
Division of Residency. This gave us three demographic variables (age, gender, and
Census Division of Residency) that may be related to participation in walking for
exercise or pleasure. In accordance with the previous assessments above, we tested if
first there is a difference in the demographic composition of refusals and the general
population using Census data. We also then tested for differences between the
respondents to the two-question survey versus those who responded to the full survey.
Refusals here answered the two-question survey and they make up the first sample group
for comparison. We want to compare “refusals” to those who responded to the full
survey through “Standard RDD” sampling, i.e. those who completed the full survey and
did not receive pre-notification letters and those who completed the full survey and
received pre-notification letters. Again, following the methods used for assessing the
pre-notification letters, we did a multivariate test to estimate participation rates for
“walking for exercise or pleasure” using a logit equation relating participation to gender,
age, Census Division of Residency, and a dummy variable for whether respondents
answered the two-question survey or the full survey. As with the other assessments
above, we then did univariate tests for differences in participation rates, but here we
limited the tests to unweighted data.
Results:
•
Age. Those who answered the Two-question Survey had a significantly different age
distribution than that of the general population from the Census (See Table 28). The
differences were statistically significant for all age categories using the Bonferroni
adjustment for experimentwise error (See Table 29).
38
•
•
•
Gender. Those who answered the Two-question Survey had a significantly different
gender distribution than that of the general population from the Census (See Table
28). The difference was significant using the Bonferroni adjustment for
experimentwise error (See Table 29).
Census Division of Residency. Those who answered the Two-question Survey were
not significantly different from the general population for Census Division of
Residency (See Tables 28 and 29).
The two-question survey estimate for walk was significantly lower than that from the
full survey. The estimate from the two-question survey was 0.6451 with a 95 percent
confidence interval of (0.6267, 0.6634) compared to the full survey of 0.8723 with a
95 percent confidence interval of (0.8658, 0.8788). The difference approach and the
logit equation also yielded the same conclusion (See Table 27 for the logit equation
results).
Table 27. Refusal Estimated Participation Functions for Walking: Logit equation.
Activity (Participation Function Coefficients) 1
Parameter
Estimated coefficient
Constant
1.2965*
Age16_24
Age25_34
0.0680
Age35_44
0.1956
Age45_54
0.0843
Age55_64
-0.1651
Age65p
-0.5573*
Male
-0.3529*
Cendiv1
-0.1241
Cendiv2
-0.4314*
Cendiv3
-0.4214*
Cendiv4
-0.6051*
Cendiv5
-0.4286*
Cendiv6
-0.4153*
Cendiv7
-0.3956*
Cendiv8
Cendiv9
-0.0667
Trt2
1.2494*
1.
2.
*=significance at .05 or less and blank means dummy category in constant.
Trt=0 is the “Answer Two-Question” Group and TRT=1 is the “Answer Full survey” Group.
Non Respondents. One of the problems with even the above analysis is that for those
who were either hard refusals (refused even the two-question survey) and those eligible,
but never contacted, we know very little about them, and are forced to extrapolate from
what we don’t know to what we do know. It is the nature of all surveys. Here we
conducted an analysis of all the eligible households from our RDD telephone numbers.
We divided the sample of RDD telephone numbers into four sample groups: 1) No
Contact, those who live in eligible households for which we received no answer to
repeated calls; 2) Refusals, those who refused all follow-up efforts (Hard Refusals); 3)
Two-question survey respondents; and 4) Respondents to the full survey (See Figure 4).
39
Here we took the telephone numbers of non respondents and derived Census Division of
where they live. We then tested for significant differences in the distributions of this one
socioeconomic/demographic factor between the four sample groups using the Chi-square
tests (See Table 28). In accordance with the previous assessments above, we then tested
for where the differences existed within the distributions using the Bonferroni method
which adjusts for experimentwise error (See Table 29). Then finally, we tested for
differences between sample groups (See Table 30).
Results for Comparison with Census Distribution (four sample groups):
•
•
There was no difference between Census distributions and distributions for the
sample that answered the two-question survey and those who were “Hard Refusals”
(See Table 28).
There was a significant difference between Census distributions and the distributions
for those who could not be contacted and for respondents to the full survey (See
Table 28).
Table 28. Comparative Profiles for Refusals for Census Division.
No
Factor
Census Answer
Answer
Age (years)
16-24
16.4
4.4
25-34
17.5
6.7
35-44
19.3
11.8
45-54
18.2
12.8
55-64
12.7
16.3
65 and older
15.9
48.1
N
643
Chi-Square
553.8
P-value
<0.000
1
40
No
Contact
-
Total
Survey
8.0
13.2
18.5
21.9
19.3
19.1
9849
1013.2
<0.0001
Factor
Gender
Male
Female
N
Chi-Square
P-value
Census Division of Residency
New England
Middle Atlantic
South Atlantic
East South Central
West South Central
East North Central
West North Central
Mountain
Pacific
N
Chi-Square
P-value
Census
Answer
48.7
51.3
37.5
62.5
No
Answer
40.9
59.1
674
33.6
<0.000
1
5.0
13.9
18.9
6.0
11.1
15.7
6.8
6.6
16.0
4.6
13.7
17.5
6.2
13.7
16.6
7.0
7.8
13.1
681
10.6
0.2264
No
Contact
303
7.3
0.0068
5.2
13.1
17.2
6.7
9.9
18.7
7.9
6.2
15.1
46.6
53.4
9994
17.7
<0.0001
5.0
15.6
14.9
4.3
20.5
4.5
10.0
4.8
20.4
535
6.6
0.5789
Total
Survey
8211
1693.5
<0.0001
5.0
12.6
18.5
7.1
10.4
15.6
7.9
7.5
15.3
10001
73.0
<0.0001
Results for Comparison with Census, where differences exist (four sample groups):
•
•
The “No Contact” sample groups had a statistically significant distribution from the
Census in all Census Divisions, except New England (See Table 29).
The “Full Survey” sample group had a statistically significant distribution from the
Census in four of the nine Census Divisions (See Table 29).
Table 29. Refusal Differences (Sample-Census) between Census and Sample Profiles for Census
Division. An * indicates significance at the experimentwise 0.05 level.
No
No
Total
Factor
Census
Answer
Answer
Contact
Survey
Age (years)
16-24
16.4
-12.0*
-8.4*
25-34
17.5
-10.8*
-4.3*
35-44
19.3
-7.5*
-0.8
45-54
18.2
-5.4*
3.7*
55-64
12.7
3.6*
6.6*
65 and older
15.9
32.2*
3.2*
Gender
Male
48.7
-11.2*
-7.8*
-2.1*
Female
51.3
11.2*
7.8*
2.1*
Census Division of Residency
New England
5.0
-0.4
0.2
0.0
0.0
Middle Atlantic
13.9
-0.2
-0.8
1.7*
-1.3*
41
Factor
South Atlantic
East South Central
West South Central
East North Central
West North Central
Mountain
Pacific
Census
18.9
6.0
11.1
15.7
6.8
6.6
16.0
Answer
-1.4
0.2
2.6
0.9
0.2
1.2
-2.9
No
Answer
-1.7
0.7
-1.2
3.0
1.1
-0.4
-0.9
No
Contact
-4.0*
-1.7*
9.4*
-11.2*
3.2*
-1.8*
4.4*
Total
Survey
-0.4
1.1*
-0.7
-0.1
1.1*
0.9*
-0.7
Results for Sample Group Comparisons (four sample groups):
•
•
•
•
•
•
There was no difference between those who answered the Two-question Survey and
those who were Hard Refusals (See Table 30).
There was a significant difference between those who answered the Two-question
Survey and those who were in the “No Contact sample group. There were significant
differences in five of the nine Census Divisions (See Table 30).
There was no difference between those who answered the Two-question Survey and
those who answered the Full Survey (Se Table 30).
There was a significant difference between those who were “Hard Refusals” and
those who were in the “No Contact” sample group. There were significant
differences in three of the nine Census Divisions (See Table 30).
There were no differences between those who were “Hard Refusals” and those who
answered the Full Survey (See Table 30).
There were significant differences between those in the “No Contact” sample group
and those who answered the Full Survey. There were differences in all Census
Divisions, except the New England Division (See Table 30).
Reasons given by Refusals for not participating in the Full Survey:
Refusals were asked a question before termination of the call.
“Why won’t you participate in the survey?”
In our refusal database we have a total of 1,216 observations. 679 of these people
answered the Two-question survey and the remaining 537 are “Hard Refusals” (wouldn’t
answer the Two-question Survey). Of the 1,216 in the refusal database, 1,206 answered
the question of why they didn’t want to participate in the full survey, while 678 of the
679 who answered the Two-question survey provided answers to the question of why
they wouldn’t participate in the full survey. This question yields additional information
relevant to non response bias.
The answers to the question of why people did not want to participate in the full survey
are summarized in Table 31. There were only a few differences between all refusals and
those who answered the Two-question survey. We estimate that most likely about 20
percent of all those that refuse to participate in the full survey do not participate in any
recreation activities due to either being “too old,” “bad health/too sick,”
42
“disabled/handicapped,” “homebound” or answered directly that they “don’t participate
in outdoor recreation.”
In the 1994-95 NSRE, an analysis was conducted that tested for the effect of a screening
question to allow more rapid exit of the survey for people who don’t participate in any
outdoor recreation and thus reduce respondent burden. Instead of going through the long
list of outdoor recreation activities to determine if a person did not participate in any
outdoor recreation, a screening question was employed that directly asked if they
participated in any outdoor recreation activities during the past 12 months. The analysis
found a significant difference in the proportion of the population that participates in
outdoor recreation using the screening question indicating that people did not understand
the definition of outdoor recreation until they go through the whole list of activities. So,
the screener was not used in NSRE 1999-2000 or NSRE 2005.
When first contact is made in NSRE 2005, people are told that the topic of the survey is
outdoor recreation, and they are given an estimate of how long the survey takes, on
average, and they are told it will take less time if they do not do much outdoor recreation.
For those who don’t participate in outdoor recreation, for whatever reason, they may see
outdoor recreation as a low salience issue and are therefore refusing to participate. Our
estimate is that 20% of all refusals are most likely not participants in any outdoor
recreation (See Table 31). Of all RDD telephone numbers, 58.66% were refusals. So we
estimate that 11.7% (58.66% times 20%) of all RDD telephone numbers are not
participants in outdoor recreation versus 3% of the 10,001 who responded to the full
survey in Versions 1 & 2 of NSRE 2005. Thus, we conclude that non response bias from
refusals will lead to overestimation of activity participation rates. To correct for this we
may have to apply an additional weight to account for this bias.
Conclusions from experiments on non response bias:
•
•
•
There appears to be significant non response bias associated with people that will not
complete the full survey. It would appear that the topic of outdoor recreation results
in people that do not participate in outdoor recreation activities to not participate in
the full survey. The result is a significant upward bias in the one activity we tested
(walking for exercise of pleasure).
Even though Census Division of Residency was a weak predictor of activity
participation, the people we are not able to contact, even after 15 calls, are
significantly different from the Census and all other sample groups.
Reason people gave for refusing to participate in the full survey indicates that about
20% of all refusals at most likely not participants in outdoor recreation versus only 3
percent of respondents to the full survey. Thus, an additional source of non response
bias that will require an additional weight to account for non response bias.
43
Table 30. Differences between Refusal Sample Census Division Profiles. An * indicates
significance at the experimentwise 0.05 level.
Factor
Age (years)
16-24
25-34
35-44
45-54
55-64
65 and older
Chi-Square
P-value
Gender
Male
Female
Chi-Square
P-value
Census Division
New England
Middle Atlantic
South Atlantic
East South
Central
West South
Central
East North
Central
West North
Central
Mountain
Pacific
Chi-Square
P-value
Answer
Versus
No Answer
Answer
Versus
No Contact
-
-
Answer
Versus
Total
Survey
No Answer
Versus
No Contact
No Answer
Versus
Total
Survey
No Contact
Versus
Total
Survey
-
-
-
-3.7*
-6.5*
-6.6*
-9.2*
-3.0
28.9*
313.7
<0.0001
-3.4
3.4
-
-9.1*
9.1*
1.0
0.3144
-
-5.7*
5.7*
20.9
<0.0001
-
3.8
0.0511
-0.7
0.6
0.3
-0.6
-0.4
-1.9
2.6
1.9
-0.5
1.1
-1.1
-1.0
0.3
-2.5
2.3
2.4
0.2
0.5
-1.3
-0.4
-0.1
3.0*
-3.6*
-2.8*
3.7
-6.8*
3.2
-10.6*
-0.5
10.1*
-2.1
12.1*
1.0
14.1*
3.1
-11.0*
-0.8
-3.0*
-0.8
-2.2
0.0
2.2*
1.6
-2.1
3.0*
-7.3*
0.3
-2.2
1.4
-5.2*
-1.3
-0.2
-2.7*
5.1*
7.1
0.5295
224.9
<0.0001
11.6
0.1694
44
236.0
<0.0001
5.2
0.7320
1093.0
<0.0001
Table 31. Reasons Given by Refusals for Not Participating in Survey
All
Two-question Survey
Refusals1
Respondents
Reason
(%)
(%)
1. Not interested
33.00
31.42
2. Don't have time
13.18
14.60
3. Survey too long
2.90
3.98
4. Don't want to participate
15.92
17.85
5. Don't do telephone surveys
2.49
2.80
6. Too old
10.20
12.98
7. Bad health/too sick
4.06
5.16
8. Disabled/handicapped
2.57
3.39
9. Homebound
0.41
0.74
10. Hung up
9.04
0.88
11. Other
3.57
2.65
12. Don't participate in outdoor recreation
2.65
3.54
2
19.89
25.81
Most Likely Not Participants in Outdoor Recreation
1.
All refusals include the 1,206 of the 1,216 in the refusal database that answered the question on the
reason for not participating in the survey. 679 of these people answered the Two-question Survey and
the rest are "Hard Refusals" (would not answer the Two-question Survey).
2.
Sum of responses 6, 7, 8, 9 and 12.
IV. Overall Conclusions
•
•
Pre-notification letters and Refusal Letters increase response rates, but they yield samples
that are more unrepresentative than Standard RDD. This results in significant biases in
activity participation rates. Thus, at any cost, pre-notification and refusal letters do not
pass a benefit-cost test if the objective is to reduce non response bias.
There are differences between those who do and do not respond to the full survey and
these differences do result in non response bias. Current sample weighting is not
accounting for all of the bias. An additional sample weight will have to be constructed to
account for the fact that refusals have a higher rate of non participation in outdoor
recreation than those that respond to the full survey.
45
References
Cordell, H. Ken. (2004). Outdoor Recreation for the 21st Century America, A Report to the
Nation: The National Survey on Recreation and the Environment. Venture Publishing, Inc.,
State College, PA. (NSRE 2000-2001 activity participation rates).
Greene, William H. (1995). LIMDEP Version 7.0. Econometric Software, Inc., Bellport, NY.
Leeworthy, Vernon R., Bowker, J.M., Hospital, Justin D., and Stone, Edward A. (2005).
Projected Participation in Marine Recreation: 2005 & 2010. National Survey on Recreation and
the Environment 2000. U.S. Department of Commerce, National Oceanic and Atmospheric
Administration, National Ocean Service, Special Projects, Silver Spring, MD. March 2005,
pp152. Available at: http://marineeconomics.noaa.gov/NSRE/NSREForecast.pdf.
SAS 9.0. Statistical Analysis System (SAS). SAS Institute, Inc., Carey, NC.
46
| File Type | application/pdf |
| File Title | Corel Office Document |
| Author | smou |
| File Modified | 2006-12-14 |
| File Created | 2006-12-14 |