OMB PreNotification Report - Version 2

Survey Response Rate and Bias Results from a Trial of Pre-notification Letters:

A Report to the Office of Management and Budget on the National Survey on Recreation and the Environment

(NSRE)

Second Submission

January 3, 2008

The Office of Management and Budget has expressed concern that low response rates for the NSRE and other public surveys may be a cause of non-response bias. To increase response rates during the 2005 through 2006 implementation of the NSRE, OMB required a test of pre-notification letters. Also required were increased numbers of call backs to persons not reached and continued implementation of the 2-question quick survey of refusing individuals. These experiments were done to gain information for use in designing future NSRE surveys. This report provides in-depth analysis of the results of testing pre-notification, more callbacks, and refuser questions. Response rates and apparent parameter estimate bias are the primary criteria for evaluating results.

Submitted by the US Forest Service, National Oceanic and Atmospheric Administration, University of Georgia, and University of Tennessee

S urvey Response Rate and Bias Results from a Trial of Pre-notification Letters:

A Report to the Office of Management and Budget on the National Survey on Recreation and the Environment (NSRE)

Science Team and Authors:

Vernon R. Leeworthy¹

Stanley J. Zarnoch

H. Ken Cordell

Gary T. Green

Becky Stephens

Table of Contents Page

I. Introduction: Overview of the Assessment 1

II. Results of Assessment of Pre-notification Letters: Response Rates, Sample

Representativeness and Non-Response Bias 4

1. Did pre-notification letters increase response rates? 4

2. Is there a relationship between response rates and sample representativeness? 5

3. The third question was, “Does improving response rates with pre-notification letters

improve representativeness of samples compared with standard RDD sampling?” 11

4. Is there a relationship between response rates, sample representativeness and

non-response bias? 11

5. Is the non-response bias significant? 16

6. Can sample weighting be used to adjust for non-response bias? 20

7. What is the impact of pre-notification letters on non-response bias? 21

8. Are the benefits of pre-notification letters worth the added costs? 21

III. Assessment of Refusal Conversions and Non-Response Bias 22

Part 1: Assess pre-call letters to refusers 22

1. Did the pre-notification letters to persons who refused (refusal letters) to complete the

survey increase response rates (refusal conversions)? 23

2. Is there a difference in sample representativeness as a result of increasing response

rates through refusal conversions? 23

3. Is there a relationship between demographic factors and activity participation? 28

4. Is there a significant difference between estimates of activity participation rates for

the No Letter and Letter sample groups? 34

5. Do the letters to refusals decrease non-response bias? 34

6. Are the benefits of the letters to refusals worth the added costs? 34

Part 2: Assess refuser two-question survey and non-response bias 34

IV. Overall Conclusions 40

References 41

List of Tables Page

1. Response Rates for each Sample Group Compared to Proportions (from Census) of the U.S.
   Population 16 or Older across Strata (e.g., age 16-24) within Demographic Factors (e.g., age) 4

2. Differences between Census and Sample Group Strata Proportions within Demographic Factors. 7

3. Statistical Test of Differences between Sample Group Demographic Profiles 8

4. Activity Participation Rates by Sample Group using Unweighted Data 12

5. Activity Participation Rates by Sample Group using Weighted Data 12

6. Definitions of Demographic Variables Included in Logit Models 13

7. Estimated Logit Participation Models for Nine Recreation Activities 15

8. Tests (P-values) on the main effects in the Logit participation models based on the Wald
Chi-Square test 16

9. Differences in unweighted and weighted estimates of activity participation rates for the full
sample 17

10. Differences in unweighted and weighted estimates of activity participation for the sample receiving pre-notification letters 17

11. Differences in unweighted and weighted estimates of activity participation rates for the Standard RDD sample (not including Respondents receiving pre-notification letters 18

12. Differences in weighted estimates of activity participation rates between pre-notification and
    standard RDD sample groups 19

13. Comparison of differences between mean activity participation rates of the pre-notification letter
    and standard RDD sample groups 20

14. Comparison of activity participation rate estimates between NSRE 2000-2001 and NSRE 2005 21

15. Comparison of Demographic Profiles between Census and Refusal Conversion Samples, including Samples Sent and Not Sent Advance Letters 25

16. Differences between Demographic Profiles for Census and Samples in the Refusal Call-Notification Letter Experiment 25

17. Differences in Demographic Profiles between Refusers Receiving and Those Not Receiving a Call-Notification Letter 27

18. Estimated Activity Participation Rates by Sample Group for the Refusal Conversion Experiment: Unweighted Data 28

19. Estimated Activity Participation Rates by Sample Group for the Refusal Conversion Experiment: Weighted Data. 30

20. Estimated Logit participation models by activity 30

21. Tests (P-values) on the main effects in the refusal-conversion logit participation models based
on the Wald Chi-Square test 31

22. Differences between Un-weighted and Weighted Estimates of Participation Rates for all Refusal Conversion Respondents 31

23. Differences between Un-weighted and Weighted Estimates of Participation Rates for Refusal Conversion Respondents Not Receiving Letters. 32

List of Tables Page

24. Differences between Un-weighted and Weighted Estimates of Participation Rates for Refusal Conversion Respondents Receiving Letters 32

25. Differences between Estimates of Participation Rates using Weighted Data for Refusal Conversion Respondents Receiving and Not Receiving Letters 33

26. Comparison of Differences in Mean Activity Participation Rates Between Refusal Conversions With and Without Refusal Letters using Weighted Data 33

27. Estimated Logit Model Relating Demographic Profiles of Refusals and their Participation in Walking (Yes or No) 35

28. Comparison with Census of Proportions of Demographic Profiles (response rates) for 2-Question Respondents, Hard Refusals, and No Contact Groups 36

29. Comparison of Differences between Census and Sample Profiles across Census Divisions 37

30. Differences between Census division profiles for refusal samples 39

31. Reasons given by refusals for not participating in survey 39

List of Figures Page

1. Assessment efforts to address non- response bias in NSRE 2005 3

2. NSRE 2005 sample groups 3

3. Sample groups for refusal letter assessment 23

4. Sample groups for non- respondent/refusal analysis 35

Survey Response Rate and Bias Results from a Trial of Pre-notification Letters with the National Survey on Recreation and the Environment

I. INTRODUCTION: Overview of the Assessment

In the review and approval process under the Paperwork Reduction Act (PRA), the U.S. Office of Management and Budget (OMB) required a number of experimental modifications within the sampling design for the National Survey on Recreation and the Environment (NSRE). These experimental modifications were aimed at improving response rates. The hypothesis was that improved response rates would increase sample representativeness of the population and reduce non-response bias. The OMB Approval Number under which the NSRE was conducted following these modifications was 0596-0127 with an expiration date of 8/31/2007.

The major driving objective for specifying the experimental sampling design modifications was to test their effects on improving response rates. Response rates for telephone surveys have been declining steadily since the early 1990s. Some of this decline in response rate has also been evident with NSRE. From 1999 and through 2004, NSRE response rates ranged from a low of 11.1 percent to a high of 21.7 percent. OMB expressed concern that low response rates increase the potential for non-response bias.

The NSRE is a national random digit dialing (RDD) telephone survey of U.S. households. The primary purpose of the NSRE survey is to obtain data describing the U.S. population’s participation in outdoor recreation. The NSRE began as the National Recreation Survey in 1958. It has been implemented periodically a number of times since its beginnings in the 1950s.

Each version of the NSRE consists of a core set of questions that take, on average, about 8 minutes. This core set of questions is asked of all respondents. Its focus is asking about recreation activity participation and a demographic description of the respondent. The total interview time per respondent is restricted to an average of 14 minutes. After administering the core, left is about 6 minutes, on average, for additional modules of questions. Additional modules are designed to meet the individual needs of the various sponsors of NSRE. A module is a set of questions designed to generate data specific to an objective or set of objectives other than the recreation participation and demographics. Different versions of the survey have different combinations of modules.

To increase response rates, the following modifications were added for conducting NSRE from 2004 through 2007:

1. Conduct an experiment on the first two versions of the survey (each version includes approximately 5,000 completed interviews) focusing on RDD telephone numbers which can be matched to mailing addresses. The experimental design specified that pre-notification letters be sent to 50 percent of those with listed numbers having matching mailing addresses. Survey Sampling, Inc. is the firm that supplies the University of Tennessee’s Human Dimensions Research Laboratory with RDD telephone numbers. SSI took all listed telephone numbers with addresses and performed a reverse append to match addresses to telephone numbers. On average, only 40 percent of RDD telephone numbers are listed with addresses.

2. Increase efforts to convert non-contacts (households never reached by phone) and refusals (households answering a call but refusing to complete an interview) by increasing call-backs from 8 to 15 before dropping a number and sending letters to 50 percent of those that refused. This letter was a pre-call notification that they would be called. Only one attempt was made after the letter mailing.

3. Attempt to conduct a two-question survey for all refusals to the full survey to assess potential non-response bias. The two questions include one demographic variable (age) related to a selected recreation activity (walking) and participation in that selected activity during the past 12 months. Gender of the respondent was recorded as perceived by the interviewer.

4. After completing versions 1 thru 3, deliver a report to OMB assessing the costs and benefits of pre-notification and refusal conversion efforts to increase response rates.

The outcome of this experiment would determine whether the NSRE would or would not continue mailing pre-notification letters, follow-up procedures aimed at converting refusals, and the two-question refusal survey to assess non-response bias due to high rates of refusals. The following criteria were specified in the supporting statement approved by OMB:

“In addition to the marginal cost comparison, computed estimates for recreation participation rates by activity for both the sample with and without the pre-notification letter will be examined. If there is a statistically significant difference in estimated participation rates between letter recipients and non-recipients and if the average cost per additional completed response (marginal cost) with an advance letter is no greater than 5 percent more than the average cost per completed interview among those not receiving the advance letter (the outer limit of the budget), then the advance letter procedure will be adopted for the duration of the NSRE 2005.”

The 2005 execution of the NSRE was slowed due to many factors, including the hurricanes Katrina and Rita. Versions 1 and 2 for a total of 10,001 interviews were completed for analysis for this report. The Assessment objectives included:

(1) Assessment of Pre-notification Letters: Response Rates, Sample Representativeness, and Non-Response Bias (reported in Section II)

(2) Assessment of Letters to Refusals (reported in Section III, Part 1)

(3) Assessment of Refusals and Non-Response Bias (reported in Section III, Part 2).

Figure 1 provides a summary of experiment tasks and related samples used for each task. Note that in the conduct of these tasks, there was limited information about non-respondents. This limited the team’s ability to “truth” the results by comparison with known parameters. The procedures used, therefore, were to extrapolate from evidence about respondents to non-respondents. The one exception to extrapolating evidence is the analysis of the class of non-respondents labeled “Hard Refusals.” These are eligible respondents who refused to answer even the two-question survey. For this group, the test for differences was to compare demographic distributions by Census Division of Residency.

Figure 1. Assessment efforts to address non-response bias in NSRE 2005

Figure 2. NSRE 2005 Sample Groups

II. Results of Assessment of Pre-notification Letters: Response Rates, Sample Representativeness, and Non-Response Bias

In assessing the benefits of pre-notification letters, we address eight questions in a sequential analysis. In addressing these questions, the sample was divided into three groups:

1. RDD Respondents with No Matching Addresses

2. RDD Respondents with Matching Addresses, but without a pre-notification letter

3. RDD Respondents with Matching addresses who received a pre-notification letter (See Figure 2).

In addition to considering them as separate groups, groups 1 and 2 were also combined to represent the full sample of respondents that would have been obtained using standard RDD telephone sampling with no stratification for receipt or non-receipt of pre-notification letters. We refer to this group as the Standard RDD Sample Group. This is an important group because it serves as the standard of comparison for the assessment of pre-notification letters. Each of the eight assessment questions is addressed below:

1. Did pre-notification letters increase response rates?

The simple answer to the above question is, yes. The response rate for the sample that received the pre-notification letter was 28.10 percent. The response rate for the sample not receiving the letter, the Standard RDD Sample Group, was 14.08 percent. Thus, the Pre-notification Letter Sample Group response rate was 14 percentage points greater than the no-letter group. This is a much larger increase than we were led to expect by project consultants and the literature. The net increase was about 3.5 percentage points for an overall response rate of 17.62 percent for the Total Sample. Again, a limitation here is that information was available only from respondents in the data currently received. This limits ability to analyze the factors related to non-response. See Table 1 for response rates by sample groups.

Table 1. Response Rates for each Sample Group Compared to Proportions (from Census) of the U.S. Population 16 or Older across Strata (e.g., age 16-24) within Demographic Factors (e.g., age).

Demographic Factors	Census	No Address No Letter	Address No Letter	Address Letter	Standard RDD	Total Sample
Age (years)
16-24	16.4	11.6	8.1	5.3	10.1	8.0
25-34	17.5	16.3	14.4	10.1	15.5	13.2
35-44	19.3	20.6	18.0	17.1	19.5	18.5
45-54	18.2	22.5	21.1	22.0	21.9	21.9
55-64	12.7	16.7	18.9	21.5	17.7	19.3
65 and older	15.9	12.2	19.5	23.9	15.4	19.1
N		3119	2454	4276	5573	9849
Chi-Square (vs. Census)		146.8	224.7	930.6	300.5	1013.2
P-value		<0.0001	<0.0001	<0.0001	<0.0001	<0.0001
Gender
Male	48.7	44.6	45.6	48.6	45.0	46.6
Female	51.3	55.4	54.4	51.4	55.0	53.4
n		3166	2496	4332	5662	9994
Chi-Square		21.6	9.4	0.0	30.4	17.7
P-value		<0.0001	0.0022	0.9350	<0.0001	<0.0001
Race/Ethnicity
White (not Hispanic)	70.6	79.4	86.0	90.2	82.3	85.7
Black (not Hispanic)	11.7	8.6	6.6	4.3	7.7	6.2
Native Am./Pac.Is. (not Hispanic)	0.7	1.9	0.9	0.9	1.5	1.2
Asian (not Hispanic)	4.4	2.3	1.4	1.1	1.9	1.6
Hispanic	12.6	7.9	5.2	3.4	6.7	5.3
n		3108	2442	4252	5550	9802
Chi-Square		206.7	298.2	818.5	464.1	1202.6
P-value		<0.0001	<0.0001	<0.0001	<0.0001	<0.0001
Education Attainment
Less than High School	19.6	8.9	7.7	6.3	8.3	7.5
High School or Equivalent	28.6	23.8	25.8	26.9	24.6	25.6
Some College or College Degree	42.9	51.2	50.3	50.2	50.8	50.5
Masters, Prof. Degree, or Doctorate	8.9	16.2	16.2	16.5	16.2	16.3
n		3128	2442	4255	5570	9825
Chi-Square		448.9	361.1	714.7	806.9	1511.9
P-value		<0.0001	<0.0001	<0.0001	<0.0001	<0.0001
Household Income ($)
0-24,999	20.8	21.7	18.2	18.1	20.2	19.3
25,000-49,999	29.1	25.8	28.3	27.0	26.9	26.9
50,000-99,999	34.8	32.4	36.4	36.7	34.1	35.3
100,000 and above	15.3	20.2	17.1	18.2	18.8	18.6
n		2380	1861	3307	4241	7548
Chi-Square		51.6	11.6	38.3	43.3	73.8
P-value		<0.0001	0.0090	<0.0001	<0.0001	<0.0001
Urban/Rural Residency
Urban	82.8	81.6	76.6	78.4	79.4	79.0
Rural	17.2	18.4	23.4	21.6	20.6	21.0
n		3169	2498	4334	5667	10001
Chi-Square		3.0	67.0	57.6	45.3	101.3
P-value		0.0821	<0.0001	<0.0001	<0.0001	<0.0001
Census Division of Residency
New England	5.0	4.5	5.3	5.3	4.9	5.0
Middle Atlantic	13.9	12.5	12.6	12.6	12.6	12.6
South Atlantic	18.9	17.5	18.8	19.2	18.1	18.5
East South Central	6.0	5.9	7.1	8.1	6.4	7.1
West South Central	11.1	10.3	11.8	9.7	11.0	10.4
East North Central	15.7	13.2	14.7	17.9	13.8	15.6
West North Central	6.8	5.3	7.0	10.3	6.1	7.9
Mountain	6.6	8.5	7.3	6.9	8.0	7.5
Pacific	16.0	22.3	15.4	10.1	19.3	15.3
n		3169	2498	4334	5667	10001
Chi-Square		129.5	14.2	230.6	81.5	73.0
P-value		<0.0001	0.0778	<0.0001	<0.0001	<0.0001
Response Rate (%)	N/A	12.67	16.33	28.10	14.08	17.62

2. Is there a relationship between response rates and sample representativeness?

This question is addressed in two steps. Within step 1, there are two tests.

Step 1, Test 1. The first comparison was between the proportional demographic factor distribution of each sample group and the proportional distribution from the U.S. Census of the non-institutionalized population age 16 years old or older. Demographic factors included age, gender, race/ethnicity, educational attainment, household income, urban/rural residency, and region of residency by Census Division. According to the U.S. Bureau of the Census, proportions of the non-institutionalized population by demographic factor is the most appropriate standard for validating representativeness of respondents to telephone surveys.

For assessing sample representativeness, we conducted several statistical tests on differences between the Census and NSRE sample group distributions across the demographic dimensions shown in Table 1. The first test was to compare the distributions of proportions for each sample group with Census across strata for each demographic factor. This same comparative test was also conducted for the “Total Sample.” A Chi-square test was run using SAS Software, Inc., and PROC FREQ with the option TESTP turned on. The sample sizes, Chi-square values, and p-values for each of these tests (statistical significance of the difference) are reported in Table 1.

Results of Test 1:

• All sample groups, except for the sample with matching address that were sent a letter, were significantly different from the Census distribution for Gender.

• All sample groups and the Total Sample were different from Census for the following demographic factors: Age, Race/Ethnicity, Educational Attainment, and Household Income.

• All sample groups and the Total Sample, with the exception of the sample group that had no matching addresses and received no letter, were different for Urban/Rural Residency.

• All sample groups and the Total Sample were different from the Census for Census Division, except for the group with Address and No Letter.

Step 1, Test 2. The second step-1 test conducted was to identify which specific strata (e.g., age 16 to 24) within a demographic factor (e.g., age) differed from Census for those demographic factors shown in Table 1 to be significantly different from Census. Because of the large number of demographic strata-by-sample group combinations possible, there were a large number of tests performed to identify specific strata differences. To protect from finding false differences, we used a conservative approach (Bonferroni adjustment), which uses an experiment-wise alpha to control error for the set of all tests within a distribution. With this approach, instead of using the 0.05 level on each test for a difference in strata proportion between sample and Census, 0.05 was divided by the number of strata for a factor minus one. This result was used as an adjusted error rate for each within-demographic factor strata test. Example: Race/Ethnicity has five strata: 1) White, not Hispanic, 2) Black, not Hispanic, 3) Native American, Pacific Islander, not Hispanic, 4) Asian, not Hispanic, and 5) Hispanic. The number 0.05 was divided by 4 (5 minus 1), which provides a strata error of 0.0125. However, 0.05 was used as the error across all strata within a single demographic factor (e.g., race/ethnicity), referred to as the experiment-wise error. These results are also summarized in Table 2.

Results of Test 2:

Results for test 2 confirm exactly the results of test 1 (Table 2). That is, there was at least one strata difference found significant in test 2 if the overall factor sample group distribution was found significant in test 1. Similarly, if the distribution was not found significant with test 1, there were no categories found significant with test 2. This consistency was expected, especially when using the Bonferroni approach. The value of these tests is to determine which factor strata are under or over represented in a sample when compared with Census. For instance, all samples had significantly different distributions from Census for age, except for age 25 – 34 in the “No Address-No Letter” sample group and age 35-44 in the Standard RDD sample group. This shows that the young ages (16-24 and 25-34) were significantly under represented as compared to Census for some sample groups, and that the older ages (45-54 and 55-64) were significantly over represented. The middle age (35-44) appears to closely match Census, except for the Address-Letter group. These results can be highly useful when examining participation rates as possible indicators of response bias.

Table 2. Differences between Census and Sample Group Strata Proportions within Demographic Factors. An * indicates significance at the experiment-wise 0.05 level.

Demographic Factors	Census	No Address No Letter	Address No Letter	Address Letter	Standard RDD	Total Sample
Age (years)
16-24	16.4	-4.8*	-8.3*	-11.1*	-6.3*	-8.4*
25-34	17.5	-1.2	-3.1*	-7.4*	-2.0*	-4.3*
35-44	19.3	1.3	-1.3	-2.2*	0.2	-0.8
45-54	18.2	4.3*	2.9*	3.8*	3.7*	3.7*
55-64	12.7	4.0*	6.2*	8.8*	5.0*	6.6*
65 and older	15.9	-3.7*	3.6*	8.0*	-0.5	3.2*
Gender
Male	48.7	-4.1*	-3.1*	-0.1	-3.7*	-2.1*
Female	51.3	4.1*	3.1*	0.1	3.7*	2.1*
Race/Ethnicity
White (not Hispanic)	70.6	8.8*	15.4*	19.6*	11.7*	15.1*
Black (not Hispanic)	11.7	-3.1*	-5.1*	-7.4*	-4.0*	-5.5*
Native Am./Pac.Is. (not Hispanic)	0.7	1.2*	0.2	0.2	0.8*	0.5*
Asian (not Hispanic)	4.4	-2.1*	-3.0*	-3.3*	-2.5*	-2.8*
Hispanic	12.6	-4.7*	-7.4*	-9.2*	-5.9*	-7.3*
Education Attainment
Less than High School	19.6	-10.7*	-11.9*	-13.3*	-11.3*	-12.1*
High School or Equivalent	28.6	-4.8*	-2.8*	-1.7*	-4.0*	-3.0*
Some College or College Degree	42.9	8.3*	7.4*	7.3*	7.9*	7.6*
Masters, Prof. Degree or Doctorate	8.9	7.3*	7.3*	7.6*	7.3*	7.4*
Household Income ($)
0-24,999	20.8	0.9	-2.6*	-2.7*	-0.6	-1.5*
25,000-49,999	29.1	-3.3*	-0.8	-2.1*	-2.2*	-2.2*
50,000-99,999	34.8	-2.4*	1.6	1.9	-0.7	0.5
100,000 and above	15.3	4.9*	1.8	2.9*	3.5*	3.3*
Urban/Rural Residency
Urban	82.8	-1.2	-6.2*	-4.4*	-3.4*	-3.8*
Rural	17.2	1.2	6.2*	4.4*	3.4*	3.8*
Census Division of Residency
New England	5.0	-0.5	0.3	0.3	-0.1	0.0
Middle Atlantic	13.9	-1.4	-1.3	-1.3	-1.3*	-1.3*
South Atlantic	18.9	-1.4	-0.1	0.3	-0.8	-0.4
East South Central	6.0	-0.1	1.1	2.1*	0.4	1.1*
West South Central	11.1	-0.8	0.7	-1.4*	-0.1	-0.7
East North Central	15.7	-2.5*	-1.0	2.2*	-1.9*	-0.1
West North Central	6.8	-1.5*	0.2	3.5*	-0.7	1.1*
Mountain	6.6	1.9*	0.7	0.3	1.4*	0.9*
Pacific	16.0	6.3*	-0.6	-5.9*	3.3*	-0.7
Response Rate (%)	N/A	12.67	16.33	28.10	14.08	17.62

Step 2. Step 2 tested for differences in strata proportion distributions between sample groups for each demographic factor (not compared in this step with Census). Sample group distributions are compared pair-wise, as opposed to comparing a sample distribution to the “known” Census distribution. Three sample group pair-wise comparisons included:

1. The “Address with Letter” sample group versus the “Standard RDD” sample group

2. The “No Address and No Letter” sample group versus the “Address and No Letter” sample group

3. The “Address with Letter” sample group versus the “Address and No Letter” sample group.

Two-way contingency tables were constructed to test for distributional differences using Chi-Square as the test statistic. Subsequent specific tests between proportions of the distribution were conducted by estimating the difference (D=p1-p2) and the associated standard error and then computing a confidence interval. Here again, the Bonferroni adjustment approach was used to control for experiment-wise error. Results are summarized in Table 3.

Table 3. Statistical Test of Differences between Sample Group Demographic Profiles. An * indicates significance at the experiment-wise 0.05 level.

Factors	Address & Letter versus Standard RDD	No Address & No Letter versus Address & No Letter	Address & Letter Versus Address & No Letter
Age (years)
16-24	-4.7*	3.5*	-2.8*
25-34	-5.4*	1.9	-4.3*
35-44	-2.3*	2.7	-0.8
45-54	0.1	1.5	0.9
55-64	3.8*	-2.3	2.6
65 and older	8.5*	-7.3*	4.4*
P-value	<0.0001	<0.0001	<0.0001
Gender
Male	3.6*	-1.1	3.0*
Female	-3.6*	1.1	-3.0*
P-value	0.0003	0.4236	0.0166
Race/Ethnicity
White (not Hispanic)	8.0*	-6.6*	4.2*
Black (not Hispanic)	-3.4*	2.1*	-2.2*
Native Am./Pac.Is. (not Hispanic)	-0.6*	0.9*	-0.1
Asian (not Hispanic)	-0.7*	0.9*	-0.2
Hispanic	-3.2*	2.7*	-1.7*
P-value	<0.0001	<0.0001	<0.0001
Education Attainment
Less than High School	-2.0*	1.2	-1.4
High School or Equivalent	2.3*	-2.0	1.1
Some College or College Degree	-0.5	0.9	-0.1
Masters, Prof. Degree or Doctorate	0.3	0.0	0.3
P-value	0.0004	0.1928	0.1730
Household Income ($)
0-24,999	-2.0	3.5*	-0.1
25,000-49,999	0.1	-2.6	-1.3
50,000-99,999	2.6	-4.0*	0.3
100,000 and above	-0.6	3.1*	1.1
P-value	0.0462	0.0003	0.6457
Urban/Rural Residency
Urban	-1.0	5.0*	1.8
Rural	1.0	-5.0*	-1.8
P-value	0.2354	<0.0001	0.0800
Census Division of Residency
New England	0.4	-0.8	0.0
Middle Atlantic	0.0	-0.1	-0.1
South Atlantic	1.1	-1.3	0.4
East South Central	1.7*	-1.2	1.0
West South Central	-1.3	-1.5	-2.1
East North Central	4.0*	-1.5	3.2*
West North Central	4.2*	-1.7	3.2*
Mountain	-1.0	1.1	-0.4
Pacific	-9.1*	6.9*	-5.2*
P-value	<0.0001	<0.0001	<0.0001

Results of Sample Group Comparisons:

Address with Letter versus Standard RDD:

• There were significant differences in overall distributions between the Address with Letter and Standard RDD sample groups across all demographic factors, except for Urban/Rural Residency.

• Age. The Address and Letter sample group is older than the Standard RDD sample group, with significantly higher concentrations of respondents in the above-55 age groups and lower concentrations in the below-44 age groups. There was no a significant difference for the age group 45-54.

• Gender. The Address and Letter sample group has a significantly higher proportion of males than the Standard RDD sample group.

• Race/Ethnicity. The Address and Letter sample group has a significantly higher proportion of respondents classified as White, Not Hispanic and significantly lower proportions of respondents in all other race/ethnicity strata than the Standard RDD sample group.

• Educational Attainment. The Address and Letter sample group has a slightly higher level of educational attainment than the Standard RDD sample group. The Address and Letter sample group had a significantly lower proportion of respondents in the “less than high school” level of education and a significantly higher proportion of respondents in the “high school or equivalent” level of education. There were no significant differences at the higher levels of education.

• Household Income. There were no significant differences across the income strata, even though the overall distribution was within the significance range at 0.0462. This occasionally occurs because the Bonferroni adjustment is only approximate and may not identify minor differences as significant.

• Urban/rural Residency. There was no a significant difference between urban and rural residency.

• Census Division of Residency. The Address and Letter sample group had significantly higher proportions of respondents in the East South Central, East North Central, and West North Central Census divisions compared with the Standard RDD sample group. Also, the Address and Letter sample group had a significantly lower proportion of respondents in the Pacific Census division of residency.

No Address and No Letter versus Address and No Letter:

These two sample groups together make up the Standard RDD group. However, there are significant differences between those who have listed telephone numbers with addresses versus those who do not. This difference is widely known and is the primary reason for RDD telephone sampling versus sampling from listings in telephone directories.

• There were significant differences in the overall sample distributions across strata for Age, Race/Ethnicity, Household Income, Urban/rural Residency and Census Division of Residency.

• There were no significant differences in the overall sample distributions for Gender and Educational Attainment.

• Age. The No Address and No Letter sample group was younger than the Address and No Letter sample group. The No Address and No Letter sample group had a significantly higher proportion of respondents in the 16-24 age strata and a significantly lower proportion of respondents in the 65 and older age strata when compared to the Address and No Letter sample group. There were no significant differences for all other age strata.

• Gender. There was no significant gender difference between the two sample groups.

• Race/Ethnicity. The No Address and No Letter sample group had a significantly lower proportion of respondents classified as White, Not Hispanic and a significantly higher proportion of respondents classified in all the other race/ethnicity categories.

• Educational Attainment. There were no significant differences in level of education.

• Household Income. The No Address and No Letter sample group had significantly higher proportions of respondents in both the lowest income strata ($0 - $24,999) and the highest income strata ($100,000 and above) when compared with the Address with No Letter sample group. The No Address and No Letter sample group had a significantly lower proportion of respondents in the $50,000 - $99,999 income category than the Address and No Letter sample group. The No Address and No Letter sample group had a lower proportion of respondents in the $25,000 - $49,999 income category as compared with the Address and No Letter sample group, but this difference was not significant.

• Urban/Rural Residency. The No Address and No Letter sample group had a significantly higher proportion of respondents who live in urban areas than the Address & No Letter sample group.

• Census Division of Residency. The No Address and No Letter sample group had a significantly higher proportion of respondents that live in the Pacific Census division than the Address and No Letter sample group. There were no other significant differences for Census division of residency.

Address and Letter versus Address and No Letter:

This comparison directly addresses the response rate effect of using pre-notification letters. Respondents with a matching phone number and mailing address but who were not sent a letter are compared with respondents with an address who were sent a letter. Use of the pre-notification letter does result in some significant differences in the demographic profiles of persons who responded to the survey by completing a phone interview.

• There were significant differences between those with an address who received a letter versus those who did not receive a letter across strata proportions for Age, Gender, Race/Ethnicity, and Census Division of Residency.

• There were no significant differences in the overall sample distributions for Educational Attainment, Household Income, or Urban/Rural Residency.

• Age. The Address and Letter sample group had a significantly lower proportion of respondents in the two age strata under 34 years old and a significantly higher proportion of respondents in the 65 and older age strata, compared with the Address and No Letter sample group. There were no significant differences for the middle age groups (35-64).

• Gender. The Address and Letter sample group had a significantly higher proportion of respondents that are male compared to the Address and No Letter sample group.

• Race/Ethnicity. The Address and Letter sample group had a significantly higher proportion of respondents classified as White-Not Hispanic, significantly lower proportions of respondents for those who were classified as Black or African American-Not Hispanic, and significantly lower for those who were classified as Hispanic.

• Educational Attainment. There were no significant differences for level of education.

• Household Income. There were no significant differences for household income.

• Urban/rural Residency. There was no significant difference for urban/rural residency.

• Census Division of Residency. The Address and Letter sample group had significantly higher proportions of respondents in the East North Central and West North Central Census divisions and a significantly lower proportion of respondents in the Pacific Census division versus the Address and No Letter sample group.

3. The third question was, “Does improving response rates with pre-notification letters improve representativeness of samples compared with standard RDD sampling?”

In the analyses presented in Tables 1-3, significant differences were found between Census and sample group distributions across all sample groups. Also, found were significant differences between the demographic distributions of the Standard RDD and Pre-notification Letter sample groups. An exception was that the Gender distribution of the Pre-notification Letter sample group was closer to the Census distribution. Generally, pre-notification letters appeared to yield a sample less representative of the population than the Standard RDD sample. Below are selected specific findings:

• The Standard RDD sample group distributions were closer to the Census distributions for Age, Race/Ethnicity, Urban/rural Residency and Census Division of Residency (based on significance of Chi-Square statistics).

• The distributions for Education and Income were slightly closer to the Census for the Pre-notification Letter sample as compared to the Standard RDD.

• The Pre-notification Letter sample group distributions were closer to the Census distribution for Gender. This indicates that pre-notification letters have the potential to correct for gender bias noted in telephone surveys.

4. Is there a relationship between response rates, sample representativeness, and non-response bias?

In answering questions 1-3, it was established that there is little relationship between response rates and sample representativeness. In fact, it appears that the net effect of pre-notification letters was to reduce sample representativeness. At the same time, it was found that all the NSRE samples are significantly different from the Census. Without post-stratification sample weighting, distributions are not representative of the population. However, sample representativeness is only a necessary, not sufficient condition for addressing potential non-response bias. Directly addressing non-response bias in the NSRE data requires determining whether there is a relationship between any of the demographic factors, for which there is over or under representativeness, and activity participation rates (the primary parameters the NSRE is designed to estimate).

To examine this potential relationship, the analysis was limited to nine activities: walking for exercise or pleasure (walk), bird watching (bird), hunting (hunt), fishing (fish), motor boating (mboat), swimming in natural water bodies (swim_nat), family gatherings (fam), day hiking (hike), and mountain biking (mtnbike). We chose these because they represent a spectrum of important natural resource-based activities and because they represent a range of participation rates from relatively high to low. These activities also are known to be of different levels of interest across a range of demographic strata in the U.S.

Tables 4 and 5 show the estimated activity participation rates for each of the nine recreation activities for the total sample and for each of the sample groups using both unweighted (Table 4) and Census-weighted (Table 5) data.

Table 4. Activity Participation Rates by Sample Group using Unweighted Data

Activity	Samples (Participation Rates)
	No Address	Address & No Letter	Address & Letter	Standard RDD1	Total Sample
Walking for exercise or pleasure	0.8672	0.8763	0.8738	0.8712	0.8723
Bird Watching	0.3932	0.4231	0.4384	0.4064	0.4202
Hunting	0.1123	0.1393	0.1271	0.1242	0.1255
Fishing	0.3376	0.3395	0.3459	0.3385	0.3417
Motor Boating	0.2872	0.2734	0.2970	0.2811	0.2880
Swimming in Natural Waterbodies	0.4771	0.4383	0.4442	0.4600	0.4531
Family Gatherings	0.7272	0.7103	0.7292	0.7193	0.7237
Day Hiking	0.3327	0.3731	0.3442	0.3520	0.3486
Mountain biking	0.2154	0.1951	0.1861	0.2057	0.1972
1 The Standard RDD sample is a combination of the sample group with no address listing and the sample group with address listings and no letter.

Table 5. Activity Participation Rates by Sample Group using Weighted Data

Activity	Samples (Participation Rates)
	No Address	Address & No Letter	Address & Letter	Standard RDD1	Total Sample
Walking for exercise or pleasure	0.8517	0.8492	0.8523	0.8507	0.8513
Bird Watching	0.3124	0.3403	0.3777	0.3240	0.3450
Hunting	0.1039	0.1287	0.1267	0.1142	0.1191
Fishing	0.3377	0.3155	0.3527	0.3286	0.3380
Motor Boating	0.2290	0.2212	0.2640	0.2257	0.2407
Swimming in Natural Waterbodies	0.4343	0.3769	0.3923	0.4106	0.4034
Family Gatherings	0.7256	0.7061	0.7233	0.7173	0.7197
Day Hiking	0.2832	0.3142	0.3019	0.2966	0.2987
Mountain biking	0.2083	0.1896	0.1751	0.2002	0.1902
1 Standard RDD sample is a combination of the sample group with no address listing and the sample group with address listings and no letter.

To estimate the relationship between demographic factors and activity participation rates, Logit equations were first computed with the unweighted data. The “dummy variable” approach was used based on previous research using NSRE 1999-2000 data for projecting participation in marine recreation (see http://marineeconomics.noaa.gov/NSRE/NSREForecast.pdf).

The demographic variables used in the Logit modeling were the same as those shown in Tables 1 through 3. Dummy variables (0, 1) were specified to represent demographic strata. For the demographic factors age, educational attainment, and household income, the lowest strata was adopted as the baseline (i.e., no dummy variable was included for this lowest strata, Hardy 1993, Regression with Dummy Variables, Sage Publications). For all other demographic factors, the baseline strata selection was based on previous marine recreation activity participation in Leeworthy et al. (2005). See Table 6 for definitions of the variables in the equations.

To test for differences between sample groups, a dummy variable was specified for the pre-notification letter treatment versus the Standard RDD treatment. The variable created was “stndrdd” which was equal to one when the observation was in the “Standard RDD” sample group. This permitted testing the effect on estimates of participation rates from sending pre-notification letters. (A similar test was established to examine effects of sending pre-notification letters to refusers and for extra effort to convert a refusal to a completed interview (in keeping with the additional OMB requirements). This latter test is given extensive discussion in Section III of this report.

Also addressed is unit (item) non-response for the household income demographic factor. Around 24.5 percent of the NSRE sample refused to respond to household income question. To retain this large percentage of the overall base of respondents, those not responding to income were treated as a special analysis group.

Table 6. Definitions of Demographic Variables included in Logit Models

Variable	Description
Age16_24	Dummy variable for age 16 to 24. Value 1=yes 0=no. Used as reference category with constant value of 0 in initial full model estimation.
Age25_34	Dummy variable for age 25 to 34. Value 1=yes 0=no.
Age35_44	Dummy variable for age 35 to 44. Value 1=yes 0=no.
Age45_54	Dummy variable for age 45 to 54. Value 1=yes 0=no.
Age55_64	Dummy variable for age 55 to 64. Value 1=yes 0=no.
Age65p	Dummy variable for age 65 and over. Value 1-yes 0=no.
Male	Dummy variable for gender. Value 1=male 0=female.
White	Dummy variable for Race/Ethnicity, White-Not Hispanic. Value 1=yes 0=no.
Black	Dummy variable for Race/Ethnicity, Black-Not Hispanic. Value 1=yes 0=no.
Asian	Dummy variable for Race/Ethnicity, Asian-Not Hispanic. Value 1=yes 0=no.
Native	Dummy variable for Race/Ethnicity, Native American, Native Hawaiian, or Pacific Islander & Not Hispanic. Value 1=yes 0=no. Used as reference category with constant value of 0 in initial full model estimation.
Hispan	Dummy variable for Race/Ethnicity, those who are Hispanic. Value 1=yes 0=no.
Educ11	Dummy variable for Education, less than High School. Value 1=yes 0=no. Reference in constant in initial full model estimation.
Educhs	Dummy variable for Education, High School, or Equivalent. Value 1=yes 0=no.
Educcoll	Dummy Variable for Education, Some College/College Grad. Value 1=yes 0=no.
Educgrad	Dummy variable for Education, Graduate/Professional Degree. Value 1=yes 0=no.
Educoth	Dummy variable for Education, Other not specified. Value 1=yes 0=no.
Inc25	Dummy variable for Household Income, less than $25,000. Value 1=yes 0=no. Used as reference category with constant value of 0 in initial full model estimation.
Inc50	Dummy variable for Household Income, $25,000 - $49,999. Value 1=yes 0=no.
Inc100	Dummy variable for Household Income, $50,000 - $99,999. Value 1=yes 0=no.
Inc100p	Dummy variable for Household Income, $100,000 & over. Value 1=yes 0=no.
Incmiss	Dummy variable for Household Income, those who did not answer. Value 1=yes 0=no.
Urban	Dummy variable for Residence, Value 1=urban 0=rural.
Cendiv1	Dummy variable for Census Division of Residence, Northeast. Value 1=yes 0=no.
Cendiv2	Dummy variable for Census Division of Residence, Mid Atlantic. Value 1=yes 0=no.
Cendiv3	Dummy variable for Census Division of Residence, S. Atlantic. Value 1=yes 0=no.
Cendiv4	Dummy variable for Census Division of Residence, E S Central. Value 1=yes 0=no.
Cendiv5	Dummy variable for Census Division of Residence, W S Central. Value 1=yes 0=no.
Cendiv6	Dummy variable for Census Division of Residence, E N Central. Value 1=yes 0=no.
Cendiv7	Dummy variable for Census Division of Residence, W N Central. Value 1=yes 0=no.
Cendiv8	Dummy variable for Census Division of Residence, Mountain. Value 1=yes 0=no. Reference category was a constant in initial full model estimation.
Cendiv9	Dummy variable for Census Division of Residence, Pacific, Value 1=yes 0=no.
Stndrdd	Dummy variable for sample treatment. Value 1=Standard RDD 0=Pre-notification letter.
Rfconv	Dummy variable for sample treatment. Value 1=Refusal conversion 0=not a refusal conversion.
Walk	Dummy variable for Activity Participation: Walking for Exercise or Pleasure. Value 1=yes 0=no.
Bird	Dummy variable for Activity Participation: Bird Watching. Value 1=yes 0=no.
Hunt	Dummy variable for Activity Participation: Hunting. Value 1=yes 0=no.
Fish	Dummy variable for Activity Participation: Fishing. Value 1=yes 0=no.
Mboat	Dummy variable for Activity Participation: Motor boating. Value 1=yes 0=no.
Swim_nat	Dummy variable for Activity Participation: Swimming in Natural Water bodies. Value 1-yes 0=no.
Fam	Dummy variable for Activity Participation: Family Gatherings. Value 1=yes 0=no.
Hike	Dummy variable for Activity Participation: Day Hiking. Value 1=yes 0=no.
Mtnbike	Dummy variable for Activity Participation: Mountain Biking. Value 1=yes 0=no.

Logit equations were estimated using both the SAS 9.0 software and LIMDEP 7.0. With SAS, the “main effect” was tested for each demographic factor and pair-wise comparisons were performed. This is analogous to what is usually done in an analysis of variance. The full results are not included in this report since they are not central to the questions asked here.

The results of the Logit equations are summarized in Tables 7 and 8. Note that all the dummy variables corresponding to each category for each factor are included, except for Gender and Urban/rural residency. These two factors are binary variables taking on values of zero or one. A person is either male or female or does or does not live in an urban area. For all other variables (factors), we include all the categorical dummy variables in the table of results. A blank in the table indicates that the category is the reference category and is in the constant.

Results of the Logit Equations:

• Age and Household Income are significant factors in all nine (9) activities tested.

• Gender and Census Division of Residency are significant factors in 8 of 9 activities tested.

• Race/Ethnicity and Educational Attainment are significant factors in 7 of 9 activities tested. The “main effects” test indicates that race/ethnicity and education attainment were significant in 8 of the 9 activities tested. The additional activity was walking for exercise or pleasure. The significance of the “main effects” is shown in Table 8.

• Urban/Rural Residency was a significant factor in 3 of 9 activity logit models tested.

• Sample treatments of pre-notification letters and refusal conversions were not significant factors for any of the nine (9) activities tested.

Conclusion: There is evidence of non-response bias. The sample contains over and under representation for most of the demographic strata. Dummy variables for these strata are significant in explaining variation in activity participation.

Table 7. Estimated Logit Participation Models for Nine Recreation Activities

Activities (Participation Function Coefficients) 2
Factor	Walk	Bird	Hunt	Fish	Mboat	Swim_nat	Fam	Hike	Mtnbike
Constant	2.0135 *	-1.7295 *	-2.4595 *	-0.1996	-2.2920 *	-0.3633	1.4254 *	-0.1291	-0.4047
Age16_24
Age25_34	-0.4028 *	0.2950 *	0.1484	-0.1381	-0.2859 *	-0.8554 *	-0.5042	-0.0015	-0.2590
Age35_44	-0.4196 *	0.6511 *	0.01750	-0.0399	-0.2073	-0.8776 *	-0.3066	0.0409	-0.5090 *
Age45_54	-0.4822 *	0.9523 *	-0.3326 *	-0.3739 *	-0.4490 *	-1.2848 *	-0.8422 *	-0.0358	-0.8405 *
Age55_64	-0.6961 *	1.0253 *	-0.5556 *	-0.7318 *	-0.5743 *	-1.7689 *	-1.0638 *	-0.2807	-1.3171 *
Age65p	-0.8667 *	0.8795 *	-1.2766 *	-1.1430 *	-1.0778 *	-2.6188 *	-1.0568 *	-0.8734 *	-2.2554 *
Male	-0.4790 *	-0.3234 *	1.9424 *	0.8941 *	0.2605 *	-0.0838	-0.1919 *	0.2922 *	0.5045 *
White	0.2184	0.3193	0.3034	-0.2343	0.5561 *	0.01489	-0.0603	-0.1265	-0.4092
Black	0.0136	-0.5101 *	-1.1572 *	-1.0233*	-1.0292 *	-1.5114 *	0.5756	-1.5053 *	-0.7284 *
Asian	-0.5876	-0.3081	-2.1376 *	-0.6259 *	-0.5952	-1.0633 *	-0.6144	-0.7431	-1.1048 *
Native
Hispan	0.3984	-0.0717	-0.2655	-0.4910 *	-0.0695	-0.2902	-0.0736	-0.3153	-0.6593
Educ11
Educhs	0.0919	0.2228 *	0.0642	0.1732	0.3356 *	0.2535 *	0.0311	0.0347	-0.2303
Educcoll	0.6621 *	0.5283 *	-0.3799 *	-0.1052	0.5563 *	0.7895 *	0.3132	0.2549	0.0179
Educgrad	1.1007 *	0.7098 *	-0.8481 *	-0.3349 *	0.5567 *	1.0293 *	0.2864	0.6239 *	0.2875
Educoth	0.5216	0.6037 *	0.3276	0.0247	0.4603	0.5477 *	0.5934	0.0713	0.0161
Inc25
Inc50	0.4378 *	0.1548 *	0.5207 *	0.2822 *	0.5972 *	0.4102 *	0.2352	0.2692 *	0.1371
Inc100	0.5850 *	0.1602 *	0.7252 *	0.4127 *	0.9219 *	0.7177 *	0.5907 *	0.4858 *	0.1431
Inc100p	0.8502 *	0.2737 *	0.4584 *	0.3627 *	1.1969 *	1.0508 *	0.5509 *	0.5160 *	0.5205 *
Incmiss	0.1983 *	-0.04255	0.3506 *	0.1057	0.6480 *	0.3695 *	0.1868	0.0671	-0.1510
Urban	0.0056	-0.1079 *	-0.9219 *	-0.3769 *	-0.05972	0.0962	-0.2239	-0.0160	-0.0016
Cendiv1	-0.1376	0.4145 *	-0.8985 *	-0.1772	0.0721	1.1323 *	0.3516	-0.6420 *	0.0255
Cendiv2	-0.3649 *	0.0564	-0.4078 *	-0.3788 *	-0.0681	0.8177 *	0.1869	-0.7786 *	-0.0032
Cendiv3	-0.3902 *	0.2478 *	-0.3567 *	0.2291 *	0.2390 *	0.8348 *	0.0070	-0.9184 *	-0.0227
Cendiv4	-0.4160 *	-0.07608	-0.0617	0.1944	0.1345	0.1971	0.1612	-1.0940 *	-0.3645
Cendiv5	-0.3848 *	-0.07841	0.4196 *	0.2287 *	0.1533	0.1306	0.2237	-1.2887 *	-0.6838 *
Cendiv6	-0.3010 *	0.1329	-0.1956	-0.0907	0.3018 *	0.3401 *	0.1661	-0.8698 *	0.1349
Cendiv7	-0.3305 *	0.1265	0.3819 *	0.3008 *	0.6234 *	0.1505	0.1893	-0.8318 *	0.0324
Cendiv8
Cendiv9	-0.0444	0.1893 *	-0.6487 *	-0.3201 *	0.0646	0.5304 *	0.4111	-0.3019 *	-0.0192
Standrdd	-0.1161	-0.0342	0.0496	-0.0295	-0.0431	-0.0823	-0.1696	-0.0157	-0.0127
Rfconv	-0.0120	-0.05902	0.0952	0.0469	0.0105	0.0268	-0.1206	-0.0244	-0.0793

Table 8. Tests (P-values) on the Main Effects in the Logit Participation Models Based on the Wald Chi-Square Test.

Factor	Walk	Bird	Hunt	Fish	Mboat	Swim_Nat	Fam	Hike	Mtnbike
Age	<0.0001	<0.0001	<0.0001	<0.0001	<0.0001	<0.0001	<0.0001	<0.0001	<0.0001
Gender	<0.0001	<0.0001	<0.0001	<0.0001	<0.0001	0.0691	0.0424	<0.0001	<0.0001
Ethrace	0.0019	<0.0001	<0.0001	<0.0001	<0.0001	<0.0001	0.0768	<0.0001	0.0305
Educ	<0.0001	<0.0001	<0.0001	0.0006	<0.0001	<0.0001	0.1182	<0.0001	0.0026
Income	<0.0001	<0.0002	<0.0001	<0.0001	<0.0001	<0.0001	0.0012	<0.0001	<0.0001
Urban	0.9509	0.0535	<0.0001	<0.0001	0.3061	0.1067	0.0624	0.8248	0.9839
Cendiv	0.0189	<0.0001	<0.0001	<0.0001	<0.0001	<0.0001	0.4671	<0.0001	0.0003
Standrdd	0.0818	0.4416	0.4996	0.5189	0.3851	0.0777	0.0805	0.7888	0.8585
Rfcon	0.8733	0.2180	0.2235	0.3818	0.8603	0.6294	0.2559	0.7377	0.3900

5. Is the non-response bias significant?

This question was tested by examining differences between unweighted and weighted estimates of participation rates for the nine selected outdoor activities. Multivariate weights were constructed for age, gender, and race/ethnicity using the 2004 Census data for the non-institutionalized population 16 years old and older for weighting the NSRE sample data. As with the NSRE 2000, also applied were multiplicative weights for educational attainment and urban/rural residency.

The multivariate weights included 60 cells (age=6 categories (strata), gender=2 categories and race/ethnicity=5 categories). Sample sizes will not support extending multivariate weighting to educational attainment and urban/rural residency. Extending to educational attainment, which includes five categories, would result in a 300-cell matrix. Sample cell densities in a 300-cell matrix would most likely not support effective computation of weights for all cells, in other words, many of the cells would not have sufficient sample sizes to be representative. Effective weighting, i.e., equalizing sample to population, requires representative samples in each cell.

For statistical tests, four approaches were used. In the first approach, 95-percent confidence intervals were computed for the estimated activity participation rates for both the unweighted and weighted sample data. Statistically significant differences are indicated by non-overlapping confidence intervals, which indicate existence of significant non-response bias. Comparisons of unweighted and weighted estimates of activity participation rates were conducted for the “Full Sample” (Table 9), the “Pre-notification Letter” sample group (Table 10) and the “Standard RDD” sample group (Table 11). Also compared were the weighted estimates of activity participation for the “Pre-notification Letter” sample group to the “Standard RDD” sample group.

Results from Comparisons Using the Overlapping Confidence Interval Approach:

• Full Sample. There were significant differences between the unweighted and weighted estimates for 5 of the 9 activities tested (walk, bird, mboat, swim_nat and hike). Unweighted estimates were always higher than weighted estimates, indicating a general upward bias (Table 9).

• Pre-notification Letter Sample. There were significant differences between the unweighted and weighted estimates for the same 5 of the 9 activities tested (walk, bird, mboat, swim_nat and hike). All the unweighted estimates were higher, except for fishing, which was lower but not significant (Table 10).

• Standard RDD Sample. There were significant differences between the unweighted and weighted estimates for the same 5 of the 9 activities tested for the full and pre-notification samples (walk, bird, mboat, swim_nat, and hike). All the unweighted estimates were higher than the weighted estimates (Table 11).

Table 9. Differences in Unweighted and Weighted Estimates of Activity Participation Rates for the Full Sample

Sample Group/Activity	Unweighted	Weighted	Statistically Significant Difference3
	95% C.I.1	95% C.I.2
Walk	0.8723 (0.8658, 0.8788)	0.8513 (0.8442, 0.8584)	Yes, +
Bird	0.4203 (0.4107, 0.4299)	0.3450 (0.3358, 0.3542)	Yes, +
Hunt	0.1255 (0.1190, 0.1320)	0.1191 (0.1128, 0.1254)	No,+
Fish	0.3417 (0.3325, 0.3509)	0.3380 (0.3288, 0.3472)	No, +
Mboat	0.2880 (0.2792, 0.2968)	0.2407 (0.2323, 0.2491)	Yes, +
Swim_nat	0.4532 0.4434, 0.4630)	0.4034 (0.3938, 0.4130)	Yes, +
Fam	0.7237 (0.7059, 0.7415)	0.7197 (0.7019, 0.7375)	No, +
Hike	0.3486 (0.3355, 0.3617)	0.2987 (0.2860, 0.3114)	Yes, +
Mtnbike	0.1972 (0.1862, 0.2082)	0.1902 (0.1794, 0.2010)	No, +
95 percent confidence interval on estimated activity participation rates using unweighted data.
95 percent confidence interval on estimated activity participation rates using weighted data.
Yes or No for statistically significant difference between unweighted and weighted estimates of activity participation rates. + or - indicating unweighted estimate of activity participation rate is greater (+) or less (-) than the weighted estimate of activity participation rate.

Table 10. Differences in Unweighted and Weighted Estimates of Activity Participation for the Sample receiving Pre-notification Letters

Activity	Unweighted	Weighted	Statistically Significant
	95% C.I.1	95% C.I.2	Difference3
Walk	0.8738 (0.8640, 0.8836)	0.8523 ( 0.8417, 0.8629)	Yes, +
Bird	0.4384 (0.4237, 0.4531)	0.3777 (0.3632, 0.3922)	Yes, +
Hunt	0.1271 (0.1171, 0.1371)	0.1267 (0.1169, 0.1365)	No, +
Fish	0.3459 (0.3318, 0.3600)	0.3527 (0.3384, 0.3670)	No, -
Mboat	0.2970 (0.2835, 0.3105)	0.2640 (0.2509, 0.2771)	Yes, +
Swim_nat	0.4442 (0.4295, 0.4589)	0.3923 (0.3778, 0.4068)	Yes, +
Fam	0.7292 (0.7025, 0.7559)	0.7233 (0.6964, 0.7502)	No, +
Hike	0.3442 (0.3242, 0.3642)	0.3019 (0.2827, 0.3211)	Yes, +
Mtnbike	0.1861 (0.1698, 0.2024)	0.1751 (0.1592, 0.1910)	No, +
95 percent confidence interval on estimated activity participation rates using unweighted data.
95 percent confidence interval on estimated activity participation rates using weighted data.
Yes or No for statistically significant difference between unweighted and weighted estimates of activity participation rates. + or - indicating unweighted estimate of activity participation rate is greater (+) or less (-) than the weighted estimate of activity participation rate.

Table 11. Differences in Unweighted and Weighted Estimates of Activity Participation Rates for the Standard RDD Sample (not including Respondents Receiving Pre-notification Letters)

Activity	Unweighted	Weighted	Statistically Significant
	95% C.I.1	95% C.I.2	Difference3
Walk	0.8712 (0.8626, 0.8798)	0.8507 (0.8415, 0.8599)	Yes, +
Bird	0.4064 (0.3937, 0.4191)	0.3240 (0.3118, 0.3362)	Yes, +
Hunt	0.1242 (0.1156, 0.1328)	0.1141 (0.1059, 0.1223)	No, +
Fish	0.3384 (0.3261, 0.3507)	0.3286 (0.3164, 0.3408)	No, +
Mboat	0.2811 (0.2693, 0.2929)	0.2257 (0.2149, 0.2365)	Yes, +
Swim_nat	0.4600 (0.4471, 0.4729)	0.4106 (0.3979, 0.4233)	Yes, +
Fam	0.7193 (0.6956, 0.7430)	0.7173 (06934, 0.7412)	No, +
Hike	0.3520 (0.3344, 0.3696)	0.2966 (0.2797, 0.3135)	Yes, +
Mtnbike	0.2057 (0.1908, 0.2206)	0.2002 (0.1855, 0.2149)	No, +
1. 95 percent confidence interval on estimated activity participation rates using unweighted data.
2. 95 percent confidence interval on estimated activity participation rates using weighted data.
3. Yes or No for statistically significant difference between unweighted and weighted estimates of activity participation rates. + or - indicating unweighted estimate of activity participation rate is greater (+) or less (-) than the weighted estimate of activity participation rate.

Difference Approach:

An alternative approach, and the one preferred here, is to estimate the difference defined as D = U – W, where U is the unweighted estimate of the participation rate and W is the weighted estimate of the participation rate. Confidence intervals on the difference are constructed as 1.96 times the square root of the variance of D (the 95 percent confidence interval) and serve as the test criterion by comparing the interval to zero. This is a more powerful test than the above overlapping, paired confidence interval test.

An alternative to the analysis of variance approach using PROC MIXED in SAS was adopted to avoid needing to assume equal variances (which is a requirement when using ANOVA). Instead, a difference test was implemented by calculating unweighted and weighted means and standard errors using PROC MEANS in SAS and constructing 95-percent confidence intervals. This latter test does not require assuming equal variances. Detailed results are not reported here since they yielded the same results as the simple, overlapping confidence interval approach reported above.

Comparison of Pre-notification Letter and Standard RDD

Here both the overlapping confidence interval and differences approaches were applied for weighted estimates of activity participation rates for the Pre-notification sample group versus the Standard RDD sample group. Differences = D = P – S, where P is the weighted estimate of the participation rate for the Pre-notification sample group and S is the weighted estimate of the participation rate for the Standard RDD sample group.

Results of the Differences in Pre-notification and Standard RDD Estimates of Weighted Activity Participation Rates:

• Using the overlapping participation rate confidence interval approach, we found significant differences for only two (2) of the nine (9) activities tested (bird and mboat). In both cases, the Pre-notification letter sample group estimates were higher than the Standard RDD sample group estimates (Table 12).

• Using the difference approach, we found significant differences for four (4) of the nine (9) activities tested (bird, fish, mboat and mtnbike). Of the four significant differences, the Pre-notification Letter sample group had higher estimates of activity participation rates than the Standard RDD sample group (bird, fish and mboat). For mountain biking (mtnbike), the Pre-notification Letter sample group estimates were lower than the Standard RDD sample group. See Table 13 for a summary of the results.

Conclusions: There is significant non-response bias in some estimates of activity participation rates. Even after applying sample weighting there are significant differences between estimates of activity participation from the Pre-notification and Standard RDD sample groups.

Table 12. Differences in Weighted Estimates of Activity Participation Rates between Pre-notification and Standard RDD Sample Groups

Activity	Pre-notification	Standard RDD	Statistically Significant
	95% C.I.1	95% C.I.2	Difference3
Walk	0.8523 (0.8417, 0.8629)	0.8507 (0.8415, 0.8599)	No, +
Bird	0.3777 (0.3632, 0.3922)	0.3240 (0.3118, 0.3362)	Yes, +
Hunt	0.1267 (0.1169, 0.1365)	0.1141 (0.1059, 0.1223)	No, +
Fish	0.3527 (0.3383, 0.3669)	0.3286 (0.3164, 0.3408)	No, +
Mboat	0.2640 (0.2509, 0.2771)	0.2257 (0.2149, 0.2365)	Yes, +
Swim_nat	0.3923 (0.3778, 0.4068)	0.4106 (0.3979, 0.4233)	No, -
Fam	0.7233 (0.6964, 0.7502)	0.7173 (06934, 0.7412)	No, +
Hike	0.3019 (0.2827, 0.3211)	0.2966 (0.2797, 0.3135)	No, +
Mtnbike	0.1751 (0.1592, 0.1910)	0.2002 (0.1855, 0.2149)	No, -
95 percent confidence interval on estimated activity participation rates using weighted data for those in the pre-notification letter sample.
95 percent confidence interval on estimated activity participation rates using weighted data for those in the sample that did not receive pre-notification letters or Standard RDD.
Yes or No for statistically significant difference between pre-notification and Standard RDD sample group estimates of activity participation rates. + or - indicating pre-notification sample group estimate of activity participation rate is greater (+) or less (-) than the estimate of activity participation rate for the Standard RDD sample group.

Table 13. Comparison of Differences between Mean Activity Participation Rates of the Pre-notification Letter and Standard RDD Sample Groups¹

Activity	Statistically Significant Difference2,3
Walk	No, +
Bird	Yes, +
Hunt	No, +
Fish	Yes, +
Mboat	Yes, +
Swim_nat	No, -
Fam	No, +
Hike	No, +
Mtnbike	Yes, -
The difference approach compares differences in weighted means for activity participation rates for two different sample groups. Yes indicates a statistically significant difference at the 0.05 significance level and + indicates the mean for the pre-notification sample is greater than the mean for the Standard RDD sample. No indicates the difference is not statistically significant at the 0.05 level of significance and - indicates that the mean for the pre-notification letter sample was less than the mean for the Standard RDD sample. Equal variance assumption relaxed in test.

6. Can sample weighting be used to adjust for non-response bias?

“True” activity participation rates were unknown and thus definitive judgments about which estimates are better were not possible. However, an alternative and widely accepted approach is to compare estimates from the earlier round of NSRE (2000-2001) with current estimates using the entire sample. Generally, current estimates for NSRE 2005 are not greatly different from those estimated in NSRE 2000-2001 (Table 14). For most of the nine activities analyzed in this assessment, the weighted estimates from NSRE 2005 are closer than the unweighted estimates to NSRE 2000-2001 results.

Activity participation rates for marine recreation, in Leeworthy et al. 2005 (see http://marineeconomics.noaa.gov/NSRE/NSREForecast.pdf), were projected to decline from 2000 to 2005 and from 2000 to 2010 based on projected changes in the same demographic factors analyzed in this assessment. One of the most important factors driving the projected declines was Race/Ethnicity. The projected decreases in the proportion of the population that is White, Not Hispanic, relative to increases in the proportions of the population that are Black or African American, Not Hispanic, and that are Hispanic were the major drivers of the projected declines in overall population activity participation rates. The White, Not Hispanic, population generally has higher activity participation rates than Black or African Americans, Not Hispanic, and Hispanics.

Given the expected future declines in activity participation rates, it was expected that NSRE 2005 estimates would be slightly lower than NSRE 2000-2001 estimates. This was true for six of the nine activities tested here using the weighted sample estimates.

Conclusions: It appears that sample weighting can adjust for non-response bias. However, it appears that pre-notification letters are contributing more unrepresentative samples on key variables, such as race/ethnicity. This lessened representativeness is introducing higher non-response bias than sample weighting can adjust.

Table 14. Comparison of Activity Participation Rate Estimates between NSRE 2000-2001 and NSRE 2005

Activity	NSRE 2000-2001	NSRE 2005 Full Sample
	Weighted	Weighted	Unweighted
Walking for exercise or pleasure	83.0	85.1	87.2
Bird Watching	32.4	34.5	42.0
Hunting	11.3	11.9	12.5
Fishing	34.1	33.8	34.2
Motor boating	24.4	24.1	28.8
Swimming in Natural Water bodies	41.7	40.3	45.3
Family Gatherings	73.5	71.2	72.4
Day Hiking	33.3	29.9	34.9
Mountain biking	21.4	19.0	19.7

7. What is the impact of pre-notification letters on non-response bias?

The rationale behind pre-notification letters is that they would increase response rates and thereby reduce non-response bias. However, the basic principle of RDD telephone sampling is that there is a difference between simple random sampling from listed numbers and RDD samples. Because RDD samples include both listed and unlisted numbers, they are more representative of the population. It appears there is a possibility that increasing the proportion of a sample with listed telephone numbers and with matching mailing addresses, relative to the proportion with unlisted numbers, has the potential to introduce bias in estimates of activity participation rates.

“True” population-wide activity participation rates are typically unknown. For a Florida study of boaters that used the reefs in Southeast Florida, however, the “true” distribution of boats by size class was available (Jones et al. 2003). In this Florida study, a stratified random sample was selected from the boat registration file, which contains the names and addresses of the boat owners, along with characteristics of the boat. Telephone numbers were not included in the boat registration files. Florida State University researchers wanted to use a computer-aided telephone instrument (CATI) system to conduct the survey. Telephone listings were used to match addresses with the boat registration files to get telephone numbers (the reverse of our problem). The result was that the sample with telephone numbers was a biased sample. A disproportionately larger proportion of the owners of boats greater than 25 foot in length were among those with unlisted numbers. Given that the reefs were generally 4 to 6 miles offshore, larger boats would have a higher probability of being able to access the reefs. The telephone survey approach was abandoned and a mail survey was used.

Conclusion: Pre-notification letters appear to introduce bias. The letters are generally not correcting biases normally found with Standard RDD sampling methods. Because the letters are going only to people with listed telephone numbers that can be matched to addresses, and these people are different from those with unlisted numbers, increasing the proportion of these people results in the NSRE sample being less representative, thus introducing more bias in estimates of activity participation rates.

8. Are the benefits of pre-notification letters worth the added costs?

The costs of pre-notification include matching telephone numbers to addresses, printing and paper for letters, stuffing and labeling envelopes, handling wrong address returns, responding to letters asking further questions, and paying postage. Overall, these steps increased the average cost per completed interview by 9.8 percent. The offsetting cost reduction of higher response rates was relatively minor in that the average completed interview required 3.5 calls for those who received pre-notification letters, while taking 4.0 calls for those who did not receive the letter. In addition, as demonstrated above, pre-notification letters resulted in less representative samples and more bias in estimates of activity participation. Therefore, according to the criteria set out at the beginning of this assessment, the increase in average costs per interview from the use of pre-notification letters exceeds the 5 percent threshold, while also introducing significant bias. It appears the pre-notification letter experiment indicated that standard RDD samples are superior.

Overall Conclusions on the use of Pre-notification Letters:

• There is evidence of non-response bias in the NSRE samples and the resulting estimates of recreation participation rates. However, there are no apparent bias reduction benefits from utilization of pre-notification letters, even though pre-notification letters increased response rates 14 percentage points (14.08 percent to 28.10 percent).

• There is no relationship between response rates and non-response bias. Non-response bias comes from the mix of people responding. Pre-notification letters result in less representative samples than simple Standard RDD samples.

• Although there is evidence of non-response bias in Standard RDD sampling, sample weighting seems adequate to adjust for this bias.

• The added cost of pre-notification letters with no corresponding bias reduction benefit indicate that the cost-benefit criteria was exceeded.

III. Assessment of Refusal Conversions and Non-Response Bias

Currently, up to 15 calls are made to a single telephone number before dropping that number. This practice will continue in future surveying. Within this pool of people is a significant proportion that are eventually contacted but refuse to complete the survey. At the end of each week’s surveying, a special calling session was set-up to call back refusals. Before these callbacks were begun, letters were sent out to 50 percent of the refusers who had listed telephone numbers and addresses.

In these callbacks to refusers, if they again refused, they were asked if they would answer two quick questions. If they agreed, they were asked their age and if they had participated in walking for exercise or pleasure during the past 12 months. Apparent gender was also recorded, but not asked. As well, Census Division of Residency was recorded for each eligible telephone number.

As with pre-calling notification letters, assessed was whether the added efforts increased refuser response rates, whether more representative samples were obtained overall, whether non-response bias exists, and if non-response bias exists, is it significant. In addition, addressed was whether sample weighting or some other correction method could be used to correct biases. Also addressed was whether refusal letters introduced bias. As with the pre-calling notification letter assessment, the analysis proceeded sequentially.

Part 1: Assess Pre-Call Letters to Refusers

To support the analysis of the pre-call letter to refusers, two sample groups were identified, those who received a letter and those who did not receive a letter (Figure 3). The analysis was organized by posing and addressing 6 questions.

Questions to address Part 1 of the analysis:

1. Did the pre-notification letters to persons who refused (refusal letters) to complete the survey increase response rates (refusal conversions)?

The answer is “Yes.” Those who did not have listed telephone numbers with matching addresses and thus did not receive a refusal letter had a response rate of 7.2 percent. Similarly, those who had matched telephone numbers with addresses, but did not receive a refusal letter had a response rate of 7.6 percent. Overall, refusals that did not receive a refusal pre-notification letter had a response rate of 7.4 percent. Those who received a refusal letter had a response rate of 14.4 percent. The overall response rate for refusals was 9.4 percent (Tables 15 and 16).

2. Is there a difference in sample representativeness as a result of increasing response rates through refusal conversions?

As with the earlier pre-notification letter assessment, the first step was to compare the two sample groups (with and without letters) and the total refusal sample to the Census distributions for all socioeconomic/demographic variables using two tests. Test 1 used SAS PROC FREQ with the TESTP option. A Chi-square test was conducted to test for significance of differences in distributions between Census and sample for each demographic factor at the 0.05 significance level. The second test used the more conservative Bonferroni adjustment to compare strata proportions within each factor and to control for experiment wise error, as described earlier.

Results of Comparison with Census, Test 1:

• Both refusal sample groups and the total sample of refusals were different from the Census for four of the seven demographic factors (Age, Race/Ethnicity, Education level and Urban/Rural residency). For Gender, the Address-No Letter sample group was different from Census with males underrepresented. Thus the letter improved sample representativeness for this factor. For Household Income and Census Division of Residency, the Address-Letter sample group was significantly different from the Census. So for these two factors, the letter resulted in less representative samples (Table 15).

Table 15. Comparison of Demographic Profiles between Census and Refusal Conversion Samples, including Samples Sent and Not Sent Advance Letters.

Factors	Census	No Letter	Letter	Total Sample
Age (years)
16-24	16.4	7.8	5.3	6.0
25-34	17.5	12.4	7.8	9.0
35-44	19.3	13.5	17.0	16.1
45-54	18.2	23.3	21.2	21.8
55-64	12.7	21.8	20.2	20.6
65 and older	15.9	21.2	28.4	26.5
N		386	1069	1455
Chi-Square		67.4	299.1	349.7
P-value		<0.0001	<0.0001	<0.0001
Gender
Male	48.7	42.5	48.3	46.8
Female	51.3	57.5	51.7	53.2
N		395	1080	1475
Chi-Square		6.0	0.1	2.2
P-value		0.0142	0.8095	0.1401
Race/Ethnicity
White (not Hispanic)	70.6	86.5	89.2	88.5
Black (not Hispanic)	11.7	5.5	5.5	5.5
Native Am./Pac.Is. (not Hispanic)	0.7	1.0	0.8	0.9
Asian (not Hispanic)	4.4	3.1	1.1	1.7
Hispanic	12.6	3.9	3.4	3.5
N		385	1061	1446
Chi-Square		51.8	184.5	233.3
P-value		<0.0001	<0.0001	<0.0001
Education Attainment
Less than High School	19.6	7.8	7.2	7.4
High School or Equivalent	28.6	26.5	30.9	29.7
Some College or College Degree	42.9	48.6	47.6	47.8
Masters, Prof. Degree, or Doctorate	8.9	17.1	14.3	15.0
N		385	1064	1449
Chi-Square		60.3	125.0	180.7
P-value		<0.0001	<0.0001	<0.0001
Household Income ($)
0-24,999	20.8	17.8	18.1	18.0
25,000-49,999	29.1	29.7	25.2	26.4
50,000-99,999	34.8	34.3	37.4	36.6
100,000 and above	15.3	18.2	19.2	19.0
N		286	800	1086
Chi-Square		2.8	16.5	17.1
P-value		0.4194	0.0009	0.0007
Urban/Rural Residency
Urban	82.8	78.0	77.6	77.7
Rural	17.2	22.0	22.4	22.3
N		395	1080	1475
Chi-Square		6.5	20.6	27.0
P-value		0.0110	<0.0001	<0.0001
Census Division of Residency
New England	5.0	5.1	5.6	5.5
Middle Atlantic	13.9	10.9	11.2	11.1
South Atlantic	18.9	17.2	18.0	17.8
East South Central	6.0	9.1	9.1	9.1
West South Central	11.1	12.2	10.3	10.8
East North Central	15.7	16.5	20.8	19.7
West North Central	6.8	8.6	9.8	9.5
Mountain	6.6	5.3	6.1	5.9
Pacific	16.0	15.2	9.1	10.7
N		395	1080	1475
Chi-Square		13.1	90.1	90.8
P-value		0.1070	<0.0001	<0.0001
Response Rate (%)	N/A	7.9	14.8	9.7

Results of Comparison with Census, Test 2:

• The results confirm the results from test 1 as expected. The primary purpose of test 2 was to find out where in the demographic distributions strata differences exist (Table 16). Generally, more strata in the Letter sample group were significantly different from Census than from the No-Letter group.

Table 16. Differences between Demographic Profiles for Census and Samples in the Refusal Call-Notification Letter Experiment. An * indicates significance at the experiment-wise 0.05 level.

Factors	Census	No Letter	Letter	Total Sample
Age (years)
16-24	16.4	-8.6*	-11.1*	-10.4*
25-34	17.5	-5.1*	-9.7*	-8.5*
35-44	19.3	-5.8*	-2.3	-3.2*
45-54	18.2	5.1*	3.0	3.6*
55-64	12.7	9.1*	7.5*	7.9*
65 and older	15.9	5.3*	12.5*	10.6*
Gender
Male	48.7	-6.2*	-0.4	-1.9
Female	51.3	6.2*	0.4	1.9
Race/Ethnicity
White (not Hispanic)	70.6	15.9*	18.6*	17.9*
Black (not Hispanic)	11.7	-6.2*	-6.2*	-6.2*
Native Am./Pac.Is. (not Hispanic)	0.7	0.3	0.1	0.2
Asian (not Hispanic)	4.4	-1.3	-3.3*	-2.7*
Hispanic	12.6	-8.7*	-9.2*	-9.1*
Education Attainment
Less than High School	19.6	-11.8*	-12.4*	-12.2*
High School or Equivalent	28.6	-2.1	2.3	1.1
Some College or College Degree	42.9	5.7	4.7*	4.9*
Masters, Prof. Degree or Doctorate	8.9	8.2*	5.4*	6.1*
Household Income ($)
0-24,999	20.8	-3.0	-2.7	-2.8
25,000-49,999	29.1	0.6	-3.8*	-2.7
50,000-99,999	34.8	-0.5	2.6	1.8
100,000 and above	15.3	2.9	4.0*	3.7*
Urban/Rural Residency
Urban	82.8	-4.8*	-5.2*	-5.1*
Rural	17.2	4.8*	5.2*	5.1*
Census Division of Residency
New England	5.0	0.1	0.6	0.5
Middle Atlantic	13.9	-3.0	-2.7	-2.8*
South Atlantic	18.9	-1.7	-0.9	-1.1
East South Central	6.0	3.1	3.1*	3.1*
West South Central	11.1	1.1	-0.8	-0.3
East North Central	15.7	0.8	5.1*	4.0*
West North Central	6.8	1.8	3.0*	2.7*
Mountain	6.6	-1.3	-0.5	-0.7
Pacific	16.0	-0.8	-6.9*	-5.3*
Response Rate (%)	N/A	7.9	14.8	9.7

Results of Comparison between Sample Groups:

Demographic differences were compared between the respondent samples of those who received the refusal letters and those who did not (Table 17).

• There were significant differences for Age and Gender.

• There were no significant differences for Race/Ethnicity, Education, Urban/Rural residency, and Census Division of residency.

• Age. The No-Letter sample group was younger than the Letter sample group. The No-Letter sample group had a significantly lower proportion in the 65 and older age group. Overall, the No-Letter sample group was closer to the Census distribution than the Letter sample group.

• Gender. There were a significantly lower proportion of males in the No-Letter respondent group. The Letter sample group was closer to the Census distribution than the No-Letter sample group.

• Race/Ethnicity. No statistically significant differences.

• Educational Attainment. No statistically significant differences.

• Household Income. No statistically significant differences.

• Urban/rural Residency. No statistically significant difference.

• Census Division of Residency. The only statistically significant difference was that the Address-No Letter sample group had a higher proportion living in the Pacific Division and the Address-No Letter sample was closer to the Census distribution than the Address-Letter sample group, i.e. the Pacific Census Division was better represented without the use of the letter.

Table 17. Differences in Demographic Profiles between Refusers Receiving and Those Not Receiving a Call-Notification Letter. (An * indicates significance at the experiment-wise 0.05 level.)

Factors		No Letter versus Letter
Age (years)
16-24		2.4
25-34		4.7
35-44		-3.6
45-54		2.1
55-64		1.6
65 and older		-7.2*
P-value		0.0025
Gender
Male		-5.8*
Female		5.8*
P-value		0.0480
Race/Ethnicity
White (not Hispanic)		-2.7
Black (not Hispanic)		-0.0
Native Am./Pac.Is. (not Hispanic)		0.2
Asian (not Hispanic)		2.0
Hispanic		0.5
P-value		0.1229
Education Attainment
Less than High School		0.6
High School or Equivalent		-4.4
Some College or College Degree		1.0
Masters, Prof. Degree or Doctorate		2.9
P-value		0.3110
Household Income ($)
0-24,999		-0.3
25,000-49,999		4.5
50,000-99,999		-3.1
100,000 and above		-1.1
P-value		0.5150
Urban/Rural Residency
Urban		0.4
Rural		-0.4
P-value		0.8760
Census Division of Residency
New England	-0.6
Middle Atlantic	-0.3
South Atlantic	-0.7
East South Central	0.0
West South Central	1.9
East North Central	-4.4
West North Central	-1.2
Mountain	-0.8
Pacific	6.1*
P-value	0.0609

Question 2 Conclusions: For five of the seven demographic factors, there were no statistically significant differences between the letter and no-letter refuser sample groups. For one factor, Gender, the letter resulted in a more representative sample. For one factor, Census Division of Residency, the letter resulted in a less representative sample, with the Pacific Census Division being under-represented. It appears that pre-call back letters generally are not improving sampling representativeness.

3. Is there a relationship between demographic factors and activity participation?

Tables 18 and 19 show the unweighted and weighted estimates of activity participation (respectively) for the refusal conversion sub-sample by treatment (letter versus no letter).

Table 18. Estimated Activity Participation Rates by Sample Group for the Refusal Conversion Experiment: Unweighted Data.

Activity	No Letter	Letter	Total Sample
Walking for exercise or pleasure	0.8582	0.8824	0.8759
Bird Watching	0.4608	0.4074	0.4217
Hunting	0.1139	0.1417	0.1342
Fishing	0.3316	0.3417	0.3390
Motor Boating	0.2886	0.2796	0.2820
Swimming in Natural Waterbodies	0.4354	0.4102	0.4169
Family Gatherings	0.6413	0.7148	0.6961
Day Hiking	0.3261	0.3479	0.3423
Mountain Biking	0.1630	0.1939	0.1859

Table 19. Estimated Activity Participation Rates by Sample Group for the Refusal Conversion Experiment: Weighted Data.

Activity	No Letter	Letter	Total Sample
Walking for exercise or pleasure	0.8415	0.8670	0.8595
Bird Watching	0.4006	0.3628	0.3741
Hunting	0.0838	0.1440	0.1261
Fishing	0.3220	0.3425	0.3364
Motor Boating	0.2082	0.2465	0.2351
Swimming in Natural Waterbodies	0.3340	0.3634	0.3547
Family Gatherings	0.7315	0.6845	0.6992
Day Hiking	0.2338	0.3308	0.3020
Mountain Biking	0.1365	0.1966	0.1788

4. Is there a significant difference between estimates of activity participation rates for the No-Letter and Letter sample groups?

As with the pre-notification assessment, the primary objective was to assess the effect of letters on estimates of activity participation. As a first test, logit equations were estimated for the responding sample of refusers focusing on the same nine selected activities as studied in the pre-notification tests. A finding of statistically significant relationships, combined with over or under representation of demographic strata, indicated likely non-response bias. To isolate the letter treatment effect, a dummy variable for receiving a refusal letter was created, with 1=received letter and 0=did not (Table 20). This test found only one significant difference, for the activity bird watching.

As a second test, we constructed 95 percent confidence intervals and used overlapping confidence intervals as the test criterion for comparing unweighted and weighted estimates of participation rates for “all refusal conversions” (Table 22), for refusal conversions that received no letters (Table 23) and for “refusal conversions that received the letter” (Table 24). This test examined the efficacy of sample weighting, whereas the Logit equation approach was an across-sample test of the effect of letters.

A third test was to compare weighted estimates of activity participation to test for differences between the “No Letter” and “Letter” sample groups (Table 25). A statistically significant difference here indicated that sample weighting may not fully adjust for non-response bias.

And as in the pre-notification letter assessment, a fourth test was a difference test on mean activity participation rates using weighted data (Table 26). This fourth test is a bit more rigorous and relaxes the assumption of equal variances used in a standard analysis of variance.

Results:

• The Logit equation approach revealed that the refusal letter resulted in only one significant difference in activity participation, holding other factors constant. This was for the activity bird watching (Table 20). As with the pre-notification letter analysis, significant relationships were found between demographic factors and activity participation. This indicates likely non-response bias. The results are summarized in Tables 20 and 21.

• Using the confidence interval approach across all refusal conversions, unweighted estimates of activity participation were generally higher than weighted estimates of activity participation. The differences were statistically significant for two of the nine activities tested (Mboat and Swim_nat). See Table 22.

• Using the confidence interval approach for those who did not receive a refusal letter, estimates of unweighted activity participation rates were higher than estimates of weighted activity participation rates. Statistically significant differences were found for only one of the nine recreation activities tested (Swim_nat). See Table 23.

• Using the confidence interval approach for those who received the refusal letter, unweighted estimates of activity participation were higher than weighted estimates of activity participation for six of the nine activities tested (Walk, Bird, Mboat, Swim_nat, Fam, and Hike). However, there were no statistically significant differences. See Table 24.

• Using the confidence interval approach and weighted data, the estimates of activity participation were higher for the “Letter” sample group than the “No Letter” sample group for seven of the nine activities tested and lower for two activities (Bird and Fam). However, only one of the differences was statistically significant (Hunt). See Table 25.

• Using the difference approach on weighted data yielded the same results as the confidence interval approach for all activities, except Hike. The group receiving the letter had a higher and statistically significant participation rate for Hike than the “No Letter” sample group. See Table 26.

Table 20. Estimated Logit Participation Models by Activity.

Activities (Participation Function Coefficients) 3
Factor	Walk	Bird	Hunt	Fish	Mboat	Swim_nat	Fam	Hike	Mtnbike
Constant	1.4407	-2.2026*	-1.1724	0.2975	-4.2839*	-1.0785	0.7636	-1.2492	-0.9908
Age16_24
Age25_34	-0.3738	0.7293	0.0768	-0.5164	0.0426	-0.6171	0.0850	0.1522	0.3552
Age35_44	-0.7977	1.1870*	-0.0546	-0.4172	0.1466	-0.7393*	-0.3651	-0.1098	0.2578
Age45_54	-0.5804	1.4495*	-0.6449	-0.8171*	-0.0600	-1.3380*	-0.7295	0.2032	-0.2665
Age55_64	-0.8117	1.6157*	-0.2863	-0.9218*	-0.2472	-1.7484*	-0.5395	-0.2376	-1.0386*
Age65p	-1.4041*	1.4199*	-1.6126*	-1.8949*	-0.7271*	-2.6214*	-0.9978	-1.0331*	-2.6487*
Male	-0.2955	-0.3193*	2.1108*	1.1035*	0.1760	0.1057	-0.0276	0.2795	0.6522*
White	0.1465	0.5635	-0.8348	-0.4573	1.6772	0.3580	-0.4226	0.3162	-0.6819
Black	-0.0708	-0.0628	-2.0659*	-1.1343	0.2065	-1.2334	-0.0542	-0.9978	-1.5405
Asian	-0.3844	0.1059	-14.5803	-0.7703	1.0722	-1.6720*	0.4758	0.8024	-0.4607
Native
Hispanic	0.8753	0.6805	-2.3564*	-0.9389	0.8944	0.1154	0.2619	0.5874	-0.8140
Educ11
Educhs	0.4283	-0.1532	0.2114	0.2263	0.4034	0.2840	-0.2061	0.0133	-0.5391
Educcoll	0.8060*	0.0257	-0.3415	0.0528	0.6038	0.7418*	0.0042	-0.1331	-0.5367
Educgrad	1.2173*	0.2236	-0.9605*	-0.2096	0.3146	0.9514*	-0.0290	0.0864	-0.1688
Educoth	14.9940	-0.1080	-0.1517	0.0393	-0.2072	0.2813	-0.3841	2.2286	1.4831
Inc25
Inc50	0.5565*	0.2571	0.3972	-0.1112	0.6408*	0.3635	-0.1279	0.6856	-0.1544
Inc100	0.6248*	0.1662	0.4740	0.2389	1.0939*	0.6594*	0.8827	0.6596	-0.5146
Inc100p	1.5171*	0.4875*	0.1048	0.0307	1.5279*	1.3020*	0.6833	0.8296*	-0.0599
Incmiss	0.4368	-0.0542	0.0519	-0.1187	0.8024*	0.4144	0.1631	0.1140	-0.6367
Urban	0.2734	-0.0494	-0.6742*	-0.1178	0.0063	0.2563	-0.4393	0.1844	0.5833
Cendiv1	-0.5241	0.9769*	-1.2166*	-0.6576	0.6616	1.2137*	1.5389*	-0.4283	1.2113*
Cendiv2	-0.1095	0.4781	-0.2562	-0.8021*	0.3726	1.0335*	0.5558	-0.2759	0.5531
Cendiv3	-0.3898	0.2480	-0.6610	0.2902	0.5424	0.9852*	0.8303	-0.4390	0.1159
Cendiv4	-0.1140	-0.0766	0.0198	0.2030	0.8038*	0.2742	1.0773	-0.2884	-0.2155
Cendiv5	-0.0686	0.2325	0.3853	0.1270	0.5086	0.0681	0.7840	-1.0831*	-0.5934
Cendiv6	-0.1435	0.3542	-0.4322	0.1094	0.8099*	0.6315*	1.0000	-0.5286	0.8934
Cendiv7	-0.2608	0.2228	0.0596	0.1982	1.2138*	0.2962	0.7574	-0.3848	0.4822
Cendiv8
Cendiv9	-0.0522	0.1705	-1.2428*	-0.4944	0.3080	0.7266*	1.3285*	0.3380	0.7608
Rfltr	0.2993	-0.2649*	0.1782	-0.0437	-0.1200	-0.0428	0.3572	0.1531	0.3039

Table 21. Tests (P-values) on the main effects in the refusal-conversion logit participation models based on the Wald Chi-Square test.

	Walk	Bird	Hunt	Fish	Mboat	Swim_Nat	Fam	Hike	Mtn bike
Age	0.0005	<0.0001	<0.0001	<0.0001	0.0008	<0.0001	0.2797	0.0010	<0.0001
Gender	0.0843	0.0052	<0.0001	<0.0001	0.1652	0.3959	0.9144	0.1144	0.0036
Race	0.6479	0.1229	0.0302	0.0957	0.0007	<0.0001	0.7945	0.0888	0.5407
Educ	0.0208	0.3899	0.0067	0.3487	0.1479	0.0025	0.9687	0.3085	0.2261
Income	0.0113	0.0728	0.3090	0.2478	<0.0001	<0.0001	0.0640	0.0499	0.3264
Urban	0.1511	0.7181	0.0005	0.4275	0.9675	0.0934	0.1867	0.4237	0.0652
Cendiv	0.9241	0.0447	0.0021	<0.0001	0.0052	<0.0001	0.5450	0.0690	0.0312
Rfltr	0.1133	0.0384	0.3965	0.7546	0.3973	0.7591	0.2063	0.4513	0.2407

Table 22. Differences between Un-weighted and Weighted Estimates of Participation Rates for all Refusal Conversion Respondents.

Activity	Unweighted 95% C. I.1	Weighted 95% C.I.2	Statistically Significant Difference3
Walk	0.8759 (0.8591, 0.8928)	0.8595 (0.8417, 0.8772)	No, +
Bird	0.4217 (0.3965, 0.4469)	0.3741 (0.3493, 0.3988)	No, +
Hunt	0.1342 (0.1168, 0.1517)	0.1261 (0.1092, 0.1431)	No, +
Fish	0.3390 (0.3148, 0.3632)	0.3364 (0.3123, 0.3606)	No, +
Mboat	0.2820 (0.2590, 0.3050)	0.2351 (0.2135, 0.2568)	Yes, +
Swim_nat	0.4169 (0.3918, 0.4421)	0.3547 (0.3303, 0.3791)	Yes, +
Fam	0.6961 (0.6485, 0.7437)	0.6992 (0.6517, 0.7467)	No, -
Hike	0.3423 (0.3073, 0.3772)	0.3020 (0.2682, 0.3359)	No, +
Mtnbike	0.1859 (0.1572, 0.2146)	0.1788 (0.1505, 0.2070)	No, +

1. 95 percent confidence interval on estimated activity participation rates using unweighted data.

2. 95 percent confidence interval on estimated activity participation rates using weighted data.

3. Yes or No for statistically significant difference between unweighted and weighted estimates of activity participation rates. + or – indicating unweighted estimate of activity participation rate is greater (+) or less (-) than the weighted estimate of activity participation rate.

Table 23. Differences between Un-weighted and Weighted Estimates of Participation Rates for Refusal Conversion Respondents Not Receiving Letters

Activity	Unweighted 95% C.I.1	Weighted 95% C.I.2	Statistically Significant Difference3
Walk	0.8582 (0.8237, 0.8928)	0.8415 (0.8054, 0.8777)	No, +
Bird	0.4608 (0.4114, 0.5101)	0.4006 (0.3521, 0.4492)	No, +
Hunt	0.1139 (0.0825, 0.1454)	0.0838 (0.0563.1112)	No, +
Fish	0.3316 (0.2850, 0.3783)	0.3220 (0.2757, 0.3683)	No, +
Mboat	0.2886 (0.2437, 0.3335)	0.2082 (0.1680, 0.2484)	No, +
Swim_nat	0.4354 (0.3863, 0.4846)	0.3340 (0.2872.3807)	Yes, +
Fam	0.6413 (0.5414, 0.7412)	0.7315 (0.6392, 0.8237)	No, -
Hike	0.3261 (0.2577, 0.3945)	0.2338 (0.1721, 0.2955)	No, +
Mtnbike	0.1630 (0.1092, 0.2169)	0.1365 (0.0864, 0.1865)	No, +

1. 95 percent confidence interval on estimated activity participation rates using unweighted data.

2. 95 percent confidence interval on estimated activity participation rates using weighted data.

3. Yes or No for statistically significant difference between unweighted and weighted estimates of activity participation rates. + or – indicating unweighted estimate of activity participation rate is greater (+) or less (-) than the weighted estimate of activity participation rate.

Table 24. Differences between Un-weighted and Weighted Estimates of Participation Rates for Refusal Conversion Respondents Receiving Letters

Activity	Unweighted 95% C.I.1	Weighted 95% C.I.2	Statistically Significant Difference3
Walk	0.8824 (0.8632, 0.9016)	0.8670 (0.8468, 0.8873)	No, +
Bird	0.4074 (0.3781, 0.4368)	0.3628 (0.3341, 0.3916)	No, +
Hunt	0.1417 (0.1208, 0.1625)	0.1440 (0.1230, 0.1650)	No, -
Fish	0.3417 (0.3133, 0.3700)	0.3425 (0.3142, 0.3709)	No, -
Mboat	0.2796 (0.2528, 0.3064)	0.2465 (0.2207, 0.2722)	No, +
Swim_nat	0.4102 (0.3808, 0.4396)	0.3634 (0.3347, 0.3922)	No, +
Fam	0.7148 (0.6606, 0.7690)	0.6845 (0.6288, 0.7403)	No, +
Hike	0.3479 (0.3071, 0.3887)	0.3308 (0.2905, 0.3712)	No, +
Mtnbike	0.1939 (0.1600, 0.2278)	0.1966 (0.1626, 0.2307)	No, -

1. 95 percent confidence interval on estimated activity participation rates using unweighted data.

2. 95 percent confidence interval on estimated activity participation rates using weighted data.

3. Yes or No for statistically significant difference between unweighted and weighted estimates of activity participation rates. + or – indicating unweighted estimate of activity participation rate is greater (+) or less (-) than the weighted estimate of activity participation rate.

Table 25. Differences between Estimates of Participation Rates using Weighted Data for Refusal Conversion Respondents Receiving and Not Receiving Letters.

Activity	With Letters 95% C. I.1	Without Letters 95% C.I.2	Statistically Significant Difference3
Walk	0.8670 (0.8468, 0.8873)	0.8415 (0.8054, 0.8777)	No, +
Bird	0.3628 (0.3341, 0.3916)	0.4006 (0.3521, 0.4492)	No, -
Hunt	0.1440 (0.1230, 0.1650)	0.0838 (0.0563, 0.1112)	Yes, +
Fish	0.3425 (0.3142, 0.3709)	0.3220 (0.2757, 0.3683)	No, +
Mboat	0.2465 (0.2207, 0.2722)	0.2082 (0.1680, 0.2484)	No, +
Swim_nat	0.3634 (0.3347, 0.3922)	0.3340 (0.2872, 0.3807)	No, +
Fam	0.6845 (0.6288, 0.7403)	0.7315 (0.6392, 0.8237)	No, -
Hike	0.3308 (0.2905, 0.3712)	0.2338 (0.1721, 0.2955)	No, +
Mtnbike	0.1966 (0.1626, 0.2307)	0.1365 (0.0864, 0.1865)	No, +

1. 95 percent confidence interval on estimated activity participation rates using weighted data for refusal conversions that received refusal letter.

2. 95 percent confidence interval on estimated activity participation rates using weighted data for refusal conversions that did not receive refusal letter.

3. Yes or No for statistically significant difference between weighted estimates of activity participation rates for those who received refusal letter and those who did not receive the refusal letter. + indicating estimate of activity participation rate for those who did receive the refusal letter is greater (+) or less (-) than the estimate of activity participation rate for those who did not receive the letter.

Table 26. Comparison of Differences in Mean Activity Participation Rates Between Refusal Conversions With and Without Refusal Letters using Weighted Data ¹

Activity	Statistically Significant Difference 2
Walk	No, +
Bird	No, -
Hunt	Yes, +
Fish	No, +
Mboat	No, +
Swim_nat	No, +
Fam	No, -
Hike	Yes +
Mtnbike	No, +

1. Difference approach compares differences in weighted means for activity participation rates for two different sample groups. Refusal conversions for those who received a refusal letter versus those who did not receive a refusal letter.

2. Yes indicates a statistically significant difference at the 0.05 level and + indicates the mean for the group that received the letter was greater than the mean for the group that did not receive the letter. No indicates the difference is not significantly different at the 0.05 level of significance and – indicates that the mean for the group that received the letter was less than the mean of the group that did not receive the letter. This test relaxes the assumption of equal variances.

5. Do the letters to refusals decrease non-response bias?

The tests conducted above indicate that the refusal letters are not generating more representative samples. The letters are yielding higher estimates of participation rates in general, but few of the differences are statistically significant. Since “true” participation rates are unknown, it is difficult to conclude whether letters are or are not reducing non-response bias.

6. Are the benefits of the letters to refusals worth the added costs?

Generally, it appears the pre-call letters to refusers increase costs without the benefit of reducing non-response bias.

Part 2: Assess Refuser Two-Question Survey and Non-Response Bias

Assess Refusals Two-question Survey: A special experiment was run by asking people who refused the survey if they would answer just two questions. Those who agreed were asked their age and if they participated in walking for exercise or pleasure over the past 12 months. Gender was recorded, but not asked. It was possible also to create a third variable, Census Division of Residency. This gave us three demographic variables (age, gender, and Census Division of Residency) to test for relationship with participation in walking for exercise or pleasure. As in the assessments described above, the first test was whether there was a difference in the demographic composition of refusers compared with the general population based on Census data. Also tested were differences between the respondents to the two-question survey and those who responded to the full survey to determine potential non-response bias.

The people who refused the full survey (refusals), but who answered the two-question survey made up the first sample group for comparison. The second sample group included those who responded to the full survey through “Standard RDD” sampling, i.e. those who completed the full survey and did not receive pre-notification letters. A third sample group compared included those who completed the full survey and received pre-notification letters. Again, following the methods used for assessing the pre-notification letters, we did a multivariate test to estimate participation rates for “walking for exercise or pleasure.” Logit modeling was used for relating participation to gender, age, and region of residency. A dummy variable was specified for whether or not respondents answered the two-question survey or the full survey. As with the other assessments described earlier in this report, univariate tests were then run for differences in participation rates, limited using unweighted data.

Results:

• Age. Those who answered the Two-question Survey had a different age distribution than that of the general population based on Census (Table 28). The differences were statistically significant for all age categories using the Bonferroni adjustment for experiment-wise error (Table 29).

• Gender. Those who answered the Two-question Survey had a statistically significant gender distribution than that of the general population (Table 28). As well, the difference was significant using the Bonferroni adjustment for experiment-wise error (Table 29).

• Census Division of Residency. Those who answered the Two-question Survey were not significantly different from the general population for Census Division of Residency (Tables 28 and 29).

• The two-question survey participation rate estimate for the activity of walking was significantly lower than that from the full survey. The estimate from the two-question survey was 0.6451 with a 95-percent confidence interval of (0.6267, 0.6634) compared to the full survey of 0.8723 with a 95-percent confidence interval of (0.8658, 0.8788). The difference approach and the logit equation yielded the same conclusion. See Table 27 for the logit equation results.

Table 27. Estimated Logit Model Relating Demographic Profiles of Refusals and their Participation in Walking (Yes or No)

Activity (Participation Function Coefficients) 1
Parameter	Estimated coefficient
Constant	1.2965*
Age25_34	0.0680
Age35_44	0.1956
Age45_54	0.0843
Age55_64	-0.1651
Age65p	-0.5573*
Male	-0.3529*
Cendiv1	-0.1241
Cendiv2	-0.4314*
Cendiv3	-0.4214*
Cendiv4	-0.6051*
Cendiv5	-0.4286*
Cendiv6	-0.4153*
Cendiv7	-0.3956*
Cendiv9	-0.0667
Trt2 (Trt = 1 = the full survey group)	1.2494*

1. *=significance at .05 or less and blank means dummy category in constant.

Non-Respondents to either Full Survey or Two-Question Survey. One of the problems with any analysis to identify potential non-response bias is very little is known about those who were either hard refusals (refused even the two-question survey) or those who were eligible as respondents, but were never contacted. The only approach is to extrapolate from what is known, to what is unknown. The approach used was to conduct an analysis of all the eligible households from the total pool of RDD telephone numbers. This sample pool of RDD telephone numbers was divided into four groups: 1) No Contact Made--those who live in eligible households for which we received no answer to repeated calls; 2) Hard Refusals--those who refused all follow-up efforts; 3) Two-question survey respondents; and 4) Respondents to the full survey (See Figure 4).

From area codes of telephone numbers of non-respondents Census Division of Residency was derived. Next testing for significant differences in the distributions across divisions of residency was conducted between the four sample groups using Chi-square tests (Table 28). Next tests were run to identify which sample groups were different in which regions using the Bonferroni method to adjust for experiment-wise error (Table 29). Finally, tests were run for differences between sample groups (Table 30).

Results for Comparison with Census Distribution (four sample groups):

• There were no significant differences between Census distributions and distributions for the sample group that answered the two-question survey and for the “Hard Refusals” group (Table 28).

• There were significant differences between Census distributions and the distributions for those who could not be contacted and for respondents to the full survey (Table 28).

Table 28. Comparison with Census of Proportions of Demographic Profiles (response rates) for 2-Question Respondents, Hard Refusals, and No Contact Groups

Factor	Census	Answered 2-Quest Survey	Hard Refusals	No Contact Made	Total Survey
Age (years)
16-24	16.4	4.4	-	-	8.0
25-34	17.5	6.7	-	-	13.2
35-44	19.3	11.8	-	-	18.5
45-54	18.2	12.8	-	-	21.9
55-64	12.7	16.3	-	-	19.3
65 and older	15.9	48.1	-	-	19.1
n		643			9849
Chi-Square		553.8			1013.2
P-value		<0.0001			<0.0001
Gender
Male	48.7	37.5	40.9	-	46.6
Female	51.3	62.5	59.1	-	53.4
n		674	303		9994
Chi-Square		33.6	7.3		17.7
P-value		<0.0001	0.0068		<0.0001
Census Division of Residency
New England	5.0	4.6	5.2	5.0	5.0
Middle Atlantic	13.9	13.7	13.1	15.6	12.6
South Atlantic	18.9	17.5	17.2	14.9	18.5
East South Central	6.0	6.2	6.7	4.3	7.1
West South Central	11.1	13.7	9.9	20.5	10.4
East North Central	15.7	16.6	18.7	4.5	15.6
West North Central	6.8	7.0	7.9	10.0	7.9
Mountain	6.6	7.8	6.2	4.8	7.5
Pacific	16.0	13.1	15.1	20.4	15.3
n		681	535	8211	10001
Chi-Square		10.6	6.6	1693.5	73.0
P-value		0.2264	0.5789	<0.0001	<0.0001

Results for Comparison Differences between Four Sample Groups and Census:

• The “No Contact” group was a significantly different than Census across all Census Divisions, except for New England (Table 29).

• The “Full Survey” sample group had statistically significant demographic differences from Census in four of the nine Census Divisions (Table 29).

Table 29. Comparison of Differences between Census and Sample Profiles across Census Divisions

* indicates significance at the experiment-wise 0.05 level

Factor	Census	Answered 2-Q Survey	Hard Refusals	No Contact Made	Total Survey
Age (years)
16-24	16.4	-12.0*	-	-	-8.4*
25-34	17.5	-10.8*	-	-	-4.3*
35-44	19.3	-7.5*	-	-	-0.8
45-54	18.2	-5.4*	-	-	3.7*
55-64	12.7	3.6*	-	-	6.6*
65 and older	15.9	32.2*	-	-	3.2*
Gender
Male	48.7	-11.2*	-7.8*	-	-2.1*
Female	51.3	11.2*	7.8*	-	2.1*
Census Division of Residency
New England	5.0	-0.4	0.2	0.0	0.0
Middle Atlantic	13.9	-0.2	-0.8	1.7*	-1.3*
South Atlantic	18.9	-1.4	-1.7	-4.0*	-0.4
East South Central	6.0	0.2	0.7	-1.7*	1.1*
West South Central	11.1	2.6	-1.2	9.4*	-0.7
East North Central	15.7	0.9	3.0	-11.2*	-0.1
West North Central	6.8	0.2	1.1	3.2*	1.1*
Mountain	6.6	1.2	-0.4	-1.8*	0.9*
Pacific	16.0	-2.9	-0.9	4.4*	-0.7

Results for Comparison between Sample Groups:

• There was no difference between those who answered the Two-question Survey and those who were Hard Refusals (Table 30).

• There was a significant difference between those who answered the Two-question Survey and those who were in the “No Contact sample group. Differences were significant in five of the nine Census Divisions (Table 30).

• There was no difference between those who answered the Two-question Survey and those who answered the Full Survey (Table 30).

• There was a significant difference between those who were “Hard Refusals” and those who were in the “No Contact” sample group. There were significant differences in three of the nine Census Divisions (Table 30).

• There were no differences between those who were “Hard Refusals” and those who answered the Full Survey (Table 30).

• There were significant differences between those in the “No Contact” sample group and those who answered the Full Survey. There were differences in all Census Divisions, except the New England Division (Table 30).

Reasons given by Refusals for not participating in the Full Survey:

Refusals were asked a question before termination of the call.

“Why won’t you participate in the survey?”

In our refusal database we have a total of 1,216 observations. 679 of these people answered the Two-question survey and the remaining 537 were “Hard Refusals” (would not answer the Two-question Survey). Of the 1,216 in the refusal database, 1,206 answered the question of why they did not want to participate in the full survey, while 678 of the 679 who answered the Two-question survey provided answers to the question of why they would not participate in the full survey. This question yields additional information relevant to non-response bias.

The answers to the question of why people did not want to participate in the full survey are summarized in Table 31. There were only a few differences between all refusals and those who answered the Two-question survey. We estimate that most likely about 20 percent of all those that refuse to participate in the full survey do not participate in any recreation activities due to either being “too old”, “bad health/too sick”, “disabled/handicapped”, “homebound” or answered directly that they “don’t participate in outdoor recreation”.

In the 1994-95 NSRE, an analysis was conducted that tested for the effect of a screening question to allow more rapid exit of the survey for people who do not participate in any outdoor recreation. The intention of this screening question was to reduce respondent burden. Instead of going through the long list of outdoor recreation activities to determine if a person did not participate in any outdoor recreation, a screening question was employed that directly asked if they participated in any outdoor recreation activities during the past 12 months. The analysis found a significant difference in the estimated proportion of the population that participates in outdoor recreation when using the screening question. This indicates that people did not initially understand the broadness of the definition of outdoor recreation, unless they were able to hear the whole list of activities. As a result of this test, the screener was omitted from use in NSRE 1999-2000 and NSRE 2005.

When first contact was made for NSRE 2005, people were told that the topic of the survey is outdoor recreation. They were then given an estimate of how long the survey takes, on average, and that it will take much less time if they do little outdoor recreation.

For those who do not participate in outdoor recreation, for whatever reason, respondents likely see outdoor recreation as a low salience issue. They therefore are more likely to refuse to complete an interview. A rough estimate is that about 20 percent of all refusals are not participants in any outdoor recreation (Table 31). Of all RDD telephone numbers, 59 percent were refusals. Thus, the estimate is that about 12 percent (59 percent times 20 percent) of all RDD telephone numbers are not participants in outdoor recreation (versus 3 percent of the 10,001 who responded to the full survey in Versions 1 & 2 of NSRE 2005). The conclusion drawn is that non-response from refusals will lead to an overestimation bias of full sample estimates of activity participation rates.

Table 30. Differences between Census Division Profiles for Refusal Samples. An * indicates significance at the experiment-wise 0.05 level.

Factor	Answer Versus No Answer	Answer Versus No Contact	Answer Versus Total Survey	No Answer Versus No Contact	No Answer Versus Total Survey	No Contact Versus Total Survey
Age (years)
16-24	-	-	-3.7*	-	-	-
25-34	-	-	-6.5*	-	-	-
35-44	-	-	-6.6*	-	-	-
45-54	-	-	-9.2*	-	-	-
55-64	-	-	-3.0	-	-	-
65 and older	-	-	28.9*	-	-	-
Chi-Square			313.7
P-value			<0.0001
Gender
Male	-3.4	-	-9.1*	-	-5.7*	-
Female	3.4	-	9.1*	-	5.7*	-
Chi-Square	1.0		20.9		3.8
P-value	0.3144		<0.0001		0.0511
Census Division
New England	-0.7	-0.4	-0.5	0.3	0.2	-0.1
Middle Atlantic	0.6	-1.9	1.1	-2.5	0.5	3.0*
South Atlantic	0.3	2.6	-1.1	2.3	-1.3	-3.6*
East South Central	-0.6	1.9	-1.0	2.4	-0.4	-2.8*
West South Central	3.7	-6.8*	3.2	-10.6*	-0.5	10.1*
East North Central	-2.1	12.1*	1.0	14.1*	3.1	-11.0*
West North Central	-0.8	-3.0*	-0.8	-2.2	0.0	2.2*
Mountain	1.6	3.0*	0.3	1.4	-1.3	-2.7*
Pacific	-2.1	-7.3*	-2.2	-5.2*	-0.2	5.1*
Chi-Square	7.1	224.9	11.6	236.0	5.2	1093.0
P-value	0.5295	<0.0001	0.1694	<0.0001	0.7320	<0.0001

Table 31. Reasons Given by Refusals for Not Participating in Survey

Reason	All Refusals1 (%)	Two-question Survey Respondents (%)
1. Not interested	33.00	31.42
2. Don't have time	13.18	14.60
3. Survey too long	2.90	3.98
4. Don't want to participate	15.92	17.85
5. Don't do telephone surveys	2.49	2.80
6. Too old	10.20	12.98
7. Bad health/too sick	4.06	5.16
8. Disabled/handicapped	2.57	3.39
9. Homebound	0.41	0.74
10. Hung up	9.04	0.88
11. Other	3.57	2.65
12. Don't participate in outdoor recreation	2.65	3.54
Most Likely Not Participants in Outdoor Recreation2	19.89	25.81
All refusals include the 1,206 of the 1,216 in the refusal database that answered the question on the reason for not participating in the survey. 679 of these people answered the Two-question Survey and the rest are "Hard Refusals" (would not answer the Two-question Survey).
Sum of responses 6, 7, 8, 9, and 12.

Conclusions from experiments on non-response bias:

• There appears to be significant non-response bias associated with people who did not complete the full survey. It would appear that the topic of outdoor recreation leads people who do not participate in outdoor recreation activities to not participate in the full survey. This appeared to result in a significant upward bias of the estimate of participation rate in the one activity we tested (walking for exercise or pleasure).

• Based on comparison population distribution across Census Divisions, it appears the people we were not able to contact, even after 15 calls, were significantly different from all other sample groups.

• Reasons people gave for refusing to participate in the full survey indicate that about 20 percent of all refusals likely are not participants in outdoor recreation. This differs substantially from the full survey estimate of only 3 percent who do not participate at all. Thus, there is an additional source of non-response bias that will require an additional weight to account for lower participation rates among people who will not complete an interview for NSRE.

IV. Overall Conclusions

• Pre-notification letters to the NSRE RDD households with matching phone number and address-increased response rates substantially. The response rate for the sample that received the pre-notification letter was 28.10 percent. The response rate for the sample not receiving the letter, the Standard RDD Sample Group, was 14.08 percent.

• Pre-call notification letters (refusal letter) to households who had refused to complete the full survey also increased response rates in attempts to convert refusals. Those who had matched telephone numbers and addresses, but did not receive a refusal letter had a response rate of 7.6 percent. Those who received a refusal letter had a response rate of 14.4 percent.

• Although Pre-notification and Refusal letters increased response rates, they yielded samples less representative of the population. Standard RDD sampling without letters produced more representative samples.

• Lowered representativeness resulting from pre-notification letters produced significant biases in estimates of activity participation rates. As a result, this study concluded that pre-notification and pre-call refusal letters do not pass a benefit-cost test if the objective is to avoid non-response bias.

• There are significant differences between those who do and those who do not respond to the full survey. These differences appear to result in significant non-response bias in estimates of recreation activity participation rates.

• Current sample weighting does not seem to account for all of the detected non-response bias. An additional sample weight will be needed to account for a higher rate of non-participation in outdoor recreation among refusals relative to those who do respond to the full survey.

References

Cordell, H. Ken. 2004. Outdoor Recreation for the 21^st Century America, A Report to the

Nation: The National Survey on Recreation and the Environment. Venture Publishing, Inc., State College, PA. (NSRE 2000-2001 activity participation rates).

Greene, William H. 1995. LIMDEP Version 7.0. Econometric Software, Inc., Bellport, NY.

Leeworthy, Vernon R., Bowker, J.M., Hospital, Justin D., and Stone, Edward A. 2005.

Projected Participation in Marine Recreation: 2005 & 2010. National Survey on Recreation and the Environment 2000. U.S. Department of Commerce, National Oceanic and Atmospheric Administration, National Ocean Service, Special Projects, Silver Spring, MD. March 2005, pp152. Available at: http://marineeconomics.noaa.gov/NSRE/NSREForecast.pdf.

SAS 9.0. Statistical Analysis System (SAS). SAS Institute, Inc., Carey, NC.

1 Dr. Vernon Leeworthy, Chief Economist, NOAA National Ocean Service, Silver Spring, MD; Dr. Stanley J. Zarnoch, Research Scientist/Mathematical Statistician, USDA Forest Service, Asheville, NC; Dr. H. Ken Cordell, Pioneering Scientist, USDA Forest Service, Athens, GA; Dr. Gary T. Green, Assistant Professor, University of Georgia, Warnell School of Forestry, Athens, GA; Ms. Becky Stephens, Senior Research Associate, University of Tennessee at Knoxville, Human Dimensions Research Lab, Knoxville, TN.

2 Activities (Participation Function Coefficients)

3 Activities (Participation Function Coefficients)

xxi