Nonresponse Follow-up Report Jan 6 2009

Date:

January 6, 2009

To:

Martin Rater, USPTO

From:

David Ferraro
Reviewer:
Tom Krenzke

Subject:

PTO Wave 6/7 Nonresponse Follow-up Report

Memo:

36.0

Introduction
The purpose of this memo is to document the results of the nonresponse follow-up study
conducted during Waves 6 and 7. The follow-up sample consisted of nonrespondents 1 that were
known to be eligible 2 in the original sample. Only nonrespondents rotating out of the sample 3
were eligible for the followup sample. The nonrespondents were sent a postcard with one
question (modification to Q7) being asked:
“Consider your experiences with USPTO Patent Examiners in the past three months. How would
you rate overall examination quality for this time period?”
The possible answers were:
1)
2)
3)
4)
5)
6)

Very Poor
Poor
Fair
Good
Excellent
Have not communicated with patent examiners in the past 3 months

Note that this is the same as Q7 in the original questionnaire except that answer #6 was only
included in the follow-up study. The intention of the follow-up study is to compare the responses
to this question between those that responded to the main survey in the outgoing panel with those
that responded to the follow-up postcard (also in the outgoing panel). The assumption is that the
respondents to the postcard follow-up are like nonrespondents to the main survey so that there is
some indications of potential bias due to nonresponse.

1

STATUSW6 = 2 and STATUSW7 = 2 for Waves 6 and 7, respectively

2

Cases with unknown eligibility status were dropped since they cannot be contacted (e.g., they are no longer at the firm)

3

C2GROUP = 6 and 7 for Waves 6 and 7, respectively

Cc: Kerry Levin, David Marker, Michele Harmon, Jennifer O’Brien, Howard King, Shelley Brock-Roth

Memorandum 36.0

-2-

January 6, 2009

In the follow-up, half of the sample was assigned to receive a white postcard and half was sent a
green postcard to see if colored cards help increase response rates to the followup survey. The set
of nonrespondents was sorted by variables related to nonresponse prior to allocation of the
colored postcard. For Wave 6 as an example, the sort variables were the response to Q7 in Wave
5, whether or not the customer was in the original panel 6 (2nd successive wave) or the
supplemental panel 6 (first and last wave) 4 , type of nonresponse based on the disposition code,
and firm ID. The resulting sort order was based on auxiliary variables that were correlated with
response propensity.
The follow-up study was conducted in order to help answer the following questions:
1.
2.
3.
4.

How different are the Wave 6/7 respondents from the followup respondents?
How different are the followup respondents from the followup nonrespondents?
Do the results impact what can be done in weighting to reduce the bias due to
nonresponse?
What is the impact of the colored postcard on followup response rates?

Methodology
Nonresponse bias is measured by two terms: the nonresponse rate, and differences between
respondents and nonrespondents. To explain further, we introduce the following expression for
nonresponse bias for a sample mean ( y R ):
Bias ( yR ) = (1 − WR )(YR − YN ),

where WR is the weighted unit response rate, YR is the population mean of the respondent
stratum, and YN is the population mean for the nonrespondent stratum. While the response rate
(first component) is universally recognized as a measure of survey quality, it is not by itself a good
indicator of nonresponse bias. The difference between respondents and nonrespondents (second
component) is just as important. Theoretically, even if the response rate is 43 percent, if there is
no difference in the mean of the characteristic y between respondents and nonrespondents, then
bias does not exist. In practice, the second component is unknown; however, typically proxies
(auxiliary data) are used to estimate the difference. Weighting adjustments are used to reduce
nonresponse bias; although, it is widely recognized that some nonresponse bias remains in survey
estimates.
However, in the case with the nonresponse follow-up sample, the bias can be written as

(

(

Bias ( y R ) = (1 − W R ) YR − YFU + YNR

))

where Y NR is the population mean of the follow-up nonrespondent stratum, and YFU is the
population mean for the follow-up respondent stratum.
4

Based on the variable GROUP

Memorandum 36.0

-3-

January 6, 2009

A bivariate analysis (response indicator versus each auxiliary variable) compares the distribution of
the participating households to the distribution of the total eligible sample of households for
several auxiliary variables. Survey base weights were used to account for the unequal withinhousehold probabilities of selection, and replicate weights were used to adequately reflect the
impact of the sample design on variance estimates. The weights for the follow-up respondents
were adjusted to account for nonrespondents to both the main survey and the follow-up. This
assumes that nonrespondents were more similar to the follow-up respondents than the main
survey respondents. Together with the main sample respondent, the weights account for the
entire eligible population. Adjustment cells were created using the Search software (WesSearch)
using the same approach as used in the normal weighting procedure.
Follow-up respondents that answered “No applications in the past 3 months” were excluded from
the analysis as ineligible. For informational purposes, the complete distributions by wave and
postcard color are shown in table 1.
To test for statistical differences, the distribution of Q7 for wave respondents was compared with
the distribution for follow-up respondents and similarly within the follow-up study for the salmon
and white postcard types. We used two approaches in the tests. To test the categorical responses,
the hypothesis of independence between the characteristic and participation status was tested
using a Rao-Scott modified Chi-square statistic at the 10 percent level. Secondly, we computed an
average score of the categorical responses treating them as continuous variables, with the larger
the average score the more favorable the response. The difference between means was tested
using a t test. Additionally, the continuous variables were tested using the Benjamin-Hochberg
procedure to control the overall false discovery rate for a family of comparisons. The B-H critical
values are shown in the appendix. See the internal PTO Memo 30 for details on the B-H
procedure. The bias and relative bias are also given in each table. The bias is the difference
between the respective estimates for the main survey respondents and the follow-up respondents
(equivalent to the formula above). The relative bias is calculated as the bias divided by the estimate
from the eligible sample. The relative bias is a measure of the size of the bias compared to the
eligible sample estimate.
Results
The results are shown for Waves 6 and 7 in tables 2 through 6 and 7 through 11, respectively. We
first present the results comparing the main survey to follow-up respondents. Q7 is shown with
the five categorical responses and with three categories by collapsing “very poor” with “poor” and
“good” with “excellent”. Since the sample size in each category is fairly small, collapsing might
show more differences. Secondly, we show the overall average response of Q7 and by selected
characteristics. We then present comparisons of the two postcard types within the follow-up
sample. The response rates by postcard type are shown first. Then Q7 is shown by postcard
similar to Q7 as categorical and as averages.
Wave 6. For Wave 6, there were no statistical differences in the categorical responses between the
follow-up and main survey respondents though some of the differences are large (table 2).
Differences were not detected due to the large standard errors on the estimates from the follow-

Memorandum 36.0

-4-

January 6, 2009

up sample. Generally though the responses were more positive for the follow-up. This can also be
seen in terms of average response where the overall average was larger for the follow-up but is not
significant (table 3). The only statistical difference by variable was the sample domain for firms
with less than 150 applications had a more favorable response for the follow-up respondents than
for the main survey (table 3).
Looking at the comparison of postcard types, the response rate was higher for the salmon-colored
(32%) postcard than for the white postcard (30%), but the difference was not significant (table 4).
In terms of the responses, there were no statistical differences in the categorical responses
between postcard colors (table 5). The overall average was higher for the white postcard but not
significant (table 6). There were two p-values less than 0.10 related to differences by characteristic,
which are the sample domain for firms with less than 150 applications and registration numbers
are less than 33229 – each having a more favorable response for the white postcard (table 6).
However, neither were significant while controlling the overall false discovery rate using the B-H
approach.
Wave 7. For Wave 7, there were no significant differences in the categorical responses between
the follow-up and main survey respondents but again the follow-up had a more favorable
response (table 7). In terms of average response, the overall difference was not significant but
again more favorable for the follow-up. There were four p-values less than 0.10 related to
differences by characteristic, however only agents and other registration numbers (those recently
registered) were significant while controlling the overall false discovery rate (table 8). In each case
the follow-up had a more favorable response.
Looking at the comparison of postcard types, the response rate was higher for the white-colored
postcard (33%) than for the salmon (25%), unlike Wave 6, and the difference was significant
(table 9). In terms of the responses, the categorical responses were significantly different (table
10). The white postcard had a much larger proportion in the fair category than the salmon
postcard but smaller proportions in all the other categories. For the average response, the overall
average was not significant but the difference was in the opposite direction from Wave 6 (table
11). The only statistically significant differences by characteristic while controlling the overall false
discovery rate was firms with registration numbers between 44155 and 50724 with the salmon
postcard having a more favorable response (table 11). There were four other characteristics with
p-values less than 0.10.
Conclusion
Based on this analysis, the conclusions are:
•
•
•
•

There are no statistically significant differences detected between the main survey and
follow-up respondents in their categorical responses to Q7 for either Wave 6 or 7.
There are, however, fairly large relative differences in both waves. These differences are
not detectable due to the large standard errors of the estimates from the follow-up study.
The responses were generally more positive for the follow-up.
For the average responses, the overall averages were not significant.
There are only a few significant differences by characteristic while controlling the overall
false discovery rate using the B-H approach. It is expected that 10% of the difference

Memorandum 36.0

•

•

-5-

January 6, 2009

would be significant by chance. In Wave 6, only one of the fifteen differences tested
(6.7%) was significant, the sample domain for firms with less than 150 applications. In
Wave 7, two of the fifteen differences tested (13.3%) were significant, agents and other
registration numbers (those recently registered).
In regards to postcard type, there were significant differences between the different colors
for response rates and categorical responses only in Wave 7.The response rate was higher
for the white postcard as was the proportion of the fair category. The direction of the
differences was not consistent in Wave 6.
For the average responses by postcard type, the overall averages were not significant.
There was only one significant difference (6.7%) by characteristic, firms with registration
numbers between 44155 and 50724, in Wave 7.

Memorandum 36.0

Table 1.

-6-

January 6, 2009

Complete distribution of Q7 by postcard type and wave

Q7

Salmon (percent)

Standard Error

White (percent)

Standard Error

5.44

3.13

4.90

2.23

2-poor

15.74

4.66

11.20

3.43

3-fair

32.23

6.18

33.80

5.95

4-good

20.82

4.38

30.95

5.59

3.19

1.85

3.71

2.45

22.57

5.44

15.45

3.80

6.51

2.95

3.34

1.64

2-poor

12.63

4.18

10.92

3.15

3-fair

20.81

4.33

39.76

6.00

4-good

28.19

6.37

25.20

4.31

5-excellent

11.41

5.50

0.83

0.86

6-no applications

20.44

5.26

19.94

5.47

Wave 6
1-very poor

5-excellent
6-no applications
Wave 7
1-very poor

-7-

Table 2.

January 6, 2009

Percentage distribution of Q7 by response status: Wave 6

Characteristic

Main Survey
(percent)

Standard Error

Follow-up
(percent)

Standard Error

Difference

Relative
difference

Q7
1-very poor

0.2610
3.93

0.99

6.41

2.47

2.478

0.630

2-poor

22.89

2.14

16.80

3.47

-6.090

-0.266

3-fair

46.51

2.50

40.80

4.92

-5.705

-0.123

4-good

24.70

2.31

31.73

4.51

7.031

0.285

1.97

0.75

4.26

1.97

2.286

1.161

5-excellent

Chi-Square
p-value

Q7 collapsed

0.2059

1/2 very poor/poor

26.83

2.33

23.21

4.05

-3.612

-0.135

3-fair

46.51

2.50

40.80

4.92

-5.705

-0.123

4/5 good/excellent

26.67

2.34

35.98

4.53

9.317

0.349

-8-

Table 3.

January 6, 2009

Average response of Q7 by selected categorical variables: Wave 6

Characteristic
Q7

Main Survey
(mean)
Standard Error

Follow-up
(mean)

Standard Error

Difference

Relative
difference

t test
p-value

2.98

0.04

3.11

0.09

0.127

0.043

0.2451

Northeast

3.05

0.08

2.99

0.17

-0.053

-0.017

0.7899

Midwest

2.91

0.10

3.19

0.16

0.278

0.096

0.1609

South

2.94

0.09

3.03

0.19

0.087

0.030

0.6732

West

3.03

0.09

3.22

0.15

0.195

0.064

0.2848

Agent

2.86

0.09

3.16

0.18

0.294

0.103

0.1494

Attorney

2.99

0.05

3.10

0.10

0.114

0.038

0.3467

Large firms, 50 customers or less

3.01

0.06

2.90

0.13

-0.102

-0.034

0.4591

Large firms, more than 50 customers
firms, number of applications between
150 and 275

3.01

0.11

3.27

0.19

0.259

0.086

0.2653

2.96

0.11

2.78

0.27

-0.181

-0.061

0.5521

firms, less than 150 applications

2.95

0.07

3.32

0.12

0.371

0.126

0.0164*

top-filer firms or independent inventors

3.66

0.75

NA

NA

NA

REG_NO < 33229

3.00

0.11

3.27

0.19

0.269

0.090

0.2489

33229 <= REG_NO <= 42055

3.06

0.09

3.01

0.16

-0.057

-0.019

0.7552

42055 < REG_NO <= 50724

2.90

0.07

3.03

0.19

0.129

0.044

0.5264

other

2.94

0.10

3.11

0.13

0.171

0.058

0.3094

Census Region (CREG)

Agent/Attorney (TYPE)

Sample Domain (DOMAIN)

-

-

Registration number (REG_NO_R)

* significant under B-H approach

-9-

Table 4.

Response rates by postcard type: Wave 6

Characteristic
Response rate

Table 5.

January 6, 2009

Salmon (percent) Standard Error
32.13

3.26

White (percent)

Standard Error

Difference

Relative
difference

t test
p-value

29.69

2.97

-2.440

-0.076

0.5545

Standard Error

Difference

Relative
difference

Chi-Square
p-value

Percentage distribution of Q7 by postcard type: Wave 6

Characteristic

Salmon (percent) Standard Error

White (percent)

Q7
1-very poor

0.7667
7.03

3.99

5.79

2.62

-1.235

-0.176

2-poor

20.33

5.82

13.25

4.01

-7.081

-0.348

3-fair

41.63

7.16

39.98

6.55

-1.648

-0.040

4-good

26.89

5.51

36.60

6.45

9.705

0.361

4.13

2.37

4.38

3.00

0.258

0.063

5-excellent
Q7 collapsed

0.4329

1/2 very poor/poor

27.36

6.67

19.04

4.79

-8.317

-0.304

3-fair

41.63

7.16

39.98

6.55

-1.648

-0.040

4/5 good/excellent

31.02

5.48

40.98

6.48

9.964

0.321

-10-

Table 6.

January 6, 2009

Average response of Q7 by selected categorical variables: Wave 6

Characteristic
Q7

Salmon
(mean)

Standard Error White (mean) Standard Error

Difference

Relative
difference
0.065

t test
p-value

3.01

0.13

3.21

0.11

0.197

0.2340

Northeast

2.78

0.28

3.17

0.16

0.390

0.140

0.2261

Midwest

3.01

0.20

3.32

0.25

0.311

0.103

0.3342

South

2.91

0.20

3.21

0.30

0.296

0.102

0.3895

West

3.40

0.25

3.06

0.14

-0.339

-0.100

0.2346

Agent

3.23

0.22

3.06

0.31

-0.166

-0.051

0.6596

Attorney

2.98

0.15

3.22

0.12

0.242

0.081

0.1839

Large firms, 50 customers or less

2.92

0.18

2.89

0.18

-0.036

-0.012

0.8862

Large firms, more than 50 customers
firms, number of applications between
150 and 275

3.51

0.26

3.00

0.29

-0.503

-0.143

0.1991

2.63

0.38

2.98

0.34

0.353

0.134

0.4748

firms, less than 150 applications

3.08

0.15

3.56

0.16

0.476

0.155

0.0331

NA

NA

NA

Census Region (CREG)

Agent/Attorney (TYPE)

Sample Domain (DOMAIN)

top-filer firms or independent inventors

-

-

-

-

Registration number (REG_NO_R)
REG_NO < 33229

2.85

0.26

3.50

0.19

0.655

0.230

0.0372

33229 <= REG_NO <= 42055

3.08

0.26

2.94

0.19

-0.144

-0.047

0.6433

42055 < REG_NO <= 50724

2.98

0.30

3.09

0.21

0.112

0.038

0.7669

other

3.07

0.14

3.22

0.30

0.155

0.051

0.6354

-11-

Table 7.

January 6, 2009

Percentage distribution of Q7 by response status: Wave 7

Characteristic

Main Survey
(percent)

Standard Error

Follow-up
(percent)

Standard Error

Difference

Relative
difference

Q7
1-very poor

0.2780
4.48

0.99

5.90

1.90

1.424

0.318

2-poor

18.19

1.72

14.60

3.10

-3.586

-0.197

3-fair

46.90

2.28

39.55

4.77

-7.351

-0.157

4-good

27.90

2.12

33.18

4.37

5.283

0.189

2.54

0.87

6.77

3.32

4.229

1.666

5-excellent

Chi-Square
p-value

Q7 collapsed

0.1672

1/2 very poor/poor

22.67

1.96

20.51

3.70

-2.162

-0.095

3-fair

46.90

2.28

39.55

4.77

-7.351

-0.157

4/5 good/excellent

30.43

2.11

39.95

4.33

9.512

0.313

-12-

Table 8.

January 6, 2009

Average response of Q7 by selected categorical variables: Wave 7
Main Survey
(mean)

Standard Error

Follow-up
(mean)

Standard Error

Difference

Relative
difference

t test
p-value

3.06

0.04

3.20

0.09

0.145

0.047

0.1471

Northeast

3.01

0.10

2.87

0.23

-0.132

-0.044

0.6077

Midwest

2.98

0.10

2.99

0.23

0.008

0.003

0.9737

South

3.04

0.09

3.45

0.14

0.413

0.136

0.0149

West

3.22

0.08

3.25

0.19

0.036

0.011

0.8503

Agent

2.87

0.12

3.76

0.29

0.884

0.308

0.0057*

Attorney

3.08

0.04

3.12

0.09

0.041

0.013

0.6834

3.08

0.05

3.08

0.11

-0.001

0.000

0.9944

Characteristic
Q7
Census Region (CREG)

Agent/Attorney (TYPE)

Sample Domain (DOMAIN)
Large firms, 50 customers or
less
Large firms, more than 50
customers
firms, number of applications
between 150 and 275
firms, less than 150
applications
top-filer firms or independent
inventors
Registration number
(REG_NO_R)

2.98

0.09

3.34

0.35

0.363

0.122

0.3286

3.00

0.09

2.76

0.28

-0.236

-0.079

0.4267

3.07

0.09

3.35

0.14

0.279

0.091

0.0987

3.47

0.35

-

-

NA

NA

NA

REG_NO < 33229

3.26

0.09

3.18

0.19

-0.076

-0.023

0.7118

33229 <= REG_NO <= 42055

2.99

0.09

3.16

0.11

0.171

0.057

0.1897

42055 < REG_NO <= 50724

2.96

0.08

2.78

0.20

-0.185

-0.062

0.4085

Other

3.01

0.08

3.64

0.22

0.631

0.210

0.0082*

* significant under B-H approach

-13-

Table 9.

Response rates by postcard type: Wave 7

Characteristic
Response rate

Table 10.

January 6, 2009

Salmon (percent) Standard Error
25.48

3.04

White (percent)

Standard Error

Difference

Relative
difference

t test
p-value

33.30

3.40

7.820

0.307

0.0850

Standard Error

Difference

Relative
difference

Chi-Square
p-value

Percentage distribution of Q7 by postcard type: Wave 7

Characteristic

Salmon (percent) Standard Error

White (percent)

Q7
1-very poor

0.0504
8.19

3.59

4.18

2.02

-4.009

-0.490

2-poor

15.88

5.18

13.64

3.86

-2.240

-0.141

3-fair

26.16

5.43

49.67

6.58

23.511

0.899

4-good

35.43

7.67

31.47

4.92

-3.957

-0.112

5-excellent

14.34

6.76

1.04

1.07

-13.304

-0.928

Q7 collapsed

0.0395

1/2 very poor/poor

24.07

6.04

17.82

4.84

-6.249

-0.260

3-fair

26.16

5.43

49.67

6.58

23.511

0.899

4/5 good/excellent

49.78

7.41

32.51

4.96

-17.262

-0.347

-14-

Table 11.

January 6, 2009

Average response of Q7 by selected categorical variables: Wave 7

Characteristic
Q7

Salmon
(mean)

Standard Error White (mean) Standard Error

Difference

Relative
difference

t test
p-value

3.32

0.18

3.12

0.09

-0.203

-0.061

0.3367

Northeast

3.30

0.36

2.62

0.23

-0.679

-0.206

0.1182

Midwest

2.47

0.45

3.38

0.17

0.910

0.369

0.0666

South

3.48

0.21

3.42

0.16

-0.066

-0.019

0.7993

West

3.68

0.39

3.06

0.11

-0.619

-0.168

0.1207

Agent

4.04

0.37

3.33

0.20

-0.717

-0.177

0.0940

Attorney

3.16

0.17

3.09

0.09

-0.063

-0.020

0.7502

Large firms, 50 customers or less

3.05

0.22

3.09

0.14

0.034

0.011

0.9012

Large firms, more than 50 customers
firms, number of applications between
150 and 275

3.44

0.60

3.21

0.40

-0.230

-0.067

0.7487

2.15

0.56

3.13

0.19

0.982

0.458

0.0951

firms, less than 150 applications

3.66

0.21

3.11

0.13

-0.544

-0.149

0.0251

-

-

-

-

NA

NA

REG_NO < 33229

3.23

0.31

3.13

0.18

-0.099

-0.031

0.7794

33229 <= REG_NO <= 42055

2.98

0.20

3.26

0.13

0.276

0.093

0.2955

42055 < REG_NO <= 50724

3.48

0.21

2.50

0.20

-0.980

-0.282

0.0010*

other

3.72

0.38

3.55

0.15

-0.177

-0.048

0.6667

Census Region (CREG)

Agent/Attorney (TYPE)

Sample Domain (DOMAIN)

top-filer firms or independent inventors

NA

Registration number (REG_NO_R)

* significant under B-H approach

Appendix

Appendix
Benjamin-Hochberg critical values
Index

B-H critical value

2 levels
1
2
4 levels
1
2
3
4

0.1000
0.0500
0.1000
0.0750
0.0500
0.0250

January 6, 2009