0057 Final Just 2009 Part B

SUPPORTING STATEMENT
UNITED STATES PATENT AND TRADEMARK OFFICE
Customer Panel Quality Survey
OMB Control Number 0651-0057

B.

COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1.

Universe and Respondent Selection

The respondent pool for this survey is made up of the businesses, organizations, and
individuals who frequently file patent applications. The USPTO plans to survey large,
medium, and small-sized domestic corporations, universities and other non-profit
research organizations, and independent inventors. Foreign entities will not be included
in the sample frame.
The target population consists of individuals associated with USPTO top filers (e.g.,
firms at a given address who have filed six or more patent applications in the past year).
The sample unit will be the USPTO-registered agents/attorneys associated with the top
filers and also independent inventors that filed six or more patents in the past 12
months. The target population typically accounts for over 85% of all patent applications
filed in a given fiscal year.
2.

Procedures for Collecting Information

The Customer Panel Quality Survey will use a longitudinal, rotating panel design. The
sampling plan is included in this submission for review (Ref. A). The sampling plan also
contains information about the respondent pool and the response rate.
The USPTO’s survey contractor, Westat, will mail the survey to all of the sampled
respondents. A personalized label will be inserted on the survey packet envelope in
order to reach the specified respondent. The survey packet will contain the paper
version of the questionnaire and a cover letter explaining that the USPTO is sponsoring
the survey and that all responses will remain confidential. The cover letter will also
contain the username, password, and the 5-digit survey ID number. The electronic and
paper surveys will mirror each other.
During the follow-up non-response prompting calls, Westat employees will use a script
developed in collaboration with the USPTO. For all nonrespondents a reminder
postcard will be sent to encourage survey participation.
In order to improve the response rate achieved in the 2002 survey, a pre-notification
letter will be sent to all groups informing them of the purpose of the survey and will
include instructions for completing the survey online.

The survey packet will include the three-page questionnaire and a postage-paid preaddressed return envelope. The cover letter will be printed on USPTO letterhead and
signed by the Commissioner of Patents.
3.

Methods to Maximize Responses

In order to maximize the number of responses received from the survey, the USPTO
plans to follow several well-established survey procedures.
First, all sampled
respondents will receive a pre-notification letter signed by the Commissioner. The letter
will explain the importance of the study and encourage respondent cooperation. Next,
all sampled respondents will receive the paper survey in the mail. Follow-up contact will
be made after the initial survey is sent. One week after the initial survey mailing, all
nonrespondents will be sent a thank you/reminder postcard in the mail. Two weeks
after the initial survey is mailed, we will telephone all of the nonrespondents to prompt
them to answer either the paper or internet version of the survey. A script has been
developed for these phone calls so that everyone conducting these interviews asks the
same questions, in the same manner.
In order to determine how the nonresponse bias affected the results of the survey,
Westat conducted a nonresponse follow-up study during Waves 6 and 7 for the survey.
The USPTO was provided with the analysis of the findings. The objective of the study
was to try to get a picture of how the nonrespondents would have responded to the
main survey if they had actually responded to the survey. The study was conducted
because the nonresponses can cause bias in the survey estimates, which is itself
affected by the response rate to the survey and the differences between those who
responded to the survey and those who did not respond.
As part of the study, Westat sent a postcard to those who did not respond to the original
survey and who now were rotating out of the survey sample. The postcard contained
one question concerning overall examination quality that was asked in the original
survey. This question was also asked in the original survey; the only difference
between the two was that the follow-up question had an additional answer that was not
included in the original survey. Half of the sample in the study received a white
postcard and the other half received a colored postcard to see if the colored card would
help increase the response rates.
The follow-up study compared the responses to the overall examination quality question
between those who responded to the question in the original survey in the outgoing
panel with those that responded to the follow-up postcard (who were also in the
outgoing panel). In this study, it is assumed that the respondents to the follow-up
survey are like the nonrespondents to the original survey and that there are indications
that the survey nonresponses are causing a potential bias. The results of the follow-up
study were used to help answer the following questions:
1. How different are the Wave 6/7 respondents from the follow-up respondents?

2

2. How different are the follow-up respondents from the follow-up nonrespondents?
3. Do the results impact what can be done in weighting to reduce the bias due to
nonresponse?
4. What is the impact of the colored postcard on the follow-up response rates?

Nonresponse bias is affected by two different items: the nonresponse rate and the
differences
between
respondents
and
nonrespondents.
While
the
response/nonresponse rate is known, the differences between those who respond and
those who do not is unknown. The follow-up study attempts to measure the difference
between the respondents and nonrespondents. The nonresponse bias is calculated
using the following equation for a sample mean:
Bias ( yR ) = (1 − WR )(YR − YN ),

where WR is the weighted unit response rate, YR is the population mean of the
respondent stratum, and YN is the population mean of the respondent stratum, and
is the population mean for the nonrespondent stratum. While the response rate is
universally recognized as a measure of survey quality, the difference between the
respondents and nonrespondents is just as important in determining the nonresponse
bias. Weighting adjustments are used to reduce the nonresponse bias (although some
nonresponse bias will remain in the survey estimates).
However, in the case with the nonresponse follow-up sample, the bias can be written as

(

(

Bias ( y R ) = (1 − W R ) YR − YFU + YNR

))

where Y NR is the population mean of the follow-up nonrespondent stratum, and YFU is
the population mean for the follow-up respondent stratum.
A bivariate analysis (response indicator versus each auxiliary variable) compares the
distribution of the participating households to the distribution of the total eligible sample
of households for several auxiliary variables. Survey base weights were used to
account for the unequal within-household probabilities of selection, and replicate
weights were used to adequately reflect the impact of the sample design on variance
estimates. The weights for the follow-up respondents were adjusted to account for
nonrespondents to both the main survey and the follow-up. This assumes that
nonrespondents were more similar to the follow-up respondents than the original survey
respondents. Together with the main sample respondent, the weights account for the
entire eligible population. Adjustment cells were created using the Search software
(WesSearch) using the same approach used in the normal weighting procedure.

3

To test for statistical differences, the distribution of the patent examination quality
question for the wave respondents was compared with the distribution for follow-up
respondents and similarly within the follow-up study for the white and colored postcards.
To test the categorical responses, the hypothesis of independence between the
characteristic and participation status was tested using a Rao-Scott modified Chi-square
statistic at the 10 percent level. The average score of the categorical responses was
computed as a continuous variable, with the larger average score the more favorable
the response. The difference between means was tested using a t test. The
continuous variables were tested using the Benjamin-Hochberg procedure to control the
overall false discovery rate for a family of comparisons.
Westat analyzed the results of this study, prepared a report and a summary of the
report, and submitted both to the USPTO. Both of these items are attachments to this
submission (Ref. B). Some of the conclusions made concerning the survey were:
•

There are no statistically significant differences detected between the main
survey and follow-up respondents in their categorical responses to the patent
examination questions for either Wave 6 or 7.

•

There are fairly large relative differences in both waves. These differences are
not detectable due to the large standard errors of the estimates from the followup study. The responses were generally more positive for the follow-up.

•

For the average responses, the overall averages were not significant.

•

There are only a few significant differences by characteristic while controlling the
overall false discovery rate using the B-H approach. It is expected that 10% of
the difference would be significant by chance. In Wave 6, only one of the fifteen
differences tested (6.7%) was significant, the sample domain for firms with less
than 150 applications. In Wave 7, two of the fifteen differences tested (13.3%)
were significant, agents and other registration numbers (those recently
registered).

•

In regards to postcard type, there were significant differences between the
different colors for response rates and categorical responses only in Wave 7.
The response rate was higher for the white postcard as was the proportion of the
fair category. The direction of the differences was not consistent in Wave 6.

•

For the average responses by postcard type, the overall averages were not
significant. There was only one significant difference (6.7%) by characteristic,
firms with registration numbers between 44155 and 50724, in Wave 7.

Westat will continue to conduct non-response analysis as necessary to evaluate the
potential for bias.

4

4.

Testing of Procedures

To ensure the survey questions are meaningful to respondents and easy to understand,
Westat conducted four cognitive interviews with customers identified by the USPTO.
These customers are similar to the sampled respondents for the Customer Panel
Quality Survey study. The wording of the survey questions was then revised based on
feedback from these customers.
Low response rates have typically been observed in previous customer surveys
administered by the USPTO. The USPTO believes that offering both a paper and a
web response option will enhance response rates for this effort. The Customer Panel
Quality Survey was designed to focus only on key aspects of examination quality to
keep the time burden to a minimum and to help response rates.
When sending the mail survey out to sampled customers, we will use the wellestablished procedures documented by Dillman (2002). After the online version of the
survey is programmed, Westat will test the web survey internally to ensure respondents’
answers are properly captured and the survey is easy to navigate online. Westat will
also ensure that all computer security requirements are met.
5.

Contact for Statistical Aspects and Data Collection

The Office of Patent Quality Assurance of the USPTO is responsible for conducting the
Customer Panel Quality Survey. Martin Rater is the point of contact for this survey and
can be reached by phone at 571-272-5966 or by e-mail at martin.rater@uspto.gov. The
names and telephone numbers for the individuals from Westat who consulted on the
statistical aspects of the survey and who are conducting the survey under the direction
of the USPTO are:
Kerry Levin, Ph.D.
Senior Study Director
Westat
301-738-3563
kerrylevin@westat.com
David Marker, Ph.D.
Senior Statistician
Westat
301-251-4398
davidmarker@westat.com

5

List of References
A.
B.

USPTO Customer Panel Quality Survey Sampling Plan
PTO Wave 6/7 Nonresponse Follow-Up Report and CPQS: Summary of the
Nonresponse Followup Analysis

6