Westat Technical Proposal

WPN 12-525 | Copy _______

GSA MOBIS Master Contract # GS-23F-8144H
RFQ # 679020

MOBIS: External Quality Survey for the
Office of Patent Quality Assurance
(OPQA) – Task Order

Technical Proposal

Submitted via email
June 8, 2012
Submitted to:
United States Patent and
Trademark Office
600 Dulany Street
Alexandria, VA 22313

Submitted by:
Westat
1600 Research Boulevard
Rockville, Maryland 20850-3129
(301) 251-1500

This proposal or quotation includes data that shall not be disclosed outside the Government and
shall not be duplicated, used, or disclosed – in whole or in part – for any purpose other than to
evaluate this proposal or quotation. If, however, a contract is awarded to this offeror or quoter as
a result of – or in connection with – the submission of this data, the Government shall have the
right to duplicate, use, or disclose the data to the extent provided in the resulting contract. This
restriction does not limit the Government’s right to use information contained in this data if it is
obtained from another source without restriction. The data subject to the restriction are contained
on all pages.

June 8, 2012

Via email
Ms. Erin Fox-Ramirez
Contract Specialist
United States Patent and Trademark Office
600 Dulany Street
Alexandria, Virginia 22313-1450
RE: MOBIS: External Quality Survey for the Office of Patent Quality Assurance (OPQA) –
Task Order
Dear Ms. Fox-Ramirez:
Westat is pleased to submit our response to the Request for Quote for the United States Patent
and Trademark Office’s (USPTO’s) “External Quality Survey” (RFQ# 679020). Our response
discusses Westat’s experience in working with USPTO over the past 10 years and our familiarity
with conducting a longitudinal, overlapping customer quality instrument.
We are excited about the prospect of continuing our highly successful working relationship on
the Customer Panel Quality Survey, which poses interesting methodological and statistical
challenges. In order to benefit from our years of experience and knowledge gained, we will use
most of the same Westat team members that have worked with USPTO over the last 10 years.
This will provide both the benefit of our understanding the complexities of the sample frame as
well as a team who understands the importance of flexibility so that the questionnaire activities
can be adapted to address ongoing needs.
If you have any questions please contact me at (301)738-3563 or kerrylevin@westat.com or
Dr. Jennifer O’Brien at (301)251-4272 or obrienj@westat.com. Thank you.
Sincerely,

Kerry Levin, Ph.D.
Vice President

Table of Contents
Chapter

Page

1

Introduction ........................................................................................................

1

2

Westat’s Technical Capabilities.........................................................................

2

2.0
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8

2
2
2
5
6
6
7
8
9

3

Requirements as Set Forth in the RFQ .............................................
Survey/Project Management ...............................................................
Design, Recruit, and Maintain Customer Panel................................
Survey Instruments ...............................................................................
OMB Clearance .....................................................................................
Data Collection......................................................................................
Data Processing .....................................................................................
Data Analysis .........................................................................................
Reporting ................................................................................................

Westat’s Past Performance ................................................................................

10

Suggested Wave 17 fielding schedule (July 2012) ..........................................

7

Table
2-1

Budget

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

iii

WPN 12-525

Introduction

1

The United States Patent and Trademark Office (USPTO) seeks proposals from qualified firms that
can provide a variety of services associated with the administration of an External Quality Survey
(EQS). The EQS is a longitudinal, rotating, and overlapping panel customer survey managed in the
Office of Patent Quality Assurance (OPQA). The overall objective of the EQS is to provide the
OPQA with reliable data to be used for point estimates of current examination quality, assess
changes in customer perceptions, and secure customer feedback on targeted training areas and
opportunities for improvement. Westat has conducted this survey on behalf of the USPTO since
October 2006, ergo Westat is uniquely qualified to undertake all the requirements set forth in the
Request For Quote (RFQ) including:


Survey/Project Management;



Design, Recruit, and Maintain Customer Panel;



Survey Instruments;



OMB Clearance;



Data Collection;



Data Processing;



Data Analysis; and



Reporting.

As a result of our close collaboration with the USPTO while conducting several customer
satisfaction surveys from 1998–2002, and conducting the current USPTO Quality Survey (QS) since
2006, Westat has developed a strong understanding of the USPTO’s patent procedures and
examination quality concerns. Also, we are very aware of the need to assure sampled customers that
their opinions will be kept confidential and their participation in the study will not affect any
pending or future patent applications they might have with the USPTO.
The remainder of our response to the USPTO’s RFQ is organized into two sections: (1) our
response to the requirements set forth in the RFQ and (2) Westat’s relevant Past Performance.

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

1

WPN 12-525

Westat’s Technical Capabilities
2.0

Requirements as Set Forth in the RFQ

2.1

Survey/Project Management

2

In order to meet the timeline for deliverables and the technical requirements of this work, Westat
proposes a strong management structure with ample technical knowledge of USPTO’s research
needs, flexibility, and staff coordination to collect, analyze, and report on the EQS. The assignment
of responsibilities to staff are based on matching appropriate corporate and individual skills to the
tasks and timeline. It emphasizes complementary skills, role relationships, and provision of sufficient
resources to maintain the work schedule at the high levels of quality that the EQS demands. Our
project plan, developed and refined over the course of 16 administrations of the EQS, details major
milestones in data collection, data processing, and data delivery. In keeping with our current
practice, discrepancies between proposed and actual dates will be documented along with remedies
for resuming project progress.

2.2

Design, Recruit, and Maintain Customer Panel

Under this task, we will design, select, and maintain a sample of rotating panels of continuous patent
customers. We understand that the current as well as intended field schedule for the EQS requires
two data collection waves per year (January and July). The sample design objectives include:

1

1.

Minimizing reporting burden;

2.

Accounting for, at a minimum, the frequency of customer contact, volume of patent
activity, and technology areas;

3.

Producing reliable estimates of wave-to-wave change;

4.

Producing estimates for each of the waves;

5.

Maintaining the customer panel to preserve representativeness;

6.

Adding new members to the panel to compensate for attrition; and

7.

Tracing lost members to maintain high response rates. 1

Since FY09-Q1, weighted response rates to the EQS have averaged about 50 percent.

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

2

WPN 12-525

Westat has completed work on the next sample frame for the EQS. Having conducted this task
twice before, we employed proven processes and procedures to execute the necessary steps to
obtaining the cleaned frame.

Step 1: Creating the FIRMS File
Below is a description of the processes and procedures involved in creating the FIRMS file. An
initial listing of 4,938 top-filing firms was provided for use in sampling frame development and
matching to a separate file of customers. From this initial file, 192 firms were dropped due to a
location outside of the United States. The sample cleaning process began by de-duplicating the file
of top-filing firms. In addition to creating a single record for each of the top-filer firms, it was
necessary to sum the number of patent applications filed by the firm across all duplicate entries.
De-duplication involved a combination of an iterative automated process using SAS and manual
cleaning. We began by running a SAS program to identify all exact duplicate entries in the firms file,
based on state, city, and organization name. However, as the information for the firms was not
entered in a standardized fashion (both the text entry for a single firm differed across duplicate
records as well as the placement of the information across the columns), extensive manual review
and editing were required to de-duplicate the firms records.
Manual cleaning involved arranging the file so information was placed in the appropriate columns,
sorting the file by state, city, and organization name and then reviewing the file to identify and
standardize duplicate firms. During this process we used the following guidelines:


If multiple locations were listed for a top filing firm, these different locations were
retained as separate records in the firms file.



If there were separate records for different people at a top filing firm, these records
were de-duplicated to a single listing for the top filing firm.

A total of 535 records were identified as duplicate firms. The final number of firms available for
sampling and matching was 4,211. A total of 807 of these firms did not have matches to the
customer file; therefore, as discussed in the next section, 3,404 firm records had at least one
matching customer record, and comprise the dataset referred to as FIRMS.

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

3

WPN 12-525

Step 2: Creating the CUSTOMERS File
An initial file of 41,981 customers was provided to match to the listing of top-filer firms. From this
initial file, 1,170 customers were dropped due to a location outside of the United States; 1,330 were
dropped due to recent registration; and 27 customers were dropped due to both issues. The final
number of customers available for sampling and matching was 39,454.

Step 3: Matching the FIRMS file to the CUSTOMERS File
The process of matching top-filing firms with registered customers also consisted of several phases.
First, a shortened version of organization name, with all commas, spaces, and other characters
removed was created. Then the FIRMS file and the CUSTOMER files were sorted by STATE,
ORG NAME, and CITY, and automated matching was done. A flag was set for any records where a
firm matched to a customer record. A total of 10,980 records were matched to a firm during this
process.
Next, the matched customer file was split into three files for research assistants to do manual
matching. Again, the matching was based on STATE, ORG NAME, and CITY, and if the matcher
found a match between the files, the value for the matching firm, FIRMID, was entered in Excel on
the customer file. An additional 4,567 customer records were matched to firms during this process,
including reviewing all non-matching firms manually in an attempt to match the records.

Sample Selection Summary
Now that the cleaned frame is ready, we are prepared to select the next four waves of customers to
contact for the next four administrations of the EQS. The USPTO sample is drawn from a frame of
PTO customers, all of whom are either associated with a particular firm or are considered
independent. There are six sampling domains for which different sampling rates are used. These are
described generally as follows:
1.

Large firms (more than 275 applications) with 50 or fewer customers, where all
customers will be sampled;

2.

Large firms (more than 275 applications) with more than 50 customers, where a sample
of customers will be drawn;

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

4

WPN 12-525

3.

Firms with 150 to 275 applications where a sample of customers will be drawn;

4.

Firms with less than 150 applications where a sample of customers will be drawn;

5.

Non-matched firms and independent inventors with an associated last name, where a
sample of customers will be drawn; and

6.

Non-matched firms and independent inventors without an associated last name, where
no sample will be drawn.

One of these six sampling domains is identified for each customer on the frame, using counts of the
number of applications within each firm in conjunction with a count of agents associated with that
firm. A sampling rate is then computed for each domain.
The USPTO uses a rotating panel design for the sample, so that sampled cases are assigned to waves
and then to two panels within each wave. The second panel from each wave is fielded in the
subsequent wave, along with a new panel.
Once the sampling rates have been determined and the sample of customers drawn, sample cases are
assigned to the waves (which occur 6 months apart) and also to the two panels within each wave.
Customers must stay out of the sample for at least 18 months. Because of this 18-month leave of
absence from the sample, it is necessary to control for when the old sample can rotate back into the
sample. Complicating this is the potential for panel conditioning effects from being in the old cycle.
Therefore, to reduce the impact from the distributional differences between frames, newly sampled
cases from old panels are spread out evenly across the new panels.

2.3

Survey Instruments

With each approaching wave of data collection, Westat staff will prepare the mail and Web versions
of the surveys. Each will be updated to reflect the current wave date and checked for accuracy in
printing. The survey currently covers examination quality concerns, importance of targeted
improvement areas, and assessment of USPTO progress from the previous wave time period. In
addition to ongoing measurement of customers’ perceptions about these core survey topics, the
USPTO plans to use the panel study to provide timely information about emerging issues. That goal
will be accomplished by adding new modules to the core survey as needed. We will follow the same
guidelines and procedures (starting with discussions with the client) used for the core survey to
develop the new modules. We recommend including new modules as a single section, located at or

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

5

WPN 12-525

near the end of the core survey, to help lower the costs of modifying the survey instrument,
database, and reporting templates. Our methodology will focus on procedures that minimize mode
effects and provide comparable data regardless of whether the respondent completes the survey on
paper or over the Web. As one way to improve survey response, we also propose a nonresponse
prompting telephone call. During this call (made to all who do not respond after the first mailing of
the survey and the mailing of a reminder postcard), respondents will be encouraged to complete the
survey using either of the modes of collection offered (i.e., paper survey or Web).

2.4

OMB Clearance

Westat routinely prepares Office of Management and Budget (OMB) clearance packages for our
Federal clients. Westat will assist the USPTO in preparing the OMB package for the EQS and any
supporting documentation. We have participated in this activity in collaboration with the OPQS in
the past and understand the materials required to the renewal package to extend OMB approval
from its current expiration date of October 2012 to 2015.

2.5

Data Collection

Although various modes, or combinations of modes, are potentially available for conducting the
EQS, we propose continuation of our current methodology of a self-administered paper survey
coupled with a Web version option. Over 16 waves of data collection, we have found that 78
percent of USPTO customers complete the EQS using the Web version. To encourage high rates of
response, we propose a data collection period of 6 weeks and a maximum number of contacts set at
four using a variety of communication methods. These contacts include: Prenotification, Survey
Invitation, Thank You/Reminder emails, Nonresponse Telephone Prompting.
In order to preserve the same fielding schedule used in past administrations of the EQS and to meet
the September 30, 2012 deadline for reporting, we present a suggested fielding schedule for
Wave 17 of the EQS (see Table 2-1).

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

6

WPN 12-525

Table 2-1.

Suggested Wave 17 fielding schedule (July 2012)

Task
Mailout sample delivered to project staff

Complete date
Wednesday, June 20th

Assemble prenotification letters and survey booklet packages

Week of July 2nd

Launch EQS Web survey

Monday, July 9th

Mail prenotification letters

Monday, July 9th

Mail survey booklet packages

Monday, July 16th

Mail reminder postcards

Monday, July 30th

Send email reminder

Monday, July 30th

Begin telephone non-response prompting

Monday, August 6th

Westat project staff deliver interim datasets (sample and survey) to
Westat statistical staff for quality control procedures

Monday, August 20th

Close data collection

Monday, August 27th

Final datasets (sample and survey) delivered to Westat statistical staff

Tuesday, August 28th

Final progress report created and delivered to USPTO

Tuesday, August 28th

Westat project staff delivers qualitative data to analysis staff

Wednesday, August 28th

Westat statisticians deliver analysis dataset to project staff

Tuesday, September 4th

Qualitative analysis delivered to project staff

Monday, September 10th

Westat project staff assemble biannual report

September 4th – September 21st

Delivery of biannual report to USPTO

Friday, September 28th

2.6

Data Processing

Before any analyses are conducted, the quantitative data will be cleaned. We will manually review
and edit all paper versions of the survey that are received, checking for completeness, extraneous
remarks or data outliers, and adherence to questionnaire skip patterns. Next, the data from cleaned
paper surveys will be entered by Westat data entry staff. To ensure data quality, we will key the data
from paper questionnaires twice, then compare the two sets of data, and resolve discrepancies. Data
entry for paper questionnaires will be performed continually as the surveys are received during the
field period.
Analogous quality control procedures will be implemented for qualitative data.

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

7

WPN 12-525

2.7

Data Analysis

After project staff clean and process the data, Westat statisticians will develop weights. The
weighting process consists of a series of adjustments. First, Westat will create base weights to
account for the probabilities of sample selection. Base weights are computed as the inverse of the
probability of selection. Second, the base weights will be adjusted to reduce potential nonresponse
bias resulting from the failure of some selected customers in the sample to respond to the survey.
For this nonresponse adjustment, we will use a classification algorithm (the Chi-squared Automatic
Interaction Detector, or CHAID) and the set of auxiliary variables on the sampling frame to form
nonresponse cells. These cells represent weighting classes that are homogeneous in terms of
response propensity. The classification algorithm divides the population into homogeneous
subgroups with respect to a target characteristic (the dependent variable). A nonresponse adjustment
factor is computed as the inverse of the weighted response rate within each weighting class, and is
applied to each respondent’s base weight.
As a final adjustment to the weights, if key population totals are available, the weights can be
calibrated to those totals to further reduce potential bias, as well as variance. Westat will use its
highly efficient standardized software, COLL_ADJ, to perform the weight adjustments. The
outcome is a data set of respondents whose information has been adjusted to compensate for the
data missing for nonrespondents.
For each wave, one set of sampling weights will be produced to serve both single wave estimates
and estimates of wave-to-wave change. This approach is a standard approach when measuring
change between successive waves at aggregate levels. The best measures of change in aggregate
levels are gained by optimizing the sample size and response rates at each wave.
For basic quantitative data analyses, we will use SPSS® software. Also, for any analytic procedures
that test for statistical significance (e.g., hypothesis or relationship testing, or other inferential
statistics), we propose using WesVar®. The specific types of analyses we conduct will continue to be
dictated by the needs of the USPTO. Should the needs of the USPTO change over the course of the
contract, we will work collaboratively to adjust the established analysis plan accordingly.
Semi-annual Wave Data. We will perform descriptive analyses on each survey item, including
generating frequency counts. We will compare results for the current and previous waves in order to
quantify the direction and extent of USPTO progress in addition to providing wave-by-wave results.

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

8

WPN 12-525

Variance Estimation. Variance estimation must take into account the sample design. In particular,
the sampling variance estimate for any statistic should account for the sample selection process and
the use of weighting adjustments. For this study, we propose using the stratified jackknife method to
estimate the variance associated with the cross-sectional estimates and the estimates of change.
Qualitative Analysis. We will continue to provide qualitative analysis of the single open-ended item
currently on the survey. We will clean the responses to this item, removing profanity and personal
names mentioned without altering the substance of the response. We will assemble all responses to
this item into a single document and convert that document into a PDF file. Further, we will
examine the response and report the 5-6 most common responses to the question and examine
whether there are any differences among the technology areas in responses to this item.

2.8

Reporting

To meet the deadline of reporting findings within 4 weeks of the close of data collection for each
wave, Westat staff will continue to use the reporting template already developed under the guidance
of USPTO staff. For quality control purposes, we will use a program to transfer the data from the
statistical program into the report template. We will visually compare the frequency printouts and
the final report to ensure accuracy. The semi-annual reports will include tables depicting overall
survey findings and comparisons of estimates by wave (when applicable). In addition, we will
document any limitations with the data.
All survey data files will be delivered to the USPTO at the end of each survey cycle. This delivery
will consist of survey data files in SPSS® format, all weights that were developed, and a codebook
and additional documentation necessary for interpreting the data files.

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

9

WPN 12-525

Westat’s Past Performance

3

This section presents three examples of our organizational experience that demonstrate our
knowledge of the USPTO, our background in survey design, longitudinal data collection, complex
data analysis, and fast turnaround reporting requirements.
U.S. Patent and Trademark Office. Under a contract with the Office of Patent Quality
Assurance, U.S. Patent and Trademark Office (USPTO), Westat is conducting the Quality Survey
(QS). The QS is a longitudinal rotating panel customer survey. It is a mail and Web survey
conducted quarterly with USPTO “top filers,” customers who have filed six or more patent
applications in the past year and who are most likely to be aware of day-to-day changes occurring
within the USPTO. Most top filers are either sole practitioners or are associated with a law firm.
Other possible affiliations include Federal Government agencies, universities or colleges, large
businesses, and individual inventors.
Westat is responsible for the overall survey design, selection and maintenance of the customer panel,
development and administration of the survey instrument, data analysis, quarterly briefings to
USPTO staff, and preparation of quarterly and annual data files with documentation. Each wave of
data collection includes top filers who have not previously been asked to respond to the QS, as well
as a subgroup that completed the survey in the previous wave. The sample size for each wave is
approximately 2,500. Web Surveyor, an off-the-shelf software package, is used to collect data on the
Web version of the QS. Quarterly response rates have ranged between 34 and 54 percent, with
approximately 78 percent of respondents choosing to respond via Web.
The results are used to produce point estimates of current examination quality, assess changes in
customer perceptions, and secure customer feedback on targeted training areas and opportunities
for improvement. Deliverables include weekly progress reports, quarterly and/or annual analytic
reports with documentation of methodology, and quarterly and/or annual data files. Our most
recent contract with USPTO for this work funds semi-annual fieldings of the QS through 2012.
Reference: Martin Rater, Office of Patent Quality Assurance, USPTO, martin.rater@uspto.gov,
(571) 272-5966.

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

10

WPN 12-525

Technology Innovation Program (formerly the Advanced Technology Program) at the
National Institute of Standards and Technology (NIST). Westat’s relationship with NIST spans
more than nine years. Across multiple contracts Westat has conducted the following surveys on
behalf of NIST 2:


The 2000, 2002, and 2004 ATP Applicant Surveys;



The 2003 Joint Venture Survey;



The 2009 and 2010 TIP Applicant Surveys;



The 2004, 2005, and 2007 Post Project Surveys; and



The 2009 Mailing List Survey.

One of the most complex pieces of work Westat conducted under contract with NIST was the
design, development, administration, and maintenance of NIST ATP’s Business Reporting System
(BRS). The ATP BRS system is a complex data collection system with multiple goals, users,
audiences, inputs, and outputs. The BRS consists of a series of surveys that each recipient of ATP
funding is obligated to complete on a quarterly basis during the course of the ATP project. These
surveys include a Baseline Survey, an Annual Survey, a Closeout Survey, and a Quarterly Report.
Beginning in early 2004, Westat played a major role in the redesign of all BRS instruments. Westat
also programmed and tested all the surveys for Web-based administration, designed and
implemented an integrated database which will allow longitudinal data analysis, designed a case
management system, and built a project manager system that will allow project managers at ATP to
monitor progress on reports. Since the Fall of 2004, Westat has implemented quarterly launches of
the BRS reports. The quarterly launch activities include (but are not limited to) preparation of
quarterly upload files, design, testing and administration of the BRS surveys, data collection and
nonresponse follow-up, data cleaning and data delivery.
Reference: Stephen Campbell, Advanced Technology Program, NIST, stephen.campbell@nist.gov,
(301) 975-3118.

2

All these surveys were some combination of mail, telephone, and Web administrations with deliverables that included, but were not limited to, a
cleaned data set, a codebook, data analysis and reporting of results.

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

11

WPN 12-525

Internal Revenue Service (IRS) Taxpayer Burden Surveys. The IRS models compliance
burdens imposed by the Federal tax system in order assess the impact of programs on taxpayer
burden, assess the role of burden in tax administration, and fulfill the IRS’s obligation to the Office
of Management and Budget (OMB) and Congress to provide information required by the
Paperwork Reduction Act. Westat has been awarded a blanket purchase agreement (BPA) by the
IRS to provide survey services for a series of taxpayer burden studies. For each of the surveys,
Westat is preparing an OMB clearance package; developing Web and mail versions of the survey and
survey administration materials; collecting and delivering survey data; and providing a small amount
of analysis and reporting. The task orders under this BPA are described below.
2010 and 2011 IRS Individual Taxpayer Burden (ITB) Surveys. Westat is developing and
administering these surveys to collect data on the time and money individual taxpayers spend filing
returns. The IRS is providing contact information for a sample of at least 20,000 taxpayers who filed
a 2011 tax return.
2010 Tax-Exempt (TE) Burden Survey. The purpose of this task is to obtain pre-filing and filing
compliance burden data for the tax exempt taxpayer population. Westat is supporting the
development and implementation of the data collection instrument and administering the survey.
For the 2010 survey, the IRS is providing contact information for a sample of at least
12,000 taxpayers identified for participation in the survey.
Taxpayer Compliance Burden Survey. Westat is supporting the development and
implementation of a data collection instrument to obtain post-filing compliance burden data for
taxpayer populations who have post-filing experiences with the IRS. The IRS is providing contact
information for a sample of approximately 8,000 taxpayers identified for participation in the survey.
2009 and 2011 IRS Business Taxpayer Burden (BTB) Survey. The 2009 BTB is a multimode
(Web, mail, and telephone) survey of 20,000 small, medium, and large businesses to assess their
burden in complying with tax regulations. The 2011 Survey will be administered to 11,000 business
taxpayers that filed a 2011 tax return.
Statistical Consulting. Under this task order, Westat is providing general statistical consulting on a
number of IRS projects.
Reference: John Guyton, Office of Research, IRS, john.guyton@irs.gov, (202) 874-0607.

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

12

WPN 12-525