USPTO Quality Metrics

Quality metrics

Top of Form

The United States Patent and Trademark Office (USPTO) considers a quality patent to be one that is correctly issued in compliance with all the requirements of Title 35 as well as the relevant case law at the time of issuance. To support the issuance of quality patents, the USPTO aims to deliver quality work products that include appropriate and clear patentability determinations for every pending claim throughout prosecution. With a focus on continuous quality improvement, the USPTO is committed to assessing the quality of its work products and processes and identifying metrics that help provide a thorough understanding of this quality. This quality metrics approach includes a focus on: (1) statutory compliance measures; (2) process measures; (3) perception measures.

Compliance measures

As part of the USPTO’s quality assurance efforts, each Technology Center (TC) has supervisors and TC reviewers who conduct numerous quality reviews of work products each year for evaluations of employee quality as well as to provide coaching and mentoring based on their findings. In addition to these TC reviews, the Office of Patent Quality Assurance (OPQA) audits a random sample of work products each fiscal year. The results from the OPQA random reviews are used to generate the USPTO’s statutory compliance measures.

Expand all | Collapse all

Sampling

At the start of each fiscal year, a goal for the total number of OPQA random reviews is set based on the USPTO’s data needs and available resources. In recent years, the goal has been to randomly review approximately 12,000 work products in a given fiscal year. This yearly volume of reviews provides sufficient data to identify corps wide trends, provide TC level insight for select topics, and allows the USPTO to answer many inquiries from our stakeholders in a timely manner.

The random sample of work products reviewed by OPQA is representative of the population of work products the Patents organization completes and mails, both by Office action type (i.e., non-final rejection, final rejection, and allowance) as well as by technology-type. To accomplish this, OPQA randomizes all Office actions mailed within a previous seven day period to create a randomized pull list. Reviewers from OPQA are then assigned Office actions for review from the randomized pull list based on their assigned technology area.

Statutory compliance standard

OPQA reviews work products based on the statutory compliance standard. Examiner Office actions are reviewed for statutory compliance by evaluating whether the Office action includes appropriate and clear determinations for every pending claim based on the four patentability statutes:

·        35 U.S.C. §102 - Novelty;

·        35 U.S.C. §103 - Non-Obviousness;

·        35 U.S.C. §112 – Specification (Enablement, Written Description, Definiteness);

·        35 U.S.C. §101 – Inventions Patentable (Subject Matter Eligibility, Utility).

To comply with the statutory compliance standard, rejections must be correct and must, at a minimum, appropriately (1) identify the claim and relevant statute and (2) set forth sufficient evidence to put a person skilled in the art on notice as to why the claim is considered unpatentable. In addition, failure to make a rejection under a statutory basis that should have been made (i.e., omitted rejection) results in the work product not complying with the statutory compliance standard.

Master Review Form

The Master Review Form (MRF) is a modular review form used by reviewers in OPQA and the TCs to evaluate work products for statutory compliance as well as to evaluate quality in other areas of prosecution (e.g. search, response to arguments, etc.). The MRF includes over 20 sections and a 330+ questions library. In addition to the sections focused on statutory compliance and other quality related information, the MRF was updated in fiscal year (FY) 2020 to include “Characteristics” sections throughout the form to capture more specific information about use of best practices and to provide better insight into Office action clarity than in the past. The Characteristics sections also serve as the basis for designating of work products as an accolade. OPQA reviewers designate select high quality work products as an accolade for including best practices that go beyond the requirements of the

Manual of Patent Examining Procedure (MPEP).

View a copy of the current Master Review Form.

Review flow

In evaluating work products under the statutory compliance standard, OPQA reviewers evaluate every rejected claim in an Office action to ensure that the rejection of the claim was proper relative to each statute under which the claim is rejected.

By evaluating all claims under each statute, the USPTO is in fact performing many statutory compliance evaluations when reviewing an individual Office action. If the review finds that any single claim has an improper determination under any statute (e.g., an improper rejection or an improper omitted rejection) then the entire Office action is identified as non-compliant regardless of how many proper determinations were made or whether the non-compliance is the result of an independent claim or dependent claim.

If all the claims treated in the Office action are treated correctly under every statute then the case is found to be compliant. Any Office action where there was at least one claim found to be non-compliant is ultimately verified by an OPQA supervisor and sent to the relevant TC for consideration. If the receiving TC disagrees with an OPQA finding of noncompliance, they may file a rebuttal and OPQA has a process to reconsider the finding of noncompliance based on the arguments presented. The appropriate course of correction for work products ultimately finalized by OPQA as noncompliant is then left to the TC. For allowances specifically, this includes correction of the Office action prior to issuing an allowance of the patent application.  

Compliance metric

The random sample of Office actions reviewed by OPQA for statutory compliance generates an overall compliance metric for each statute. The metrics are calculated by dividing the total number of Office actions that properly evaluated all pending claims in light of the patentability statutes by the total number of Office actions reviewed. The statutory compliance metrics are then expressed as a percentage of Office actions reviewed that properly handled all claims in the application and can be further broken by relevant statute.

The compliance metrics generated by OPQA are validated in part through TC feedback as part of the rebuttal process, by comparisons to quality reviews performed in the TCs and data collected from external perception surveys. In addition, OPQA does not have any targets or incentives based on the findings of noncompliance, but does monitor the consistency of OPQA reviews as an additional validation.

The granularity of data obtained by reviewing all claims in an Office action for statutory compliance provides meaningful feedback to TC management and quality assurance specialists and facilitates the identification of quality trends, training opportunities, as well as an evaluation of recent training at the corps level and below.

Review results

This section includes a summary of data collected in the MRF as a result of the OPQA random reviews over the last fiscal year.

MRF Data Summary Table – Fiscal Year (FY) 2021

Other reviews

In addition to the random reviews that underpin the statutory compliance metric, OPQA conducts numerous other reviews throughout each fiscal year, often in partnership with the TCs. This allows for investigation of specific quality issues relevant to our stakeholder community as well as to the specific needs of each TC.

 

Process measures

The USPTO’s quality metrics also include process measures that assist the agency in tracking the efficiency and consistency of the examination processes. With respect to the examination process, the USPTO’s current focus is on preventing reopening of prosecution, reducing rework, and ensuring consistency of decision-making. The USPTO does not set consistency targets for particular transactions, but instead focuses on conducting a root-cause analysis on the trends and behaviors to either capture identified best practices or correct issues, as appropriate.

Monitoring

The USPTO leverages various data sources to monitor for consistency. This includes evaluating certain types of transactions in the Patent Application Location and Monitoring (PALM) system as well as the Master Review Form (MRF) to identify trends and examiner behaviors indicative of either best practices or potential quality concerns.

In addition to providing consistency data to supervisors, examiners are provided with on-demand access to their own examiner statistics for a better understanding of their prosecution trends and decision-making compared to the average in their art unit, TC, and the overall corps.

 

Perception measures

The USPTO has conducted both internal and external stakeholder perception surveys semi-annually since 2006. The results of these surveys are a vital quality indicator and they are useful for validating other USPTO quality related metrics. For example, the results of the external perception surveys assure alignment of the data underlying our metrics and our stakeholders’ perceptions and assure that the quality metrics we report are useful for our stakeholders.

Sampling

Internal Stakeholder Perception Survey: the internal survey is sent to 750 randomly selected patent examiners on a semi-annual basis.

External Stakeholder Perception Survey: the external survey is sent to 3,000 of our frequent-filing customers on a semi-annual basis. A rotating panel effect is used to minimize respondent burden but still facilitate comparisons across surveys. Respondents are stratified into panels and asked to participate in two consecutive survey periods, with two unique panels participating in each survey. Each collection period, 50% of the respondents (or one panel) are rotated out and replaced with a new panel. Customers rotating out of the survey are not selected for further participation until the sample frame is refreshed and new panel assignments are made, which occurs approximately every three years.

To maintain objectiveness and anonymity of respondents, the surveys are administered and analyzed by a third-party contractor that performs the work under the guidance of the USPTO’s Chief Statistician.

Survey overviews

The internal quality survey administered to patent examiners focuses on internal and external factors impacting examiners’ ability to provide high-quality patent examination. Internal factors address examiner satisfaction with topics such as patent examination tools, training, and the coaching and mentoring they receive. External factors address examiner satisfaction with incoming patent applications and applicant interactions during prosecution.

The external quality survey is designed to provide customer perspectives on the correctness, clarity, and consistency of rejections made by examiners. The survey also gathers perceptions about examiners’ adherence to rules and procedures and satisfaction with search and prior art. While the survey questions remain static to facilitate longitudinal analyses, a single open-ended question is incorporated during each enumeration to explore current topics of interest to the USPTO, such as specific effects of recent quality efforts or considerations for pending quality initiatives.

Survey results

External Survey – Fiscal Year 2021, Quarter 3

Internal Survey – Fiscal Year 2021, Quarter 4

 

We want your feedback

If you have questions or comments about quality metrics, please send an email to QualityMetrics@uspto.gov. For general inquiries about patent quality, please send an email to PatentQuality@uspto.gov.

Bottom of Form