Enhancements

U.S. Department of Veterans Affairs
Office of Acquisition and Logistics

Supplier Perception Survey
Response Analysis Plan

Submitted to:

Office of Management and Budget
(OMB Control Number: 2009-0751)

Supplier Perception Survey Response Analysis Enhancements

Contents
SECTION 1.0 EXECUTIVE SUMMARY ............................................................................................................. 3
About this Document ............................................................................................................................ 3
Findings ................................................................................................................................................. 3
SECTION 2.0 INTRODUCTION ........................................................................................................................ 4
Previous Response Analyses ................................................................................................................. 4
VA OAL’s Activities to Assess Enhancements to Response Analysis ..................................................... 4
3.0 CENTRAL CONTRACTOR REGISTRATION (CCR) DATABASE ..................................................................... 5
3.1 Matching CCR to Supplier Perception Survey Sampling Frame ...................................................... 5
3.2 Variables to be used from CCR........................................................................................................ 5
3.2.1 Business Type Category ............................................................................................................... 5
3.2.2 Organizational Type Category ...................................................................................................... 6
3.3.3 Firm Size ....................................................................................................................................... 7
3.3.4 Annual Receipts ........................................................................................................................... 7
3.3.5 Business Tenure Distribution ....................................................................................................... 8
4.0 VA ELECTRONIC CONTRACT MANAGEMENT SYSTEM (ECMS) DATABASE .............................................. 9
4.2 eCMS Data Elements to be Leveraged ............................................................................................ 9
4.2.1 Contracting Support by VA Office ................................................................................................ 9
4.2.2 Contracting Activity.................................................................................................................... 10
4.2.3 VA Contract Value ...................................................................................................................... 11
4.4.4 NAICS Code ................................................................................................................................ 11
5.0 CONCLUSION ......................................................................................................................................... 12

2

Supplier Perception Survey Response Analysis Enhancements

SECTION 1.0 EXECUTIVE SUMMARY
The Supplier Perception Survey is a component of the U.S. Department of Veterans Affairs (VA) Office of
Acquisition and Logistics (OAL) Supplier Relationship Transformation (SRT) initiative. This survey is one
of many enterprise feedback initiatives through which quantitative and qualitative information is
systematically collected to holistically assess VA Acquisitions Supply Chain Management policies and
practices.
About this Document
Subsequent to the most recent iteration of the Supplier Perception Survey, VA OAL presented the Office
of Management and Budget with the results of its analysis of supplier representativeness and nonresponse bias. The major take-away from that discussion was for VA OAL to investigate options to
incorporate objective data sources into response analyses going forward. This document presents the
results of VA OAL’s investigation.
Findings
VA OAL’s investigation identified nine new data elements that will be incorporated into Supplier
Perception Survey response analysis going forward. These data elements will be extracted from two
data sources:
 Central Contractor Registration (CCR) Database
 VA Electronic Contract Management System (eCMS) Database
Table I.1 below outlines the list of variables that will be utilized from each database.
Table I.1 Data Elements to be Included in Supplier Response Analysis
Field / Dimension
Business Type
Organization Type
Business Size
Annual Receipts
Business Tenure / Age
Contracting Support
Contracting Activity
VA Contract Value
Primary NAICS Code

Source
CCR
CCR
CCR
CCR
CCR
eCMS
eCMS
eCMS
eCMS

These nine data elements are expected to provide for a robust assessment of suppliers who respond to
this survey, as well as the entire target population suppliers. The remainder of this document provides a
more detailed assessment of these data elements.

3

Supplier Perception Survey Response Analysis Enhancements

SECTION 2.0 INTRODUCTION
The Supplier Perception Survey is a component of the U.S. Department of Veterans Affairs (VA) Office of
Acquisition & Logistics (OAL) Supplier Relationship Transformation (SRT) initiative. This survey is one of
many enterprise feedback initiatives, through which VA OAL collects quantitative and qualitative
feedback from VA Acquisitions stakeholders to holistically assess VA Acquisitions Supply Chain
Management policies and practices.
Previous Response Analyses
For the most recent iteration of the Supplier Perception Survey, chi-square goodness of fit testing was
performed on key supplier attributes to assess representativeness of respondents. Supplier selfreported attributes (e.g., business category) were compared with self-reported attributes from the
baseline Supplier Perception Survey conducted in 2010 due to unavailability of external data sources
containing detailed information on supplier attributes. Subsequent to this iteration of the survey, VA
OAL has investigated options for enhancing the data leveraged for response analysis.
VA OAL’s Activities to Assess Enhancements to Response Analysis
Table II.1 below lists the data sources that have been identified as potential sources for future response
analysis.
Table II.1 Data Sources Investigated by VA OAL
Data Source Investigated
Central Contractor Registration Database
VA Electronic Contract Management System
Federal Procurement Data System
Hoovers (Commercial) Corporate Data

Outcome
Will be used for general supplier attributes
Will be used for VA specific supplier information
Will not be used
Will not be used

VA OAL ultimately determined that two of the four data sources investigated – the Central Contractor
Registration (CCR) database managed by the General Services Administration (GSA); and the VA
Electronic Contract Management System (eCMS) managed by the VA Enterprise Acquisition Systems
(EAS) Service team – would provide ample data for more robust response analysis.

4

Supplier Perception Survey Response Analysis Enhancements

3.0 CENTRAL CONTRACTOR REGISTRATION (CCR) DATABASE
The Central Contractor Registration (CCR) Database contains rich information on federal government
contractors, such as organizational and business type codes. This database is managed by the General
Services Administration (GSA), and is updated daily with new suppliers and updates to information of
existing suppliers. All suppliers wishing to do business with the federal government outside of purchase
card transactions must register with CCR.
3.1 Matching CCR to Supplier Perception Survey Sampling Frame
CCR was evaluated by matching CCR records to the sampling frame from the most recent Supplier
Perception Survey. This was done in order to ensure the CCR assessment was directly applicable to VA
suppliers as opposed to all federal government suppliers.
3.2 Variables to be used from CCR
The following five fields will be leveraged from CCR for Supplier Perception Survey Response analysis
going forward:






Business Type
Organization Type
Business Size (Average number of employees)
Annual Receipts
Business Tenure (age)

The remainder of this section provides summary statistics on each of these variables as appended to the
Supplier Perception Survey supplier file.
3.2.1 Business Type Category
There are 74 business type codes. A firm may have one or more business type code. As many of the
business type codes in CCR were either irrelevant to VA OAL’s analysis, or provided too much detail –
logic was applied to consolidate relevant codes into major groups. Table III.1 below provides a summary
of how business type codes were mapped to major groups. Figure III.1 provides a distribution of these
groupings.
Table III.1 Business Type Codes
Business Type Grouping
Business Type Codes Included
Minority-Owned Firm
23, PI, OY, QZ, OW, FR
Government Entity
12, 2F, 2R, C6, C7, C8, NG, FO,
Woman-Owned
8A, 8C, 8D, 8E, A2
Veteran-Owned
A5, QF
Small Disadvantaged
27
Non-Profit
A7, 2U, A8

5

Supplier Perception Survey Response Analysis Enhancements

Figure III.1 Business Type Distribution

3.2.2 Organizational Type Category
The processing of organizational type codes was less challenging as these codes provide for nonoverlapping categorization of each record. Table III.2 provides a listing of these codes. Figure III.2 is a
percent distribution of organizational type codes for survey respondents as well as for the target
population.
Table III.2 Organizational Type Codes
Organizational Type Code
Description
2J
SOLE PROPRIETORSHIP
2K
PARTNERSHIP OR LIMITED LIABILITY
2L
CORPORATE ENTITY (NON TAX EXEMPT)
8H
CORPORATE ENTITY (TAX EXEMPT)
2A
U.S. GOVERNMENT ENTITY
CY
COUNTRY - FOREIGN GOVERNMENT
X6
INTERNATIONAL ORGANIZATION
ZZ
OTHER

6

Supplier Perception Survey Response Analysis Enhancements

Figure III.2 Organization Type Distribution

3.3.3 Firm Size
Seven firm size buckets were derived from the CCR’s ‘Average Number of Employees’ field. Figure III.3
shows a comparison of the percent distribution of survey respondents vs. the target population.
Figure III.3 Firm Size (Average Employees) Bucket Distribution

3.3.4 Annual Receipts
The ‘annual receipts’ field – provided in continuous form in CCR – is bucketed into six categories for
comparison of survey respondents with the target population in Figure III.4 below.

7

Supplier Perception Survey Response Analysis Enhancements

Figure III.4 Annual Receipts Bucket Distribution

3.3.5 Business Tenure Distribution
Business tenure was derived from the ‘Business Start Date’ field included on the CCR database. Figure
III.5 below contains the business tenure distribution.
Figure III.5 Business Tenure Distribution

8

Supplier Perception Survey Response Analysis Enhancements

4.0 VA ELECTRONIC CONTRACT MANAGEMENT SYSTEM (ECMS)
DATABASE
The VA Electronic Contract Management System (eCMS) is VA Acquisitions’ Enterprise Resource
Planning tool. Contracting staff use eCMS to enter information on procurement activity, contractors,
and internal customers. As of the Q2 FY 2012 Supplier Perception Survey, eCMS has also functioned as
the source from which VA supplier names are extracted for inclusion in the survey sampling frame.
The evaluation of eCMS data began with discussions between VA OAL and VA’s Enterprise Acquisition
Systems (EAS) Service team, the team that manages eCMS. The goal of this discussion was for VA OAL to
gain a better understanding of data elements available through eCMS. This discussion ultimately led to
the provision of an eCMS data extract by EAS for use by VA OAL in investigating the utility to be had by
using eCMS data elements for the Supplier Perception Survey response analysis.
The data analyzed from eCMS consisted of all records entered into eCMS between January 2012 and
May 2012. The objectives of this analysis were to: (1) Identify eCMS data fields that are expected to
provide utility in the analysis of Supplier Perception Survey responders and non-responders alike and (2)
Evaluate the quality of eCMS data fields deemed useful for survey response analysis.
4.2 eCMS Data Elements to be Leveraged
Based on OAL’s analysis, the following eCMS data elements – inherent to eCMS or derived - will be
incorporated into all subsequent Supplier Perception Survey response analyses:





Contracting Support Indicator
Contracting Activity Indicator
VA Contract Value
Primary NAICS Code

The remainder of this section provides summary statistics on these data elements for suppliers active in
eCMS between January 2012 and May 2012.
4.2.1 Contracting Support by VA Office
In prior surveys, visibility into which VA offices have provided suppliers with support during a given
evaluation period has been limited. VA OAL has identified new fields in eCMS that will allow for
attributing contracting support at the head of contracting activity (HCA) level (e.g., Veterans Health
Administration [VHA]), as well as down to the field office level. For instance, Figure IV.1 below
demonstrates the distribution of contracting activity as supported by VHA’s Veterans Integrated Service
Network (VISN) offices. VA OAL will be able to leverage this information to compare the HCA distribution
of suppliers who respond to the survey with that of the entire target population (e.g., are we getting a
representative sample of VHA customers).

9

Supplier Perception Survey Response Analysis Enhancements

Figure IV.1 VHA Contract Activity Support by VISN Office

4.2.2 Contracting Activity
The transactional-level eCMS data will allow us to derive a number of ‘activity’ related metrics. This data
will allow us to derive the number of eCMS transactions, the number of distinct orders, and number of
distinct contracts for each supplier in eCMS for a given timeframe. Moreover, data in eCMS will allow for
controlling by ‘type’ of action or contracting phase. And while it is expected that these metrics will prove
useful, care will be required when using this data to account for anomalous values appropriately.
Figures IV.2 and IV.3 below provide the contract and order distribution, respectively, for suppliers
included in this analysis.
Figure IV.2 Supplier Percent Distribution by Number of VA Contracts

10

Supplier Perception Survey Response Analysis Enhancements

Figure IV.3 Supplier Percent Distribution by Number of VA Orders

4.2.3 VA Contract Value
eCMS also provides a continuous contract value figure, which can be manipulated and bucketed for
purposes of distribution analysis and comparison between survey responders and non-responders.
Figure IV.1 below provides a frequency distribution of maximum contract value buckets.
Figure IV.4 Supplier Percent Distribution by Maximum Contract Value Bucket

4.4.4 NAICS Code
The two-digit NAICS code provides sufficient granularity in which to compare the distribution of
suppliers who respond to the survey versus that of the target population. NAICS codes as entered in
eCMS are based on the VA contracting officer’s determination of the supplier NAICS code applicable to
11

Supplier Perception Survey Response Analysis Enhancements

the product or service renders via the VA contract. This will allow us to assess the distribution of
suppliers - and to assess satisfaction differentials - on the basis of product / service provided. Figure IV.5
below provides the NAICS major group distribution for this file.
Figure IV.5 eCMS Actions by NAICS Major Group Distribution

5.0 CONCLUSION
The nine variables identified through CCR and eCMS are expected to provide VA OAL with a long-term,
systematic solution to incorporating objective data sources into analysis of survey responders and nonresponders alike. This will aid tremendously in enhancing our ability to detect and address response bias
in this survey, and will also further VA OAL’s ability to systematically evaluate factors beyond selfreported survey data that contribute to the satisfaction and perception of suppliers.

12