2900-0701_Part_B. Awards & ROI (MO 9.8.15)

Awards & ROI Survey

VA Form 10-21081(NR)

2900-xxxx

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Provide a numerical estimate of the potential respondent universe and describe any sampling or other respondent selection method to be used. Data on the number of entities (e.g., households or persons) in the universe and the corresponding sample are to be provided in tabular format for the universe as a whole and for each strata. Indicate expected response rates. If this has been conducted previously include actual response rates achieved.

The Awards and Return-on-Investment Survey (Awards & ROI) will collect information from small and large businesses attending the National Veterans Small Business Engagement (NVSBE) during each year of this project. The table below shows data for 2013 and 2014. During each of these years an average of 1,100 small businesses and 220 large businesses attended the NVSBE. The VA Office of Small and Disadvantaged Business Utilization (OSDBU) was able to achieve an average response rate of 15% response among small business and 13% among large business which yielded to an average of 157 completed small business surveys and 30 completed large business per year.

Based on the 2013 and 2014 collected data, OSDBU estimates a response rate of 17% for small business for the 2015 NVSBE and 14% for large business. It is also expects a slight increase in the number of small and large business attending the NVSBE and an increase in responses rates through the next years.

Historical Universe of potential respondents, response rates, and completed surveys (Years 2013, 2014)

	Universe	QTRs	Annual total	Response Rate	Completed surveys
2013-SB	1356	n/a	1356	13.0%	177†
2013-LB	283	n/a	283	13.4%	38†
2014-SB	857	n/a	857	16.0%	137†
2014-LB	162	n/a	162	13.6%	22†

Projected Universe of potential respondents, response rates, and completed surveys (Years 2015, 2016, 2017)

	Universe	QTRs	Annual total	Response Rate	Completed surveys
2015-SB	950	n/a	950	17.0%	162∆
2015-LB	250	n/a	250	14.0%	35∆
2016-SB	1050	n/a	1050	18.0%	189∆
2016-LB	350	n/a	350	15.0%	53∆
2017-SB	1150	n/a	1150	19.0%	219∆
2017-LB	450	n/a	450	16.0%	72∆

†actual number of completed surveys

∆ projected number of completed surveys

2. Describe the procedures for the collection of information, including:

• Statistical methodology for stratification and sample selection

• Estimation procedure

• Degree of accuracy needed

• Unusual problems requiring specialized sampling procedures

• Any use of less frequent than annual data collection to reduce burden

The Awards & ROI data is collected one time approximately 8 months following the NVSBE. Given the length of time between the NVSBE and the date when respondents receive the survey, low response rates are expected to continue. The 8 month time frame is necessary to allow businesses time to develop awards or explore networking connections from the NVSBE and provide OSDBU with long-term feedback on effectiveness of the event. Considering the small size of the total population and the small number of businesses expected to respond, VA OSDBU will use a whole-population survey of the business attendees rather than a sampling procedure. The survey will be deployed to all small and large businesses attending the NVSBE. The use of a whole-population survey is particularly important to capture the unique aspects of responses from small set-aside groups within the target population, such as Woman-Owned Small Businesses and HUBZone Small Businesses.

The responses of this web-based survey are collected over secured, encrypted SSL/TLS connections. Secure Sockets Layer (SSL) and Transport Layer Security (TLS) technology protect communications using both server authentication and data encryption. This ensures that user data in transit is safe, secure, and available only to intended recipients. All responses are stored in secure data centers in accordance with industry best practices. The data repository is protected from unauthorized access.

3. Describe methods to maximize response rate and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

The Awards & ROI Survey is sent to just one point of contract from each company; that person is asked to forward it the corresponding person in case he or she is not the person that in addition to attend to the NVSBE, has procurement or contracting information related to this event. The following actions will be taken in order to maximize response rate and to ensure accuracy and reliability of the data:

• Reminders and follow-ups are employed to minimize non-response bias.

• Data and results will be fully documented along with the methodology employed.

• Consistent systems of data collection will be employed throughout the data collection process to ensure year-to-year monitoring is comparable.

• Quality control checks and internal peer review are performed on data and analysis products to ensure they are clear, concise, and accurate representations of the survey results.

• Monitor for and account for blank or "N/A only" non-responses in the data set to ensure effectively empty responses are not recorded as a response.

• Use tokenized anonymous links that ensure only a single response is submitted by each respondent.

• Employ technical measures to ensure the survey is completed correctly, such as to prevent multiple answers to multiple choice questions.

• Ensure questions have a null response, such as "N/A", where a respondent might not have a meaningful response.

The questionnaire employed for the Awards & ROI survey has been tested in the previous two years and the responses analyzed to understand the data collected and verify alignment with VA OSDBU research objectives.

The intended use of the data is to improve the long-term ROI for businesses that attend the NVSBE and identify areas for improvement and areas of success. Non-responses will be compared to the demographics of the target population to determine if any known demographics factors may be correlated with non-responses. The demographics, registration data, and other administrative data available to VA OSDBU from all NVSBE attendees will allow careful comparison of the response demographics to the demographics of all attendees to identify any bias.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions of 10 or more individuals.

OSDBU has conducted an analysis to test the accuracy of the 2013 collected data; specifically tested the percentage of all Veteran-owned small business (VOSB) that attended the NVSBE and received a VA procurement award. OSBDU’s Acquisition Support Team (AST) determined that during the 10 months after the 2013 NVSBE, 23% of the VOSB who attended the Fiscal Year (FY) 2013 NVSBE were awarded contracts from the VA. The Awards & ROI 2013 survey showed that the same percentage of VOSB that claimed to have received an award based on connections made or opportunities about at the 2013 event. While the information provided by AST states that the conditions exist for this specific population (receiving an award after the NVSBE and attending the NVSBE), the Award & ROI survey is able to measure the number of businesses that claim to receive an award because of their attendance to the NVSBE. The same analysis will be conducted for the 2014 and 2015 Awards & ROI data. OSDBU does not anticipate major changes in either the administration procedure or the survey itself as it looks to gather comparable data through the years.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Matthew Roy
Statistician

Turn Around Factor

7847 Riverside Dr.

Richmond, Virginia 23225

Tel: 804-554-2149

Email: matthew@turnaroundfactor.com