Supporting Statement for Evaluation of the Veterans Business Outreach Centers
Part B
(OMB Control Number: 3245- XXXX)
Prepared for:
Small Business Administration, Office of Veterans Business Development
409 3rd Street, S.W. Suite 5700
Washington, DC 20416
Phone: 202-205-6773
COR: Jonathan Jones
jonathan.jones@sba.gov
CS: Barbara Carson
barbara.carson@sba.gov
Prepared by:
Optimal
Solutions Group, LLC
M Square Research Park
5825 University Research Court, Suite 2800
College Park, MD 20740-9998
Point of Contact:
Jaime Wood, Director of Policy and Engagement, OVBD
Email: jaime.wood@sba.gov
Phone: 202-619-0151
PART B. Collections of Information Employing Statistical Methods 3
1. Universe and Sampling Respondent Selection 3
2. Describe the procedures for the collection of information 6
3. Describe methods to maximize response rates and to deal with issues of non-response. 8
4. Describe any tests of procedures or methods to be undertaken. 10
5. Expert contact information 13
Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.
Director Survey
The completion of the director survey, although not mandatory, will be highly recommended by the Office of Veterans Business Development (OVBD). Up to 15 directors will be invited to participate1 in a web-based survey, which will be hosted on a restricted web-based survey tool. However, VBOC directors are not obligated by their cooperative agreement with SBA to complete the survey. SBA will send an e-mail to each VBOC director with a secure link to the survey and directions for completing the survey. (See Appendix B-1 for screenshots of the director survey and Appendix B-3 for the director survey invitation e-mail.) The consent form will be embedded in the survey. Directors will be sent up to three e-mail reminders by Optimal before the deadline, and the director survey will be deployed once in November 2015. The reminder e-mail is included in Appendix B-4. As a result, a 100% completion rate is expected. This data collection effort has not been done previously.
Client Outcome Survey
The client outcome survey will be sent to a stratified sample of VBOC clients, and the survey will not be mandatory. This survey has not been conducted previously. The client sample will be stratified by center to maintain geographic representation because there is variability in client flow across centers; after stratifying, clients will then be randomly selected from each center. The client population has been broken down into two cohorts. Cohort 1 is composed of veterans from the first granting period (2010-2015) and who received at least one VBOC service in calendar year 2014. Cohort 2 is made up of veterans from the second granting period (starting April 2015) and who have received at least one VBOC service from May 2015 – August 2015. The data from the WebCATS/Neroserra database will be used to determine the clients comprising these two populations as well as the VBOC that provided a service to each client. An e-mail address is required for a client to be included in the survey sample. If a randomly chosen client does not possess an e-mail account or if the e-mail address is invalid, the client will be replaced by another randomly selected client. The screenshots for the client outcome web surveys are presented in Appendix C-1.
The electronic client survey will be hosted on a restricted web-based survey tool. SBA will announce the data collection effort to the VBOC directors; VBOC directors will announce the survey to the clients. Optimal will send an e-mail to each client with a secure link to the survey and directions for completing the survey. The consent form is embedded in the survey. Clients will be sent up to three e-mail reminders by Optimal to complete the survey. The reminder e-mail is included in Appendix C-6.
Cohort 1: The cohort 1 sample will include 2,000 VBOC clients who received services during calendar year 2014 from the first granting period. Because some VBOCs will no longer exist in the second granting period starting April 2015, the veterans from the first granting period that attended VBOCs no longer in operation under the second granting period are expected to have a lower rate of survey completion. OVBD expects a response rate of approximately 45% from this sample (N=900).2 All respondents are required to have a working e-mail address. If a client is chosen and does not have a working e-mail address, the client will be replaced by another randomly selected client. Based on the 78,124 clients served during 2014, the minimum sample size for a 99% confidence level and 5% margin of error is 658. Additionally, we will conduct inferential statistical analyses and estimate the minimum sample size required to detect statistically significant differences based on 80% power and a small effect size. For a t-test with unequal groups, 820 clients are required. For a regression analysis with up to 5 predictors, 647 clients are required. For analysis of variance with 2 groups, 788 clients are required. Accounting for the 45% response rate, a minimum of 820 responding clients are needed to create reliable estimates and detect statistically significant differences.
Cohort 2: The cohort 2 sample will include 2,000 clients who received services from May to August 2015, under the second grant cycle. OVBD expects a response rate of approximately 65% from this sample (N=1300).3 All respondents are required to have a working e-mail address. If a client is chosen and does not have a working e-mail address, the client will be replaced by another randomly selected client. Based on approximately 18,229 clients being served from May to July 2015, the minimum sample size for a 99% confidence level and 5% margin of error is 641. Additionally, we will conduct inferential statistical analyses and estimated the minimum sample size required to detect statistically significant differences based on 80% power and a small effect size. For a t-test with unequal groups, 820 clients are required. For a regression analysis with up to 5 predictors, 647 clients are required. For a chi-square test with 4 groups, 1091 clients are required. For analysis of variance with 4 groups, 1096 clients are required. Accounting for the 65% response rate, a minimum of 1096 responding clients are needed to create reliable estimates and detect statistically significant differences.
Site Visit and Interviews
SBA expects 100% response rates for site visits and in-person interviews. Four VBOCs will be purposively selected to provide an appropriate geographic distribution and represent “typical” centers. If the chosen center does not wish to host a site visit, another location will be chosen for the site visit to ensure that all four site visits are completed. SBA will send an introduction e-mail to VBOC directors explaining the purpose and methodology of the site visit (see Appendix D-1). Permission to visit these sites will be verified in site visit consent forms provided by Optimal and signed by the director of each site (see Appendix D-1). Once a director has signed and electronically returned the site visit consent forms, Optimal and the VBOC that is to be visited will work together to schedule the exact dates and times for the visit; and identify appropriate candidates for the client interviews. (Please see Appendix D-1 for the e-mail from Optimal to VBOC directors to schedule the site visit and set-up interviews with staff and clients.) Optimal will not interrupt normal proceedings at the site or participate directly in any activities. Optimal will keep all interview notes and recordings secured in order to ensure respondent confidentiality.
Exhibit 1: Expected Response Rate by Survey Population
Type of sample |
Respondent Universe |
Sample size |
Expected response rate |
Expected non-participants |
Site visit |
15 |
4 |
100% |
0 |
Director interview |
15 |
4 |
100% |
0 |
Staff interview |
161 |
16 |
100% |
0 |
Client interview (cohort 2) |
18,2292 |
16 |
100% |
18,213 |
Director survey |
15 |
15 |
100% |
0 |
Client survey (cohort 1) |
78,124 |
2,000 |
45% |
77,224 |
Client survey (cohort 2) |
18,229 |
2,000 |
65% |
16,929 |
1 Estimate approximates up to four staff members per VBOC. Many VBOCs only have one or two staff members.
2 This approximation is based on the average number of clients in the previous granting period per month and center, multiplied by the number of VBOCs in the new granting period and then multiplied by the three-month time period (May – August 2015) that will be surveyed.
Statistical methodology for stratification and sample selection, estimation procedure, degree of accuracy needed for the purpose described in the justification, unusual problems requiring specialized sampling procedures, and any use of periodic (less frequent than annual) data collection cycles to reduce burden.
Director Survey
There is no sampling for the director survey; each director will be invited to complete the web-based survey.
Client Outcome Survey
Based on the obtained client lists for cohorts 1 and 2, a stratified sampling approach will be used to select a nationally representative sample of VBOC clients (in each cohort) with probability proportional to the number of clients served per center. This is a single-stage stratification design utilizing VBOCs as the primary sampling strata. From that strata, clients will be chosen at random but will be removed from the sample and replaced if they do not have a working email address.
Sampling procedures will result in the selection of 2,000 clients per cohort. For cohort 1, based on the 78,124 clients served during 2014, the minimum sample size for a 99% confidence level and 5% margin of error is 658. Accounting for the 45% response rate, a minimum of 820 responding clients are needed to create reliable estimates and detect statistically significant differences. For cohort 2, based on approximately 18,229 clients being served from May to August 2015, the minimum sample size for a 99% confidence level and 5% margin of error is 641. Accounting for the 65% response rate, a minimum of 1096 responding clients are needed to create reliable estimates and detect statistically significant differences. Non-sampling errors arising from unit and item nonresponses will be dealt with through weighting and imputation, where appropriate. The client data will be weighted by calculating the sampling weight as the inverse of the probability of selection.
Site Visit and Interview
Four VBOCs will be purposively selected for the site visits and interviews in order to provide for appropriate geographic distribution and to represent “typical” centers. If a chosen center does not wish to host a site visit, another location will be chosen for the site visit to ensure that all four site visits are completed.
There is no sampling for the director interview. Each director of the selected VBOCs will be invited to participate in a qualitative interview. Following the site visit invitation e-mail and the receipt of a signed site visit consent form, Optimal will send VBOC directors a follow-up e-mail (see Appendix D-1) that provides details on the director, staff, and client interview. This e-mail will also include a copy of the director interview consent form (Appendix D-1). Directors do not need to send back the consent form ahead of time. An Optimal researcher will provide a copy of the consent form for directors to sign before beginning the interview. Interviews will be conducted in-person during the site visit.
There is no sampling for the staff interview; all staff from each center will be interviewed. Optimal plans to conduct up to four staff interviews per site; VBOCs typically have one or two staff members. Optimal will obtain consent from each interview participant (see Appendix E-1). Copies of the staff interview consent form will be provided in Optimal’s scheduling e-mail to VBOC directors. These forms do not need to be returned ahead of time. An Optimal researcher will provide a copy of the consent form for the respondents to complete before beginning the interview. Interviews with available staff will be conducted in-person during the site visit.
From each center, four clients that received services after April 2015 and are described by directors as “typical” clients will be purposively selected to participate in the client interviews. Optimal will interview two clients who received counseling services and two clients who received only training services. For the counseling clients, at least one pre-venture and one in-business client will be selected who represent “typical” clients in terms of services received, business needs, and business outcomes. Training clients will be selected based on the topic of training attended and no receipt of counseling services; one client will be selected from each of the two most highly attended training topics offered. Center directors will assist in selecting the clients to be interviewed to ensure that all criteria are met. Directors will be asked to identify clients in Optimal’s scheduling e-mail to directors (see Appendix D-1.) The goal is to interview clients (a total of twelve across all four site visits) in-person during Optimal’s site visits; however, telephone interviews will be offered as an alternative if selected clients cannot meet in-person. VBOC directors will obtain permission from clients for Optimal to contact the clients directly to schedule the interviews. Optimal will invite clients via e-mail (see Appendix F-1 for e-mail script) and/or telephone (see Appendix F-1 for phone script) to arrange the interview. Optimal will provide copies of client interview consent forms to directors in the scheduling e-mail (Appendix D-1). Optimal will provide another copy of the consent form (see Appendix F-1) to clients when scheduling interviews. Clients will not be required to fill out the form ahead of time. An Optimal research will provide a copy of the form for clients to sign in-person before beginning the interview.
The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.
The following data collection techniques will be used to maximize response rates and minimize respondent burden.
Director Survey
Surveys will be introduced to VBOC directors via an official SBA introduction e-mail. The e-mail to VBOC directors will explain that the survey is designed to collect information on how and why VBOCs operate in particular ways as well as their challenges, lessons learned, and best practices. Directors will not be required to complete the survey. Up to three reminder e-mails will be sent by Optimal to the directors to remind them of the survey deadline.
Client Survey
Surveys will be introduced to the VBOC directors by SBA; surveys will be introduced to clients via an official introduction e-mail from the VBOC director. After this introduction, Optimal will send an email to clients with the direct survey link. The e-mails to clients will explain that the survey is designed to collect information on their experiences interacting with a VBOC, their business goals, the progress they have made with achieving their business goals, and their attitudes about the VBOC assistance they received. Clients will not be required to complete the survey. Optimal will send up to three reminder e-mails encouraging them to complete the survey before the deadline.
Site Visits and Interviews
Plans for conducting site visits will be coordinated through OVBD.
Site visits with in-person interviews will be introduced to VBOC directors via an official SBA introduction letter and e-mail. The letter to VBOC directors will explain the purpose of the site visit and how the data collected will be used for performance evaluation purposes and finding ways to improve the program. Optimal will follow-up with directors that agree to the site visit for scheduling purposes. Directors will also be asked to identify clients to be interviewed. Directors will identify clients that they believe would be good interview candidates and who represent “typical” clients. Therefore, convenience sampling will be used to identify interview candidates.
Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from ten or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.
Across all five instruments, a total of nine respondents completed pretesting.
Director Survey
The director survey was pretested with a cognitive interview with one VBOC director prior to preparation of the Paperwork Reduction Act (PRA) package. The purpose of the cognitive interview was to identify respondents’ difficulties in understanding directions or interpreting and responding to the questions. This approach utilized a methodology of allowing respondents to offer their impressions of directions and questions, as well as having the researcher ask the respondent questions about directions and items, to determine whether respondents were having difficulty understanding the directions, meaning of the questions or words, problems remembering the information needed to answer questions, issues with unnecessary questions or response options, or trouble with missing questions or response categories. The items and/or directions that were reported by the respondents or were noticed by the interviewers as being problematic, difficult, or time-consuming to answer, were further discussed to determine the reasons for the difficulties and if they should be re-worded, shortened, or eliminated. Additionally, response burden was estimated using contractor staff as respondents, who were unfamiliar with the project and survey.
Based on the pretesting results, the survey was revised to minimize the respondents’ burden. The revisions involved rewording some questions to improve comprehension and the respondents’ ability to respond more quickly, omitting some items that were confusing or difficult to answer, adding skip question options to avoid asking questions that were not applicable, revising some of the response scales, adding a few specific questions and items, revising the surveys’ overall layout and format, and clarifying the directions.
Additional Pretesting
After OMB approval, additional pretesting of the director survey will occur. One cognitive interview and one web-based usability test with VBOC directors will be conducted. Based on the results of the additional pretesting, final revisions to the questions, response options, directions, and format may be made. SBA will report to OMB any such revisions to the survey.
Client Survey
The client survey was pretested with a cognitive interview of one in-business client prior to preparing the PRA package. The purpose of the cognitive interview was to identify respondents’ difficulties with interpreting and responding to the questions as well as understanding directions. This approach utilized a methodology of allowing respondents to offer their impressions of directions and questions, as well as having the researcher ask the respondent questions about directions and items, to determine whether respondents were having difficulty understanding the meaning of the directions, questions or words, problems remembering the information needed to answer questions, issues with unnecessary questions or response options, or trouble with missing questions or response categories. The items and/or directions that were reported by the respondents or were noticed by the interviewers as being problematic, difficult, or time-consuming to answer, were further discussed to determine the reasons for the difficulties and whether they should be re-worded, shortened, or eliminated. Additionally, the response burden was estimated using contractor staff as respondents, who were unfamiliar with the project and survey.
Based on the pretesting results, the survey was revised to minimize respondent burden. The revisions involved rewording some questions to improve comprehension and the ability to more quickly respond, omitting some items that were confusing or difficult to answer, adding skip question options to avoid asking questions that were not applicable, revising some of the response scales, adding a few specific questions and items, revising the surveys’ overall layout and format, and clarifying the directions.
Additional Pretesting
After OMB approval, additional pretesting of the client survey will occur; Optimal completed one cognitive interview with an in-business client. Three additional cognitive interviews (including one in-business and two pre-venture clients) and one web-based usability test with VBOC clients will be conducted. Based on the results of the additional pretesting, final revisions to the questions, response options, directions, and format may be made. SBA will report to OMB any such revisions to the survey.
Site Visit and Interviews
The director, staff, and client interview protocols were tested during two-day site visits at two VBOCs from the first granting period. Across the two centers, two directors and five clients were interviewed. Modifications, such as re-wording questions, making questions shorter, changing the order of questions, and eliminating questions, were made to the interview protocols based on these initial tests. These modifications should reduce respondent burden.
Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
Expertise |
Name |
Position |
Affiliation |
|
Labor economics, program evaluation |
Mark Turner |
President & CEO |
Optimal Solutions Group, LLC |
mturner@optimalsolutionsgroup.com |
Veteran families, program evaluation |
Allison Holmes |
Research Associate |
Optimal Solutions Group, LLC |
aholmes@optimalsolutionsgroup.com |
Program evaluation |
Brian Cramer |
Research Associate |
Optimal Solutions Group, LLC |
bcramer@optimalsolutionsgroup.com |
Statistical Contact
For questions regarding the study or questionnaire design or statistical methodology, contact:
Mark Turner, Ph.D., President and CEO
Optimal Solutions Group, LLC
M Square Research Park
5825 University Research Court, Suite 2800
College Park, MD 20740-9998
Telephone: 301-918-7301; e-mail: OVBD@optimalsolutionsgroup.com
1 The previous VBOC granting period ends in April 2015. OVBD is in the process of choosing new centers, and the new granting period will begin in April. As of now, OVBD is still considering applications and deliberating on the number of VBOCs. The maximum number of VBOCs is 15 so this document is working under the assumption that there will be 15 centers in the new granting period.
2 Response rates for web-based surveys with entrepreneurs or veterans are typically around 20% (e.g., NASPA Research and Policy Institute (2013) “Measuring the success of student veterans and active duty military students” http://www.naspa.org/images/uploads/main/NASPA_vets_13(1).pdf). However, OVBD expects a higher response rate due to the extensive relationships formed with clients.
3 OVBD expects a higher response rate in cohort 2 because the members of cohort 2 will be closer to the time of service and potentially still have ongoing contact with the VBOC.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Kelsye Turner |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |