0625-XXXX Supprtg Stmnt Part B-Surveys for User Satisfaction Impact & Needs rev 030415

SUPPORTING STATEMENT

U.S. Department of Commerce

International Trade Administration

Surveys for User Satisfaction, Impact, and Needs

OMB Control No. 0625-XXXX

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

 

1.  Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

 

All clients who have received assistance from ITA will have the opportunity to fill out a transactional survey (i.e. comment card).  The transactional comment card survey is not mandatory or predicated on receiving ITA assistance.  The estimated number of clients that will have the opportunity to fill out a transaction comment card survey will be 30,000.  Moreover, the estimated potential respondent universe will be around 6,900 based on a historic average response rate of 23 percent. However, ITA will note that this estimated figure is much higher than our historic averages show because ITA will be expanding the use of surveys to all services. Previous historic data were only from select services that were fee-based. Please see the table below for respondent universe and historic response rates.

 

Fiscal Year	Surveys Sent	Responses	Percentage
2014	6,030	1,462	24%
2013	5,062	1,528	30%
2012	6,904	1,427	21%
2011	9,107	2,108	23%
2010	20,187	4,758	24%
2009	19,190	4,486	23%
2008	10,878	2,551	23%

In addition, clients that have received in-depth assistance will be randomly selected to receive an annual survey.  The sampling methodology will be explained in further detail in question 2 below. The annual survey is also voluntary and estimated to be sent to 6,000 clients.  The estimated response rate will be 10 percent, based on last year’s annual survey results. Last year, the survey was sent electronically via e-mail to 4,291 U.S. companies and 239 responded, for a response rate of 6%. In addition, ITA called 1,985 U.S. companies, of which 250 participated in the survey, for a response rate of 13%. This is the only available data for the annual survey.

 

2.  Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

 

Only those selected to receive the annual survey will consist of a sample selection. Clients that have been identified by ITA trade specialists to receive in-depth assistance (assistance requiring multiple hours of client engagement/interaction) will be selected to receive the annual survey.  This survey is sent once a year and those that have received the annual survey in the previous year will automatically be exempted from being selected to receive the annual survey again.

The sampling frame will use a random sample at the overall organizational level and then use a stratified, random sample plan that takes into account the relevant components of its clients such as industry, size of business (perhaps number of employees or revenues), and/or geographic location. To provide a simple example, please see Table 1 (page 9). Assume that the relevant components are industry and size. Further, assume that there are four industries and two sizes of business per industry. ITA will need to know what the proportions of these four industries and two business sizes are across its sampling frame. For purposed of this example, assume that Industry 1 represents 50% of the client base and Industries 2, 3, and 4 represent 25%, 15% and 10%, respectively.

Perhaps another relevant indicator of interest may be the size of the business or organization. Size could be an indication of the number of employees a client has, or it could be total revenues, assuming clients are willing to report the number or it is publicly available. ITA will need to know the distribution of business sizes by industry – or if it chooses other variables to segment its clients. For purposes of this simplified example, assume the distribution is as shown in Table 1.

The extent to which ITA will be in a position to enhance the accuracy of the estimates depends on the selection of the segments it defines. Serious consideration needs to be given to a geographic component in the segmentation scheme since ITA manages its accounts and cases from geographically dispersed locations. Externalities unique to geographic areas may have an effect on accounts/cases and should be taken into consideration. In addition, having data elements such as the congressional district identified and associated with each account will facilitate decisions about how to respond to requests for results at the congressional district level.

The sample pool will be randomly selected and stratified using the following parameters: industry, size of business (number of employees or revenues), and geographic location. Each parameter will have equal weight and be proportional to the overall sample population. Additionally, the sampling procedure will adjust for over sampling to ensure that the appropriate segments (industry, business size and geographic location) are statistically represented in the respondent sample size. The annual survey will conclude once the sample size is statistically valid at the 95 percent confidence level.

All other surveys will not be using any sampling procedures and ITA hopes to achieve a 23 percent response rate, which is in line with its historic averages. ITA will be taking the following steps to improve its response rates: reducing the length of the Comment Card and ensuring that all other surveys are as short as possible, revising the content of the messages that will contain the survey links, changing the subject line and sending a pre-survey message prior to the survey being distributed.

3.   Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

 

ITA will be committed to employing the following best practices to increase response rates:

• Letting clients know that ITA will be soliciting their feedback at the beginning of a “case” because ITA values their perspective to inform its continuous improvement efforts.

• Engaging the client in determining an appropriate timeframe for capturing their feedback after a meaningful service has been delivered.

• Facilitating the client providing the feedback via a method of their choice – for example, via telephone interview, or email invitation to take a web survey.

• Providing pre-notification that a customer feedback survey is coming – this can be via a phone call (if the client has indicated a preference for doing the telephone interview) and/or via an advance email letting the client know when to expect the invitation and what the SUBJECT of the email will be and what FROM address will be indicated. (In the private sector, this pre-notification has been shown to be effective when it comes from a known or trusted source.)

• Ensuring that the ‘invitation’ language explains why ITA is conducting the survey, why it is important the selected client participate in the survey, how it intends to use the data collected and when the aggregated results will be shared.

• Sending reminders to encourage the client to participate.

• Sharing the aggregated results with all clients – for example, the results can be shared periodically on a website or via an email.

• Communicating changes/improvements that are being made in response to customer feedback surveys.

For the annual survey, the sampling procedure will adjust for over sampling in order to yield "reliable" data that can be generalized to the universe studied. That is, ITA will continue to send additional surveys from its sample pool until the appropriate response rate is attained to be statistically valid at the 95 percent confidence level for the universe studied.

ITA will conduct an analysis of nonresponse bias for the survey and for each important question on the survey. If significant bias is identified, ITA will propose changes to the survey to accommodate this bias and/or correct the results in a systematic manner. Since the annual survey is new, ITA has not compiled enough data, therefore cannot currently provide an analysis of nonresponse bias for this survey. However, ITA will conduct this analysis once we have obtained enough data.

4.  Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

 

A cross-tabulation and regression analysis will be employed for the annual survey results only.  The purpose of this test is to determine drivers of customer satisfaction and will be for internal use only. More specifically, ITA will quantitatively test the questionnaire itself in terms of:

• gaining insight into whether there are apparent relationships between/among satisfaction/use again/recommend questions and the needs and impact questions

• the effectiveness of trying to capture all three types of data (satisfaction/use again/ recommend, needs and impact) in one survey questionnaire

• utility of individual questions,

• understanding if there are topics that customers think should be included that were not in the pilot questionnaire

• obtaining feedback from respondents about the questionnaire experience itself

• gaining understanding about the willingness/ability of respondents to report on impact results

 

5.  Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Individuals who will be or have been consulting on the statistical aspects of the design of the advocacy user satisfaction survey, as well as those individuals who will actually collect and/or analyze the information, are:

Susan Grow, Federal Consulting Group: (202) 513-7687

Cheryl Oros, Federal Consulting Group: (202) 513-7687

Jon Cioffi, CFI Group: (734) 623-5424 

Joe Carter, International Trade Administration, U.S. Department of Commerce: (303) 844-5656

John Seo, International Trade Administration, U.S. Department of Commerce: (202) 809-5977

LEGAL AUTHORITY: PUBLIC LAW 15 U.S.C. et seq and 15 U.S.C. 171 et seq

Table 1

(NOTE: All data are for illustrative purposes only)

	Industry 1		Industry 2		Industry 3		Industry 4
Proportion of All Clients (A)	50%		25%		15%		10%
Business Size	% Large	% Small	% Large	% Small	% Large	% Small	% Large	% Small
Proportion within the Industry (B)	90	10	80	20	13	87	8	92
Proportion for Stratified, Random Sample (C)	45.00%	5.00%	20.00%	5.00%	1.95%	13.05%	0.80%	9.20%
Proportion of Respondents Reporting Increased Revenues (D)	35.00%	30.00%	40.00%	52.00%	32.00%	18.00%	22.00%	17.00%
Total Amt of Increased Revenues Reported (E)	$110,000,000	$20,000,000	$15,000,000	$8,000,000	$75,000,000	$25,000,000	$8,817,000	$27,000,451	$288,817,451
Total Respondents Reporting Revenue Increases (F)	20	30	19	31	18	32	17	37	204
Average Amt of Increased Revenues for this Segment (G)	$5,500,000	$666,667	$789,474	$258,065	$4,166,667	$781,250	$518,647	$729,742
Total Number of Clients for this Segment or "Cell" (H)	4500	500	2000	500	195	1305	80	920	(S) 10,000
Total Number to Apply Avg to (J)	1575	150	800	260	62	235	18	156
Total Estimated Revenue Increased for Segment (K)	$8,662,500,000	$100,000,000	$631,578,947	$67,096,774	$260,000,000	$183,515,625	$9,128,188	$114,131,636	$10,027,951,171

Calculations in support of Table 1

Total number of clients in sampling frame x proportion industry is of total sampling frame x proportion business size is of the industry = number of clients in that segment (S x A x B = H)

Total amount of Increased Revenue (or number of Jobs Created or number of Jobs Safeguarded) reported / number or respondents reporting Increased Revenue (or Jobs Created or Jobs Safeguarded) = average Increased Revenue (or Jobs Created or Jobs Safeguarded) among those reporting Increased Revenue (or Jobs Created reported or Jobs Safeguarded reported)(E/F=G)

Total number of clients for the segment x proportion of reporting Increased Revenues (or Jobs Created or Jobs Safeguarded) = total number to apply the average amount of Increased Revenue (or Jobs Created or Jobs Safeguarded) to (H x D = J)

Average amount of Increased Revenues for this segment (or Jobs Created or Jobs Safeguarded) x total number to apply the average to = total estimated Increased Revenue (or Jobs Created or Jobs Safeguarded) (G x J = K)

Once the total estimated Increased Revenues (or Jobs Created or Jobs Safeguarded) are determined for each segment (or cell), sum those to have Total Estimated Revenue Increased (or Jobs Created or Jobs Safeguarded)

4