Part B
Customer Satisfaction Survey Supplemental Supporting Statement
Title: Using the American Customer Satisfaction Index to Measure Customer Satisfaction in the Senior Community Service Employment Program
Abstract:
The 2006 amendments to Title V of the Older Americans Act (OAA-2006) (Pub. L.109-365) require that customer satisfaction surveys be conducted for all three customer groups: participants, host agencies, and employers. The Employment and Training Administration (ETA) is using the American Customer Satisfaction Index (ACSI) to meet the customer satisfaction measurement needs of several ETA programs including the Senior Community Service Employment Program (SCSEP). The survey approach allows grantees flexibility and, at the same time, captures common customer satisfaction information that can be aggregated and compared among national and state grantee. The measure is created with a small set of core questions that will form a customer satisfaction index. The index is created by combining scores from three specific questions that address different dimensions of customers' experience. Additional questions that do not affect the assessment of performance are included to allow grantees to effectively manage the program.
The ACSI is a widely used customer satisfaction measurement approach. It is used extensively in the business communities in Europe and the United States, including over 150 Fortune 500 companies. In addition, twenty-nine agencies of the Federal government have used the ACSI.
The ACSI will allow the SCSEP program to not only look at performance within the system, but also to gain perspective on SCSEP’s performance by benchmarking against organizations and industries outside of the workforce system. The ACSI also has a history of being useful in tracking change in customer satisfaction over time, making it an ideal way to gauge grantees’ progress toward continuously improving performance.
Since the ACSI trademark is the property of the University of Michigan and the Claes Fornell International Group (CFI), ETA established a license agreement with the University of Michigan that will allow grant recipients the use of the ACSI for a statewide sample of participants and employers.
In the first year, PY2004, baseline data were collected. The following year, PY2005, was the first year when evaluation and sanctions were possible. For PY2007, with the just-published SCSEP Interim Final Rule, the customer satisfaction measures, while still required by the 2006 amendments to the Older Americans Act, have had sanctions removed.
Compliance with 5 CFR 1320.8 Yes _X_ No ____
Consultation with persons outside the Department of Labor:
Name Telephone No. Agency/Company
Barry A. Goff, Ph.D. (860) 659-8743 The Charter Oak Group, LLC
Pretest Conducted: 18 people
Assurances of Confidentiality: A statement assuring confidentiality is included as follows:
For employers: “The Older Worker Program, also known as the Senior Community Service Employment Program (SCSEP), wants to provide the highest quality services to its customers. You can help us improve our services by answering the following questions. Please be completely honest. Your answers will be strictly confidential. Unless the question directs you otherwise, please answer each question on the basis of your most recent experience with the Older Worker Program.”
For participants: “Older Worker Program, also known as the Senior Community Service Employment Program (SCSEP), wants to provide the highest quality services to its customers. You can help us improve our services by answering the following questions. Please be completely honest. Your answers will be strictly confidential. No one in the agency will see your individual responses.”
For host agencies: “The Older Worker Program, also known as the Senior Community Service Employment Program (SCSEP), wants to provide the highest quality services to its customers. You can help improve services by answering the following questions. Please be completely honest. Your answers are strictly confidential. No one in the agency will see your individual responses. Unless directed otherwise, please answer based on your most recent experience with the Older Worker Program.”
Federal Cost:
Activity Category |
Cost |
Mailing, scanning, and processing |
$45,000 |
Administration, analysis, and reporting |
$308,000 |
TOTAL: |
$353,000 |
Burden Estimates:
There are three customer groups being surveyed. The surveys vary slightly in length depending on the customer group. The estimates are based on two different methods of administration. The grantees mail surveys to the participant and host agency respondents. The sub-grantees hand-deliver the employer surveys. Details of these methods are included in the methodology section.
Participants and Host Agencies
Frequency: Annually
Average Minutes/Hours per response: 6 minutes
Estimated burden hour costs: 0
Total burden hours: 2,700
Employers
Frequency: Ongoing throughout the year
Average Minutes/Hours per response: 6 minutes
Estimated burden hour costs: 0
Total burden hours: 440
Requested expiration date: Three years from approval of ICR package
Statistical Methodology
A. Measuring SCSEP Participant and Host Agency Customer Satisfaction
Participants
The weighted average of participant ratings on each of the three questions regarding overall satisfaction is reported on a 0-100 point scale. The score is a weighted average, not a percentage.
Host Agencies
The weighted average of host agency ratings on each of the three questions regarding overall satisfaction is reported on a 0-100 point scale. The score is a weighted average, not a percentage.
Who Will Be Surveyed?
Participants
All SCSEP participants who are active at the time of the survey or have been active in the preceding 12 months are eligible to be chosen for inclusion in the random sample of records.
Host Agency Contacts
Host agencies are public agencies, units of government and non-profit agencies that provide employment and related services to SCSEP participants. All host agencies that are active at the time of the survey or that have been active in the preceding 12 months are eligible for inclusion in the survey.
2. How Many (number to be surveyed)?
For each state grantee, two hundred and fifty completed surveys should be obtained each year for both participants and host agencies. At least 250 completed surveys for both customer groups should be obtained for each national grantee, depending on the number of states in which each national grantee is operating. It is anticipated that a sample of 370 will yield 250 completed interviews at a 70% response rate. In the event the number eligible for the survey is small (where 250 completed interviews are not attainable), attempts should be made to contact all participants or host agencies. The surveys of participants and host agencies will be conducted once each program year.
3. How Will the Data be Collected?
The responses are obtained using a uniform mail methodology. The rationale for using mail surveys includes: individuals and organizations that have a substantial relationship with program operators, in this case, with the SCSEP sub-grantees, are highly likely to respond to a mail survey; mail surveys are less expensive when compared to other approaches; and mail surveys are easily and reliably administered to potential respondents.
As with other data collected on the receipt of services, the responses to the customer satisfaction surveys must be held confidential as required by applicable state law. Before promising respondents confidentiality of results, grantees must ensure that they have legal authority for that promise. Such authority can be found in state privacy laws, for example.
To ensure ACSI results are collected in a consistent and uniform manner, the following standard procedures are used by grantees to obtain participant and host agency customer satisfaction information:
ETA’s survey research contractor, The Charter Oak Group, will determine sample sizes based on data in the SCSEP Performance and Results QPR (SPARQ) system. As with WIA, there may be smaller state and national grantees where 250 completed surveys will not be achievable. In such cases, no sampling will take place and the grantee will survey the population.
Grantees should ensure sub-grantees notify customers of the customer satisfaction survey and the potential for being selected for the survey.
Inform participants at the time of enrollment and exit.
Inform host agencies at the time of assignment of a participant.
Inform via mail selected participants that they will be receiving a survey in approximately one week.
When discussing the surveys with participants for any of the above reasons, refresh contact information, including mailing address.
Grantees should ensure that sub-grantees prepare pre-survey letters to those participants selected for the survey.
Grantees will provide a list of names to sub-grantees about 3 weeks prior to the date of the mailing of the surveys.
Letters should be personalized using a mail merge function and a standard text.
Each letter should be printed on the sub-grantee’s letterhead and signed in blue ink by the sub-grantee’s director.
Grantees are responsible for the following activities:
Provide letterhead, signatures, and correct return address information to DOL for use in the survey cover letters.
Send participant sample to sub-grantees with instructions on preparing and mailing pre-survey letters.
Contractors to the Department of Labor are responsible for the following activities:
Provide sub-grantees with list of participants to receive pre-survey letters.
Print cover letters for first mailing of survey.
Select signatory from list provided and obtain that individual’s permission for first mailing and follow-up mailings.
Obtain sufficient quantities of letterhead for first and second mailings.
Personalize letters for first mailing using mail merge function.
Each letter should be printed on the signatory’s letterhead and signed in blue ink with the signatory’s signature.
Generate mailing envelopes.
Enter preprinted survey number into worksheet for each customer.
Assemble survey mailing packets: cover letter, survey, and stamped reply envelope.
Mail surveys on designated day. Enter date of mailing into worksheet.
Send survey worksheet to the Charter Oak Group.
From list of customers who responded to first mailing, generate list for second mailing.
Print second cover letter using mail merge function and standard text (different text from the first letter). Letters should be personalized as in the first mailing.
Enter preprinted survey number into worksheet for each customer to receive second mailing.
Assemble second mailing packets: cover letter, survey, stamped reply envelope.
Mail surveys on designated day. Enter date of mailing into worksheet.
Send survey worksheet to the Charter Oak Group.
Repeat tasks 7-12 if third mailing is required.
4. What are the Core Questions?
The core questions to be included in the mail surveys for participants and host agencies are detailed below. The other questions may be viewed in the survey forms submitted through ROCIS.
Utilizing the scale of 1 to 10 below, what is your overall satisfaction with the services provided by the Older Worker Program? (Choose one number)?
Dissatisfied Satisfied
1 2 3 4 5 6 7 8 9 10 11 12
Considering all of the expectations you may have had about the services, to what extent have the services met your expectations? A 1 now means “Falls Short of Your Expectations” and 10 means “Exceeds Your Expectations.”
Falls Short of Exceeds DK REF
Expectations Expectations
1 2 3 4 5 6 7 8 9 10 11 12
Now think of the ideal program for people in your circumstances. How well do you think the services you received compare with the ideal set of services? A 1 now means “Not very close to the Ideal” and 10 means “Very Close to the Ideal.”
Not Close Very Close DK REF
To Ideal To Ideal
1 2 3 4 5 6 7 8 9 10 11 12
B. Measuring SCSEP Employer Customer Satisfaction
The weighted average of employer ratings on each of the three questions regarding overall satisfaction is reported on a 0-100 point scale. The score is a weighted average, not a percentage.
Who Will Be Surveyed?
Employers that hire SCSEP participants and employ them in unsubsidized jobs. To be considered eligible for the survey, the employer: 1) must not also be a host agency; and 2) must have had substantial contact with the sub-grantee in connection with placing the participant; and 3) must not have accepted another placement in this program during the current program year.
All employers that meet the criteria in B1 will be surveyed at the time the sub-grantee conducts the first case management follow-up, which occurs 30 days after the date of placement.
2. How Many (number to be surveyed)?
For grantees of less than 1250 participants, nearly all employers will need to be surveyed. For larger grantees, it will be necessary to survey sufficient employers to assure that at least 250 employers respond.
3. How Will the Data be Collected?
The responses are obtained using a uniform mail methodology. The rationale for using mail surveys includes: employers that have a substantial relationship with program operators are highly likely to respond to a mail survey; mail surveys are less expensive when compared to other approaches; and mail surveys are easily and reliably administered to potential respondents.
As with other data collected on the receipt of services, the responses to the customer satisfaction surveys must be held confidential as required by applicable State law. Before promising respondents confidentiality of results, grantees must ensure that they have legal authority for that promise. Such authority can be found in State privacy laws, for example.
To ensure ACSI results are collected in a consistent and uniform manner, the following standard procedures are to be used by grantees to obtain employer customer satisfaction information:
Provide counts of employers (required for first year of survey only) to ETA’s survey research contractor, the Charter Oak Group.
Grantees should ensure that sub-grantees notify employers of the customer satisfaction survey and the potential for being selected for the survey. Employers should be informed at the time of placement of the participant.
Grantees and sub-grantees are responsible for the following activities:
Sub-grantee identifies employer for surveying the first time there is a placement with that particular employer in the program year. Employer is selected only if it is not also a host agency and the sub-grantee has had substantial communication with the employer in connection with the placement. Each employer is surveyed only once each year.
Sub-grantee enters preprinted survey number into SCSEP database.
Sub-grantee hand delivers survey packet (cover letter, survey, stamped reply envelope) to employer contact in person at time of 30-day follow-up (Follow-up 1). Mail may be used if hand delivery is not feasible.
Sub-grantee enters date of delivering packet into database.
The contractor sends weekly e-mail to all grantees and sub-grantees listing numbers of all employer surveys that have been completed. Sub-grantee reviews e-mails for three weeks following the delivery of the survey to determine if survey was completed.
If completed survey received by the contractor, sub-grantee updates database with date. If survey not received, sub-grantee calls employer contact and uses an appropriate script to either encourage completion of the first survey (if the employer still has it) or indicate that it will send another survey for completion.
If needed, sub-grantee generates second cover letter using same procedures as for first cover letter.
Sub-grantee enters preprinted survey number into database.
Sub-grantee assembles survey packet (cover letter, survey, return envelope) and mails to employer contact.
Sub-grantee enters date of mailing into database.
Sub-grantee sends notification to grantee that survey has been delivered: employer name, survey number, date of delivery.
Sub-grantee reviews e-mails from vendor for three weeks following the mailing of the second survey (or the employer’s assurance that it will complete the original survey) to determine of survey has been completed.
If survey received, sub-grantee updates database with date.
If third mailing needed, sub-grantee repeats steps 9-16.
Grantee monitors process to make sure that all appropriate steps have been followed and to advise sub-grantee if third effort at obtaining completed survey is required.
4. What are the Core Questions?
The core questions to be included in the mail surveys for participants and host agencies are detailed below. The other questions may be viewed in the survey forms submitted through ROCIS.
Utilizing the scale of 1 to 10 below, what is your overall satisfaction with the services provided by the Older Worker Program? (Choose one number)?
Dissatisfied Satisfied
1 2 3 4 5 6 7 8 9 10 11 12
Considering all of the expectations you may have had about the services, to what extent have the services met your expectations? A 1 now means “Falls Short of Your Expectations” and 10 means “Exceeds Your Expectations.”
Falls Short of Exceeds DK REF
Expectations Expectations
1 2 3 4 5 6 7 8 9 10 11 12
Now think of the ideal program for people in your circumstances. How well do you think the services you received compare with the ideal set of services? A 1 now means “Not very close to the Ideal” and 10 means “Very Close to the Ideal.”
Not Close Very Close DK REF
To Ideal To Ideal
1 2 3 4 5 6 7 8 9 10 11 12
C. Definition of Terms
Sample. A group of cases selected from a population by a random process where everyone has an equal probability of being selected.
Response rate. All of those who respond to all three of the core questions on the survey divided by the percentage of people contacted and eligible.
D. Calculation of the ACSI
The ACSI scores represent the weighted sum of the three ACSI questions’ values, which are transformed into 0 to 100 scale value. The weights are applied to each of the three questions to account for differences in the characteristics of the state’s customer groups.
For example, assume the mean values of three ACSI questions for a state are:
1. Overall Satisfaction = 8.3
2. Met Expectations = 7.9
3. Compared to Ideal = 7.0
These mean values from raw data must first be transformed to the value on a 0 to 100 scale. This is done by subtracting 1 from these mean values, dividing the results by 9 (which is the value of range of a 1 to 10 raw data scale), and multiplying the whole by 100:
1. Overall Satisfaction = (8.3 -1)/9 x 100 = 81.1
2. Met Expectations = (7.9 -1)/9 x 100 = 76.7
3. Compared to Ideal = (7.0 -1)/9 x 100 = 66.7
The ACSI score is calculated as the weighted averages of these values. Assuming the weights for the example state are 0.3804, 0.3247 and 0.2949 for questions 1, 2 and 3, respectively, the ACSI score for the state would be calculated as follows:
(0.3804 x 81.1) + (0.3247 x 76.7) + (0.2949 x 66.7) = 75.4
Weights were calculated by a statistical algorithm to minimize measurement error or random survey noise that exists in all survey data. State-specific weights are calculated using the relative distribution of ACSI respondent data for non-regulatory Federal agencies previously collected and analyzed by CFI and the University of Michigan.
Specific weighting factors have been developed for each state. New weighting factors will be published annually. It should be noted that the national grantees will have different weights applied depending on the state in which their sub-grantees’ respondents are located.
E. Response Rate Estimate
The initial response rate for each of the three customer groups is anticipated to be between 50 and 70 percent, with multiple mailings being used to obtain the desired 70 percent rate. In recent years, mail surveys that follow Dillman’s5 TDM methodology have achieved response rates of 50-70 percent. For example, a review of mailed surveys to physicians between 1985 and 1995 found the average response rate to be 61 percent. In an article from the Urban Institute on response rates for mail surveys among non-profits6, they reported that among 17 studies there was a grand response rate of 42 percent. These studies varied in the degree to which they followed the Dillman recommendations, however.
1 DK = Don’t Know
2 REF = Refused to Answer
3 DK = Don’t Know
4 REF = Refused to Answer
5 Dillman, Don A. (1978) Mail and Telephone Surveys. John Wiley & Sons. New York, NY.
6 Forthcoming, Nonprofit and Voluntary Sector Quarterly
| File Type | application/msword |
| File Title | OMB 83-I |
| Author | rbenedict |
| Last Modified By | hostetter.phil |
| File Modified | 2007-06-27 |
| File Created | 2007-06-25 |