1205-0040 Supporting Statement Part B_FINAL 8.7.18

Senior Community Service Employment Program (SCSEP)

OMB Control No. 1205-0040

August 2018

Supporting Statement B, General Instructions:

Collections of Information Employing Statistical Methods

The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When the question “Does this ICR contain surveys, censuses or employ statistical methods” is checked, "Yes," the following documentation should be included in the Supporting Statement Part B to the extent that it applies to the methods proposed. Make a copy of this format, Supporting Statement Part B, 1-5, and provide your response below each numbered topic. This is a separate document from Part A since they are uploaded in different sections in ROCIS.

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

Title V of the Older Americans Act requires that Senior Community Service Employment Program (SCSEP) include in its core measures “Indicators of effectiveness in serving employers, host agencies, and project participants.’’ In its Interim Final Rule published on December 1, 2017, the Department defined this new core measure as “the combined results of customer assessments of the services received by each of these three customer groups.” See section 641.710. In the preamble to this section of the regulations, the Department stated that although WIOA was piloting a new measure of effectiveness in serving employers, that measure does not have obvious application to SCSEP’s other two customer groups. As a result, the Department announced that it would continue to conduct the existing customer satisfaction surveys for each of its three customer groups “as an interim measure at least until the WIOA pilot is complete and a WIOA measure is defined in final form.” SCSEP has been conducting these three customer satisfaction surveys nationwide since 2004.

The interim measure adopted by SCSEP to satisfy the statutory requirement is the ACSI. The survey approach employed allows the program flexibility and, at the same time, captures common customer satisfaction information that can be aggregated and compared among national and state grantees. The measure is created with a set of three core questions that form a customer satisfaction index. The index is created by combining scores from three specific questions that address different dimensions of customers' experience. Additional questions that do not affect the assessment of grantee performance are included to allow grantees to effectively manage the program.

The ACSI is a widely used customer satisfaction measurement approach. It is used extensively in the business communities in Europe and the United States, including more than 200 companies in 44 industries. In addition, over 100 Federal government agencies have used the ACSI to measure citizen satisfaction with more than 200 services and programs.

As stated in the preamble to section 641.710, during this interim period, the Department will explore with grantees, and with its three customer groups, options for best measuring the effectiveness of SCSEP’s services. The Department will also explore ways to improve the efficiency of the current customer surveys (including the use of online surveys and changes to the administration of the employer survey) and will examine what, if any, new or revised questions would support an index of effectiveness as an alternative to the current index of satisfaction.

SCSEP currently funds 75 grantees, (19 national grantees (one of which has two separate grants) operating in 47states, DC, and Puerto Rico; and 52 state grantees, including DC and Puerto Rico; the three overseas territories and USVI do not participate in the surveys):

NATIONAL GRANTEES

AARP Foundation

Asociación Nacional Pro Personas Mayores

Associates for Training and Development, Inc. (A4TD)

Easter Seals, Inc.

Experience Works, Inc.

Goodwill Industries International, Inc.

Institute for Indian Development

Mature Services, Inc.

National Able Network

National Asian Pacific Center on Aging

National Caucus and Center on Black Aged, Inc.

National Council on Aging, Inc.

National Indian Council on Aging

National Older Worker Career Center

National Urban League

Operation A.B.L.E of Greater Boston, Inc.

Senior Service America, Inc.

SER – Jobs for Progress National, Inc.

Workplace, Inc.

STATE GRANTEES

Alabama Department of Senior Services

Alaska Department of Labor and Workforce Development

Arizona Department of Economic Security

Arkansas Department of Human Services, Division of Aging and Adult Services

California Department of Aging

Colorado Department of Human Services, Aging and Adult Services

Connecticut Department of Social Services

Delaware Division of Services for Aging and Adults with Physical Disabilities

District of Columbia Department of Employment Services

Florida Department of Elder Affairs

Georgia Department of Human Services

Hawaii Department of Labor And Industrial Relations

Idaho Commission on Aging

Illinois Department on Aging

Indiana Family and Social Services Administration

Iowa Department on Aging

Kansas Department of Commerce, Business Development Division

Kentucky Division of Aging Services

Louisiana Governor's Office of Elderly Affairs

Maine Office of Elder Services

Maryland Department of Aging

Massachusetts Executive Office of Elder Affairs

Michigan Office of Services to the Aging

Minnesota Department of Employment and Economic Development

Mississippi Department of Employment Security

Missouri Department of Health and Senior Services

Montana Department of Labor and Industry

Nebraska State Unit on Aging

Nevada Aging and Disabilities Services Division

New Hampshire Department of Resources and Economic Development

New Jersey Department of Labor and Workforce Development

New Mexico Aging and Long-Term Services Department

New York State Office for the Aging

North Carolina Department 0f Human Services, Division of Aging And Adult

North Dakota Department of Human Services

Ohio Department of Aging

Oklahoma Employment Security Commission

Oregon Department of Human Services, Aging and People with Disabilities

Pennsylvania Department of Aging and Long Term Living

Puerto Rico Department of Labor

Rhode Island Department of Elderly Affairs

South Carolina Lt. Governor's Office on Aging

South Dakota Department of Labor and Regulation

Tennessee Department of Labor And Workforce Development

Texas Workforce Commission

Utah Department of Human Services

Vermont agency of Human Services

Virginia Department for Aging and Rehabilitative Services

Washington Department of Social and Health Services, State Unit on Aging

West Virginia Bureau of Senior Services

Wisconsin Department of Health Services, Bureau of Aging and Disability Resources

Wyoming Department of Workforce Services

A. Participant and Host Agency Surveys

1. Who is Surveyed

Participants - All SCSEP participants who are active at the time of the survey or have been active in the previous 12 months are eligible to be chosen for inclusion in the random sample of records. In the latest survey, for PY 2017, there were approximately 60,000 participants eligible for the survey, of whom 21,409 were selected for the participant sample.

Host Agency Contacts - Host agencies are public agencies, units of government, and non-profit agencies that provide subsidized employment, training, and related services to SCSEP participants. All host agencies that are active at the time of the survey or that have been active in the preceding 12 months are eligible for inclusion in the sample of records. In the latest survey, for PY 2017, there were approximately 30,000 host agencies eligible for the survey, of which 14,929 were selected for the host agency sample.

2. Sample Size and Procedures

The national and state grantees have parallel, but complementary procedures. The national grantees have a target of 370 at the first stage of sampling and at the second stage, a target of 70 for each state in which the grantee provides services. The state grantees have only one stage to their sampling procedure. It is anticipated that a sample of 370 will yield 222 completed interviews at a 60 percent response rate. Following the sampling procedure, for each state grantee, 222 completed surveys should be obtained each year for both participants and host agencies. At least 222 completed surveys for both customer groups will be obtained for each national grantee, depending on the number of states in which each national grantee is operating. For all but the smallest national grantees, the number of completed surveys exceeds 222. In cases where the number eligible for the survey is small and 222 completed interviews are not attainable (a situation common among state grantees), the sample is actually the population of all participants or host agencies. The surveys of participants and host agencies are conducted through a mail house once each program year. The target of 222 completed surveys produces a sufficiently narrow confidence interval to provide an accurate estimate of the population values for the ACSI.

The actual number of participants sampled in PY 2015, the year of last completed survey analysis, was 21,566. There were 12,944 returned surveys that had answers to all three ACSI questions, for a response rate of 60.0%. 15,091 host agencies were surveyed in PY 2015. There were 9096 returned surveys, for a response rate of 60.1%%. For both surveys, all participants and host agencies that were active within the 12 months prior to the drawing of the samples were eligible to be surveyed.

The response rate tables below for the PY 2004-PY 2015 participant and host agency surveys show the number of returned surveys that had answers to all three ACSI questions and the response rate based on those complete responses. The number of actual responses differs from the number expected by the sampling procedure because the large national grantees operate in many states and sampling 70 from each of those states yields far more than the minimum sample of 370 would yield. At the same time, most state grantees do not have sufficient participants or host agencies to permit sampling, so all participants and host agencies are surveyed.

3. Response Rates

Response rates achieved for the participant and host agency surveys since 2004 have ranged from 56 percent to 70 percent. Both participant and host agency response rates for PY 2015 are 60 percent.

We have received a total of 257,212 responses since PY 2004, 158,676 from the participant survey and 98.526 from the host agency survey. Below is a table of responses and response rates. (Note: There were no surveys for PY 2007, and in PY 2011, there was no host agency survey. There were no surveys in PY 2016, and the survey currently in the field is designated as PY 2016-2017.)

Participant ACSI Response Rate

	PY04	PY05	PY06	PY07	PY08	PY09	PY10	PY11	PY12	PY13	PY14	PY15
Response Rate	69.2%	62.9%	61.7%		56.1%	63.8%	62.7%	60.6%	63.9%	59.9%	61.1%	60.0%
# of Responses	15,662	15,806	14,317		13,522	15,535	15,969	14,822	13,844	12,804	13,451	12,944

Host Agency ACSI Response Rate

	PY04	PY05	PY06	PY07	PY08	PY09	PY10	PY11	PY12	PY13	PY14	PY15
Response Rate	69.8%	67.9%	64.5%		62.3%	64.6%	60.4%		62.3%	58.9%	58.0%	60.1%
# of Responses	10,655	10,314	10,679		10,569	11,331	10,606		9,345	8,077	7864	9096

B. Employer Surveys

1. Who Is Surveyed

Employers that hire SCSEP participants and employ them in unsubsidized jobs are included in the Employer Survey. To be considered eligible for the survey, the employer: 1) must not have served as a host agency in the past 12 months; and 2) must have had substantial contact with the sub-grantee in connection with hiring of the participant; and 3) must not have received another survey from this program during the current program year.

All employers that meet the three criteria described above are surveyed at the time the sub-grantee conducts the first case management follow-up, which typically occurs 30-45 days after the date of placement.

2. Sample Size and Procedures

All employers meeting the three criteria described above will be surveyed. No sampling is used.

For the employer surveys, all qualified employers are surveyed because the number of qualified employers is relatively low. Employers qualify for the survey if they did not also serve as host agencies, if the grantee was directly involved in making the placement, and the employer was aware of the grantee’s involvement. Employers are only surveyed for the first hire they make in each 12 month period. The number of qualified employers in any given program year is estimated to be approximately 1650. That should yield 1000-1100 returned surveys. Although there are many more placements than 1650, roughly 30% of them are with host agencies, and most of the remainder were self-placements by the participant. In a 12-month period from March 2016-March 2017, there were 271 returned employer surveys. It was not possible to calculate a response rate because of grantee non-compliance with the procedures for survey administration. We are developing new procedures an enhanced management report, and new edits to the data collection system to improve compliance with survey administration and increase the number of employers surveyed.

3. Response Rates

Response rates for the employer survey have been difficult to track because of grantee non-compliance with the requirement to enter the survey number and date of mailing into the system. Where the survey administration requirements have been followed, employer response rates have been very high. To improve compliance with the procedures, several additional changes will be made to the administration of the employer survey. The changes include: 1) an enhanced management report that will tell grantees when each qualified employer needs to be surveyed; 2) additional edits in the data collection system that will warn grantees when they have failed to deliver surveys as required;3) new edits that will prevent the entry of inaccurate data, such as the entry of the same survey number more than one time; and 4) revised procedures that require closer monitoring of survey administration by the grantees. As a result of these changes, we expect greatly increased compliance with the survey administration and an increase in the numbers of employers surveyed. As stated in Section 1 above, we will work with the grantees to explore changes to survey administration if needed.

2. Describe the procedures for the collection of information including:

* Statistical methodology for stratification and sample selection,

* Estimation procedure,

* Degree of accuracy needed for the purpose described in the justification,

* Unusual problems requiring specialized sampling procedures, and

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

A random sample is drawn annually from all participants and host agencies that were active at any time during the prior 12 months. The data come from the data collection and reporting system.

A. Design Parameters:

• There are 19 national grantees operating in 47 states, DC, and Puerto Rico; one national grantee has two separate grants and thus two separate samples

• There are 52 state grantees, including DC and Puerto Rico

• There are three customer groups to be surveyed (participants, host agencies, and employers)

• Surveying each of these customer groups should be considered a separate survey effort.

The major difference between the three groups is the number of participants served in any given year by the national, state, and territorial grantees. Using the sampling from PY 2015, 15 of the 16 national grantees funded at that time had sufficient participants to provide a sample of 370. Only 8 state grantees had sufficient participants to be able to provide a sample of 370. In regard to the host agency sampling, 8 national grantees had sufficient numbers of host agencies to allow a sample of 370. None of the state grantees had a sufficient number of host agencies to permit sampling. Where sampling is not possible, all participants or host agencies are surveyed. None of the territories had sufficient numbers for the participant or host agency sampling. . American Samoa, Guam, the Northern Mariana Islands and the Virgin Islands did not participate in the surveys due to the inability to conduct time-sensitive mail surveys overseas. The methodology requires tight turnaround times for three waves of surveys.

• Participant Surveys:

  • A point estimate for the ACSI score is required for each national grantee, both in the aggregate and for each state in which the national grantee is operating. The calculations of the ACSI are made using the formulas presented in Section 4, page 13-14, near the end of this document.

  • A point estimate for the ACSI score is required for each state grantee.

  • A sample of 370 participants from each national and state grantee will be drawn from the pool of participants who are currently active or have exited the program during 12 months prior to the survey period.

  • Some state grantees may not have a total of 370 participants available to be surveyed. In those cases, all participants who are active or who have exited during the 12 months prior to the survey will be surveyed.

  • As indicated above, 370 participants will be sampled from each national grantee. With an expected response rate of 60 percent, this should yield 222 usable responses. However, a single grantee sample may not be distributed equally across the states in which a national grantee operates. We, therefore, aim for a sample of 70 in each state, with a potential of 42 responses. Where there are fewer than 70 potential respondents in the sample for a state, we select all participants. If the overall sample for a national grantee is less than 370 and there are additional participants in some states who have not been sampled, we will over-sample to bring the sample to 370 and the potential responses to at least 222.

  • To determine the impact on the standard deviation of the ACSI for differing sample sizes, a series of samples was drawn from existing participant data. The samples were drawn using the random sampling function within SPSS Version 19 that uses random numbers table to draw each sample. The function allows the analyst to choose the number of records to be sampled for each run. The average standard deviation for samples of 250 was 20.6 in PY2013. The average standard deviation for samples of 50 was 18.6 for the same year. The analysis of standard deviations was used to provide greater confidence that grantees with smaller numbers of participants or host agencies would not have significantly different variability.

• Host Agency Surveys:

  • A point estimate for the ACSI score is required for each national grantee, both in the aggregate and for each state in which the national grantee is operating.

  • A point estimate for the ACSI score is required for each state grantee.

  • A sample of 370 host agency contacts from each national and state grantee will be drawn from the pool of agencies hosting participants during 12 months prior to the survey period.

  • Most state grantees do not have a total of 370 host agency contacts available to be surveyed. In those cases, all agencies hosting participants during the 12 months prior to the survey will be surveyed.

  • As indicated above, 370 host agencies will be sampled from each national grantee. With an expected response rate of 60 percent, this should yield 222 usable responses. However, a single grantee sample may not be distributed equally across the states in which a national grantee operates. We, therefore aim for a sample of 70 in each state, with a potential of 42 responses. Where there are fewer than 70 potential respondents in the sample for a state, we select all host agencies. If the overall sample for a national grantees is less than 370 and there are additional host agencies in some states that have not been sampled, we will over-sample in those states to bring the sample to 370 and the potential responses to at least 222. To determine the impact of different sample sizes on standard deviations, a series of samples was drawn from existing host agency data. The samples were drawn using the random sampling function (Data, Select Cases, Random sample of Cases) within SPSS Version 19. The procedure for 250 and 50 were each run several times, since the statistician observed that different random runs of each quantity of records yielded slightly different standard deviations. The standard deviations of the different runs of each sample size were than averaged. The average standard deviation for samples of 250 was 20.8 for PY2013. The average standard deviation for samples of 50 was 22.8 for the same year. The standard deviation formula used in SPSS is:

The reason that the number surveyed is as high as it is stems from the fact that the sampling for grantees with large number of participants and host agencies is a two stage sample. The first stage seeks a total grantee sample of 370. The second stage seeks to ensure that, where possible, 70 participants or host agencies from each state served by a national grantee are included in order to provide data from all states served without unduly increasing the overall sample size. Since much of our customers’ experience is determined by local management of the program, surveying each state captures that dimension of the customer experience. The oversampling at this second stage also contributes to large numbers being surveyed among those national grantees that serve participants in many different states. National grantees currently operate in as few as one state and as many as 14 states.

We routinely conduct analyses to determine if any of the standard demographic variables have a significant influence on the participants’ responses. This weighting is, first of all, taking place in a setting in which there are random samples taken; for the smaller grantees where there is no sampling, the total population of participants eligible for surveying is used. Although there is some evidence of influence from gender, the only consistent factor influencing the overall satisfaction score has been education.

For the employer surveys, all qualified employers are surveyed because the number of qualified employers is relatively low. Employers are only surveyed if they did not also serve as host agencies and if the grantee was involved in making the placement and the employer was aware of the grantee’s involvement. Employers are only surveyed for the first hire they make in each 12 month period. The number of qualified employers in any given program year is estimated to be approximately 1650. In 2013, there were 552 returned employer surveys; from March 2016-March 2017, there were 271. It was not possible to calculate a response rate for either period because of grantee non-compliance with the procedures for survey administration. Revised procedures, an enhanced management report, and new edits to the data collection system are expected to improve compliance with survey administration and result in an increase in employers surveyed.

• Employer Surveys:

  • All qualified employers are surveyed. There is no sampling. To be considered eligible for the survey, the employer: 1) must not have served as a host agency in the past 12 months; and 2) must have had substantial contact with the sub-grantee in connection with hiring of the participant; and 3) must not have received another survey from this program during the current program year.

  • Qualified employers are surveyed following the first time they hire a participant in the program year. The surveys are delivered by the sub-grantees at the time the sub-grantees conduct the first case management follow-up, which typically occurs 30-45 days after the date of hire.

  • Once an employer has been surveyed, the employer will only be surveyed again when it has another placement and at least one year has passed since the last survey.

The statute and regulations include the surveys of all three customer groups as a core measure of performance for SCSEP that must be negotiated with the grantees and reported each year. Survey procedures ensure that no customer is surveyed more than once each year. Less frequent data collection would require a change in the law.

B. Degree of Accuracy Required

The 2000 amendments to the OAA designated customer satisfaction as one of the core SCSEP measures for which each grantee had negotiated goals and for which sanctions could be applied. In the first year of the surveys, PY 2004, baseline data were collected. The following year, PY 2005, was the first year when evaluation and sanctions were possible. Because of changes made by the 2006 amendments to the OAA, starting with PY 2007, the customer satisfaction measures became additional measures (rather than core measures), for which there are no goals and, hence, no sanctions. With the OAA amendments of 2016, the surveys are used as indicators of effective in serving employers, host agencies, and participants and are again core measures for which goals must be established each year.

As set forth on page 4, where the population of participants or host agencies for a particular grantee is less than 350, the whole population is surveyed. Where the population of participants or host agencies is greater than 350, a random sample is taken. The sample size of 350 per grantee and the projected 60 percent response rate, yields 222 useable responses for the ACSI (participants or host agencies are only considered responsive if they return the survey and all three questions that make up the ACSI are completed). The 222-respondent target is important for at least two reasons First, the scores obtained for any random sample are assumed to reflect the score if one surveyed the whole population. To test for this, confidence intervals around the ACSI are regularly calculated to ensure that the target provides “an estimated range of values with a given high probability of covering the true population value” (Yays, 1988, p. 206). The confidence interval for the ACSI for the last survey report (PY2015) was 77.45 ± 3.51 for a random sample of 222 respondents with ACSI scores.

The second reason for the target of 222 is to have a sufficient number of cases for conducting multivariate analyses to better understand how the way the program delivers its services interacts with the measure of overall program quality (ACSI).

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

A. Maximizing Response Rates

• Participant and Host Agency Surveys:

The responses are obtained using a uniform mail methodology. The rationale for using mail surveys includes: individuals and organizations that have a substantial relationship with program operators, in this case, with the SCSEP sub-grantees, are highly likely to respond to a mail survey; and mail surveys are easily and reliably administered to potential respondents. The experience in administering the surveys by mail since 2004 has established the efficacy of this approach. Use of the Internet for conducting the host agency surveys has proven to be impracticable at this time because an insufficient number of host agency contact persons have reliable, unique email addresses and contact persons have been unresponsive to a URL provided to them in a mailed letter. The Department intends to explore ways in which Internet surveys can be more successful with this client group.

As with other data collected on the receipt of services, the responses to the customer satisfaction surveys must be held private as required by applicable state law. Before promising respondents privacy of results, grantees must ensure that they have legal authority under state law for that promise.

The survey procedures set forth below include a set of contact strategies that have been shown to boost response rates. First, participants receive up to five contacts if needed, starting with oral notification by sub-grantee personnel that the survey will be coming in the mail soon Second, participants included in the sample receive a personalized letter signed by the sub-grantee director informing them that they will receive a survey within a week and urging them to complete the survey and mail it back in the postage-paid envelope that will be included. Third, potential respondents receive personalized letters from the grantee director with the survey instrument and return envelope. Fourth, a second personalized letter accompanies the second mailing of the survey (if the potential respondent’s first survey is not received within four weeks). Fifth, another personalized letter with the third survey mailing is sent if required. Host agency contact persons receive all but the pre-survey letter because testing has established that the pre-survey letter does not significantly increase the response rate for host agencies. This contact strategy follows closely the strategies recommended by Dillman, Smyth, and Christian (2014)

SCSEP has also experimented with three different legends on the mailing envelope that were designed to encourage recipients to open the envelope and complete the survey. These legends were also found not to increase response rates. The Department will continue to follow the research on survey administration and explore features that can be added to the current survey administration to enhance response rates.

To ensure ACSI results are collected in a consistent and uniform manner, the following standard procedures are used by grantees to obtain participant and host agency customer satisfaction information:

• ETA’s survey research contractor, The Charter Oak Group, determines the samples based on data in the SCSEP Performance and Results QPR (SPARQ) system. There are smaller grantees where 370 potential respondents will not be achievable. In such cases, no sampling takes place and the entire population is surveyed. See the Design Parameters in Section 2 above for details.

• Grantees are required to ensure that sub-grantees notify customers of the customer satisfaction survey and the potential for being selected for the survey. Sub-grantees are required to:

• Inform participants at the time of enrollment and exit.

• Inform host agencies at the time of assignment of a participant.

• Mail a customized version of a standard letter prepared by the Charter Oak Group to all participants selected for the survey informing them that they will be receiving a survey in approximately one week.

• When discussing the surveys with participants for any of the above reasons, refresh contact information, including mailing address.

• Grantees are required to ensure that sub-grantees prepare and send pre-survey letters to those participants selected for the survey.

• Grantees provide the participant sample list to sub-grantees about 3 weeks prior to the date of the mailing of the surveys.

• Letters are personalized using a mail merge function and a standard text.

• Each letter is printed on the sub-grantee’s letterhead and signed in blue ink by the sub-grantee’s director to provide the appearance of a personal signature.

Grantees are responsible for the following activities:

• Provide letterhead, signatures, and correct return address information to DOL for use in the survey cover letters and mailing envelopes.

• Send participant samples to sub-grantees with instructions on preparing and mailing pre-survey letters.

Contractors to the Department of Labor are responsible for the following activities:

• Provide sub-grantees with list of participants to receive pre-survey letters.

• Print personalized cover letters for first mailing of survey. Each letter is printed on the grantee’s letterhead and signed in blue ink with the signatory’s electronic signature.

• Generate mailing envelopes with appropriate grantee return addresses.

• Generate survey instruments with bar codes and preprinted survey numbers.

• Enter preprinted survey numbers for each customer into worksheet.

• Assemble survey mailing packets: cover letter, survey, pre-paid reply envelope, and stamped mailing envelope.

• Mail surveys on designated day. Enter date of mailing into worksheet.

• Send survey worksheet to the Charter Oak Group.

• From list of customers who responded to first mailing, generate list for second mailing.

• Print second cover letter with standard text (different text from the first letter). Letters are personalized as in the first mailing.

• Enter preprinted survey number into worksheet for each customer to receive second mailing.

• Assemble second mailing packets: cover letter, survey, pre-paid reply envelope, and stamped mailing envelope.

• Mail surveys on designated day. Enter date of mailing into worksheet.

• Send survey worksheet to the Charter Oak Group.

• Repeat tasks above if third mailing is required.

• Employer surveys:

  • Grantees are required to ensure that sub-grantees notify employers of the customer satisfaction survey and the potential for being selected for the survey. Employers should be informed at the time of placement of the participant.

• Grantees and sub-grantees are responsible for the following activities:

• Sub-grantee uses the new management report to identify an employer for surveying the first time there is a placement with that employer in the program year. Employer is selected only if it is not also a host agency and the sub-grantee has had substantial communication with the employer in connection with the placement. Each employer is surveyed only once each year. The management report in the reporting system provides grantees and sub-grantees a list of all qualified employers that should receive the survey.

• Sub-grantee generates customized cover letter using standard text.

• Sub-grantee hand-delivers survey packet (cover letter, survey, stamped reply envelope) to employer contact in person at time of first case management follow-up. Mail may be used if in-person delivery is not practical.

• Sub-grantee enters pre-printed survey number and date of delivering packet into the system. (Survey instruments with pre-printed bar codes and survey numbers are provided to the grantees by DOL.)

• A contractor, responsible for processing the surveys, sends weekly e-mail to all grantees and sub-grantees listing the survey numbers of all employer surveys that have been completed.

• Sub-grantee reviews e-mails for three weeks following the delivery of the survey to determine if survey was completed.

• If the contractor lists a survey number in the weekly email, no further action is required. If survey not received, sub-grantee calls employer contact and uses a standard script to either encourage completion of the first survey (if the employer still has it) or indicate that the sub-grantee will send another survey for completion.

• If needed, sub-grantee generates second cover letter using same procedures as for first cover letter.

• Sub-grantee follows procedures as for first survey.

• If survey received, no further action taken; if third mailing needed, sub-grantee repeats steps above.

• Grantee monitors process to make sure that all appropriate steps have been followed and to advise sub-grantee if third effort at obtaining completed survey is required.

B. Nonresponse Bias

The potential problem of missing data was a concern; even with the relatively high response rates presented in Section 1, non-response bias still must be addressed. A study of non-response bias and the impact of missing data was conducted at the University of Connecticut Statistics Department in 2013-2014.

Missing data are carefully controlled in regard to the ACSI. The index is composed of three separate questions. However, the ACSI score is not calculated unless valid responses are recorded to all three questions (the scale is from 1-10 and “Don’t know.” Only if the respondent records one of the 1-10 scores on all three questions is the ACSI score complete.

The objective was to obtain bias-adjusted point estimates of:

1. Overall ACSI score

2. ACSI score for each national grantee

3. ACSI score for each state grantee

The author of the study used two different modeling approaches to determine the degree to which nonresponse biased the ACSI results. Below are the predicted nationwide ACSI scores by each of the models and without any adjustment:

• No adjustment: 82.40

• Ordinal logistic adjustment: 85.58

• Multinomial logistic adjustment: 84.73

Although the ordinal and multinomial logistic adjustments use different methodologies for estimating non-response bias, their estimates are within three-quarters of a point of each other. While they are both regression models, the major distinction is that the multinomial method assumes independence of choices and the ordinal method does not. As the statistician notes in his report: “Both of the regression models considered have assumptions that are difficult to validate in practice. However, the good news is that since the primary objective is score prediction (and not interpretation of regression coefficients), the models considered are quite robust to departures from ideal conditions.”

The fact that the scores change very little indicates that the non-respondents’ satisfaction is very similar to that of those who responded to the survey.¹ Adjusting for nonresponse would slightly increase the scores of some but not all grantees. Given the relatively minor impact of nonresponse, we consider the adjustments unnecessary.

In order to continue monitoring of the potential for nonresponse bias, we will annually compare respondents and non-respondents on variables that might differentially influence the ACSI score. In the past, we have noted that some demographic characteristics have not had an impact on ACSI scores for a number of years. For example, neither racial nor ethnic differences have had a significant impact on ACSI scores. Other characteristics, such as education, have had a significant impact on the ACSI score. We will test for those characteristics that have a significant impact, assess the potential for differences on those characteristics between respondents and non-respondents, and made adjustments where necessary.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

The core questions that yield the single required measure for each of the three surveys are from the ACSI and cannot be modified. The supplemental questions included in this submission were revised in 2015 for the first time since OMB approved them in 2004. No additional changes are requested with this submission; however, as stated in Section 1 above, we intend to investigate changes to existing questions or additional questions that may better capture effectiveness in serving SCSEP’s three customer groups, as well as the possibility of constructing an index of effectiveness to replace the ACSI. No changes will be implemented without OMB approval.

The ACSI model (including the weighting methodology) is well documented. (See http://www.theacsi.org/about-acsi/the-science-of-customer-satisfaction ) The ACSI scores represent the weighted sum of the three ACSI questions’ values, which are transformed into 0 to 100 scale value. The weights are applied to each of the three questions to account for differences in the characteristics of the state’s customer groups.

For example, assume the mean values of three ACSI questions for a state are:

1. Overall Satisfaction = 8.3

2. Met Expectations = 7.9

3. Compared to Ideal = 7.0

These mean values from raw data must first be transformed to the value on a 0 to 100 scale. This is done by subtracting 1 from these mean values, dividing the results by 9 (which is the value of range of a 1 to 10 raw data scale), and multiplying the whole by 100:

1. Overall Satisfaction = (8.3 -1)/9 x 100 = 81.1

2. Met Expectations = (7.9 -1)/9 x 100 = 76.7

3. Compared to Ideal = (7.0 -1)/9 x 100 = 66.7

The ACSI score is calculated as the weighted averages of these values. Assuming the weights for the example state are 0.3804, 0.3247 and 0.2949 for questions 1, 2 and 3, respectively, the ACSI score for the state would be calculated as follows:

(0.3804 x 81.1) + (0.3247 x 76.7) + (0.2949 x 66.7) = 75.4

Weights are calculated by a statistical algorithm to minimize measurement error or random survey noise that exists in all survey data. State-specific weights are calculated using the relative distribution of ACSI respondent data for non-regulatory Federal agencies previously collected and analyzed by CFI and the University of Michigan.

Specific weighting factors have been developed for each state. New weighting factors are published annually. It should be noted that the national grantees have different weights applied depending on the states in which their sub-grantees’ respondents are located.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The Charter Oak Group, LLC:

Barry A. Goff, Ph.D., (860) 659-8743, bgoff@charteroakgroup.com; Bennett Pudlin, J.D., (860) 324-3555, bpudlin@charteroakgroup.com

Barry Goff was consulted on the statistical aspects of the design; Barry Goff and Bennett Pudlin will collect the information.

Ved Deshpande, Department of Statistics, University of Connecticut, (860) 486-3414, ved.deshpande@uconn.edu, was also consulted on the statistical aspects of the design.

OIRA has produced a number of documents that may serve as useful reference material for completing Supporting Statement B. These can be found at:

http://www.whitehouse.gov/omb/inforeg_statpolicy/

1 Ved Deshpande, Department of Statistics, University of Connecticut, “Bias-adjusted Modeling of ACSI scores for SCSEP” (2014)

16