1205-0040 Supporting Statement Part B Revised 100421

Senior Community Service Employment Program (SCSEP)

OMB Control No. 1205-0040

August 2021

Supporting Statement B, General Instructions:

Collections of Information Employing Statistical Methods

The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When the question “Does this ICR contain surveys, censuses or employ statistical methods” is checked, "Yes," the following documentation should be included in the Supporting Statement Part B to the extent that it applies to the methods proposed. Make a copy of this format, Supporting Statement Part B, 1-5, and provide your response below each numbered topic. This is a separate document from Part A since they are uploaded in different sections in ROCIS.

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

Title V of the Older Americans Act requires that Senior Community Service Employment Program (SCSEP) include in its core measures “Indicators of effectiveness in serving employers, host agencies, and project participants.’’ In its Final Rule published on June 30, 2018, the Department defined this new core measure as “the combined results of customer assessments of the services received by each of these three customer groups.” (See section 641.710.) In the preamble to this section of the regulations, the Department stated that although WIOA was piloting a new measure of effectiveness in serving employers that measure does not have obvious application to SCSEP’s other two customer groups. As a result, the Department announced that it would continue to conduct the existing customer satisfaction surveys for each of its three customer groups “as an interim measure at least until the WIOA pilot is complete and a WIOA measure is defined in final form.” SCSEP has been conducting these three customer satisfaction surveys nationwide since 2004.

The interim measure adopted by SCSEP to satisfy the statutory requirement is the American Customer Satisfaction Index (ACSI). The survey approach employed allows the program flexibility and, at the same time, captures common customer satisfaction information that can be aggregated and compared among national and state grantees. The measure is created with a set of three core questions that form a customer satisfaction index. The index is created by combining scores from three specific questions that address different dimensions of customers' experience. Additional questions that do not affect the assessment of grantee performance are included to allow grantees to effectively manage the program.

The ACSI measures customer experience in 10 economic sectors and 47 key industries that together represent a broad swath of the national economy. (https://www.theacsi.org/national-economic-indicator/national-sector-and-industry-results) In addition, over 100 Federal government agencies and services have used the ACSI to measure citizen satisfaction. (https://www.theacsi.org/industries/government)

As stated in the preamble to section 641.710, during this interim period, the Department will explore with grantees, and with its three customer groups, options for best measuring the effectiveness of SCSEP’s services. The Department will also explore ways to improve the efficiency of the current customer surveys (including the use of online surveys and changes to the administration of the employer survey) and will examine what, if any, new or revised questions would support an index of effectiveness as an alternative to the current index of satisfaction. Based on feedback from its customers and its grantees, the Department is requesting approval of several non-material changes to the participant and host agency surveys. See Attachment X, Justification for Non-Material Change, for an explanation of the proposed non-material changes.

SCSEP currently funds 77 grantees, (21 national grantees (two of which have both general and set-aside grants) operating in 47 states, the District of Columbia (DC) and Puerto Rico; and 52 state grantees, including DC and Puerto Rico; the three overseas territories and USVI do not participate in the surveys):

NATIONAL GRANTEES

AARP Foundation

Asociación Nacional Pro Personas Mayores

Associates for Training and Development, Inc. (A4TD)

Center for Workforce Inclusion (formerly SSAI, Senior Services America, Inc.)

Easter Seals, Inc.

Goodwill Industries International, Inc.

Institute for Indian Development

International Pre-Diabetes Center Inc.

National Able Network

National Asian Pacific Center on Aging (General and Set-Aside)

National Caucus and Center on Black Aged, Inc.

National Council on Aging, Inc.

National Indian Council on Aging (General and Set-Aside)

National Older Worker Career Center

National Urban League

Operation A.B.L.E of Greater Boston, Inc.

SER – Jobs for Progress National, Inc.

Vantage Aging (formerly Mature Services)

The Workplace, Inc.

STATE GRANTEES

Alabama Department of Senior Services

Alaska Department of Labor and Workforce Development

Arizona Department of Economic Security

Arkansas Department of Human Services, Division of Aging and Adult Services

California Department of Aging

Colorado Department of Human Services, Aging and Adult Services

Connecticut Department of Aging and Disability Services

Delaware Division of Services for Aging and Adults with Physical Disabilities

District of Columbia Department of Employment Services

Florida Department of Elder Affairs

Georgia Department of Human Services

Hawaii Department of Labor And Industrial Relations

Idaho Commission on Aging

Illinois Department on Aging

Indiana Department of Workforce Development

Iowa Department on Aging

Kansas Department of Commerce

Kentucky Cabinet for Health and Family Services

Louisiana Governor's Office of Elderly Affairs

Maine Department of Health and Human Services

Maryland Department of Labor, Workforce Development & Adult Learning

Massachusetts Executive Office of Elder Affairs

Michigan Aging and Adult Services Agency

Minnesota Department of Employment and Economic Development

Mississippi Department of Employment Security

Missouri Department of Health and Senior Services

Montana Department of Labor and Industry

Nebraska Department of Labor

Nevada Aging and Disabilities Services Division

New Hampshire Business and Economic Affairs

New Jersey Department of Labor and Workforce Development

New Mexico Aging and Long-Term Services Department

New York State Office for the Aging

North Carolina Dept. of Health and Human Services, Div. of Aging and Adult Services

North Dakota Department of Human Services

Ohio Department of Aging

Oklahoma Department of Human Services

Oregon Department of Human Services, Aging and People with Disabilities

Pennsylvania Department of Aging, Bureau of Aging Services

Puerto Rico Department of Labor & Human Resources

Rhode Island Department Labor and Training

South Carolina Department on Aging

South Dakota Department of Labor and Regulation

Tennessee Department of Labor And Workforce Development

Texas Workforce Commission

Utah Department of Human Services

Vermont Department of Disabilities, Aging and Independent Living

Virginia Department for Aging and Rehabilitative Services

Washington Department of Social and Health Services, State Unit on Aging

West Virginia Bureau of Senior Services

Wisconsin Department of Health Services, Bureau of Aging and Disability Resources

Wyoming Department of Workforce Services

A. Participant and Host Agency Surveys

1. Who is Surveyed

Participants - All SCSEP participants who are active at the time of the survey or have been active in the previous 12 months are eligible to be chosen for inclusion in the random sample of records. In the survey for PY 2019, there were approximately 60,000 participants eligible for the survey, of whom 19,249 were selected for the participant sample.

Host Agency Contacts - Host agencies are public agencies, units of government, and non-profit agencies that provide subsidized employment, training, and related services to SCSEP participants. All host agencies that are active at the time of the survey or that have been active in the preceding 12 months are eligible for inclusion in the sample of records. In the survey, for PY 2019, there were approximately 21,784 host agencies eligible for the survey, of which 13,482 were selected for the host agency sample.

2. Sample Size and Procedures

The national and state grantees have parallel, but complementary procedures. The national grantees have a target of 370 at the first stage of sampling and, at the second stage, a target of 70 for each state in which the grantee provides services. The state grantees have only one stage to their sampling procedure. It is anticipated that a sample of 370 will yield 185 completed interviews at a 50 percent response rate. Following the sampling procedure, for each state grantee, 185 completed surveys should be obtained each year for both participants and host agencies. At least 185 completed surveys for both customer groups will be obtained for each national grantee, depending on the number of states in which each national grantee is operating. For all but the smallest national grantees, the number of completed surveys exceeds 185. In cases where the number eligible for the survey is small and 185 completed interviews are not attainable (a situation common among state grantees), the sample is actually the population of all participants or host agencies. The surveys of participants and host agencies are conducted through a mail house once each program year. The target of 185 completed surveys produces a sufficiently narrow confidence interval to provide an accurate estimate of the population values for the ACSI. See table below for illustrative example.

	Population	Stage One Sample Target	Sample Size	Achieved Sample Size
National Grantee	2956	370 +	420	210
California	300		70	35
Colorado	600		70	35
Kansas	550		70	35
Missouri	350		70	35
Texas	451		70	35
Wisconsin	705		70	35
State Grantee Large	400	370	370	185
State Grantee Small	200	200	200	100

The actual number of participants sampled in PY 2019, the year of the last completed survey analysis, was 19,249. There were 9,693 returned surveys that had answers to all three ACSI questions, for a response rate of 50.4%. 13,482 host agencies were surveyed in PY 2019. There were 6,841 returned surveys, for a response rate of 50.7%. For both surveys, all participants and host agencies that were active within the 12 months prior to the drawing of the samples were eligible to be surveyed. See table below.

	Population	Sample Size	Percent Sampled	Achieved Sample Size	Response Rate
Participants	51,221	19,249	37.6%	9,693	50.4%
Host Agencies	21,784	13,482	61.9%	6,841	50.7%

The number of actual responses differs from the number expected by the sampling procedure because the large national grantees operate in many states and sampling 70 from each of those states yields far more than the minimum sample of 370 would yield. At the same time, most state grantees do not have sufficient participants or host agencies to permit sampling, so all participants and host agencies are surveyed.

3. Response Rates

Response rates achieved for the participant and host agency surveys since 2004 have ranged from 50 percent to 70 percent. Both participant and host agency response rates for PY 2019 are 50 percent.

(Note: There were no surveys for PY 2007, 2016, and 2020 due to the national grantee competition in those program years, and in PY 2011, there was no host agency survey.)

B. Employer Surveys

1. Who Is Surveyed

Employers that hire SCSEP participants and employ them in unsubsidized jobs are included in the Employer Survey. To be considered eligible for the survey, the employer: 1) must not have served as a host agency in the past 12 months; and 2) must have had substantial contact with the sub-grantee in connection with hiring of the participant; and 3) must not have received another survey from this program during the current program year.

All employers that meet the three criteria described above are surveyed at the time the sub-grantee conducts the first case management follow-up, which typically occurs 30-45 days after the date of placement.

2. Sample Size and Procedures

All employers meeting the three criteria described above will be surveyed. No sampling is used.

For the employer surveys, all qualified employers are surveyed because the number of qualified employers is relatively low. Employers qualify for the survey if they did not also serve as host agencies, if the grantee was directly involved in making the placement, and the employer was aware of the grantee’s involvement. Employers are only surveyed for the first hire they make in each 12 month period. The number of qualified employers in any given program year is estimated to be approximately 1500. That should yield 750 returned surveys. Although there are many more placements than 1500, roughly 30% of them are with host agencies, and most of the remainder were self-placements by the participant. In a 12-month period from March 2016-March 2017, there were 271 returned employer surveys. It was not possible to calculate a response rate because of grantee non-compliance with the procedures for survey administration. We have developed new procedures, an enhanced management report, and new edits to the data collection system to improve compliance with survey administration and increase the number of employers surveyed. As set forth below, however, these improvements have not proved adequate and additional changes are now being explored.

3. Response Rates

Response rates for the employer survey have been difficult to track because of grantee non-compliance with the requirement to enter the survey number and date of mailing into the system. Where the survey administration requirements have been followed, employer response rates have been very high. To improve compliance with the procedures, several additional changes were made to the administration of the employer survey. The changes include: 1) an enhanced management report that tells grantees when each qualified employer needs to be surveyed; 2) additional edits in the data collection system that warn grantees when they have failed to deliver surveys as required; 3) new edits that prevent the entry of inaccurate data, such as the entry of the same survey number more than one time; and 4) revised procedures that require closer monitoring of survey administration by the grantees. As a result of these changes, we expected greatly increased compliance with the survey administration and an increase in the numbers of employers surveyed. Unfortunately, these changes did not result in an increase in usable surveys for individual grantees. As a result, administration of the employer survey has been suspended. As stated in Section 1 above, we have been working with the grantees to explore changes to survey administration and will be submitting a later request to revise the employer survey and their administration based on grantee and customer input.

2. Describe the procedures for the collection of information including:

* Statistical methodology for stratification and sample selection,

* Estimation procedure,

* Degree of accuracy needed for the purpose described in the justification,

* Unusual problems requiring specialized sampling procedures, and

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

A random sample is drawn annually from all participants and host agencies that were active at any time during the prior 12 months. The data come from the data collection and reporting system.

A. Design Parameters:

• There are currently 21 national grantees operating in 47 states, DC, and Puerto Rico; two national grantees have two separate grants and thus two separate samples

• There are 52 state grantees, including DC and Puerto Rico

• There are three customer groups to be surveyed (participants, host agencies, and employers)

• Surveying each of these customer groups should be considered a separate survey effort.

The major difference between the three groups is the number of participants served in any given year by the national, state, and territorial grantees. Using the sampling from PY 2019, 18 of the 20 national grantees funded at that time had sufficient participants to provide a sample of 370. Only 7 state grantees had sufficient participants to be able to provide a sample of 370. In regard to the host agency sampling, 12 national grantees had sufficient numbers of host agencies to allow a sample of 370. Only one of the state grantees had a sufficient number of host agencies to permit sampling. Where sampling is not possible, all participants or host agencies are surveyed. American Samoa, Guam, the Northern Mariana Islands and the Virgin Islands did not participate in the surveys due to the inability to conduct time-sensitive mail surveys overseas. The methodology requires tight turnaround times for three waves of surveys.

• Participant Surveys:

  • A point estimate for the ACSI score is required for each national grantee, both in the aggregate and for each state in which the national grantee is operating. The calculations of the ACSI are made using the formulas presented in Section 4, pages 13-14, near the end of this document.

  • A point estimate for the ACSI score is required for each state grantee.

  • A sample of 370 participants from each national and state grantee will be drawn from the pool of participants who are currently active or have exited the program during 12 months prior to the survey period.

  • Some state grantees may not have a total of 370 participants available to be surveyed. In those cases, all participants who are active or who have exited during the 12 months prior to the survey will be surveyed.

  • As indicated above, 370 participants will be sampled from each national grantee. With an expected response rate of 50 percent, this should yield 185 usable responses. However, a single grantee sample may not be distributed equally across the states in which a national grantee operates. We, therefore, aim for a sample of 70 in each state, with a potential of 35 responses. Where there are fewer than 70 potential respondents in the sample for a state, we select all participants. If the overall sample for a national grantee is less than 370 and there are additional participants in some states who have not been sampled, we will over-sample to bring the sample to 370 and the potential responses to at least 185.

  • The returns from PY 2019 were analyzed using the random sampling function within SPSS to determine the impact on the standard deviation of the ACSI for differing number of returns. The function allows the analyst to choose the number of records to be sampled for each run. . The average standard deviation was calculated for grantees with 200 or fewer returns and compared to those grantees that had more than 200 returns. The average standard deviation for grantees with less than 200 was 23.6. The average standard deviation for participant grantee with returns of 200 or more was 22.7. The analysis of standard deviations was used to provide greater confidence that grantees with smaller numbers of participants or host agencies would not have significantly different variability.

• Host Agency Surveys:

  • A point estimate for the ACSI score is required for each national grantee, both in the aggregate and for each state in which the national grantee is operating.

  • A point estimate for the ACSI score is required for each state grantee.

  • A sample of 370 host agency contacts from each national and state grantee will be drawn from the pool of agencies hosting participants during 12 months prior to the survey period.

  • Most state grantees do not have a total of 370 host agency contacts available to be surveyed. In those cases, all agencies hosting participants during the 12 months prior to the survey will be surveyed.

  • As indicated above, 370 host agencies will be sampled from each national grantee. With an expected response rate of 50 percent, this should yield 185 usable responses. However, a single grantee sample may not be distributed equally across the states in which a national grantee operates. We, therefore, aim for a sample of 70 in each state, with a potential of 35 responses. Where there are fewer than 70 potential respondents in the sample for a state, we select all host agencies. If the overall sample for a national grantees is less than 370 and there are additional host agencies in some states that have not been sampled, we will over-sample in those states to bring the sample to 370 and the potential responses to at least 185. To determine the impact of different sample sizes on standard deviations, a series of samples was drawn from existing host agency data, the returns from PY 2019 were analyzed. The average standard deviation was calculated for grantees with 200 or fewer returns and compared to those grantees that had more than 200 returns. The average standard deviation for grantees of less than 200 was 19.0. The average standard deviation for those with returns of more than 200 was 19.2. The standard deviation formula used in SPSS is:

The reason that the number surveyed is as high as it is stems from the fact that the sampling for grantees with a large number of participants and host agencies is a two stage sample. The first stage seeks a total grantee sample of 370. The second stage seeks to ensure that, where possible, 70 participants or host agencies from each state served by a national grantee are included in order to provide data from all states served without unduly increasing the overall sample size. Since much of our customers’ experience is determined by local management of the program, surveying each state captures that dimension of the customer experience. The oversampling at this second stage also contributes to large numbers being surveyed among those national grantees that serve participants in many different states. National grantees currently operate in as few as one state and as many as 14 states.

We routinely conduct analyses to determine if any of the standard demographic variables have a significant influence on the participants’ responses. This weighting is, first of all, taking place in a setting in which there are random samples taken; for the smaller grantees where there is no sampling, the total population of participants eligible for surveying is used. Although there is some evidence of influence from gender, the only consistent factor influencing the overall satisfaction score has been education.

For the employer surveys, all qualified employers are surveyed because the number of qualified employers is relatively low. Employers are only surveyed if they did not also serve as host agencies and if the grantee was involved in making the placement and the employer was aware of the grantee’s involvement. Employers are only surveyed for the first hire they make in each 12 month period. The number of qualified employers in any given program year is estimated to be approximately 1500. In 2013, there were 552 returned employer surveys; from March 2016-March 2017, there were 271. It was not possible to calculate a response rate for either period because of grantee non-compliance with the procedures for survey administration. As set forth above, revised procedures, an enhanced management report, and new edits to the data collection system did not yield the expected compliance with survey administration and did not result in an increase in employers surveyed. As a consequence, the Department is in the process of revising the employer surveys and their administration and will be submitting a future request to OMB for approval of the changes. For now, we are seeking continued approval of the existing employer survey.

• Employer Surveys:

  • All qualified employers are surveyed. There is no sampling. To be considered eligible for the survey, the employer: 1) must not have served as a host agency in the past 12 months; and 2) must have had substantial contact with the sub-grantee in connection with hiring of the participant; and 3) must not have received another survey from this program during the current program year.

  • Qualified employers are surveyed following the first time they hire a participant in the program year. The surveys are delivered by the sub-grantees at the time the sub-grantees conduct the first case management follow-up, which typically occurs 30-45 days after the date of hire.

  • Once an employer has been surveyed, the employer will only be surveyed again when it has another placement and at least one year has passed since the last survey.

The statute and regulations include the surveys of all three customer groups as a core measure of performance for SCSEP that must be negotiated with the grantees and reported each year. Survey procedures ensure that no customer is surveyed more than once each year. Less frequent data collection would require a change in the law.

B. Degree of Accuracy Required

The 2000 amendments to the OAA designated customer satisfaction as one of the core SCSEP measures for which each grantee had negotiated goals and for which sanctions could be applied. In the first year of the surveys, PY 2004, baseline data were collected. The following year, PY 2005, was the first year when evaluation and sanctions were possible. Because of changes made by the 2006 amendments to the OAA, starting with PY 2007, the customer satisfaction measures became additional measures (rather than core measures), for which there are no goals and, hence, no sanctions. With the OAA amendments of 2016, the surveys are used as indicators of effective in serving employers, host agencies, and participants and are again core measures for which goals must be established each year.

As set forth on page 4, where the population of participants or host agencies for a particular grantee is less than 350, the whole population is surveyed. Where the population of participants or host agencies is greater than 350, a random sample is taken. The sample size of 350 per grantee and the projected 50 percent response rate, yields 185 useable responses for the ACSI (participants or host agencies are only considered responsive if they return the survey and all three questions that make up the ACSI are completed). The 185-respondent target is important for at least two reasons. First, the scores obtained for any random sample are assumed to reflect the score if one surveyed the whole population. To test for this, confidence intervals around the ACSI are regularly calculated to ensure that the target provides “an estimated range of values with a given high probability of covering the true population value” (Yays, 1988, p. 206). The confidence interval for the ACSI for the last participant survey report (PY 2019) was 83.2± 1.7 for a random sample of 185 respondents with ACSI scores.

The second reason for the target of 185 is to have a sufficient number of cases for conducting multivariate analyses to better understand how the way the program delivers its services interacts with the measure of overall program quality (ACSI).

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

A. Maximizing Response Rates

• Participant and Host Agency Surveys:

The responses are obtained using a uniform mail methodology. The rationale for using mail surveys includes: individuals and organizations that have a substantial relationship with program operators, in this case, with the SCSEP sub-grantees, are highly likely to respond to a mail survey; and mail surveys are easily and reliably administered to potential respondents. The experience in administering the surveys by mail since 2004 has established the efficacy of this approach. However, in order to obtain higher response rates, ease the burden on respondents, and reduce cost, the Department is exploring ways in which Internet surveys can be more successful with these client groups.

As with other data collected on the receipt of services, the responses to the customer satisfaction surveys must be held private as required by applicable state law. Before promising respondents privacy of results, grantees must ensure that they have legal authority under state law for that promise.

The survey procedures set forth below include a set of contact strategies that have been shown to boost response rates. First, participants receive up to five contacts if needed, starting with oral notification by sub-grantee personnel that the survey will be coming in the mail soon. Second, participants included in the sample receive a personalized letter signed by the sub-grantee director informing them that they will receive a survey within a week and urging them to complete the survey and mail it back in the postage-paid envelope that will be included. Third, potential respondents receive personalized letters from the grantee director with the survey instrument and return envelope. Fourth, a second personalized letter accompanies the second mailing of the survey (if the potential respondent’s first survey is not received within four weeks). Fifth, another personalized letter with the third survey mailing is sent if required. Host agency contact persons receive all but the pre-survey letter because testing has established that the pre-survey letter does not significantly increase the response rate for host agencies. This contact strategy follows closely the strategies recommended by Dillman, Smyth, and Christian (2014).

SCSEP has also experimented with three different legends on the mailing envelope that were designed to encourage recipients to open the envelope and complete the survey. These legends were also found not to increase response rates. The Department will continue to follow the research on survey administration and explore features that can be added to the current survey administration to enhance response rates.

To ensure ACSI results are collected in a consistent and uniform manner, the following standard procedures are used by grantees to obtain participant and host agency customer satisfaction information:

• ETA’s survey research contractor, The Charter Oak Group, determines the samples based on data in the SCSEP Performance and Results QPR (SPARQ) system. There are smaller grantees where 370 potential respondents will not be achievable. In such cases, no sampling takes place, and the entire population is surveyed. See the Design Parameters in Section 2 above for details.

• Grantees are required to ensure that sub-grantees notify customers of the customer satisfaction survey and the potential for being selected for the survey. Sub-grantees are required to:

• Inform participants at the time of enrollment and exit.

• Inform host agencies at the time of assignment of a participant.

• Mail a customized version of a standard letter prepared by the Charter Oak Group to all participants selected for the survey informing them that they will be receiving a survey in approximately one week.

• When discussing the surveys with participants for any of the above reasons, refresh contact information, including mailing address.

• Grantees are required to ensure that sub-grantees prepare and send pre-survey letters to those participants selected for the survey.

• Grantees provide the participant sample list to sub-grantees about 3 weeks prior to the date of the mailing of the surveys.

• Letters are personalized using a mail merge function and a standard text.

• Each letter is printed on the sub-grantee’s letterhead and signed in blue ink by the sub-grantee’s director to provide the appearance of a personal signature.

Grantees are responsible for the following activities:

• Provide letterhead, signatures, and correct return address information to DOL for use in the survey cover letters and mailing envelopes.

• Send participant samples to sub-grantees with instructions on preparing and mailing pre-survey letters.

Contractors to the Department of Labor are responsible for the following activities:

• Provide sub-grantees with list of participants to receive pre-survey letters.

• Print personalized cover letters for first mailing of survey. Each letter is printed on the grantee’s letterhead and signed in blue ink with the signatory’s electronic signature.

• Generate mailing envelopes with appropriate grantee return addresses.

• Generate survey instruments with bar codes and preprinted survey numbers.

• Enter preprinted survey numbers for each customer into worksheet.

• Assemble survey mailing packets: cover letter, survey, pre-paid reply envelope, and stamped mailing envelope.

• Mail surveys on designated day. Enter date of mailing into worksheet.

• Send survey worksheet to the Charter Oak Group.

• From list of customers who responded to first mailing, generate list for second mailing.

• Print second cover letter with standard text (different text from the first letter). Letters are personalized as in the first mailing.

• Enter preprinted survey number into worksheet for each customer to receive second mailing.

• Assemble second mailing packets: cover letter, survey, pre-paid reply envelope, and stamped mailing envelope.

• Mail surveys on designated day. Enter date of mailing into worksheet.

• Send survey worksheet to the Charter Oak Group.

• Repeat tasks above if third mailing is required.

• Employer surveys:

  • Grantees are required to ensure that sub-grantees notify employers of the customer satisfaction survey and the potential for being selected for the survey. Employers should be informed at the time of placement of the participant.

• Grantees and sub-grantees are responsible for the following activities:

• Sub-grantee uses the new management report to identify an employer for surveying the first time there is a placement with that employer in the program year. Employer is selected only if it is not also a host agency and the sub-grantee has had substantial communication with the employer in connection with the placement. Each employer is surveyed only once each year. The management report in the reporting system provides grantees and sub-grantees a list of all qualified employers that should receive the survey.

• Sub-grantee generates customized cover letter using standard text.

• Sub-grantee hand-delivers survey packet (cover letter, survey, stamped reply envelope) to employer contact in person at time of first case management follow-up. Mail may be used if in-person delivery is not practical.

• Sub-grantee enters pre-printed survey number and date of delivering packet into the system. (Survey instruments with pre-printed bar codes and survey numbers are provided to the grantees by DOL.)

• A contractor, responsible for processing the surveys, sends weekly e-mail to all grantees and sub-grantees listing the survey numbers of all employer surveys that have been completed.

• Sub-grantee reviews e-mails for three weeks following the delivery of the survey to determine if survey was completed.

• If the contractor lists a survey number in the weekly email, no further action is required. If survey not received, sub-grantee calls employer contact and uses a standard script to either encourage completion of the first survey (if the employer still has it) or indicate that the sub-grantee will send another survey for completion.

• If needed, sub-grantee generates second cover letter using same procedures as for first cover letter.

• Sub-grantee follows procedures as for first survey.

• If survey received, no further action taken; if third mailing needed, sub-grantee repeats steps above.

• Grantee monitors process to make sure that all appropriate steps have been followed and to advise sub-grantee if third effort at obtaining completed survey is required.

B. Nonresponse Bias

The potential problem of missing data was a concern; even with the relatively high response rates presented in Section 1, non-response bias still must be addressed. A study of non-response bias and the impact of missing data was conducted at the University of Connecticut Statistics Department in 2013-2014.

Missing data are carefully controlled in regard to the ACSI. The index is composed of three separate questions. However, the ACSI score is not calculated unless valid responses are recorded to all three questions (the scale is from 1-10 and “Don’t know.” Only if the respondent records one of the 1-10 scores on all three questions is the ACSI score complete.

The objective was to obtain bias-adjusted point estimates of:

1. Overall ACSI score

2. ACSI score for each national grantee

3. ACSI score for each state grantee

The author of the study used two different modeling approaches to determine the degree to which nonresponse biased the ACSI results. Below are the predicted nationwide ACSI scores by each of the models and without any adjustment:

• No adjustment: 82.40

• Ordinal logistic adjustment: 85.58

• Multinomial logistic adjustment: 84.73

Although the ordinal and multinomial logistic adjustments use different methodologies for estimating non-response bias, their estimates are within three-quarters of a point of each other. While they are both regression models, the major distinction is that the multinomial method assumes independence of choices and the ordinal method does not. As the statistician notes in his report: “Both of the regression models considered have assumptions that are difficult to validate in practice. However, the good news is that since the primary objective is score prediction (and not interpretation of regression coefficients), the models considered are quite robust to departures from ideal conditions.”

The fact that the scores change very little indicates that the non-respondents’ satisfaction is very similar to that of those who responded to the survey.¹ Adjusting for nonresponse would slightly increase the scores of some but not all grantees. Given the relatively minor impact of nonresponse, we consider the adjustments unnecessary.

In order to continue monitoring of the potential for nonresponse bias, we will annually compare respondents and non-respondents on variables that might differentially influence the ACSI score. In the past, we have noted that some demographic characteristics have not had an impact on ACSI scores for a number of years. For example, neither racial nor ethnic differences have had a significant impact on ACSI scores. Other characteristics, such as education, have had a significant impact on the ACSI score. We will test for those characteristics that have a significant impact, assess the potential for differences on those characteristics between respondents and non-respondents, and make adjustments where necessary.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

The core questions that yield the single required measure for each of the three surveys are from the ACSI and cannot be modified. The supplemental questions included in this submission were revised in 2015 for the first time since OMB approved them in 2004. As set forth above, based on grantee and customer input we are requesting a few non-material changes to the participant and host agency surveys. We are also exploring the use of digital surveys for these customer groups.

The ACSI model (including the weighting methodology) is well documented. (See http://www.theacsi.org/about-acsi/the-science-of-customer-satisfaction ) The ACSI scores represent the weighted sum of the three ACSI questions’ values, which are transformed into 0 to 100 scale value. The weights are applied to each of the three questions to account for differences in the characteristics of the state’s customer groups.

For example, assume the mean values of three ACSI questions for a state are:

1. Overall Satisfaction = 8.3

2. Met Expectations = 7.9

3. Compared to Ideal = 7.0

These mean values from raw data must first be transformed to the value on a 0 to 100 scale. This is done by subtracting 1 from these mean values, dividing the results by 9 (which is the value of range of a 1 to 10 raw data scale), and multiplying the whole by 100:

1. Overall Satisfaction = (8.3 -1)/9 x 100 = 81.1

2. Met Expectations = (7.9 -1)/9 x 100 = 76.7

3. Compared to Ideal = (7.0 -1)/9 x 100 = 66.7

The ACSI score is calculated as the weighted averages of these values. Assuming the weights for the example state are 0.3804, 0.3247 and 0.2949 for questions 1, 2 and 3, respectively, the ACSI score for the state would be calculated as follows:

(0.3804 x 81.1) + (0.3247 x 76.7) + (0.2949 x 66.7) = 75.4

Weights are calculated by a statistical algorithm to minimize measurement error or random survey noise that exists in all survey data. State-specific weights are calculated using the relative distribution of ACSI respondent data for non-regulatory Federal agencies previously collected and analyzed by CFI and the University of Michigan.

Specific weighting factors have been developed for each state. New weighting factors are published annually. It should be noted that the national grantees have different weights applied depending on the states in which their sub-grantees’ respondents are located.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The Charter Oak Group, LLC:

Barry A. Goff, Ph.D., (860) 659-8743, bgoff@charteroakgroup.com; Bennett Pudlin, J.D., (860) 324-3555, bpudlin@charteroakgroup.com

Barry Goff was consulted on the statistical aspects of the design; Barry Goff and Bennett Pudlin will collect the information.

Ved Deshpande, Department of Statistics, University of Connecticut, (860) 486-3414, ved.deshpande@uconn.edu, was also consulted on the statistical aspects of the design.

OIRA has produced a number of documents that may serve as useful reference material for completing Supporting Statement B. These can be found at:

http://www.whitehouse.gov/omb/inforeg_statpolicy/

1 Ved Deshpande, Department of Statistics, University of Connecticut, “Bias-adjusted Modeling of ACSI scores for SCSEP” (2014)

16