Summary of Comments Received

Summary of Comments Received

in Response to February 8, 2007 Federal Register Notice

“Implementation of Senior Community Service Employment Program Performance Measures under Public Law 109-365”

The 2006 Amendments to the Older Americans Act (OAA-2006) were enacted on October 17, 2006. Title V of OAA-2006 reauthorizes the Senior Community Service Employment Program (SCSEP). The law calls for the Department of Labor (DOL) to establish and implement new SCSEP measures of performance by Program Year 2007 after consultation with stakeholders. Therefore, DOL published a notice in the Federal Register which solicited public input on the definitions and implementation of the performance indicators. (Volume 72, Number 26, pages 5999-6001).

The notice specifically requested input on six topics:

1) Core indicators;

2) Retention in unsubsidized employment for one year;

3) Customer satisfaction;

4) Other additional indicators of performance, including possibly retaining the SCSEP placement measure;

5) Performance outcomes; and

6) Other comments related to SCSEP performance measures.

The stakeholder input provided in response to the notice informs the DOL as it develops its policies and regulations for the SCSEP performance indicators, which were published through an Interim Final Rule (IFR). The IFR incorporates many of these comments, and specifies DOL’s response to this stakeholder input in formulating its policies and regulations.

The summary below, which follows the organization of the Federal Register notice, is intended to represent a general overview of the input received in response to each subject area in the notice, and is published in accordance with DOL’s commitment to do so in the notice. This summary does not attempt to capture each comment DOL received. DOL notes that some comments received suggested actions that are contrary to the provisions in OAA-2006. Those comments are included in the summary as well.

1. Core Indicators

DOL received a wide range of comments on the core indicators. A large portion of the comments focused on three key topics: the first core indicator which focuses on hours of community service employment, the fifth core indicator which focuses on the number of individuals served and interpretation of the terms that define those “most in need” which includes characteristics from the list of eligible for priority of service and the characteristics of those eligible for a waiver of the 48 month limit on program participation. (OAA sec. 513(b)(1)(E)).

DOL received a variety of comments on the first core indicator, hours of community service employment. There were a number of comments expressing general support for this indicator because it relates to the community service goals of the SCSEP, and because community service employment assignments are a unique and vital aspect of the SCSEP that is valuable to participants as well as to communities. Several commenters encouraged DOL to continue its new practice of transforming the raw data provided by grantees into a rate that can be the subject of a performance goal.

A few respondents suggested definitions for this indicator; these comments included recommendations that (1) the performance measure for hours of community service employment be determined by comparing the number of community service hours provided to the potential number of community service hours based on the average current participants; and (2) community service performance could be measured in relation to the dollar amounts of Enrollee Wages and Fringe Benefits within a grantee’s budget. Another commenter recommended that hours of community service should not be based on a per participant number of hours, but on the aggregate number of community service employment hours. Another stakeholder contended that this measure should not assume that each participant slot would be full and the incumbent present 20 hours per week throughout the year. One commenter suggested that participant training for computer skills and soft skills (e.g. pre-vocational skills such as time management and interviewing techniques) be counted as community service hours because of the importance of developing such skills in the workplace.

DOL received numerous comments about whether the fifth core indicator should be split into two indicators. The fifth core indicator is the number of eligible individuals served, including those individuals included in the categories specified in the law for providing a priority for services, and in the categories specified as eligible for extension of the individual time limits. Many commenters supported establishing two separate measures, with one measure for total persons served and one for serving those “most-in-need”. Respondents provided a variety of rationales for this position. For example, several of these respondents noted that this would be a better measure of services provided to those individuals with barriers to employment; one noted that this would ensure service to “high barrier populations;” one noted that combining measures would not give an accurate depiction of the individuals being served; and one noted that two measures can be beneficial for effective program management.

Several commenters expressed support for one combined measure, rather than two separate measures. These respondents also provided a range of rationales for their positions. For example, one noted that grantees cannot control who enters the program; one stated that one measure would promote more effective services and ensure services to those “at great risk” but falling outside the priority of service list; and one noted that, based on income eligibility requirements, all individuals eligible for the SCSEP are effectively most-in-need and so a separate measure for “most-in-need” is not necessary.

Many commenters offered views on the characteristics that the statute cites to define those “most in need.” These comments touched on a range of issues. For example, a few commenters asked DOL to set aside additional time for broad community discussion before definitions are set. One commenter noted repetitiveness among several of the “most-in-need” categories. Another noted that most of the terms are self-explanatory except for “lives in an area with persistent unemployment and are individuals with severely limited employment prospects.” One stated that “although “frail” and severely disabled” individuals deserve appropriate services, from the SCSEP perspective, one must consider whether SCSEP is appropriate and if so, would the effort required to prepare these individuals for unsubsidized employment have the potential of being detrimental to performance measurement calculations.” Several offered definitions for specific “most-in-need” characteristics, including the terms frail, homeless, low employment prospects, rural area, and severely limited employment prospects.

In addition to the large number of comments on the first and fifth indicators and definitions for “most-in-need” characteristics, DOL also received a number of general comments on the core measures. Some commenters emphasized that the core indicators need to reflect SCSEP’s focus on community service as well as employment. Another recommended that the placement rate reflect the age of the participant at the time of placement, not at the time of enrollment.

Several commenters offered comments on the earnings measure. Some commenters recommended eliminating this measure. One commenter suggested that this measure should be calculated based on wages earned at any time during a quarter (rather than a specific date). Another suggested that earnings increase should be measured from entry in the program to the earnings at six months.

2. Retention in Unsubsidized Employment for One Year

The Federal Register notice afforded the opportunity for persons to provide input on the timing of the one-year retention measure and offered three possible instances when one-year retention could be determined: the 365^th day after placement, during the 4^th quarter after the exit quarter, or at the 5^th quarter after the exit quarter. DOL also invited suggestions for other points in time for measuring one-year retention.

Many respondents commented on this measure. No commenters recommended measuring one-year retention during the 5^th quarter after the exit quarter. A few recommended the 365^th day, and a number encouraged DOL to measure this indicator during the 4^th quarter after the exit quarter. One commenter recommended measuring one-year retention during the fourth quarter but not later than the 365^th day. Another suggested that this indicator be measured during the 12th month after the date the unsubsidized employment began.

Many commenters expressed the view that obtaining follow-up data one year after program exit will be a challenge. One commenter noted that while participants are generally grateful for the services provided through the SCSEP, participants sometimes feel “a sort of infringement” about providing follow-up information. Another commenter noted that employers sometimes hesitate to release employment and wage information despite being provided with a release signed by the participant. One commenter observed that in her experience it is difficult to maintain participant contact more than six months after program exit; similarly, another pointed out that securing follow-up information becomes more difficult the longer the participant has been exited from the SCSEP. Another commenter maintained that it may be difficult for participants who satisfy the new priority characteristics to remain employed for one year. One commenter argued that grantees should not be penalized if follow-up data is unobtainable; another contended that grantees should not be punished if a participant must exit the workforce due to failing health before one-year retention gets measured. A few encouraged DOL to retain its current exclusion policy under which in certain circumstances, such as the death of a participant, grantees may remove those participants from the pool of participants analyzed for performance outcomes. One commenter expressed concern that the law’s establishment of an additional indicator for retention at one year made the SCSEP inconsistent with other Federal workforce development programs. One commenter suggested that DOL delay implementation of the one-year retention measure until the SCSEP gains access to Unemployment Insurance wage records.

3. Customer Satisfaction

The Federal Register notice requested suggestions for ways to measure the satisfaction of participants, host agencies, and employers, including whether the current methodology should be carried forward or changed. Several commenters recommended that the current system be maintained as the information received is valuable and the system appears to be working satisfactorily. A few commenters suggested that the surveys not be done in Program Year (PY) 2006 because the recent national grant competition would lead to data that is not useful. There were several individual suggestions on how to improve the survey instrument, including convening focus groups to refine survey questions, shortening the surveys, and adding more questions to the surveys. One commenter recommended modifying the definition of employers and host agencies to increase the number of eligible individuals to be surveyed. Another commenter suggested similar surveys of grantees and sub-grantees to assess the data collection system and grantees’ ability to run the program. A few commenters suggested that DOL eliminate customer satisfaction surveys or at least not use them as a performance measure, which is contrary to the requirements of the statute. Several commenters suggested that customer satisfaction surveys were either too costly to administer, did not add value to the program, or should be administered less frequently than every year.

4. Other Additional Indicators of Performance

DOL solicited comments on other additional indicators to measure SCSEP performance, particularly the continued use, modification, or elimination of the current SCSEP placement rate measure. This measure counts as a placement each participant with at least 30 days of employment in the first 90 days after exiting the program.

A number of commenters addressed this question. Several stakeholders argued against keeping the SCSEP placement measure. Some expressed the view that the SCSEP has too many measures already and that the program should be evaluated on only the common measures, as is the case with other DOL programs. Other respondents stated that the common measure “entered employment” indicator is sufficient to track placement and that the SCSEP placement rate measure is duplicative. Several suggested keeping the current SCSEP placement measure as an additional indicator, including a small number who recommended keeping it with some modifications. Several commenters noted that retaining the SCSEP placement measure will enable grantees to compare program performance trend prior to 2003. One commentor who recommended keeping the SCSEP placement measure suggested that it was a good tool for staff to monitor a participant’s job satisfaction to better ensure job retention.

Suggestions for other additional indicators included a measurement of recruitment efforts, and concentrating any additional measures on SCSEP as community service employment and service to a specific population. Other respondents recommended including no additional indicators. Another commenter objected to the perception that DOL can at any time determine additional criteria without giving grantees an opportunity to respond with comments.

5. Performance Outcomes

The Federal Register notice provided the opportunity for persons to comment on how DOL should determine whether a grantee fails, meets or exceeds expected levels of performance.

DOL received many comments concerning the evaluation of performance outcomes. Several commenters recommended staying with the same methodology that is currently used, i.e., giving equal weight to all core indicators and requiring that grantees achieve at least 80 percent of the negotiated levels of performance for the aggregate of all the core indicators. Others suggested that any grantee that meets or exceeds the negotiated (i.e., expected) levels of performance in the aggregate for a majority of the core indicators should be considered to have met the expected levels of performance. A small number of commenters suggested lowering the threshold for achievement of negotiated levels of performance in the aggregate for all core indicators from 80 percent to 65 percent, due to the unique population that the SCSEP serves and the perception that the majority of the core indicators focus on employment, not community service. One commenter recommended that DOL use the same method to determine whether a SCSEP grantee fails, meets or exceeds performance that is used for the Workforce Investment Act of 1998 (WIA).

6. Other Comments

The Federal Register notice provided the opportunity for persons to comment on items that are in addition to the five specified categories discussed above.

There were a number of comments on the process for establishment of performance goals. One commenter suggested that DOL establish the same performance goals for all national grantees or that DOL make public the specific criteria and rationale used to justify different goals for each grantee. One commenter noted that during the negotiation of performance levels, more consideration needs to be given to the recommendations of the affected grantee, reflecting the current economic situation in a particular State rather than relying on historical data. A few commenters requested that DOL maintain its practice of basing negotiated performance levels for service level on the number of modified authorized slots. A few recommended that separate performance outcome goals be established for separate population groups, one for those “most in need” and one for those that are not “most in need.” A small number of commenters suggested separate performance goals for priority populations and participants who did not have priority characteristics.

Other commenters addressed broader issues related to performance accountability. A few suggested that the same criteria for application of sanctions be applied to state and national grantees. Another acknowledged the need for performance measures to evaluate the program, but requested more support in tracking retention, including technical assistance on completion of follow-ups, and eventual access to unemployment insurance data to confirm wage data

Several commenters urged DOL to continue to allow grantees with their own data collection systems to collect and submit performance data from these systems. The same commenters also urged that (1) they not be compelled to collect and submit data not directly relating to performance measures, and (2) a meeting of stakeholders be convened to discuss the SCSEP performance measures before they are put in place.

A small number of individuals commended DOL for establishing July 1, 2007 as the beginning date for the calculation of the 48-month period rather than the original date of enrollment. One commenter suggested “grandfathering” participants who entered the SCSEP prior to PY 2004 – the year when an increased emphasis on placement in unsubsidized employment began. Another commenter asked that more training and more precise definitions and explanations be provided before the beginning of PY 2007.

DOL also received single comments on the following topics: involving the SCSEP “network” in establishing the waiver process; allowing SCSEP grantees to directly place participants in employment without receiving community service training; challenges with WIA co-enrollment; possible exclusions for the common measures; support for developing a database that captures the priorities or most in need characteristics; and general feedback on the common measures.

6