State Exchange on Employment and Disability Initiative Evaluation
ICR Reference Number: 201612-1230-001
May 2017
Supporting Statement for the
Paperwork Reduction Act of 1995
Part A. Justification
Evaluation of the State Exchange on Employment and Disability (SEED) Initiative
OMB No. XXXX-XXX
Expiration Date XX/XX/20XX
U.S. Department of Labor
200
Constitution Ave. NW
Washington DC 20210
TABLE OF CONTENTS
Page
JUSTIFICATION 1
A.1 Circumstances Necessitating Collection of Information 1
A.2 Purposes and Uses of the Data 2
A.3 Use of Improved Information Technology 2
A.4 Efforts to Identify Duplication 3
A.5 Collection of Data from Small Businesses 3
A.6 Consequences of Less Frequent Data Collection 3
A.7 Special Circumstances of Data Collection 3
A.8 Consultations Outside the Agency 4
A.9 Payments to Respondents 4
A.10 Assurance of Confidentiality 4
A.11 Sensitive Questions 4
A.12 Estimated Response Burden 4
A.13 Cost to Respondents 5
A.14 Cost to the Federal Government 5
A.15 Reasons for Program Changes 5
A.16 Publication Plans and Project Schedule 5
A.17 Approval to Not Display the Expiration Date for OMB Approval 5
A.18 Exceptions to the Certification Statement 6
List of Tables
Table
1 Response burden for SEED Initiative Implementation Evaluation Survey 5
JUSTIFICATION
Part A
Clearance is being requested for a brief on-line survey, the “State Exchange on Employment and Disability (SEED) Initiative Implementation Evaluation Survey” that will aid in the formative evaluation of U.S. Department of Labor (DOL) Office of Disability Employment Policy’s (ODEP) SEED initiative. The SEED initiative is designed to advance policy development at the state and local levels to promote employment opportunities for people with disabilities. This survey will be distributed to a sample of State legislators and their staff who have had the opportunity to participate in SEED related activities and/or learn about SEED through various dissemination activities of two intermediary organizations, Council of State Governments (CSG) and National Conference of State Legislatures (NCSL). Intermediaries serve as value-added interfaces between and among various levels of government and entities with overlapping interests. The information collected in this survey will be used as part of the formative evaluation of the SEED initiative, which evaluation will identify barriers and needs to inform the SEED implementation team on what SEED could do to assist states interested in adopting disability employment policies. It will also provide feedback from State legislators and staff regarding their perceptions of SEED activities and resources to date as well as identify where State legislators are currently getting their information about disability employment policy issues to improve outreach.
In June of 2015, DOL’s ODEP, together with two intermediaries, launched the SEED initiative designed to advance policy development at the state and local level that promotes employment opportunities for people with disabilities. The goal of SEED is to foster a nationwide workforce more inclusive of people with disabilities by working with intermediary organizations to advance policy at the state level. SEED was instituted in response to ODEP’s challenge in promoting disability employment policies at the State and local levels. It also responds to states’ need for insight and guidance to build on their understanding of disability employment policy. This is especially important because, while ODEP has experienced success engaging individuals and organizations within the disability community, ODEP’s typical approach of using demonstration projects is considered protracted and does not necessarily reach those in the position to develop policy at the State level backed with legal, executive, or regulatory force. Moreover, ODEP’s experience is that state policymakers are often circumspect regarding what they consider as “one-size fits all” Federal recommendations that are not tailored to the nuances of their state.
Thus, SEED’s core approach is to engage member-based state policymaker organizations to serve as “intermediaries” between ODEP and state and local policymakers (e.g., NCSL, CSG, and Women in Government [WIG]). These intermediaries can help foster better understanding regarding the needs of state and local policymakers. Based on these needs, SEED will offer an array of support, including research, policy options, and technical assistance to help state and local policymakers establish, adopt, and implement policies that promote employment opportunities for people with disabilities. This core approach was based on a prior successful collaboration between ODEP and the U.S. Department of Transportation to promote accessibility policy that leveraged intermediaries in this way.
Because SEED is a relatively new initiative, its model and processes continue to evolve. The CEO anticipates that when SEED is fully developed, implemented, evaluated, and refined, it may serve as a model for other DOL and Federal agencies tasked with promoting policy from the Federal to State and local levels.
In September 2015, DOL’s CEO selected the team of Coffey Consulting, LLC (prime contractor) and the American Institutes for Research (AIR) to evaluate the new SEED initiative. Division G, Title I, Section 107 of Public Law 114-113 Sec. 107. (a), the “Consolidated Appropriations Act, 2016” authorizes the Secretary of Labor to reserve not more than 0.75 percent from specific budget accounts for transfer to and use by the Office of the Chief Evaluation Officer for departmental program evaluation. The team was contracted to determine the evaluability of the initiative and to design and conduct an appropriate evaluation. Following the evaluability assessment, the evaluation team – in consultation with a Technical Working Group of subject matter experts and the DOL Contracting Officer Representative (COR) – determined that a formative evaluation was most appropriate. Formative evaluations take place before or during a project’s implementation with the aim of improving its design and performance. Given the infancy of SEED and the use of intermediaries in this way is a relatively new approach for DOL to promote policy to States, a formative evaluation was determined important to provide on-going feedback and research support to help make the SEED program as efficacious as possible, and thus the most efficient use of public funds.
An important aspect of this formative evaluation is to gather information and feedback from the target audience of the SEED initiative: specifically State legislators and their staff. Therefore, it is critical to conduct a survey of State legislators and their staff that have engaged in SEED activities or with SEED materials. The survey will serve multiple important purposes. First, it will determine the extent to which the legislators are aware of the range of SEED materials and services (e.g., a National Policy Framework and technical assistance sessions). Second, the survey will include a “needs analysis” that will help identify barriers State legislators face in adopting disability employment related policy and suggest the means by which the initiative might address those barriers. Third, the survey will provide feedback from participants on their perceptions of SEED activities and materials and how they could be improved. Finally, the survey will inform a network analysis that will identify where participating legislators and their staff are obtaining information related to disability employment policy. This will enable SEED to more effectively target their outreach efforts.
The SEED Initiative Implementation Evaluation Survey will be supplemented with a limited number of interviews (less than 10) with SEED staff and State legislators, but only a survey of the target legislators and staff can fulfill the varied needs and provide the information required for improving the program design—the goal of the formative evaluation.
The Coffey Consulting/AIR evaluation team will collect the data on behalf of DOL’s ODEP and CEO. The evaluation team will analyze the data and share the results with DOL and the SEED implementation team, which includes representatives from ODEP, a communications contractor, and members of the intermediary organizations participating in SEED. (Survey participants will be members of these intermediary organizations and aware of their involvement.)
The data will be aggregated and analyzed, with results included in the Evaluation Final Report, and one or more briefings that will be provided to DOL, ODEP and CEO, as well as the SEED implementation team.
The data will only be used to address research questions pertaining to the formative evaluation of the SEED initiative. Specifically, the data will be used to assess the awareness of the range of SEED activities and materials among the target population of State legislators and governors and their staff to assess outreach. Data collected will also be used to assess reported changes in knowledge or motivation related to disability employment policy as a result of SEED. It will identify participants’ perceived barriers to implementing disability employment policy in their State as well as the means by which SEED could assist in overcoming these barriers. In addition, data will be used to evaluate participants’ perceptions of the SEED activities and materials with which they are familiar, including strengths and areas for improvement. Finally, the data will be used to determine any common sources of information on disability employment policy among participants to assess and improve outreach efforts of SEED.
The evaluation team will also be collecting data through interviews with key stakeholders to supplement the survey information in order to elucidate the survey findings and supplement the survey data collection. This effort will involve interviews with a limited number of participants from the intermediary organizations working with ODEP as part of SEED, as well as a small sample of State legislators or their staff to learn more about their implementation of disability employment policy. However, in no case will the evaluation team collect substantially the same information from more than 9 participants as part of these interviews. Since information collected from less than 10 respondents is not covered by the PRA, that interview protocol is not part of this request.
The instrument used in the SEED Initiative Implementation Evaluation survey will be conducted using an on-line survey platform for efficiency and to minimize burden to participants. Participants will be sent a secure link to a survey in an electronic message that includes a brief description of the study and the purpose of the survey. Clicking the link will take them to the web-based survey where they are provided a further brief explanation and instructions.
Survey Design. The instrument consists of a maximum of 27 items. A branching technique is used to make the survey more efficient for participants. Moreover, those who indicate they are not aware of particular SEED activities or resources are not asked about them (yielding a survey of as few as 11 items for these participants). With few exceptions, items are multiple choice with either Likert-type scales or “select all that apply” responses. Where appropriate, items provide response options with “check all that apply” and an opportunity to write-in a short response to clarify an answer of “other.” Three items specifically request short text-based responses.
The first five items capture background information about participants (i.e., position, tenure, opportunity to make disability employment policy decisions, membership in an intermediary organization, and participation in SEED activity). The final question in this first section is used to branch within the survey. Those who have participated in SEED activities receive items about their perceptions of the SEED initiative and SEED implementation. Next, participants are asked four items for the network analysis regarding to whom they turn for information related to disability employment policy and the influence of intermediary organizations and other States on employment policy in their own state. Finally, participants rate the importance of four possible barriers to adopting disability employment policy in their State on Likert-type scales, and respond to a write-in item to suggest technical assistance that would assist them adopting or implementing disability employment policy in their State.
If participants indicated in the first section that they were not aware of SEED-related resources or activities, branching ensures they are only asked questions related to the needs analysis and network analysis, thereby shortening the survey.
Data Capture. The web-based survey and electronic response format allow data capture to be instant at the moment the participants submit the survey. All data are automatically captured in a secure database for analysis.
Not applicable.
This request for clearance is for a single administration of the SEED Initiative Implementation Evaluation survey. The consequences of not collecting the information at all are that DOL ODEP and CEO will lack the critical feedback from their target audience needed to continue to refine and improve the SEED initiative, potentially making less efficient use of resources, having less efficacy in assisting states to adopt disability employment policy, and ultimately limiting SEED’s capacity for improving opportunities for employment for persons with disabilities..
None of the special circumstances listed in the instructions for completing the supporting statement apply to the SEED Initiative Implementation Evaluation survey.
There have been several consultations outside of DOL in preparing this survey. More specifically, a Technical Working Group, a panel of subject matter experts from academe and government experience, reviewed the survey instrument (as was the whole evaluation design). The work group members were asked to provide input and guidance to ensure the data are not duplicative and were collected as efficiently as possible to minimize burden. The panel includes experts on evaluation, knowledge transfer, diffusion of innovation, and working with intermediary organizations to promote policy to States.
On November 23, 2016, the agency published a Federal Register Notice to invite public comments on the ICR. (See 81 FR 84618.) The CEO received one comment. A commenter, who identified herself as “Jean Public,” raised general program concerns, noting her belief that the work to find people with disabilities could be done more effectively in state offices.
Respondents will not receive payment. Participation is voluntary.
There is no assurance of confidentiality. Respondents will be informed of the voluntary nature of the survey and of the privacy provision in the initial cover letter and on the questionnaires, stating that their responses may be used for statistical purposes only and may not be disclosed, or used, in identifiable form for any other purpose except as required by law. Respondents are also informed of the general risk inherent in supplying information over the Internet. In addition, participants will be provided with the telephone number and e-mail addresses of the Project Manager and Principal Investigator, as well as the AIR Institutional Review Board, if they have any questions or concerns.
The SEED Initiative Implementation Evaluation survey is voluntary, and no person is required to respond to it. There are no items of a sensitive nature. In addition, respondents may decline to answer any question in the survey. Respondents are informed of the voluntary nature of the survey in the instruction that accompanies the questionnaire, as well as on the cover letter.
The SEED Initiative Implementation Evaluation Survey will be sent to State legislators and their staff who are members of the intermediary organizations mentioned above (NCSL and CSG). Representatives of these intermediary organizations will assist in contacting potential respondents and requesting their participation. Representatives of these intermediaries indicate that in prior surveys conducted electronically with their members, they obtained average response rates around 80% percent. As this survey frame is expected to be the expected 500 persons with prior participation with SEED or its materials, a survey response rate of approximately 80 percent is assumed, bringing the total number of expected participants to 400. A detailed description of the planned sampling design is provided in this submission in Supporting Statement Part B.
The response burden per instrument and the total response burden are shown in Table 1. The administration time for the web-based survey ascertained via pilot testing in practice administrations is a maximum of 13 minutes. The expected number of respondents and number of responses are based on the expected numbers of completed surveys of each type, discussed in section B.1.3. The hourly rate for each Survey Respondent Type is based on 2016 average annual salaries divided by 2,080 hours for full-time positions, and 1,040 (rounded) for part-time. Salary information was obtained based on research from Council of State Governments and National Conference of State Legislators, and publicly available Bureau of Labor Statistics wage data.1
Table 1. Estimated response burden for the State Exchange on Employment and Disability (SEED) Initiative Implementation Evaluation Survey |
|||||
Survey Respondent Type |
Estimated time (minutes) |
Number sampled 1 response each |
Hourly Rate |
Total time (hours) |
Value of Respondent time
|
State Legislator |
13 |
200 |
$43.64 |
43.33 |
$1,891 |
Staff to State Legislator |
13 |
150 |
$31.25 |
32.5 |
$1,016 |
Governor |
13 |
25 |
$66.06 |
5.42 |
$358 |
Staff to Governor or Lt. Governor |
13 |
25 |
$31.25 |
5.42 |
$358 |
State Executive Branch Employee |
13 |
100 |
$24.30 |
21.67 |
$527 |
Study Total |
13 |
500 |
|
108 (rounded) |
$4,150 |
There are no costs to respondents beyond the value of time to participate, as presented in table 1 above.
There Federal Government cost associated with the purchase of services for the SEED Evaluation contract including costs, for initiating this data collection as well as generating, maintaining, and disclosing or providing the information, are included in the statement of work of this contract. The total value for this contract is $450,000 over 2 years, and this includes all aspects of work including project management, knowledge development about SEED, the evaluability assessment of SEED, the evaluation plan, obtaining OMB PRA Clearance, data collection, reports, and briefings. Contract oversight costs are approximately $119,000 for a Grade 14 Step 4 senior analyst. The Department estimates total annualized Federal costs to be $284,500. ($450,000 + $119,000 = $5669,000)/2 = $284,500.
This is a new data collection.
The OMB authorization number and expiration date will be displayed in the cover e-mail and web survey.
There are no exceptions to the Certification Statement.
1 Information for Governors’ salary can be found at: http://knowledgecenter.csg.org/kc/content/governors-salaries-2016; information for Lt. Governor salaries can be found at: http://knowledgecenter.csg.org/kc/system/files/4.11%202016.pdf. Information for State Legislators obtained from National Conference of State Legislators 2016 Survey: State Legislative Compensation, Session Per Diem and Mileage, and BLS data http://www.bls.gov/oes/current/oes111031.htm based on 1040 hours. Information on staff to legislators is based on an NCSL survey of legislative staff, assuming 2080 hours. Where salary is per legislative session, two sessions were used for these estimates. Data for state executive branch employees from http://www.bls.gov/oes/current/naics4_999300.htm#11-0000 BLS data for Local Government (excluding schools and hospitals), all occupations.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | WAITS_T |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |