SEED OMB Package Part B 12-1 ch FINAL

Supporting Statement for the

Paperwork Reduction Act of 1995

Part B. Collections of Information Employing Statistical Methods

ICR #

Evaluation of the State Exchange on Employment and Disability (SEED) Initiative

December 2016

US Department of Labor

200 Constitution Ave. NW
Washington DC 20210

TABLE OF CONTENTS

Page

B.1 Respondent Universe and Statistical Design and Estimation 1

B.2 Statistical Procedures for Collection of Information 2

B.3 Methods for Maximizing Response Rates 3

B.4 Test of Procedures or Methods to be Undertaken 4

B.5 Individuals Responsible for Study Design and Performance 4

REFERENCES 5

List of Tables

Table

1 Estimated Universe and Sample Size by Intermediary and Participants 2

2 Sample Survey Name, Year, and Response Rate Conducted by Intermediaries 2

3 Margins of error for the survey by respondent type 5

DESCRIPTION OF STATISTICAL METHODOLOGY

PART B

The US Department of Labor (DOL) is requesting clearance for information collection to conduct a survey as part of a formative evaluation of the State Exchange on Employment and Disability (SEED) initiative. This survey will be conducted to provide needs analysis and feedback from state legislators and their staff who are the target audience of the initiative. This survey will be a brief, one time data collection that will provide valuable information to DOL and our partners on the SEED implementation team to improve the effectiveness of the initiative.

B.1 Respondent Universe and Statistical Design and Estimation

The purpose of the SEED survey is to provide feedback to the implementation team of the SEED initiative on SEED activities and materials among the target population, including State legislators, governors and their staff who are involved in the SEED initiative so far. The data will also be used to assess reported changes in knowledge or motivation related to disability employment policy as a result of SEED. The data will identify participants’ perceived barriers to implementing disability employment policy in their states along with means by which SEED could assist in overcoming the barriers. In addition, the data will be used to evaluate participants’ perceptions of the SEED activities and resources with which they are familiar, including strengths and areas for improvement. Lastly, the data will be used to determine any common sources of information on disability employment policy among participants to assess and improve outreach efforts of SEED.

The respondent universe for this survey will be state legislators, governors, Lt. Governors, and their respective staff who have had the opportunity to participate in SEED related activities and/or learn about SEED through various dissemination activities of two intermediary organizations, Council of State Governments (CSG) and National Conference of State Legislatures (NCSL). The size of the target population by job category is presented in column 1 of Table 1. The estimated universe is 500. The implementation team have identified these individuals as the participants for SEED and will provide a list of contact of information to the evaluation team.

An expected response rate of approximately 80 percent is assumed for this survey based on results from previous years’ surveys conducted by NCSL and CSG. Under this assumption, the total number of respondents is anticipated to be 400. In table 2, we provided sample survey names, years and response rates conducted by the two intermediary organizations as a reference. The average response rate for surveys conducted by CSG and NCSL is 86%. We have used a conservative estimate given the on-line surveying method, target population, and the early stage of SEED.

As the purpose of the survey is to understand SEED participants’ knowledge and awareness shift and identify behavior changes after participation of the SEED initiative we will focus on using descriptive analysis. We do not plan – nor would we attempt – to draw statistical comparisons from the data. Nevertheless, the data resulting from this survey will provide valuable information to DOL.

Currently, there is no information about how the SEED initiative is being received by the state legislators and their staff at whom SEED is directed, beyond very limited anecdotal evidence of those few directly involved. The proposed survey of the SEED participants is the only opportunity within the first two years of SEED development to gather baseline information about SEED participants’ perception of the initiative and its products; perceptions of their utility, and how they may be improved moving forward.

Table 1. Estimated Universe and Sample Size by Intermediary and Participants.

Survey Respondent Type	Estimated Universe	Anticipated Response Rate	Estimated Number of respondents
NCSL
State Legislator	200	80%	160
Staff to State Legislator	150	80%	120
CSG
Governor	25	80%	20
Staff to Governor or Lt. Governor	25	80%	20
State Executive Branch Employee	100	80%	80
Total	500		400

Table 2. Sample Survey Name, Year, and Response Rate Conducted by Intermediaries.

Name	Description	Year	Response Rate
NCSL
Public Transportation Funding Survey	Survey of state DOT agencies	2015	96%
CSG
Annual Survey of State Trade Directors	State International Development Organizations Annual Survey of State Trade Directors	2016	78%
Survey of State Administrative Officials Salaries		2016	94%
Participant Evaluation Survey	Medicaid Policy Academy, Participant Evaluation Survey	2016	79%

B.2 Statistical Procedures for Collection of Information

Statistical Methodology for Stratification and Sample Selection. No statistical methods are needed for stratification and sample selection for the survey. The universe of participants will be invited to participate in the survey.

Estimation Procedures. Data will be analyzed using descriptive methods. Simple tabulations and analysis will be conducted to analyze the survey constructs. We will, for example, tabulate scale scores, means, standard deviations, and percentages.

Survey Constructs. The key constructs of the state survey includes knowledge about SEED; network analysis; needs assessment; and perceptions of SEED. The evaluation team will calculate the percentage of respondents who answer positively or negatively about each question and compute scale scores using sums of the items within each construct. For instance, the state survey will allow the evaluator to gather information on the percentage of respondents that indicate a positive change in knowledge about disability employment policies after the implementation of SEED, percentage of respondents that indicate the need for educational resources for future technical assistance, and percentage of respondents that indicate areas the SEED implementation team can improve to better facilitate policy adoption. The survey is for feedback and improvement purposes and descriptive in nature; it is not intended to draw statistically accurate inferential comparisons between subgroups. Therefore, power analysis was not conducted. In addition, variance estimation calculations and clustering are not needed in the analysis.

Reliability.

The reliability of the estimates by survey respondent type is shown in Table 3. Margins of error for a 95% confidence interval around the response proportion (i.e., proportion responding a particular way) are computed as

where the “value of p” represents the sample proportion, q = (1- p), n corresponds to the estimated number of respondents for each respondent type (See Table 1 for estimated Respondent Type n sizes), and 1.96 is the z-value for a 95% confidence interval.. As indicted in Table 3 below, if a sample response is 70 percent, the margin of error will be below 0.08 for state legislators and below 0.09 for staff to state legislators given anticipated sample sizes. The margins of errors for Governors and staff to Governor and Lt. Governors will slightly exceed 0.2. The margin of error for state executive branch employees is expected to be approximately 0.1. These margins of error are sufficient for the intended use of providing feedback to SEED and identifying areas to improve.

Table 3. Margins of error for the survey by respondent type

	Value of p
Survey Respondent Type	0.7	0.6	0.5
State Legislator	0.071	0.076	0.077
Staff to State Legislator	0.082	0.088	0.089
Governor	0.201	0.215	0.219
Staff to Governor or Lt. Governor	0.201	0.215	0.219
State Executive Branch Employee	0.100	0.107	0.110

Unusual problems requiring specialized sampling procedures. There are no unusual problems requiring specialized sampling procedures.

Periodic data collection cycles to reduce burden. There will be only one administration of the SEED survey.

B.3 Methods for Maximizing Response Rates and Addressing Nonresponse

The SEED survey design incorporates a number of features to maximize response rates. This section discusses those features. These include multiple contacts with the targeted participants, an advance letter and up to two reminder e-mails and one phone reminders for the survey. Survey results will be monitored on a daily basis to assess progress of data collection. Respondent contact materials are provided in Appendix A and survey instruments in Appendix B.

Advance letter. The SEED survey data collection will begin with an advance notification e-mail in early February 2017. Studies have showed that using an advance letter was effective at raising response rates which reduces the amount of follow-up contact required (Kaplowitz et al. 2011).

Reminders and nonresponse follow-up. The data collection protocol includes several stages of nonresponse follow-up at each phase. In addition to the number of contacts, changes in method (e-mail reminder and phone reminder) are designed to capture the attention of potential respondents. A thank you/reminder e-mail will be sent to all sampled participants approximately one week after the launch of the survey. A second e-mail reminder will be sent to non-responding participants approximately two weeks after the launch of the survey is sent out. The evaluation staff will conduct phone reminders to non-responding participants approximately one week after the second e-mail reminder. We estimate being able to devote up to 5 FTEs for one week to phone follow up, reaching out to as many as 250 people for up to 2 rounds of follow-up phone surveys (assuming 6-8 calls per person per hour). If there are more non-respondents than this, we will select randomly from among the non-respondents for follow-up calls, dividing proportionally among the respondent types as needed.

Total Design Method/Respondent-Friendly Design. Surveys that take advantage of respondent-friendly design have demonstrated increases in survey response (Dillman, Smyth, and Christian 2008; Dillman, Sinclair, and Clark 1993). Team researchers have honed the design of the SEED survey through multiple iterations of cognitive interviewing and field-testing. These efforts have focused on the design and content of all participant contact materials.

Engaging Respondent Interest and Cooperation. The content of advance e-mail and instructions for the survey focused on communicating the legitimacy and importance of the study. Survey participants are affiliated members of the two intermediary organizations that have been working with the SEED initiative. Researchers will conduct the survey with the assistance of the intermediary organizations to engage respondents’ interests and cooperation. Additionally, the sampling frame has been limited to only those who are engaged in committees whose work is related to disability employment to ensure the survey is relevant to their work and states.

Addressing Nonresponse Bias. Although we expect the response rate to be 80%, we will conduct nonresponse bias analysis by comparing the characteristics of the respondents to the population universe as a whole to test whether nonresponse is at random. These characteristics may include job category, years of experience in the position and intermediary group affiliation. Demographic information other than experience are not collected by the survey nor the intermediaries and hence will not be used for nonresponse bias analyses.

Nonresponse bias will be assessed by comparing respondents and non-respondents. The nonresponse bias analysis will examine characteristics of the population members available on the sampling frame. Moreover, the study will compare early respondents to late respondents to determine whether they differ on frame characteristics and survey variables. We will also conduct a non-response follow-up with only a few key survey questions from a random sample of non-respondents and compare respondents in the initial contacts with the respondents from the non-response follow-up.

Based on the above analysis, modern weighting strategies (e.g., calibration) will be used to adjust for nonresponse bias. After the weight adjustments, the weighted estimates will be compared to population/external estimates to determine whether the nonresponse adjustments have effectively reduced nonresponse bias.

The information to be collected is not personally sensitive and there are no consequences to participants associated with survey responses. Moreover, the results will not include any personally identifiable information. Therefore, there is no risk to respondents or incentive to report inaccurate information. Since scale score computation requires use of all items within a construct, missing values for any item within a construct will result in a missing scale score. Therefore, to produce reliable and accurate scale scores for each construct and to mitigate the impact of missing items we will do the following: If the missing rates for the items within the survey exceeds 30 percent multiple imputations will be employed if needed. If multiple imputations is employed, analysis models that take multiple imputations into account will be used as described in Rubin (1987).

B.4 Test of Procedures or Methods to be Undertaken

In order to understand the time-burden and user experience associated with completing the survey, SEED asked partnering intermediaries to engage a number of staff members to respond to the pilot survey. Three individuals began the pilot survey, and two individuals completed the pilot survey. Based on the pilot, the research team learned that the time-burden involved with completing the survey is approximately 13 minutes. Participants did not experience difficulties understanding the questions and did not provide additional suggestions to improve the survey.

B. 5. Individuals Responsible for Study Design and Performance

The persons listed below participated in the study design and are responsible for the collection and analysis of the data:

Michelle Yin, AIR

202-403-5580

Deeza Mae Smith, AIR

202-403-5127

Cynthia Overton, AIR

202-403-5058

Daniel Weissbein, Coffey Consulting LLC

202-423-5267

Lester Coffey, Coffey Consulting LLC

301-907-0900

References

DeVellis, R. (2003).  Scale development: theory and applications: theory and application. Thousand Okas, CA: Sage.

Dillman, D.A., Sinclair, M.D., and Clark, J.R. (1993). Effects of questionnaire length, respondent-friendly design, and difficult questions on response rates for occupant-addressed Census mail surveys. Public Opinion Quarterly, 57, 289-304.

Dillman, D.A., Smyth, J.D., and Christian, L.M. (2008). Internet, mail, and mixed mode surveys: The Tailored Design Method. New York: Wiley.

Kaplowitz, M., Lupi, F. Couper, M., Thorp, L. (2011). The Effect of Invitation Design on Web Survey Response Rates. Social Science Computer Review August 2012 vol. 30 no. 3 339-349.

Rubin, Donald B. (1987) Multiple Imputation for Nonresponse in Surveys. New York: John Wiley & Sons

i