B. Collections of Information Employing Statistical Methods
The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When the question “Does this ICR contain questionnaires, censuses or employ statistical methods” is checked, "Yes," the following documentation should be included in the Supporting Statement to the extent that it applies to the methods proposed:
1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.
Currently, there are 30 states participating in the EFSLMP CoP. The sampling frame for the CoP survey will be one representative (primary respondent) from each of the following agencies:
Department of Rehabilitation Services;
Department of Mental Health;
Department of Intellectual and Developmental Disabilities;
Workforce Development system;
Department of Education (Special Education Division);
Medicaid agencies.
The Evaluation Team will compile a list of respondents from each of the participating six agencies participating in the CoP. The unit of analysis for this data collection is the “agency” participating in the EFSLMP CoP. ODEP, its subject matter experts, and members of the CoP will select one representative from each agency to complete the CoP questionnaire on behalf of their agency. The goal of the data collection is to determine how each of the six agencies in each state participates in the CoP, its approach to implementing EF policies and practices, and the progress each agency has made since joining the CoP.
2. Describe the procedures for the collection of information including:
* Statistical methodology for stratification and sample selection,
* Estimation procedure,
* Degree of accuracy needed for the purpose described in the justification,
* Unusual problems requiring specialized sampling procedures, and
* Any use of periodic (less frequent than annual) data collection cycles to reduce burden.
The respondents asked to complete the questionnaire will be identified by ODEP, with input from subject matter experts and the leaders of each participating state, as those that should have “a seat at the table” in discussions related to EF in their states. The selected primary respondents are State agency employees and leaders of their state’s EF activities, and therefore can appropriately represent their State agency in providing the requested information. However, “secondary” respondents are available in the event that a primary respondent from an agency is unavailable. These staff work closely with the primary respondent and can, if necessary, complete the questionnaire with support and discussion from other staff. The primary or secondary respondent from each of the six agencies in each of the 30 states will participate in the data collection for a total sample of N = 180 respondents.
Because primary respondents are engaged in the CoP, hold leadership positions in their agency system, and are responsible for the implementation of their agency’s EF policies and practices, they can effectively represent the agency in which they are employed better than other CoP participants employed by the same agency system. The downside to this approach is that relying on one respondent per agency may result in a point of view that may not represent the perspective of an entire agency. However, because the primary respondent is in a leadership position unlike other CoP participants and is the EF decision maker for her/his agency, selecting this person as the primary respondent for the questionnaire is the best approach to identify agency needs and activities. Agency leaders are more likely to take a broader view on what is going on in different parts or groups within the agency. In addition, primary respondents will be encouraged to discuss her/his responses to each item on the questionnaire with other staff members in the agency and take this information into consideration when completing the questionnaire.
EFSLMP CoP States |
Agencies represented |
Number of respondents |
|
|
N=180 |
3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.
Plans to Limit Non-Response
There are six components to our approach for limiting non-response. First, response rates will be calculated in real-time on a weekly basis during the four-week data collection period. The Evaluation Team will follow up with respondents who do not complete the questionnaire via telephone in an attempt to increase response rates. Respondents will be contacted three times, first by email and then, every 72 hours thereafter, by telephone throughout the data collection period until a response is received. Second, the questionnaire will be 508-compliant. Large print lettering and a screen reader will be used to ensure that people with disabilities can access the questionnaire online. Third, an introductory e-mail will be sent to the primary respondent from each agency. The e-mail will be signed by an official from ODEP and will introduce the study, emphasize confidentiality, explain the respondent’s rights, and alert the respondent that she/he will be receiving an e-mail from Social Dynamics with a link to the questionnaire and to mention how the questionnaire will be used by the sponsoring agency. The introductory statement also will state the voluntary nature of the study and assure respondents that all of their responses will be kept private. Fourth, Social Dynamics will use quality-control procedures so that all staff contacting non-respondents via email and/or telephone are monitored, evaluated, and provided with instant feedback on their performance to eliminate interaction patterns or demeanors that might be detrimental to achieving cooperation. Fifth, in the event of non-response due to a primary respondent being unavailable, another respondent, referred to as a secondary respondent, will be selected based on recommendations from ODEP, subject matter experts, and senior staff members of the agency whose primary respondent has not responded to the questionnaire. We are confident that secondary respondents will be able to complete the questionnaire on behalf of their agency due to their involvement in the CoP and position of authority within the agency. As with primary respondents, secondary respondents will be encouraged to discuss their questionnaire responses with other senior members of their agency system who participate in the CoP. Finally, a pilot test was conducted with nine states in April 2013. The pilot test collected information on the time it takes to complete the questionnaire, the relevance of the questions and question wording, and a number of suggestions for revising the questionnaire. Based on the pilot test, the questionnaire was revised and a second, less formal pilot test was conducted internally.
Although this data collection is not based on statistical sampling and respondents are not obligated to complete the questionnaire, we expect to achieve an 80 percent response rate. A similar survey of technical assistance grantees in 2011 in Minnesota received an 82 percent response rate (Minnesota Department of Health 2011). The Minnesota survey targeted 29 grantees, while our CoP will focus on 30 states. The level of engagement in the CoP grantees, and our thorough follow up procedures will also contribute to the response rate. .
4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.
The data collected from the ODEP CoP questionnaire will be used to prepare descriptive statistics tables and graphs on the issues related to EFSLMP CoP efforts in the 30 states participating in the initiative. The descriptive statistics will detail whether efforts are being undertaken at the State level to promote integrated employment, the public agencies that are in charge of those efforts, the extent to which there is interagency collaboration, perceived benefits of this collaboration, and barriers to the adoption and implementation of EF policies and practices. A comparison of agencies’ barriers and efforts to implement EF strategies will be highlighted. Finally, the CoP questionnaire also includes questions designed to assess the technical assistance received by each State agency through the EFSLMP and recommendations for the EFSLMP CoP, as well as ratings of development/implementation of EF strategies.
The analysis will provide descriptive information with bivariate statistical presentations in tabular and graphical form. Initially, the analysis will focus on respondent characteristics (i.e., education level, position title, years employed in position), EFSLMP participation, training and technical assistance (T&TA) needs, and development/implementation of EF activities at the State level. Descriptive information will highlight differences among groups of respondents and the factors associated with variability in development/implementation of EF activities.
The analysis will begin with a description of the respondent sample, including the following information: agency affiliation, length of time at agency, job title, and length of time in current position:
Question 1: At what agency are you employed?
Question 2: How long have you worked at this agency?
Question 3: What is your job title?
Question 4: How long have you held this position?
Next, the analysis will provide graphical displays and bivariate cross-tabulation tables using Questions 1–4 (as independent variables) and Questions 5 and 6, which include information on external agencies that provide T&TA, as dependent variables.
Question 5: Excluding ODEP’s EFSLMP CoP, what additional agencies or organizations have you and your colleagues working on your state’s Employment First efforts received technical assistance from in the last two years?
Question 6: Please indicate all the topics that you and your colleagues working on your state’s Employment First efforts received T&TA from in the last two years from the sources you selected in question 5.
The cross-tabulation analyses will highlight differences in the distribution among groups on the variables of interest (e.g., types of training received and training sources).
Interval-level response categories are used in Question 7 to rate respondents’ level of satisfaction with each type of T&TA practice on a 5-point interval-level scale. This question will be presented using descriptive analyses, including frequency distributions and cross-tabulations to examine group differences using respondent characteristics from Questions 1–6.
Question 7: Listed below are 12 features of the ODEP’s CoP T&TA provided to CoP states. Please rate your level of satisfaction for each of the T&TA characteristics.
a.) Information presented during webinars
b.) Quality of speakers during webinars
c.) Level of interaction among participants
d.) Opportunities to ask questions and get feedback
e.) Frequency and duration of webinars
f.) Relevance of topics covered in monthly webinars
g.) Real-world examples from CoP states
h.) Quality of contacts made with other states through participation in the CoP
i.) ePolicyWorks systems
j.) Monthly CoP Electronic Bulletins
k.) Annual In-Person National EFSLMP CoP meeting (held in November 2012)
l.) Opportunities to provide feedback to Federal policymakers via participation
Questions 8 and 9 also use a 5-point rating scale. A graphical display will be used to identify group differences on mean satisfaction ratings using respondent characteristics from Questions 1–6 as independent variables.
Question 8: Below please indicate whether you have used what you have learned from participating in the CoP to influence your state’s Employment First planning and implementation efforts.
Yes
No
No Opportunity
Question 9 of the EFSLMP CoP questionnaire collects information on the “state of development/implementation” for 5 types of State EF activities:
Strategic Planning
Policy Development & Operational Alignment
Interagency Collaboration
Funding and Reimbursement
Outcomes and Measures
Question 9: Listed below are six components of Employment First that have been topics of discussion during various CoP technical assistance events. Please select the "stage of development/implementation" in your state agency for each of these components using the following scale:
1. Not yet present
2. In its infancy (e.g., early conversations are happening)
3. Under development (e.g., beginning to formulate, conceptually planning, drafting language, etc.)
4. In existence (e.g., approved, formalized, beginning to be operationalized)
5. Fully implemented (e.g., fully established, monitored with a focus on improvement, and used to guide policy development)
6. Status unknown
Question 9 will provide DOL with a self-rated estimate of the progress that each state has made to date, which will help to target future T&TA efforts. The development/implementation indicator for each of the components will also be used to create quantitative scores (e.g., average/median scores, and an individual score for each component).
The qualitative data for the EF evaluation derive from questions 10 and 11, which are open-ended questions that ask respondents to describe their most valuable experiences as an EFSLMP CoP participant, as well as recommendations for improving the EFSLMP CoP.
Question 10: Please describe what you thought was most valuable in your experience with the EFSLMP CoP.
Question 11: Please provide recommendations for how you would improve the EFSLMP CoP in the future.
Using this information, a content analysis will be conducted using a rigorous approach to qualitative data analysis using principles of Grounded Theory modified to fit the scope and purpose of the EF Evaluation (Glaser & Strauss, 1967). As is the case with most qualitative analyses, the data require reduction. Key to this effort is the development of a coding scheme that is aligned with the conceptual framework of the overall evaluation. We will create a hierarchy of conceptual categories linked to the evaluation research questions. The categories, or “codes,” at the top of the hierarchy will, at a minimum, align with the three research questions presented in section 2 of Part A of this package. Also included as “top codes” will be the T&TA activities provided to the EFSLMP CoP.
The DOL will utilize information from the CoP questionnaire to inform its strategic planning process, promote best and promising practices, and help states in adopting and implementing EF initiatives. Any published materials will be used to inform policymakers and other stakeholders on EF policy and technical assistance. A final report will be made available at the DOL’s Web site. Additionally, materials may be prepared for journals, conferences, and associates interested in the subject. Findings will be used to improve technical assistance materials and methods sponsored by the DOL. We anticipate that the Web survey will be implemented on February 3, 2014 and will end on February 28, 2014. The report on questionnaire findings for ODEP will be completed on March 30, 2014.
5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
ODEP contracted with Social Dynamics, LLC to undertake the data collection and analysis of the questionnaire of states participating in the EFSLMP CoP.
CONTACT for Social Dynamics, LLC:
Dr. Douglas Klayman, President
481 N Frederick Ave, Suite 410
Gaithersburg, MD 20878
Telephone: 301-990-1105
dklayman@socialdynamicsllc.com
References
Minnesota Department of Heath (2011) Eliminating Health Disparities Initiative (EHDI). Evaluation Technical Assistance and Support: Feedback Survey Results. Retrieved from: http://www.health.state.mn.us/ommh/grants/ehdi/tasurvey060311.pdf
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Meredith DeDona |
| File Modified | 0000-00-00 |
| File Created | 2021-01-28 |