Section A

National Occupational Research Agenda (NORA) 2016 Decade Review

Request for Office of Management and Budget Review and

Approval for Federally Sponsored Data Collection

Supporting Statement Part A

Project officer:

Sidney C. Soderholm, PhD

National Occupational Research Agenda (NORA) Coordinator

National Institute for Occupational Safety and Health

395 E Street, SW, Suite 9100, MS P-06

Washington, DC 20201

202-245-0665 (voice)

202-245-0628 (fax)

Email: SSoderholm@cdc.gov

October 1, 2014

Table of Contents

1. Circumstances Making the Collection of Information Necessary

2. Purpose and Use of the Information Collection

3. Use of Improved Information Technology and Burden Reduction

4. Efforts to Identify Duplication and Use of Similar Information

5. Impact on Small Businesses or Other Small Entities

6. Consequences of Collecting the Information Less Frequently

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside

the Agency

9. Explanation of Any Payment or Gift to Respondents

10. Assurance of Confidentiality Provided to Respondents

10.1. Privacy Impact Assessment Information

11. Justification for Sensitive Questions

12. Estimates of Annualized Burden Hours and Costs

13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

14. Annualized Cost to the Federal Government

15. Explanation for Program Changes or Adjustments

16. Plans for Tabulation and Publication and Project Time Schedule

17. Reason(s) Display of OMB Expiration Date is Inappropriate

18. Exceptions to Certification for Paperwork Reduction Act Submissions

References

Appendices

Appendix A: Occupational Safety and Health Act of 1970

Appendix B: 60-Day Federal Register Notice

Appendix C: IRB Determination of Non-Research

Appendix D: Sector Council Member Questionnaire

Appendix E: Sector Council Member Questionnaire - Survey Monkey Screen Shots

Appendix F: Pre-Survey Email to Potential Respondents

Appendix G: First Email with Survey Link

Appendix H: Second Email with Survey Link

Appendix I: Final Email with Survey Link

A. Justification

1. Circumstances Making the Collection of Information Necessary

This is a new information collection request. The National Institute for Occupational Safety and Health (NIOSH) is responsible for conducting research and making recommendations to prevent worker injury and illness, as authorized in Section 20(a)(1) of the Occupational Safety and Health Act (29 U.S.C. 669) (Appendix A). In 1995-6, NIOSH saw an opportunity to enhance its ability to accomplish its mission through partnerships that involved a broad national stakeholder base in occupational safety and health. With stakeholder input, NIOSH developed and launched a decade-long partnership program titled the National Occupational Research Agenda (NORA) in 1996. Participation in NORA includes stakeholders from universities, large and small businesses, professional societies, government agencies, and worker organizations. After an internal management review of the first decade of NORA, conducted in 2005, NIOSH launched the second decade of NORA (2006 – 2016) structured for even greater national impact.

The second decade of NORA was based on a new structure of ten NORA sectors to better move research to practice within workplaces. The work of the sectors is managed through a partnership structure of sector councils. Each council is co-led by a NIOSH co-chair and a co-chair from another organization. Most of the councils consist of approximately 1/3 NIOSH members with the rest from other organizations, including some from other Federal agencies. The corresponding NIOSH sector program provides logistical support for council meetings and activities. Each of the ten councils develops and maintains an agenda for the decade for its sector. The sector agendas become part of the national agenda for improvements in occupational safety and health through research and partnerships. Representing all stakeholders, the councils use an open process to set goals, develop strategies, encourage partnerships, and promote improved workplace practices.

NIOSH is conducting a management review of the second decade of NORA. This includes collecting data from NIOSH personnel about NIOSH program activities and NORA sector council activities, and NIOSH will also survey Federal employees who are or were leaders or members of a NORA sector council. In addition, we are seeking permission to open the same questionnaire to current and former council leaders and members who are not Federal employees. Permission is sought to collect the information over a period of no more than one year (12 months).

The web-based questionnaire requests information on satisfaction with the efficiency of the council and its processes, on impacts made in the sector during the second decade, and suggestions for improving the effectiveness and impact of NORA in the future.

The requested collection of information is necessary for NIOSH management to fully assess the efficiency and effectiveness of the NORA sector councils. Without this data collection, NIOSH’s internal management review of NORA would lack critical stakeholder input from its many non-Federal partners.

2. Purpose and Use of the Information Collection

The study is an internal, summative evaluation with a management-oriented approach. This approach will provide key decision makers with evaluation data that will help inform program improvement of the second decade of NORA (Fitzpatrick, Sanders & Worthen, 2004). Evaluation findings will be utilized by NIOSH leaders in the development of the third decade of NORA, which will begin in 2016. The evaluation results will be summarized and included in a comprehensive report in aggregate. A sub-set of evaluation summaries will be prepared for specific targeted audiences.

The purpose of the evaluation is to assess the effectiveness of the second decade of NORA, based on the program’s alignment with overall goals and objectives. This evaluation will include an assessment of the leveraging of resources to achieve outcomes related to addressing the most pertinent problems in the field of occupational safety and health.

The collection of information is necessary for NIOSH management to assess the efficiency and effectiveness of the NORA sector councils that were central to the work of the second decade of NORA. With this information, NIOSH will be able to structure the third decade of NORA to maximize impact. Without this data collection, NIOSH’s internal management review of NORA would lack critical stakeholder input from its many non-Federal partners. This is a one-time information collection effort. Future evaluation plans may include a similar effort during the next decade. No additional data collection is planned at this time.

3. Use of Improved Information Technology and Burden Reduction

100% of the responses will be through Survey Monkey and the internet. We know each respondent has access to the internet, because they routinely receive and respond to NORA sector council communications sent by email. In addition, the process of completing an online survey imposes only a small burden. This process consists of opening the invitation email, clicking on a link to their questionnaire, clicking on responses, having the ability to close the questionnaire and come back to it later and submitting the survey with a click. Such a process reduces the burden on the respondent compared to other survey administration approaches.

Additional efforts in designing the survey and its administration reduce the burden. Reminder emails will not be sent to those who have already responded. Correspondence about the survey is concise. As a result of pilot testing the instrument, the number of survey items is reduced to the minimum required to achieve the purpose of the review. Most responses are yes/no or multiple-choice. There are only two questions inviting narrative entries, and there is no limitation on response length.

4. Efforts to Identify Duplication and Use of Similar Information

NORA is a partnership program organized by NIOSH. There is no other review of NORA ongoing or planned within NIOSH. No other entity has identified itself as collecting information to review the National Occupational Research Agenda program during NORA sector council meetings, other NORA discussions or in the literature known to NIOSH. Some individual NORA sector councils have internal discussions periodically about how to improve their effectiveness and impact, but those discussions do not provide the type of information NIOSH management needs to evaluate the program overall and plan for the next decade of NORA.

5. Impact on Small Businesses or Other Small Entities

This data collection is intended for volunteer members of NORA sector councils. Some may be owners or employees of a small business, but the questions are only about their volunteer activities. The number of questions has been limited to the absolute minimum for the intended use of the data.

6. Consequences of Collecting the Information Less Frequently

This is a one-time information collection.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

This request fully complies with the regulation 5 CFR 1320.5.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

A. A 60-day Federal Register Notice was published in the Federal Register on May 27, 2014, vol. 79, No. 101, pp. 30147-8 (Appendix B). There were no public comments.

B. The scope and approaches for reviewing the second decade of NORA were discussed early in the review design process with the NIOSH Board of Scientific Counselors, a Federal Advisory Committee, and participants in a public meeting of NORA partners. Their input has helped shape the overall review, but no comments were received on the NORA sector council member questionnaire that is the subject of this request.

The following contact information is for the Chair of the NIOSH Board of Scientific Counselors:

Bonnie Rogers, M.P.H., Dr.P.H., C.O.H.N.-S.

Professor and Director of Occupational Safety and Health

University of North Carolina School of Public Health

E-mail: rogersb@email.unc.edu

Phone: (919) 966-1765

This is an internal management review of a NIOSH partnership program. NIOSH is using agency expertise to design the questionnaire and overall review process. The Program Evaluation Fellow who is leading the design and execution of the review has extensive evaluation training and experience and was not involved with the NORA partnership Program or NIOSH prior to joining CDC to lead this effort.

No one outside NIOSH was consulted on the details of this data collection. Instead, a group of NIOSH leaders of NORA sector councils was consulted, since they are also part of the cohort that will receive the questionnaire, albeit as Federal employees. This group contributed input on the clarity of the survey instructions, survey content and wording of the questions. They did not identify any major problems that could not be resolved. Their contact information follows:

Vern Anderson

Public Health Advisor

NIOSH

Email: vep1@cdc.gov

Phone: 513-533-8319

Pietra Check

Public Health Analyst

NIOSH

Email: gfe8@cdc.gov

Phone: 202-245-0660

Maryann D’Alessandro

Director, National Personal Protective Technology Laboratory

NIOSH

Email: bpj5@cdc.gov

Phone: 412-386-4033

9. Explanation of Any Payment or Gift to Respondents

Respondents will not receive any payments or gifts for participating.

10. Assurance of Confidentiality Provided to Respondents

The Privacy Act does not apply to this data collection, since no individually identifiable information is being collected.

The email providing the link to the questionnaire states the following regarding privacy and security of the information submitted:

Your participation is completely voluntary and there will be no adverse effects for you if you choose not to participate. You can discontinue participation at any time and choose whether to complete the survey at a later date. Your previous responses will be saved. Your responses will be secured and not disclosed unless otherwise compelled by law. All survey results will be reported in aggregate form, except for some narrative information that may be summarized, or quoted in whole or in part. All identifiers will be removed from narrative text. Completing and submitting this survey will imply your consent.

The requested information collection has been determined to be exempt from IRB review. See Appendix C for the IRB determination of non-research.

10.1 Privacy Impact Assessment Information

Overview of the Data Collection System

One web-based questionnaire is administered via a subscription with Survey Monkey (Appendices D and E). Respondents will receive:

• An introduction or pre-survey email one week prior to survey administration from a recognizable influential person, the NIOSH Director (Appendix F)

• A personalized email one week following the introductory email from the NIOSH Director, providing the link to the survey (Appendix G)

• A reminder email with the survey link to non-respondents one week later encouraging completion of the questionnaire (Appendix H)

• A final follow up email with the survey link to non-respondents one week later encouraging participation (Appendix I)

This standard web-based survey approach follows the Tailored Design Method of Dillman et al. (2009).

Items of Information to be Collected

All information will be collected in one web-based questionnaire with 16 items. The four general constructs of data to be collected are:

• Background

• Satisfaction with the effectiveness of the Council

• Impact of the Council

• Suggestions for how to structure the third decade of NORA.

The background section will ask respondents to identify which of the 10 councils the respondent is evaluating, whether the respondent was a NIOSH employee or associated with one of a number of other types of organizations while on the council and whether the respondent was a short-term or long-time member or co-leader of the council. The respondent’s satisfaction with the council and its leadership may depend on these characteristics, so having this information will improve NIOSH management’s ability to interpret the results and respond appropriately when planning the third decade of NORA.

In addition, the background section of the questionnaire asks for information about the approximate level of resources the respondent’s organization contributed to the council’s partnership activities as an annual average. Without this information, NIOSH management would only know the NIOSH resource contributions and could not assess the degree of leveraging of NIOSH funds to reach partnership goals.

The key information obtained from the questionnaire will come from the following section, which focuses on satisfaction with the effectiveness of the council in holding productive meetings and responding to the needs of the workers in the sector. Equally important is satisfaction with the effectiveness of the council leadership. NIOSH management will structure the third decade of NORA to improve in any areas of weakness that are identified.

In a separate internal questionnaire, NIOSH leaders of sector councils will be asked to identify the impact the council has had in improving the safety and health of the workers in the sector. However, it will be important to know whether other members of the council perceived an impact. Stakeholder members of the council may know of impacts achieved that were not known previously by NIOSH. The impacts section of the questionnaire will provide NIOSH management with valuable additional information.

As a partnership effort, NIOSH has always solicited the input of stakeholders when planning the next decade of NORA. The questions in the last section provide an opportunity for council members and leaders to provide input in an unstructured open-ended response format that is expected to be very helpful as NIOSH prepares for the third decade of NORA.

Appendix D contains a copy of the individual questions as a Word file. Appendix E includes copies of the Survey Monkey screenshots.

The data will be maintained by Survey Monkey until it is downloaded for analysis. The individual questionnaire responses will be coded and analyzed by the NIOSH staff and CDC Evaluation Fellows, under the supervision of senior leadership. Aggregate results will be shared in reports. Narratives provided by respondents may be summarized or quoted. All identifiers will be removed from narrative text before publication or distribution. Respondents are informed of the purpose and use of the survey data, and that completing and submitting the survey implies consent.

No individually identifiable information is being collected. It is possible that due to the small number of Council leaders who have served in some of the Sector Councils, it may be possible for some NIOSH staff to guess the identity of individual respondents based on their answers. Steps will be taken to make this very unlikely. Survey Monkey will not associate the email address of the respondent with the individual response data. The analysis of individual narrative data will be performed by CDC Evaluation Fellows who are not personally familiar with members of individual Sector Councils. They will remove any identifiers before quoting narrative text in draft reports.

How Information will be Shared and for What Purpose

The individual questionnaire responses will be viewed only by the CDC Evaluation Fellow and her supervisors. Aggregate data will be included in reports, except that individual narratives might be quoted after identifying information is removed.

A diverse range of products will originate from the evaluation of the second decade of NORA, which includes this information collection. Materials will include a lengthy formal report and shorter informal and user-friendly educational and marketing materials containing evaluation findings. In particular, products of this decade evaluation will include:

• Formal, comprehensive final printed reports

• Sector-specific reports

• Electronic information for easy download and quick messaging

• Bibliometrics of the second decade of NORA

• Short pieces (either group or focus area-specific)

• Stories, aural histories, and other evidence of impact

• Internal infrastructure for lasting performance evaluation across NIOSH.

In addition, NIOSH leaders will use the information in the reports as an important input when deciding how to structure the third decade of NORA. Will it be a sector-based structure, like the second decade, or a topic structure, like the first decade, or something else? Will NIOSH manage its interactions with NORA stakeholders through a similar structure of managers, coordinators and assistant coordinators or another approach? How can NIOSH improve its interaction with stakeholders to increase impact?

Impact of Proposed Collection on Respondent’s Privacy

No individually identifiable information is being collected. The email that provides the survey link includes that data will be secured and not shared beyond the data analysts unless compelled by law, and any identifying information will be removed before narrative information is summarized or quoted in a report.

Individuals Informed that Providing Information is Voluntary or Mandatory

Individuals are informed that providing the information is voluntary. The email that provides the survey link includes the information that participation is voluntary.

Opportunities to Consent

Individuals are informed that data will be presented in aggregate form. They are informed that completing and submitting the survey implies consent for NIOSH to summarize or quote narrative information after removing identifying information.

How Information will be Secured

Submitted information will be secured in a password-protected Survey Monkey account that is included in a subscription paid by NIOSH. After the information is downloaded in an Excel file for analysis, the Program Evaluation Fellow will password protect the file and store it only in government file space. The passwords will only be available to her supervisors and a second Program Evaluation Fellow.

Whether a System of Records is being Created under the Privacy Act

No individually identifiable information is being collected. A system of records is not being created under the Privacy Act.

11. Justification for Sensitive Questions

None of the questions requests sensitive information.

12. Estimates of Annualized Burden Hours and Costs

A. Estimates of the Hour Burden of the Collection of Information

A 16 item questionnaire has been developed and will be sent to all NORA sector council members or leaders. Some members are NIOSH or other Federal employees. Only those who are not Federal employees are included in the burden estimates.

A pilot test of the questionnaire was conducted by asking 8 NIOSH employees who are a leader of a NORA sector council to complete the questionnaire and provide feedback. Respondents to the pilot test estimated the questionnaire requires approximately 15 minutes to complete.

Type of Respondent	Form Name	Number of Respondents	Number of Responses per Respondent	Average Burden per Response (in Hours)	Total Burden (in Hours)
Non-federal NORA Sector Council members or leaders	Council Questionnaire	352	1	15/60	88
Total					88

B. Estimates of Annualized Cost to Respondents for the Burden Hours for Collections of Information

The following table uses the Bureau of Labor Statistics Occupational Employment and Wages results for May 2013. The respondents are typically senior occupational safety and health professionals who have moved into program management positions. The estimated hourly wage rate used is the 75^th percentile for the category Natural Sciences Managers (11-9121).

Type of Respondents	Form Name	Total Burden Hours	Hourly Wage Rate	Total Respondent Costs
Non-federal NORA Sector Council members or leaders	Council Questionnaire	88	$77.15	$6,789
	Total			$6,789

13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

1. There is no capital or start-up costs for collection information in this project.

2. There is no cost to respondents for operation or maintenance in this project. Respondents are not asked or expected to purchase any services.

14. Annualized Cost to the Federal Government

Costs for conducting the survey are summarized in Table A1. The request is for one year, so all costs are automatically annualized costs. There will be no new overhead, support staff, or construction required for the survey administration and data analysis.

Table A1. Annualized cost to the Federal Government

Personnel--1 Evaluation Fellow, 15% time	$8,848
Personnel--1 Evaluation Fellow, 4% time	$7,871
Personnel--1 GS-15, 5% time	$2,400
Survey Monkey subscription	$ 780
Total of Annualized estimate of federal cost	$19,899

15. Explanation for Program Changes or Adjustments

This is a new data collection.

16. Plans for Tabulation and Publication and Project Time Schedule

Project Time Schedule
Activity	Time Schedule
Pre-survey email sent to respondents	1 week after OMB approval
Data collection ends	5 weeks after OMB approval
Data downloaded and validation completed	9 weeks after OMB approval
Data analyses completed	15 weeks after OMB approval
NIOSH review of draft report completed	20 weeks after OMB approval
Report reviewed and revisions incorporated	29 weeks after OMB approval
Final document prepared and released to the public	36 weeks after OMB approval

All quantitative material in this study, including the council survey, will be organized in a table format in Microsoft Excel for data analysis and analyzed using descriptive statistics (frequency count, ranges, mean [X] and standard deviation [sd]). Means and standard deviations will be reported for satisfaction questions, which have a numerical score response, while nominal survey data such as demographic information, multiple choice items, and questions with a dichotomous “yes/no” response will be depicted through frequency counts.

Qualitative data will be recorded, transcribed and organized using qualitative analysis software. Data will be analyzed using qualitative content analysis (QCA). This approach involves exploring multiple pieces of qualitative information (e.g., narrative) to develop general themes of the data, determined by research questions or study objectives (Schreier, 2012). A pre-determined code book, based on the research questions, will be developed for standardized analysis of qualitative responses.

Both the quantitative and qualitative survey data will be summarized overall and by each of the 10 NORA sectors. Qualitative narrative will be highlighted for each sector, featuring impact stories and other accomplishments.

17. Reason(s) Display of OMB Expiration Date is Inappropriate

The display of the OMB expiration date is not inappropriate.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification.

References

Dillman, D., Smyth, J. & Christian, L. (2009). Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method, New York: Wiley.

Fitzpatrick, J. L., Sanders, J. R. & Worthem, B. R. (2004). Program evaluation: Alternative approaches and practical guidelines. Boston, MA: Pearson.

Manfreda, K. L., Bosnjak, M., Berzelak, J., Haas, I., & Vehovar, V. (2008). Web surveys versus other survey modes A meta-analysis comparing response rates. International Journal of Market Research, 50(1), 79-104.

Schreier, M. (2012). Qualitative content analysis in practice. Thousand Oaks, CA: Sage.

Vehovar, V., & Manfreda, K. L. (2008). Overview: Online surveys. In N. G. Fielding, R. M. Lee, & G. Blank (Eds.). The handbook of online research methods. (pp. 177-194). Thousand Oaks, CA: Sage.