0990-Evaluation of the IT Professionals in Health Care (Workforce) Program - OMB 040411

Department of Health & Human Services
Office of the National Coordinator for Health Information Technology
Evaluation of the IT Professionals in Health Care Program:

University-Based Training

Supporting Statement

A. Justification

A.1. Circumstances That Make the Collection of Information Necessary

The Office of the National Coordinator for Health Information Technology’s Office of Economic Analysis, Evaluation, and Modeling in the Department of Health & Human Services requests OMB approval for the evaluation of the IT Professionals in Health Care (“Workforce”) Program. The Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 seeks to improve American health care delivery and patient care through an unprecedented investment in health information technology. The provisions of the HITECH Act are specifically designed to work together to provide the necessary assistance and technical support to providers, enable coordination and alignment within and among states, establish connectivity to the public health community in case of emergencies, and assure the workforce is properly trained and equipped to be meaningful users of EHRs. Combined, these programs build the foundation for every American to benefit from an electronic health record, as part of a modernized, interconnected, and vastly improved system of care delivery.

The Workforce Program, created under Section 3016 of the Public Health Service Act (PHSA), as added by Title XIII in Division A of the American Recovery and Reinvestment Act of 2009, directed the Secretary of Health and Human Services to provide “assistance to institutions of higher education (or consortia thereof) to establish or expand medical health informatics education programs, including certification, undergraduate, and masters degree programs, for both health care and information technology students to ensure the rapid and effective utilization and development of health information technologies.” Funded by ONC to support the development of a robust workforce dedicated to the rapid and widespread adoption and use of health information technology (IT), the Workforce Program includes four components:

Community College Consortia (CCC) to Educate Information Technology Professionals in Health Care. Provides funds to a total of 84 community colleges in five consortia to establish intensive certificate courses, designed to be completed within six months.

Program of Assistance for University-Based Training (UBT). Provides funds to nine universities to target health care roles that require a higher level of training by creating or expanding training programs, most of which can be completed by enrolled students within a year or less.

Curriculum Development Centers. Funds five institutions to develop health IT curricula and educational materials aimed primarily at the community-college level.

Competency Examination for Individuals Completing Non-Degree Training. Supports the development and initial administration of competency exams in health IT.

In support of the Workforce Program, ONC’s Division of Evaluation is conducting a separate program evaluation study. The Workforce Program is likely to face significant challenges, including integrating evolving and newly developed curricula, recruiting and training faculty and prospective students, coordinating among the four program components and coordinating with other HITECH-funded efforts (such as Health Information Technology Regional Extension Centers) and other prospective employers of students trained through the program. The Workforce Program evaluation is a new information collection activity that will explore these challenges through both formative and summative evaluation approaches, provide critical formative feedback to the grantee institutions on their activities, and offer perspectives on the Program’s contributions in helping to build a skilled workforce equipped to meet the needs of employers. While current plans for this evaluation do not include a comprehensive, quantitative analysis of employers’ opinions about the Program, they do include informal discussions with selected employers in a number of markets/regions to gauge their familiarity with the Program and their exposure to its graduates. If it is determined at a later date that a different approach is necessary, an amendment may be offered that modifies the evaluation plan accordingly.

The overarching goals of the evaluation are: 1) to support improvements as the program evolves, 2) to assess the success of each of the Program’s components, and 3) to assess the success of the Program overall. Specifically, the evaluation will:

Identify best-practice models and approaches to the development of a well-trained and highly qualified workforce.

Provide insight into the adequacy of the Workforce Program to support the necessary scale of training and to target employer needs.

Generate real-time feedback and analyses that can be used by ONC and the awardees themselves in assessing and improving Workforce Program offerings.

Provide a valid and comprehensive assessment of Workforce Program and its grantees’ success in establishing and expanding health informatics education programs to ensure the rapid and effective utilization and development of health information technologies.

More specifically, the evaluation will consist of the following data collection efforts:

Web-Based Survey of Community College Students. This data collection consists of two surveys: a baseline survey completed shortly before or after the conclusion of the students’ enrollment in Workforce courses (or, if they dropped out, six-months after enrolling in the program); and a follow-up survey that will be administered six months after the baseline.

Baseline Survey. The purpose of the Web-based baseline survey of students is to collect information on community college students’ experiences in the program, including students’ attitudes and satisfaction with the learning environment, perceptions about work/skill readiness, students’ motivation for entering the program and the health IT profession, students’ level of engagement with faculty, satisfaction with support systems available within and outside of the college environment, areas for program improvement, and awareness of the competency exam. Baseline surveys will be completed shortly before or after the conclusion of the students’ enrollment in Workforce courses.
Follow-up Survey. The purpose of the Web-based follow-up survey of students is to collect information on student outcomes, including their experience of the competency exam, experiences and success in obtaining a health IT job, and their opinions as to how the Workforce Program prepared them to enter the health IT field. Students will be surveyed using the follow-up instrument six months after completing the baseline survey.

Web-Based Survey of University Students. This survey will be sent to students at the approximate time they are expected to complete their certificate or degree program at the university. The purpose of the survey is to learn about how prepared students feel to join the health IT workforce. It will also collect information on post-graduation employment. Because of the different structure of the community college and university programs, the university student survey instrument will be different than the one used with community college students.

Baseline Survey. Similar to the Survey of Community College Students, the purpose of the Web-based baseline survey of university students is to collect information on students’ experiences in the program, including opinions on the learning environment, perceptions about work/skill readiness, students’ motivation for entering the program and the health IT profession, and areas for program improvement. Baseline surveys will be completed shortly before or after the conclusion of the students’ enrollment in Workforce courses.
Follow-up Survey. The purpose of the Web-based follow-up survey of students is to collect information on student outcomes, including their experiences and success in obtaining a health IT job and their opinions as to how well the Workforce Program prepared them for the health IT field. Students will be surveyed using the follow-up instrument six months after completing the baseline survey.

Course Evaluation Forms. Course evaluation forms will be available to the community colleges to distribute at the end of each community college course to all students enrolled in the course. The purpose of the evaluation forms is to collect students’ opinions and attitudes about the materials in individual courses. This information will provide critical information on the relevance of course materials and how effective the materials developed by the curriculum development grantees were in preparing students for work in the health IT sector.
Focus Groups. We plan to hold small focus groups with distinct stakeholders: students, faculty members, and individuals who have taken the competency exam. Our discussions with students will take place during unstructured site visits to up to eight colleges. The site visits, and therefore the focus groups, will occur within each year for up to three years. The focus groups will explore their perceptions of the quality of instruction, motivations for enrolling in the program, and plans for taking (or not taking) the competency exam. This information will be used to provide feedback to programs about how to better recruit students, which classes are most useful to the students, and how the program can be enhanced to better train students. Focus groups with faculty members will be conducted on-line, and we will discuss how they are using the curricula developed as part of the Workforce program and whether they have identified any gaps in the available materials. This information will provide relevant feedback to the grantees developing the curriculum to help "tweak" their materials to better meet the needs of the programs. Focus groups with faculty within the five consortia will occur once per year for the almost three years of the project. We will also conduct on-line focus groups with individuals who took the exam, but did not attend one of the ONC-funded training programs. These focus groups will occur annually for three years. The questions for this group will seek to understand their motivations for taking the exam and their perception of the value of the credential in seeking employment in the field.
Web-Based Survey of Community College Faculty. The purpose of the Web-based survey of faculty is to collect information on instructors’ opinions of the curriculum developed, the extent to which the instructor adhered to the curriculum, and their impressions of the implementation of the program at their institution. The University-Based instructors will be using their own curriculum rather than the curricula and educational materials developed for the community colleges, and, therefore, will not be surveyed.

This OMB package is for approval of the baseline and follow-up surveys of university students. The baseline and follow-up community college student surveys, course evaluation forms, focus group protocols, and faculty survey received approval as part of a previously submitted OMB package.

In addition to understanding the average outcomes of UBT program participants by surveying the first and second year graduates of the program, ONC is interested in understanding the outcomes of students that graduate in the third year of the program. It is theorized that labor market needs and conditions may evolve during the funding of this program, especially when eligible providers are focused on trying to meet Stage 2 and Stage 3 Meaningful Use requirements. Therefore, ONC is interested in learning if students have significantly different outcomes across the program years and why. However, in order to conduct these comparisons, ONC would need to survey the third year graduates in 2013. For this reason, ONC is currently exploring the possibility of adding a third year to the data collection and potentially extending this project into 2014. If such a modification occurs, an additional OMB package for surveying the third year graduates will be submitted to address research questions that explore the differences between the three years of graduates.

A.2. Purpose and Use of Information Collection

ONC is at the forefront of the administration’s health IT efforts and is a resource to the entire health system to support the adoption of health information technology and the promotion of nationwide health information exchange to improve health care. ONC is the principal Federal entity charged with coordination of nationwide efforts to implement and use the most advanced health information technology and the electronic exchange of health information.

This evaluation is designed to contribute to a comprehensive understanding of the planning, implementation, and effectiveness of the Workforce Grant Program. The evaluation will determine how the Workforce Grant Program has contributed to the development of comprehensive, integrated health IT training programs across community colleges, universities, and other programs. In order to answer this overarching question, the evaluation will focus on identifying results of the established key evaluation questions provided in Exhibit 1.

Exhibit 1: Key Evaluation Questions

To what extent did the grantees meet the requirements of the Workforce program? (e.g., implementing new educational programs, matriculating and training the expected number of students, developing adequate curriculum materials, and developing and administering a competency exam)
What processes did the grantees use to implement the programs and meet program goals? (e.g., barriers, lessons learned, successful strategies, coordination, program satisfaction)
To what extent did participants in the Program gain and maintain employment in health IT? (e.g., job placement, job retention, salary, promotion, job readiness, employer needs)

ONC anticipates that the findings from the evaluation will be used for continuous improvement of the Workforce Program and inform ONC on the progress of this program in training and introducing more health IT workers into the labor force.

A.3. Use of Improved Information Technology and Burden Reduction

This study will rely on data gathered from self-administered, Web-based baseline and follow-up surveys of students enrolled and/or having completed a Workforce program. We assume individual respondents will have access to and be familiar with the necessary technology to complete the surveys due to the fact that they are entering the information technology field. Surveys will be administered electronically to alleviate burden on the respondents. The Web-based survey permits respondents to complete each instrument at their preferred time. Respondents who begin the survey and are unable to complete it in one attempt will be able to save their responses and resume work on the survey at a later time. The Web-based format will incorporate skip patterns that ensure that respondents automatically skip past sections of the survey that are not relevant to their experiences.

The study will have a centralized case management system (CMS), linked to the Web survey as well as the locating, prompting, and receipt control systems, which will allow for the review of case status at any time. This will allow for effective follow-up, including ensuring no sample member is prompted for a survey response once they have completed the Web survey. Depending on the type of contacting information available, all cases will be mailed or emailed an invitation letter with Web survey access, including a unique Personal Identification Number (PIN) and password. This initial contact will be followed-up with additional emails encouraging participation and eventually with telephone prompting. By indicating complete/incomplete status of the Web survey, the CMS will allow for prompting efforts to be directed only at non-respondents. Also, in order to maintain contact with students and learn updated contacting information, there will be email and/or USPS mail contacts between the baseline and follow-up surveys.

A.4. Efforts to Identify Duplication

The information needed for this evaluation is not routinely recorded in any electronic information medium that could be adapted to obtain the data required to address the research objectives of the evaluation. This data collection is part of an evaluation of a new Workforce program, so no students have been enrolled in the program and the program has not yet been evaluated. To avoid duplication, ONC has engaged in discussions with grantees to ensure that these data are not being collected by other means. ONC will continue to collaborate with the grantees to make sure there will not be duplication in the future.

The data requested for this evaluation are unique to this Workforce program. Given our thorough review of existing data, this information is not available elsewhere. No other data are currently being collected to answer the research questions about the Workforce program. However, existing data can and will be used whenever possible.

A.5. Impact on Small Businesses or Other Small Entities

No small businesses are involved as respondents in the proposed data collection effort. Respondents are all students who are associated with the Workforce program.

A.6. Consequences if Information Collected Less Frequently

Without collecting these data, ONC will not have access to a comprehensive assessment of the current planning and implementation of the Workforce Program. Specifically, the evaluation will help determine if the funding provided through the program is helping to develop a health IT workforce capable of meeting the growing demand for skilled employees. The evaluation will be gathering information from students throughout the three-year span of the evaluation in order to be able to determine which factors contribute to successful training programs and to be able to feed that information back to the program and grantees in a timely fashion to foster continuous quality improvement.

It will be critically important to survey students at baseline and follow-up. The baseline survey will allow the evaluation to capture students’ impressions of their programs immediately following their completion of the program. The follow-up survey will provide an opportunity to learn about the employment outcomes of the students and whether the training they received as part of the Workforce Program was sufficient to prepare them for the labor market.

A.7. Consistency with Guidelines of 5 CFR 1320.8(d)

This data collection request is fully consistent with the guidelines in 5 CFR 1320.8(d). There are no special circumstances required for the collection of information in this data collection.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

A. In accordance with the paperwork Reduction Act of 1995, the notice required in 5 CFR 1320.8(d) has been published in the Federal Register announcing ONC’s intention to request an OMB review of data collection activities. This notice was published on January 27, 2011 in volume 76, number 18, on pages 4906 - 4907 and provided a 60-day period for public comments. There were no public comments submitted to ONC in response to the Federal Register notice.

The student surveys were developed by ONC’s contractor, the National Opinion Research Center (NORC) at the University of Chicago, with input from the grantees of the Workforce program.

Since January 2010, the Agency has consulted with the following persons regarding this information collection.

Michael Furukawa

Director, Evaluation Division

Office of Economic Analysis, Evaluation, and Modeling, ONC

U.S. Department of Health and Human Services

Phone: (202) 690-3950 Email: michael.furukawa@hhs.gov

Vaishali Patel

Senior Researcher

Economic Analysis, Evaluation, and Modeling, ONC

U.S. Department of Health and Human Services

Phone: (202) 690-3912 Email: vaishali.patel@hhs.gov

A.9. Explanation of any Payment or Gift to Respondents

No payments or gifts will be offered for participation in the university student baseline or follow-up surveys.

A.10. Assurance of Confidentiality Provided to Respondents

Participation in any study is voluntary. Responses by individuals will be kept private to the extent allowed by law (Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c)). Students will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose. Data will be collected in conformance with the Privacy Act of 1974. Invitation letters sent to the selected sample members will advise them that the information they provide is confidential. Similar information will appear on the introduction screen of the student Web.

The data collection plan has been reviewed and approved by the Institutional Review Board of ONC’s contractor, NORC. Data collection procedures will incorporate numerous safeguards for the data. While collecting Workforce data, information that could identify a particular sample member will be stored in a separate file from survey data collected from that person. Each sample member will be assigned a unique identifier, and this identifier will be used to store identifying information (such as name, address, etc.) in a separate database from the survey response data.

The electronic systems for data collection and data storage are on a local area network (LAN). All systems used to store electronic survey data are secure by design and protected by passwords only available to authorized study staff. Special steps will be taken to ensure that data collected via the Web questionnaire are secure. First, access to the Web instrument is only allowed with a valid Personal Identification Number (PIN) and password correctly entered in combination. Second, data will be transmitted by the Secure Sockets Layer (SSL) protocol that uses powerful encryption during transmission through the Internet.

If a respondent keeps a Web survey open without any activity, the Web server will close the survey after a short period of inactivity, thus preserving the data up to the break-off point and securely closing the connection. The Web system architecture process has been designed in a way that places authentication information and response data on physically separate servers. This strategy provides an extra layer of security to protect response data. Both development and production servers are backed up nightly. All data and analysis are reported in aggregate form only and measures are taken so that the identity of individuals or organizations is not disclosed.

A.11. Justification for Sensitive Questions

The university student baseline survey will ask students to self-identify their race and ethnicity using the federally approved questions on the U.S. Census. This question is necessary in order to conduct subgroup analyses and to understand the characteristics of students enrolling in courses and entering the health IT field.

No other questions of a sensitive nature are asked in this data collection.

A.12. Estimates of Annualized Hour Burden and Costs

ONC estimates contact with up to 520 university students to complete the baseline Web survey in the first year of this data collection effort and 634 students in the second year. These same students will be extended the opportunity to complete a follow-up survey six months after graduating from the program. The total burden for the individual is estimated at 20 minutes, based on experience with similar instruments. In addition, the Web instrument was pre-tested on a group of six university students. For both the baseline and follow-up surveys, not all questions will be asked of each respondent. Different sets of questions will be administered to a respondent depending on his/her background and level of work experience in health IT. Each data collection instrument includes flow charts demonstrating which questions will be asked of each type of respondent. See section B.1 for more detail.

Exhibit 2 presents estimates of the reporting burden for respondents. A census of university students will be approached to participate in the baseline and follow-up surveys. The expected response rates are 80%. Please see Exhibit 2 for the breakdown of the burden by student type. The total cost to respondents for the hour burden is estimated to be $20.90 per hour for students. Because we assume a diverse group of individuals will enroll in these health IT programs, the student estimate is based on the mean hourly wage of all occupations in the U.S. according to the Bureau of Labor Statistics of $20.90.

If a lower than expected response rate results, we will conduct nonresponse bias tests to determine if any bias resulted from the lower response rate. If these tests provide evidence of bias, we will make adjustments to our results with the use of weight adjustments and/or response imputation.

The following table identified the annualized burden estimate:

Exhibit 2: Estimated Burden Hours

Forms (If necessary)	Type of Respondent	Number of Respondents per year	Number of Responses per Respondent	Average Burden hours per Response	Total Burden Hours	Cost per hour	Total Cost For All Respondents
Web-based UBT Student Baseline Survey	Students enrolled in university-based Workforce program	634	1	20/60	211	$20.90	$4409.90
Web-based UBT Student Follow-up Survey	Students enrolled in university-based Workforce program	634	1	20/60	211	$20.90	$4409.90
Total					422		$8,819.90

A.13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

There are no annualized capital/startup or ongoing operation and maintenance costs involved in collecting the information. Other than their time to complete the surveys, which is estimated in Exhibits 2 and 3, there are no direct monetary costs to respondents.

A.14. Estimates of Annualized Costs to the Federal Government

The estimated cost to the Federal Government for the Evaluation of Information Technology Professionals in Health Care Program data collection activities is $310,100. This is the cost to our Federal contractor, NORC, for data collection activities associated with this submission.

A.15. Explanation for Program Changes or Adjustments

No change in burden is requested. This submission to OMB is for an initial request for approval.

A.16. Plans for Tabulation and Publication and Project Time Schedule

Data for the evaluation will be collected in each of the three years of the evaluation. Analysis will begin shortly after data are collected from the Web-based surveys, and will continue periodically throughout the project period to produce annual reports, as well as a final report in July 2013 upon completion of the evaluation. Exhibit 3 provides an estimated schedule of data collection for the Web-Based Survey, contingent on receiving OMB approval. These dates are based on the estimated program start schedule and may change slightly.

Exhibit 3: Timetable for Data Collection for Web-Based Surveys

Cohort	Program Completion Date(s)	Baseline Survey		Follow-up Survey
Cohort	Program Completion Date(s)	Estimated Start Date	Estimated End Date	Estimated Start Date	Estimated End Date
Cohort 1 (N=413)	June and August 2011	July 2011	September 2011	January 2012	March 2012
Cohort 2 (N=107)	December 2011	December 2011	February 2012	June 2012	August 2012
Cohort 3 (N=634)	June and August 2012	July 2012	September 2012	January 2013	March 2013

Our interim and final reports will provide an analysis of the quantitative data in the student surveys, as well as qualitative information gathered through previously OMB-approved data collections, including site visits and focus groups.

Quantitative analysis will primarily be descriptive. We will provide aggregate descriptions of the following types of information:

Students’ satisfaction with the learning environment (e.g., with faculty/courses);
Students’ perceptions about work/skill readiness; and
Employment outcomes of students who complete the educational programs.

In addition to producing descriptive statistics of the measures associated with student satisfaction and employment outcomes and conducting longitudinal analyses of how key measures vary over time, we will analyze the data by student race/ethnicity, gender, enrollment status, and curriculum focus. We will also run tests of association on such variables as student satisfaction with training programs and their employment outcomes.

A.17. Exception for Display of Expiration Date

All data collection materials will display the OMB expiration date.

A.18. Certifications

ONC certifies that the collection of information encompassed by this request complies with 5 CFR 1320.9 and the related provisions of 5 CFR 1320.8(b)(3).

Page | 4

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Patrick Moulding
File Modified	0000-00-00
File Created	2021-02-01