0990-Evaluation of IT Professionals In Health Care Program for HITECH ACT - Supporting Statements (2)

Department of Health & Human Services
Office of the National Coordinator for Health Information Technology
Evaluation of the IT Professionals in Health Care Program

Supporting Statement

A. Justification

A.1. Circumstances That Make the Collection of Information Necessary

The Office of the National Coordinator for Health Information Technology’s Office of the Chief Scientist in the Department of Health & Human Services requests OMB approval for the evaluation of the IT Professionals in Health Care (“Workforce”) Program. The Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 seeks to improve American health care delivery and patient care through an unprecedented investment in health information technology. The provisions of the HITECH Act are specifically designed to work together to provide the necessary assistance and technical support to providers, enable coordination and alignment within and among states, establish connectivity to the public health community in case of emergencies, and assure the workforce is properly trained and equipped to be meaningful users of EHRs. Combined, these programs build the foundation for every American to benefit from an electronic health record, as part of a modernized, interconnected, and vastly improved system of care delivery.

The Workforce Program, created under Section 3016 of the Public Health Service Act (PHSA), as added by Title XIII in Division A of the American Recovery and Reinvestment Act of 2009, directed the Secretary of Health and Human Services to provide “assistance to institutions of higher education (or consortia thereof) to establish or expand medical health informatics education programs, including certification, undergraduate, and masters degree programs, for both health care and information technology students to ensure the rapid and effective utilization and development of health information technologies.” Funded by ONC to support the development of a robust workforce dedicated to the rapid and widespread adoption and use of health information technology (IT), the Workforce Program includes four components:

Community College Consortia (CCC) to Educate Information Technology Professionals in Health Care. Provides funds to a total of 84 community colleges in five consortia to establish intensive certificate courses, designed to be completed within six months.

Program of Assistance for University-Based Training (UBT). Provides funds to nine universities to target health care roles that require a higher level of training by creating or expanding training programs, most of which can be completed by enrolled students within a year or less.

Curriculum Development Centers. Funds five institutions to develop health IT curricula and educational materials aimed primarily at the community-college level.

Competency Examination for Individuals Completing Non-Degree Training. Supports the development and initial administration of competency exams in health IT.

In support of the Workforce Program, ONC’s Division of Evaluation is conducting a separate program evaluation study. The Workforce Program is likely to face significant challenges, including integrating evolving and newly developed curricula, recruiting and training faculty and prospective students, coordinating among the four program components and coordinating with other HITECH-funded efforts (such as Health Information Technology Regional Extension Centers) and other prospective employers of students trained through the program. The Workforce Program evaluation is a new information collection activity which will explore these challenges through both formative and summative evaluation approaches, providing critical formative feedback to the grantee institutions on their activities, and determining whether the Workforce Program overall was successful in helping to build a skilled workforce equipped to meet the heightened demands of the current environment.

The overarching goals of the evaluation are: 1) to support improvements as the program evolves, 2) to assess the success of each of the Program’s components, and 3) to assess the success of the Program overall. Specifically, the evaluation will:

Identify best-practice models and approaches to the development of a well-trained and highly qualified workforce.

Provide insight into the adequacy of the Workforce Program to support the necessary scale of training and to target employer needs.

Generate real-time feedback and analyses that can be used by ONC and the awardees themselves in assessing and improving Workforce Program offerings.

Provide a valid and comprehensive assessment of Workforce Program and its grantees’ success in establishing and expanding health informatics education programs to ensure the rapid and effective utilization and development of health information technologies.

More specifically, the evaluation will consist of the following data collection efforts:

Web-Based Survey of Community College Students. This data collection consists of two surveys: a baseline survey completed shortly before or after the conclusion of the students’ enrollment in Workforce courses (or, if they dropped out, 6-months after enrolling in the program); and a follow-up survey that will be administered between 6 and 8 months after graduation.

Baseline Survey. The purpose of the Web-based baseline survey of students is to collect information on community college students’ experiences in the program, including students’ attitudes and satisfaction with the learning environment, perceptions about work/skill readiness, students’ motivation for entering the program and the health IT profession, students’ level of engagement with faculty, satisfaction with support systems available within and outside of the college environment, areas for program improvement, and awareness of the competency exam. Baseline surveys will be completed shortly before or after the conclusion of the students’ enrollment in Workforce courses.
Follow-up Survey. The purpose of the Web-based follow-up survey of students is to collect information on student outcomes, including their experience of the competency exam, experiences and success in obtaining a health IT job, and their opinions as to how the Workforce Program prepared them to enter the health IT field. Students will be surveyed using the follow-up instrument between 6 and 8 months after completing the baseline survey.

Web-Based Survey of University Students. This survey will be sent to students after they complete their certificate or degree programs at the university. The purpose of the survey is to learn about how prepared students feel to join the health IT workforce. It will also collect information on post-graduation employment. Because of the different structure of the community college and university programs, the university student survey instrument will be different than the one used with community college students and will be administered starting later in the course of the evaluation and therefore is not included in this package.
Course Evaluation Forms. Course evaluation forms will be distributed at the end of each community-college course to all students enrolled in the course. The purpose of the evaluation forms is to collect students’ opinions and attitudes about the materials in individual courses. This information will provide critical information on the relevance of course materials, whether the products developed by the curriculum development grants were utilized, and how effective the materials were in preparing students for work in the health IT sector.
Focus Groups. We plan to hold small focus groups with distinct stakeholders: students, faculty members, and individuals who have taken the competency exam. Our discussions with students will take place during unstructured site visits to up to eight colleges. The site visits, and therefore the focus groups, will occur within each year for up to three years. The focus groups will explore their perceptions of the quality of instruction, motivations for enrolling in the program, and plans for taking (or not taking) the competency exam. This information will be used to provide feedback to programs about how to better recruit students, which classes are most useful to the students, and how the program can be enhanced to better train students. Focus groups with faculty members will be conducted on-line, and we will discuss how they are using the curricula developed as part of the Workforce program and whether they have identified any gaps in the available materials. This information will provide relevant feedback to the grantees developing the curriculum to help "tweak" their materials to better meet the needs of the programs. Focus groups with faculty within the five consortia will occur once per year for the almost three years of the project. We will also conduct on-line focus groups with individuals who took the exam, but did not attend one of the ONC-funded training programs. These focus groups will occur annually for three years. The questions for this group will seek to understand their motivations for taking the exam and their perception of the value of the credential in seeking employment in the field.
Web-Based Survey of Community College Faculty. The purpose of the Web-based survey of faculty is to collect information on instructors’ opinions of the curriculum developed, the extent to which the instructor adhered to the curriculum, and their impressions of the implementation of the program at their institution. The University- Based instructors will be using their own curriculum rather than the curricula and educational materials developed for the community colleges, and, therefore, will not be surveyed.

This OMB package is for approval of the baseline community college student survey, course evaluation forms, focus group protocols, and faculty survey. ONC will submit an additional package for approval of the follow-up community college student survey and the survey of university students at a later date.

A.2. Purpose and Use of Information Collection

ONC is at the forefront of the administration’s health IT efforts and is a resource to the entire health system to support the adoption of health information technology and the promotion of nationwide health information exchange to improve health care. ONC is the principal Federal entity charged with coordination of nationwide efforts to implement and use the most advanced health information technology and the electronic exchange of health information.

This evaluation is designed to contribute to a comprehensive understanding of the planning, implementation, and effectiveness of the Workforce Grant Program. The evaluation will determine how the Workforce Grant Program has contributed to the development of comprehensive, integrated health IT training programs across community colleges, universities, and other programs. In order to answer this overarching question, the evaluation will focus on identifying results of the established key evaluation questions provided in Exhibit 1.

Exhibit 1: Key Evaluation Questions

1. The extent to which the grantee institutions met the expectations of the Workforce Program in creating training programs capable of generating a health IT workforce equipped to meet the demands of the current environment

2. Whether the five Community College Consortium, the Curriculum Development Centers, and the Competency Exam developers met their respective responsibilities in a spirit of collaboration with one another to support the overarching goals of the Program

3. The extent to which the training programs met the needs and expectations of those involved (including students, faculty, employers, and IT professionals electing to take the competency exam.)

ONC anticipates that the findings from the evaluation will be used for continuous improvement of the Workforce Program and inform ONC on the progress of this program in training and introducing more health IT workers into the labor force.

A.3. Use of Improved Information Technology and Burden Reduction

This study will rely on data gathered from self-administered, Web-based baseline and follow-up surveys of students enrolled and/or having completed a Workforce program and of their instructors (Please see our sampling plan in Section B.1 for more detail). We assume individual respondents will have access to and be familiar with the necessary technology to complete the surveys due to the fact that they are entering the information technology field. Surveys will be administered electronically to alleviate burden on the respondents. The Web-based survey permits respondents to complete each instrument at their preferred time. Respondents who begin the survey and are unable to complete it in one attempt will be able to save their responses and resume work on the survey at a later time. The Web-based format will incorporate skip patterns that ensure that respondents automatically skip past sections of the survey that are not relevant to their experiences.

The study will have a centralized case management system (CMS), linked to the Web survey as well as the locating, prompting, and receipt control systems, which will allow for the review of case status at any time. This will allow for effective follow-up, including ensuring no sample member is prompted for a survey response once they have completed the Web survey. Depending on the type of contacting information available, all cases will be mailed or emailed an invitation letter with Web survey access, including a unique Personal Identification Number (PIN) and password. This initial contact will be followed-up with additional emails encouraging participation and eventually will be moved into telephone prompting. By indicating complete/incomplete status of the Web survey, the CMS will allow for prompting efforts to be directed only at non-respondents. Also, in order to maintain contact with students and learn updated contacting information, there will be email and/or USPS mail contacts between the baseline and follow-up surveys.

Focus group members will be recruited via email. All students, faculty, and test examiners for whom we have email addresses supplied by the programs will be mailed or emailed an invitation letter. If they respond that they are interested in participating, there will be follow up on this initial contact with additional information on logistics for attending a focus group. We will also be conducting a series of focus groups with faculty members and exam takers using on-line technology to alleviate burden for participants and to avoid the need for individuals to travel to participate in the discussions.

A.4. Efforts to Identify Duplication

The information needed for this evaluation is not routinely recorded in any electronic information medium that could be adapted to obtain the data required to address the research objectives of the evaluation. This data collection is part of an evaluation of a new Workforce program, so no students have been enrolled in the program and the program has not yet been evaluated. To avoid duplication, ONC has engaged in discussions with grantees to ensure that these data are not being collected by other means. ONC will continue to collaborate with the grantees to make sure there will not be duplication in the future.

The data requested for this evaluation are unique to this Workforce program. Given our thorough review of existing data, this information is not available elsewhere. No other data are currently being collected to answer the research questions about the Workforce program. However, existing data can and will be used whenever possible.

A.5. Impact on Small Businesses or Other Small Entities

No small businesses are involved as respondents in the proposed baseline data collection effort. Respondents are all students, faculty members, and exam takers who are associated with the Workforce program.

A.6. Consequences if Information Collected Less Frequently

Without collecting these data, ONC will not have access to a comprehensive assessment of the current planning and implementation of the Workforce Program. Specifically, the evaluation will help determine if the funding provided through the program is helping to develop a health IT workforce capable of meeting the growing demand for skilled employees. The evaluation will be gathering information from students and faculty members throughout the three-year span of the evaluation in order to be able to determine which factors contribute to successful training programs and to be able to feed that information back to the program and grantees in a timely fashion to foster continuous quality improvement.

With regard to the student survey, it will be critically important to survey students at baseline and follow-up. The baseline survey will allow the evaluation to capture students’ impressions of their programs immediately following their completion of the program. The follow-up survey will provide an opportunity to learn about the employment outcomes of the students and whether the training they received as part of the Workforce Program was sufficient to prepare them for the labor market.

A.7. Consistency with Guidelines of 5 CFR 1320.8(d)

This data collection request is fully consistent with the guidelines in 5 CFR 1320.8(d). There are no special circumstances required for the collection of information in this data collection.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

A. In accordance with the paperwork Reduction Act of 1995, the notice required in 5 CFR 1320.8(d) has been published in the Federal Register announcing ONC’s intention to request an OMB review of data collection activities. This notice was published on August 25, 2010 in volume 75, number 164, on pages 52345 - 52346 and provided a 60-day period for public comments.

A 60-day Federal Register Notice was published in the Federal Register on August 25, 2010, vol. 75, No. 164; pp. 52345-52346. One public comment was received.

Summary of Comment:

Comment

Commenter indicated support for this proposed collection and recommended that ONC broaden the scope of the Information Technology Professionals in Healthcare evaluation to include all health information management, health IT, and informatics training programs in areas associated with electronic health records and health information exchange goals.

Response

While additional information on the universe of programs providing professional training in health information management, health IT, and informatics could be useful for a variety of purposes, the suggestion is outside the scope of the proposed data collection and the evaluation for which it is needed. The purpose and scope of the Information Technology Professionals in Healthcare (“Workforce”) Program evaluation is to assess the overall efficacy of the Workforce Program as a whole and each of its components (e.g. Community College Consortia, Curriculum Development Centers) in meeting the program’s goals of establishing and expanding training programs that will rapidly expand and subsequently maintain the availability of individuals with the skills needed to support the nationwide meaningful use of certified EHR technology and exchange and use of electronic health information to improve the quality, safety, and value of health care for every individual in the United States. The evaluation is designed to furnish both critical formative feedback and a summative assessment, to support and inform Workforce Program awardees’ development and improvement of programs established or expanded under the grants, and inform ONC oversight of, and policy formulation related to, the Workforce Program. Evaluating the entire universe of education programs potentially associated with overall electronic health records use and health information exchange is outside the scope of this evaluation and its data collection activities.

The student and faculty surveys, evaluation forms, and focus group protocols were developed by ONC’s contractor, the National Opinion Research Center (NORC) at the University of Chicago, with input from the grantees of the Workforce program.

Since January, 2010, the Agency has consulted with the following persons regarding this information collection.

Rachel Nelson

Acting Director

Division of Evaluation, ONC

U.S. Department of Health and Human Services

Phone: (202) 205-4416 Email: rachel.nelson@hhs.gov

Alok Doshi

Program Analyst

Division of Evaluation, ONC

U.S. Department of Health and Human Services

Email: alok.doshi@hhs.gov

A.9. Explanation of any Payment or Gift to Respondents

To offset the burden on participants, Web survey respondents will be offered a one year discounted membership to the Healthcare Information and Management Systems Society (HIMSS), a professional organization for individuals in the health care information and management field. ONC’s contractor, National Opinion Research Center (NORC) at the University of Chicago, has negotiated a reduced membership rate for survey respondents of $30 per person, and project funds will be used to offset the $30 fee. Thus, respondents will receive a free one year membership to HIMSS upon completion of the student survey. In addition, due to the sizable burden of participating in the 1.5 hour student, faculty, and text examiner focus groups, these participants will also be offered a gift certificate of $50 per person.

A.10. Assurance of Confidentiality Provided to Respondents

Participation in any study is voluntary. Responses by individuals will be kept private to the extent allowed by law. Under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose. Data will be collected in conformance with the Privacy Act of 1974. Invitation letters sent to each selected sample member will advise them that the information they provide is confidential. Similar information will appear on the introduction screen of the student and faculty Web questionnaire (see Attachment 1 – Proposed Invitation Contacts; Attachment 2–Proposed Student Web Questionnaire; Attachment 3 –Proposed Faculty Web Questionnaire).

The data collection plan has been reviewed and approved by the Institutional Review Board of ONC’s contractor, NORC. Data collection procedures will incorporate numerous safeguards for the data. While collecting Workforce data, information that could identify a particular sample member will be stored in a separate file from survey data collected from that person. Each sample member will be assigned a unique identifier, and this identifier will be used to store identifying information (such as name, address, etc.) in a separate database from the survey response data.

The electronic systems for data collection and data storage are on a local area network (LAN). All systems used to store electronic survey data are secure by design and protected by passwords only available to authorized study staff. Special steps will be taken to ensure that data collected via the Web questionnaire are secure. First, access to the Web instrument is only allowed with a valid Personal Identification Number (PIN) and password correctly entered in combination. Second, data will be transmitted by the Secure Sockets Layer (SSL) protocol that uses powerful encryption during transmission through the Internet.

If a respondent keeps a Web survey open without any activity, the Web server will close the survey after a short period of inactivity, thus preserving the data up to the break-off point and securely closing the connection. The Web system architecture process has been designed in a way that places authentication information and response data on physically separate servers. This strategy provides an extra layer of security to protect response data. Both development and production servers are backed up nightly. All data and analysis are reported in aggregate form only and measures are taken so that the identity of individuals or organizations is not disclosed.

A.11. Justification for Sensitive Questions

The student Web-based survey will ask students to self-identify their race and ethnicity using the federally approved questions on the U.S. Census. This question is necessary in order to conduct subgroup analyses and to understand the characteristics of students enrolling in courses and entering the health IT field.

No other questions of a sensitive nature are asked in this data collection.

A.12. Estimates of Annualized Hour Burden and Costs

ONC estimates contact with up to 1,233 Community College Consortia students to complete the baseline Web survey in the first year of this data collection effort. The total burden for the individual is estimated at 20 minutes, based on experience with similar instruments. In addition, the Web instrument was pre-tested on a group of seven Community College Consortia students.

ONC estimates contact with up to 300 faculty members at Community College Consortia programs to complete the Web survey. The total annual burden for the individual is estimated at 10 minutes. In addition, the Web instrument was pre-tested on a group of nine similar education faculty members.

Exhibit 2 presents estimates of the reporting burden for respondents. A stratified random sample of Community College Consortia students and a census of Community College Consortia course instructors will be approached to participate. The expected response rates are 80%. Please see Exhibit 2 for the breakdown of the burden by student type. The total cost to respondents for the hour burden is estimated to be $20.90 per hour for student and $36.62 for faculty. Because we assume a diverse group of individuals will enroll in these health IT programs, the student estimate is based on the mean hourly wage of all occupations in the U.S. according to the Bureau of Labor Statistics of $20.90. The faculty estimate is based on an estimate of instructor salaries from one community college. They reported that on average their adjunct instructors receive $3,626 per quarter for teaching a 3-hour, 11-week course. With an additional fives hours per week spent preparing for instructing and holding office hours, the average hourly rate is $41.20.

The following table identified the annualized burden estimate:

Exhibit 2: Estimated Burden Hours – Web Survey

Forms (If necessary)	Type of Respondent	Number of Respondents per year	Number of Responses per Respondent	Average Burden hours per Response	Total Burden Hours	Cost per hour	Total Cost For All Respondents
Web-based Student Baseline Survey	Students enrolled in Workforce program	1,233	1	20/60	411	$20.90	$8,589.90
Focus groups with students	Students enrolled in Workforce program	256	1	1.5	384	$41.24	$15,836.16
Focus groups with faculty	Instructors from Workforce program	50	1	1.5	75	$20.90	$1,567.50
Focus groups with Exam takers	Competency exam takers not enrolled in Workforce program	32	1	1.5	48	$109.89	$5,274.72
Web-based Faculty Survey	Instructors from Workforce program	300	1	10/60	50	$20.90	$1,045.00
					968		$32,313.38

Additionally, students will be asked to complete a course evaluation form for the courses they attend. However, we are not including burden here because course instructors will be administering the course evaluation as part of the normal class procedure.

A.13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

There are no annualized capital/startup or ongoing operation and maintenance costs involved in collecting the information. Other than their time to complete the surveys, which is estimated in Exhibits 2 and 3, there are no direct monetary costs to respondents.

A.14. Estimates of Annualized Costs to the Federal Government

The estimated cost to the Federal Government for the Evaluation of Information Technology Professionals in Health Care Program data collection activities is $506,710. This is the cost to our Federal contractor, NORC, for data collection activities associated with this submission.

A.15. Explanation for Program Changes or Adjustments

No change in burden is requested. This submission to OMB is for an initial request for approval.

A.16. Plans for Tabulation and Publication and Project Time Schedule

Data for the evaluation will be collected in each of the three years of the evaluation. Analysis will begin shortly after data are collected from the Web-based surveys, and will continue periodically throughout the project period to produce annual reports, as well as a final report in March 2013 upon completion of the evaluation. Exhibit 3 provides an estimated schedule of data collection for the Web-Based Survey, contingent on receiving OMB approval. These dates are based on the estimated program start schedule and may change slightly.

Exhibit 3: Timetable for Data Collection for Web-Based Survey

Cohort	Baseline Survey		Follow-up Survey
Cohort	Estimated Start Date	Estimated End Date	Estimated Start Date	Estimated End Date
Cohort 1	January 2011	April 2011	July 2011	December 2011
Cohort 2	May 2011	August 2011	November 2011	April 2012
Cohort 3	January 2012	April 2012	July 2012	December 2012

Exhibit 4 provides an estimated schedule for collecting data through other mechanisms, contingent on receiving OMB approval, as well as the reporting schedule for the entire evaluation. The remainder of this section describes the analytic techniques that will be employed.

Exhibit 4: Timetable for Data Collection And Publication For Other Data Collection Efforts

Activity	Estimated Start Date	Estimated End Date
Course evaluations	February 2011	December 2012
Focus groups with students	1 month following OMB approval	October 2012
Focus groups with faculty	1 month following OMB approval	October 2012
Web-based faculty survey	January 2011	February 2012
On-line focus groups with exam takers	May 2011	October 2012
Development of first annual report	April 2011	May 2011
Development of second annual report	April 2012	May 2012
Development of final report	January 2013	March 2013

Our interim and final reports will provide an analysis of the quantitative data in the student surveys and the course evaluations, as well as the qualitative information gathered through those two data-collection efforts and the focus groups.

Quantitative analysis will primarily be descriptive. We will provide aggregate descriptions of the following types of information:

Students’ satisfaction with the learning environment (e.g., with faculty/courses/resources/curriculum materials);
Students’ perceptions about work/skill readiness;
Satisfaction with support systems available within and outside of the college environment;
Students’ level of engagement with faculty; and
Employment outcomes of students who complete the educational programs.

In addition to producing descriptive statistics of the measures associated with student satisfaction and employment outcomes and conducting longitudinal analyses of how key measures vary over time, we will analyze the data by student race/ethnicity, gender, enrollment status, and curriculum focus. We will also run tests of association on such variables as student satisfaction with training programs and their employment outcomes.

The reports will also include a discussion of the qualitative information gleaned through the focus groups and open-response survey questions. This will allow us to explore:

Students’ motivations for entering the program and the health IT profession;

Faculty members’ perceptions of student engagement and availability of appropriate curricula; and
Areas for program improvement.

A.17. Exception for Display of Expiration Date

All data collection materials will display the OMB expiration date.

A.18. Certifications

ONC certifies that the collection of information encompassed by this request complies with 5 CFR 1320.9 and the related provisions of 5 CFR 1320.8(b)(3).

B. Statistical Methods

B.1. Respondent Universe and Sampling Methods

ONC’s contractor, the National Opinion Research Center (NORC), will collect information for the Evaluation of Information Technology Professionals in Health Care Program on behalf of ONC. The contractor is responsible for the design and administration of the survey that will be used to collect information about students, faculty, and exam takers associated with the Workforce program.

As discussion in Part A, the data collection is part of a mixed method (surveys and focus groups) evaluation design that address the complex nature of the Workforce program and will ensure that all aspects of the program are captured. The design will therefore require multiple contacts with students and other program stakeholders at various times during the evaluation. This is necessary to capture the different types of information and inform subsequent stages of the assessment.

However, the primary data collection for which OMB approval is being sought is for baseline data collection and includes the following activities:

Web-Based Survey of Students in Community Colleges Programs

In order to allow for stratification and subgroup analyses of key variables, such as gender, race, and professional roles, we will derive a growing panel of community college students that will total approximately 1,850 unique sample members by the end of the program’s third year across five Community College Consortia made up of 84 colleges. The universe consists of students enrolled in the Workforce program and the sampling frame will encompass most of the universe, excluding a small number of students without email addresses. As outlined in the table below, for the baseline survey in the first year, we would sample 616 enrollees in the community college program. For each subsequent cohort, we will add an additional 617 community college students to the panel. This will allow us to conduct robust analyses of specific subgroups as well as meaningful cohort comparisons over time. We expect an 80% response rate each round. The Community College Consortia have agreed to provide lists of students enrolled in their programs during the evaluation from which we will draw the sample. We offer more detail on our sampling plan in section B.2.3.

Student Survey Sample	Sample	Baseline		Follow-Up
Student Survey Sample	Sample	2011	2012	2011	2012
Community College Enrollees
Cohort 1	616	X	--	X	--
Cohort 2	617	X	--	--	X
Cohort 3	617	--	X	--	X
Total Respondents	1,850	1,233	617	616	1,234

Web-Based Survey of Faculty from Community Colleges Programs. A census of the estimated 300 faculty instructors participating in the Workforce program will be invited to participate in the Web-based faculty survey via email. The email addresses of faculty in their programs will be provided by the Community College Consortia.

Course Evaluation Forms. The community colleges will be required to collect student evaluation forms of Workforce courses. All students in the Workforce program will receive an email invitation to complete a course evaluation form. This will not add any additional burden as evaluations are already part of the normal class procedures. We will provide a set of core questions to which each college may add their own specific questions.

Focus Group with Students Discussion Guide. During unstructured site visits to the programs, we plan to hold up to four focus groups with eight students at each site (approximately eight sites per year). Participation in the focus groups will be voluntary and students will be selected from individuals who respond to an email invitation to participate.

Focus Groups with Faculty Discussion Guide. Once per year for the three years of the project, we plan to hold up to five focus groups with ten faculty members selected from across multiple community colleges within the same Community College Consortia. These Web-based focus groups will likely consist of faculty from more than one school and participation will be voluntary. Faculty will be invited to participate via email.

Focus Groups with Exam Takers Discussion Guide. We plan to conduct four Web-based focus groups per year with up to eight exam takers. Focus group participants will be selected from individuals who respond to a question on the competency exam inviting them to participate in a focus group on health IT certification and careers.

B.2. Information Collection Procedures

ONC will coordinate with the staff of Community College Consortia participating in the grant program to obtain lists of students enrolled in Workforce course(s). To obtain baseline information, a self-administered Web-based survey will be made available to respondents. Respondents will be contacted via email using addresses provided by the Community College Consortia and asked to complete the 20 minute survey. In addition, evaluation forms will be distributed to students enrolled in Workforce courses, and a subset of students, as well as faculty and exam takers, will be asked to participate in small, in-depth focus group discussions.

B.2.1. Statistical Methodology for Stratification and Sample Selection

Web-based Survey

The purpose of the Web-based survey of students is to collect information on students’ experiences in the program including, students’ attitudes and satisfaction with the learning environment (e.g., with faculty/courses/resources/curriculum materials), perceptions about work/skill readiness, students’ motivation for entering the program and the health IT profession, students’ level of engagement with faculty, satisfaction with support systems available within and outside of the college environment , areas for program improvement, employment outcomes, and experiences with the competency exam.

To minimize burden and cost to the government, ONC will collect data from community college students via a Web survey. The Web survey will include skip patterns, range checks, and other quality control measures; and will be hosted on a secure server. Students’ email addresses and other contact information will be provided by the community colleges. Email addresses, or postal addresses where the former are not available, will be used to correspond with members of the sample. This communication will include sending members their unique personal identification numbers (PINs) and passwords, as well as the URL of the survey. If no email or postal address is available for a student, the student will be contacted via telephone and asked for his or her email address.

Course Evaluation Forms

The purpose of the course evaluation forms is to collect students’ opinions and attitudes about the materials in individual courses. The Community College Consortia programs will distribute these course evaluation forms via email to all workforce students at the end of each course. The programs will be responsible for processing the course evaluation forms and delivering the data to ONC.

Focus Groups

The purpose of the focus groups is to gain insights from the students, faculty members, and exam takers. Our discussions with students will explore their perceptions of the quality of instruction, motivations for enrolling in the program, and plans for taking (or not taking) the competency exam. With faculty members, we will discuss how they are using the curricula developed as part of this Program and whether they have identified any gaps in the available materials. We will also conduct focus groups with individuals who took the exam, but did not attend one of the ONC-funded training programs in order to understand their motivations.

To minimize burden and cost to the government, we will conduct focus groups with students during site visits and with faculty and test takers using an on-line forum. Students will be recruited via email invitations sent to all students. An experienced researcher will facilitate each focus group, using the discussion guide that will ensure that key topics are covered, while allowing for sufficient flexibility to follow the natural flow of the conversation. Faculty will be recruited via email invitations sent to all instructors in the program. For the focus groups with exam takers, we will schedule some of the discussions shortly after the individuals have sat for the exam in order to discuss the clarity, validity, and relevance of the questions while they are still fresh in participants’ minds. A question at the end of the exam will ask if the exam taker is interested in being contacted in the future for focus groups, and we will then draw our sample from those individuals who respond “yes.”

The table below provides an estimated timeline of data collection activities.

Activity	Estimated Start Date	Estimated End Date
Course evaluation forms	February 2011	December 2012
Web-based faculty survey	January 2011	February 2012
Focus groups with students	1 month following OMB approval	October 2012
Focus groups with faculty	1 month following OMB approval	October 2012
Web-based survey of students – cohort 1, baseline	January 2011	April 2011
Web-based survey of students – cohort 2, baseline	May 2011	August 2011
Focus groups with exam takers	May 2011	October 2012
Web-based survey of students – cohort 1, follow-up	July 2011	December 2011
Web-based survey of students – cohort 2, follow-up	November 2011	April 2012
Web-based survey of students – cohort 3, baseline	January 2012	April 2012
Web-based survey of students – cohort 3, follow-up	July 2012	December 2012

B.2.2. Estimation Procedure

For producing population-based estimates of totals, percentages and means, each respondent member will have a sampling weight. This weight combines the base weight that is the inverse of the probability of selection of a member and an adjustment for non-response. This adjustment is to account for members who are in the sample but do not respond to the follow-up surveys. Every effort will be made to minimize the non-sampling errors in the estimates by maximizing the response rates and taking steps to reduce response errors.

Quantitative analysis will primarily be descriptive to answer research questions on program satisfaction and student employment outcomes. We will provide aggregate descriptions of the following information:

Students’ satisfaction with the learning environment (e.g., with faculty/courses/resources/curriculum materials);
Students’ perceptions about work/skill readiness;
Satisfaction with support systems available within and outside of the college environment;
Students’ level of engagement with faculty; and
Employment outcomes of students who complete the educational programs.

In addition to descriptive statistics of the measures associated with student satisfaction and employment outcomes and conducting longitudinal analyses of how key measures vary over time, we will analyze the data by key subgroups, such as student race/ethnicity, gender, enrollment status, and curriculum focus.

The reports will also include a discussion of the qualitative information gleaned through the focus groups and open-response survey questions. This will allow us to answer key research questions on:

Students’ motivations for entering the program and the health IT profession;
Faculty members’ perceptions of student engagement and availability of appropriate curricula; and
Areas for program improvement.

B.2.3. Degree of Accuracy Needed for the Purpose Described in the Justification

For the Web-based survey of students, the sample size of 1,850 was selected to enable us to conduct our analysis. The selection will be done in two stages: 1) at the first stage a sample of community colleges with the Workforce program will be chosen; 2) at the second stage a sample of students within each of the selected community college programs will be selected. The goal of our sampling plan is to select a sample that is representative of the population of students either currently or previously enrolled in a Workforce program. The sample size, which will be representative of the population, will then be used to generate frequencies and means, by important characteristics.

Sample Size Needs. A sample size of 1,850 students across three cohorts gives a sufficient size to be representative of the population of students enrolled in the workforce program across the five consortia. In determining the sample size, a 95% confidence level and a margin of error of 3.5% were our assumed rates to determine the sample size needs. Based on these rates, a simple random sample would require a sample size of 755, which includes the finite population correction (fpc) for an enrollment of 20,000 students at the time of sample selection. The use of 20,000 enrolled students is an estimate of the universe of students. The sampling frame will contain the number of students currently or previously enrolled in the program at the time of sample selection for each cohort.

Since the proposed sample design is a cluster design, the number of effective completes will be less than the number of completed surveys (i.e., the design effect will be greater than one). The design effect can be estimated from a standard formula, using an assumed intraclass correlation coefficient (ICC). For this analysis, we are assuming an ICC of .05, which equates to a design effect of 1.5 that will be assumed for this study. Note that this is an estimate, and the actual intraclass correlation and design effect will not be known until after the survey is complete. With a baseline sample of 1,850 students and an 80% response rate, the result would be 1,480 completed surveys, which results in a 986 effective sample size. Since the minimum required sample size is 755, this will be a sufficient size to be representative of the population, while also providing sufficient additional sample for subgroup analysis by gender, race, and role (i.e., program type).

Stratification of Community Colleges. The colleges will be stratified by consortium and three community colleges within each consortium will be selected. We will select the colleges from each consortium using Probability Proportional to Size (PPS) sampling. Therefore, the three colleges chosen will be those with the largest number of students enrolled in the workforce program at the time of selection for each cohort. Below is a table containing the estimated number of community colleges within each consortium that are expected to have a workforce program; however, these numbers could change slightly at the time of sample selection for each cohort.

Consortium	Community Colleges with Workforce Program
Bellevue	8
Los Rios	15
Cuyahoga	17
Pitt	21
Tidewater	23

Sampling of Students. The allocation of the 1,850 students across the community colleges will be based on the size of the community college program (i.e., number of students enrolled) as well as the enrollment within each consortium. The allocation will be in two stages. First, the students will be allocated proportional to the size of the program (PPS) within each consortium based on the total enrollment for the workforce program across all of the consortia. A second step will be done to allocate the consortium’s sample needs across the three community colleges chosen for the sample, based on workforce program enrollment size at the time of sample selection for those three community colleges. This sampling method will be applied during each cohort timeframe; so that the colleges chosen and allocation calculated for cohort 1 may not be the same as for cohort 2. This will allow for accurate estimates based on current enrollment levels during each cohort. At any time of sample selection, if there are fewer enrolled students than the sample size requires, a census of all students will be done.

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

There are no unusual problems requiring specialized sampling procedures.

B.2.5. Use of Periodic (Less Frequent Than Annual) Data Collection Cycles

The baseline survey is a one-time data collection necessary to establish a baseline from which to measure outcomes on program participants. The course evaluation forms and focus groups will occur up to three times over the 24-month data collection period.

B.3. Methods to Maximize Response Rates

During the study, contact with students will be maintained between survey rounds in order to track them successfully over the period of subsequent follow-ups. A dedicated study Website will be made available to students, as well as a dedicated 800-line and email address. The research team may also explore the use of additional tools to track students such as online networking (e.g., Facebook and MySpace). ONC also plans to offer a free one year membership to a professional organization for students participating in the Web survey and a $50 gift certificate to a retail store for students, faculty, and text examiners participating in focus groups to offset the burden of study participation. Overall, the questions in the survey and focus groups were developed to not only meet analytic goals, but also to encourage the interest and investment of program students.

B.4. Tests of Procedures

The survey instrument has been drafted and has undergone two reviews: (1) an internal review conducted by NORC’s Institutional Review Board and (2) a pre-test with seven students. In order to accurately determine the burden placed on respondents as well as further test the clarity of the survey questions, a pre-test was conducted in which a total of seven students from diverse backgrounds responded to the survey to assess the reliability of the instrument. Slight revisions were made to the order and wording of a small number of questions based on comments received from both of these reviews.

Modifications to the length, content, and structure of the survey have been made based on the results of the survey pre-test interviews. Respondents provided generally positive feedback indicating that they could readily answer the questions and that the time to complete the survey was not onerous (approximately 20 minutes). The respondents also had an enthusiastic response to the idea of a professional membership to offset the burden of participating in the survey and registered their likelihood to respond to the baseline and subsequent student surveys if presented this offer.

B.5. Statistical Consultants

The information for this study is being collected by NORC, a research and consulting firm, on behalf of ONC. With ONC oversight, NORC is responsible for the study design, instrument development, data collection, analysis, and report preparation.

The instrument for this study and the plans for statistical analyses were developed by NORC. The staff team is composed of Dr. Kristina Lowell, Project Director; Dr. Carrie Markovitz, Senior Research Scientist; and a team of senior-level staff including Karen Grigorian. Contact information for these individuals is provided below.

Name	Number
Kristina Lowell	301-634-9488
Carrie Markovitz	301-634-9388
Karen Grigorian	312-759-4025

Page | 3

File Type	application/msword
File Title	REQUEST FOR CLEARANCE FOR
Author	DHHS
Last Modified By	DHHS
File Modified	2010-11-30
File Created	2010-11-30