Supporting Statement B

Supporting Statement B

Assessing the Effectiveness of eLearning to Improve Birth Data Quality

for the

National Center for Health Statistics

Generic IC:

NCHS Customer Surveys

OMB No. 0920-0729

(Expires September 30, 2020)

Contact Information

Joyce A. Martin

Division of Health Care Statistics

National Center for Health Statistics

3311 Toledo Rd

Hyattsville, MD 20782

301-458-4369

jcm9@cdc.gov

November 8, 2018

Table of Contents

Table of Contents i

LIST OF ATTACHMENTS ii

Part B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS 1

B.1 Respondent Universe and Sampling Methods 1

B.2 Procedures for Collection of Information 4

B.3 Methods to Maximize Response Rates and Deal with Nonresponse 9

B.4 Tests of Procedures or Methods to be Undertaken 10

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 10

LIST OF ATTACHMENTS

Attachment A1a. Introductory/Invitation Email for Hospitals

Attachment A1b. Hospital Staffing List

Attachment A2. Confidentiality FAQ

Attachment A3. Data Collector Confidentiality Agreement

Attachment B1a. Baseline Survey Invitation Email

Attachment B1b. Baseline Survey Password Email

Attachment B1c. Baseline Survey Reminder Email

Attachment B2. Online Baseline Survey

Attachment C1a. eLearning Course Invitation Email

Attachment C1b. eLearning Course Reminder Email

Attachment D1a. Follow-Up Survey Invitation Email

Attachment D1b. Follow-Up Survey Password Email

Attachment D1c. Follow-Up Survey Reminder Email

Attachment D2. Online Follow-Up Survey

Attachment E. Types of Outcomes

Attachment F. Thank-You Email

Part B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Respondent Universe and Sampling Methods

This study will focus on evaluating the effectiveness of the online eLearning course entitled “Applying Best Practices for Reporting Medical and Health Information on Birth Certificates.” To do so, four different respondent types will be invited to participate: (1) birth registrars (BR); (2) BR managers (BRM); (3) labor and delivery nurses, including mid-wives (L&D nurses); and (4) obstetricians and gynecologists (OB/GYNs). Up to 645 individuals from each of the 4 respondent types in 9 selected hospitals in Massachusetts will be invited to complete the Online Baseline Survey and Online Follow-up Survey to obtain up to 359 responses (i.e., 37 BR, 12 BRM, 216 L&D nurses, and 94 OB/GYNs). Participation in the web-based surveys is voluntary. A statement that participation is voluntary and that refusal to participate involves no loss of benefits and can be discontinued at any time is included in the Introductory/Invitation Email for Hospitals (Attachment A1a) and the Baseline Survey Invitation Email to participants (Attachment B1a, B1b, and B1c). In addition, subsequent letters (Attachments C1a, C1b, D1a, D1b, and D1c) inviting participants to complete the eLearning course and the Online Follow-Up Survey will reiterate that participation is voluntary and that refusal to participate involved no loss of benefits and can be discontinued at any time. See the Figure below for the overall data collection procedure.

This study does not require a nationally representative sample to determine responses to the eLearning course; therefore, quantitative data collected with the Online Baseline Survey and Online Follow-Up Survey will not be weighted. The potential respondent universe is the total number of BR, BRM, L&D nurses, and OB/GYNs in the 46 birthing hospitals in the State of Massachusetts. On average, there are 2 BR, 1 BRM, 18 OB/GYNs, and 51 L&D nurses per hospital equaling approximately 3,456 individuals in the respondent universe. Nine of the 46 hospitals were selected for inclusion in this study by the Massachusetts Department of Public Health/Registry of Vital Records and Statistics (DPH/VR), based on the following criteria:

• Annual number of births

• Data quality, ranked on a scale from 1 to 10

• Hospital characteristics (e.g., teaching, high-risk, urban location)

• Data collection process for birth certificate data

• Type of birth certificate records (e.g., electronic, paper)

These nine hospitals were approached by the Massachusetts DPH/VR and have agreed to participate in this study.

Of the 1,043 individuals who are responsible for the collection and/or reporting of birth certificate data at the 9 selected hospitals in Massachusetts, a sample of up to 645 will be invited to participate in the Online Baseline Survey (Attachment B2) and Online Follow-Up Survey (Attachment D2).

Inclusion criteria for participants include the following:

• Employment in one of nine selected hospitals in Massachusetts (described above)

• Involvement in the birth certificate data documentation and reporting process

• Willingness/availability to participate

• Ability to read and understand English (surveys and eLearning course are only in English)

Exclusion criteria for participants include the following:

• Employment at a Massachusetts hospital other than one of the nine selected for inclusion in the study

• Not being included on the list of staff involved in the birth data collection process provided by the hospitals

• Exposure to the eLearning course prior to completion of the baseline survey

Enough individuals will be selected to achieve up to 359 total completed Online Baseline Survey (Attachment B2). Specifically, it is estimated that up to 216 L&D nurses (approximately 31 percent of eligible L&D nurses), 94 OB/GYNs (approximately 31 percent of eligible OB/GYNs), 37 BR, and 12 BRM will complete the survey. The total population of eligible BR and BRM are expected to complete the survey.

To ensure the target number of completes in the L&D nurses and OB/GYN strata are met, a stratified random sample, by hospital, will be created (using the random number generator in Excel) and initial survey invitations (Attachment B1a and Attachment B1b) will be sent to 277 random L&D nurses and 119 random OB/GYNs. A reminder email (Attachment B1c) will be sent to the nonrespondents after one week. A second reminder email (Attachment B1c) will be sent to the remaining nonrespondents two weeks after the initial invitation email. After four weeks, response rates will be evaluated and if the goal of 216 L&D nurses and 94 OB/GYN completes has not occurred, new survey invitations will be sent to an additional 100 random L&D nurses and 100 random OB/GYNs. Only participants who complete the Online Baseline Survey (Attachment B2), the eLearning Course, and the Online Follow-Up Survey (Attachment D2) will be considered completed responses. If the final survey response rate drops below the target, the contractor will perform weighting adjustments to correct for the potential biases.

B.2 Procedures for Collection of Information

As previously noted, the information will be collected from individuals representing four types of hospital staff who are involved in the collection and/or reporting of birth certificate data: (1) BR; (2) BRM; (3) L&D nurses; and (4) OB/GYNs. Administrators from recruited hospitals will be sent introductory emails from Massachusetts DPH/VR (Attachment A1a) requesting that they provide staffing and email information (Attachment A1b) that will be used to identify potential participants. Once these potential participants have been identified, the nine hospitals will send the contact information to the contractor via FedEx. The contractor will then contact potential participants for inclusion in the study.

Up to 645 staff involved in the collection of birth certificate data will be invited to participate in completing the Online Baseline Survey until up to 359 responses (i.e., 37 BR, 12 BRM, 216 L&D nurses, and 94 OB/GYNs) have been obtained. All staff who completed the Online Baseline Survey (Attachment B2) and the eLearning course will be invited to participate in the Online Follow-Up Survey (Attachment D2) within one week of completing the eLearning course. The details on each data collection point are provided in the following sections.

This one-time study is not nationally representative; therefore, concerns regarding the statistical methodology for both stratification and sample selection, the degree of accuracy, the estimation procedure, and periodicity of data collection cycles is not applicable. Likewise, the National Center for Health Statistics (NCHS) does not anticipate unusual problems that require the usage of specialized sampling procedures.

Baseline Data Collection

Online Baseline Survey

Up to 645 staff involved in the collection and/or reporting of birth certificate data at the nine selected Massachusetts hospitals will be invited to participate in the Online Baseline Survey (Attachment B2). A link to the Online Baseline Survey will be sent in the invitation email (Attachment B1a), which will also contain information on confidentiality (Attachment A2). A second email (Attachment B1b) will be sent with a unique password for accessing the Online Baseline Survey. Passive informed consent is provided when participants click on the survey link and enter their password. If a participant does not complete the Online Baseline Survey within one week after the initial email was sent, a reminder email will be sent (Attachment B1c). If a participant does not complete the Online Baseline Survey after the first reminder email, a second reminder email will be sent approximately one week after the first reminder email was sent (Attachment B1c).

Online eLearning Course

All individuals who complete the Online Baseline Survey will then receive an email with a link to the eLearning course (Attachment C1a). A reminder email with a link to the eLearning course will be sent one week after the initial email to remind participants to complete the eLearning course (Attachment C1b). If a participant does not complete the eLearning course after the first reminder email, second reminder email will be sent approximately one week after the first reminder email was sent (Attachment C1b).

Follow-Up Data Collection

Online Follow-Up Survey

After participants complete the eLearning course, they will receive an email with a link to the Online Follow-Up Survey (Attachment D1a) and an email with a unique password for accessing the survey (Attachment D1b). If a participant does not complete the Online Follow-Up Survey (Attachment D2) one week after the initial email was sent, a reminder email will be sent (Attachment D1c). If a participant does not complete the Online Follow-Up Survey after the first reminder email, a second reminder email will be sent approximately one week after the first reminder email was sent (Attachment D1c). Once a participant completes the Online Follow-Up Survey, a Thank-You Email for participation in the study (Attachment F) will be sent.

The figure below provides an overview of data collection and communication materials.

Data Collection and Communication Materials Flow Chart

Participants will be sent a unique link to each of the surveys (Attachments B1a and D1a) and a unique, randomly generated password (Attachments B1b and D1b) that will be required to access the survey. Links and passwords will be sent in separate emails. The link, generated by Qualtrics (described below), uses a long string of letters and numbers generated for each data record. The contractor will create randomized passwords using random number and random letter generators in Excel.

Qualtrics, which stores responses in an internal database, will be used as the data collection and survey software. Surveys completed during business hours will be immediately downloaded and deleted from the Qualtrics server; surveys completed outside of business hours will be downloaded and deleted at the start of business hours and twice daily on weekends. Downloaded survey data will be stored in a network folder on a quarantined virtual server with access limited by a security group of authorized domain users. For quality control and assurance, the data will be directly exported into Statistical Package for Social Sciences (SPSS) (version 24) as an SPSS file or will be exported as a CSV and imported into Stata (version 14) for analysis (The Python programming language may be used for more advanced commands in SPSS; NVivo 11 Plus will be used to analyze qualitative data from the surveys). These automated processes will ensure data accuracy. Any differences found during this process will be discussed by the contractor and reconciled.

The contractor will use SharePoint and Microsoft Excel for information management. PII (names and email addresses only) will be encrypted (in Excel) and stored separately from survey responses and will only be seen by NCHS and the contractor for purposes of contacting and tracking the participants for the study. Data will not be shared or made available to any unauthorized personnel or other third party and will only be accessible to the minimum number of staff necessary to perform the work. Any contact information or other data from paper records will be transferred to electronic files for use during projects. Any paper records will be stored in a locked file drawer. After all reports have been completed, all data files will be encrypted and transferred to NCHS. The Contractor will archive all data files for 3 years in a secure file server.

Record-level data will not be shared with Massachusetts DPH/VR or the hospitals. Massachusetts DPH/VR will receive summarized reports (created by NCHS) in which information about staff category, job experience, and barriers faced will not be shown by hospital. Where respondents are fewer than five in any category, data will be suppressed. No personally identifiable information (PII) will be included on any of the surveys.

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

The following procedures will be used to maximize the completion rates for the Online Baseline Survey and Online Follow-Up Survey that will be administered online:

• The letters inviting participants to complete both the Online Baseline Survey and Online Follow-Up Survey were carefully developed to emphasize the importance of this study and how the information will help NCHS improve and refine the eLearning course.

• Designated NCHS and Massachusetts DPH/VR staff will be kept closely informed of the project so they will be able to answer questions from hospitals, and employees at hospitals, and to encourage participation.

• Up to two email reminders will be sent to participants who have not yet completed either the Online Baseline Survey or the Online Follow-Up Survey.

B.4 Tests of Procedures or Methods to be Undertaken

No pretesting of the instruments occurred. However, all instruments were internally tested by the Contractor and NCHS for flow, understandability, and length. All instruments were created specifically for this study, and have not been used in previous studies. Therefore, there is no information on the validity nor reliability of the instruments, the sensitivity and specificity of the instruments, nor how well they yield reproducible results. However, questions were created based on the Health Belief Model¹ and the Theory of Planned Behavior.² There are no controversial methods being used to collect information. It is anticipated that there will be no changes to the instruments once data collection has begun. Any changes to the data collection instruments will occur, as needed, during the OMB review process.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Questionnaires and protocols were developed in collaboration with, and were reviewed by, staff of NCHS and staff of the Massachusetts DPH/VR. The Contractor, 2M Research, will conduct this study.

Name	Role	Telephone Number	E-mail
Molly Matthews-Ewald	Project Director	469-453-0888	mmatthewsewald@2mresearch.com
Laurie Hinnant	Corporate Monitor	470-586-7050	lhinnant@2mresearch.com
Cindy Romero	Project Coordinator	817-856-0873	cromero@2mresearch.com
Moyo Kimathi	Quantitative Data Specialist	817-856-0868	mkimathi@2mresearch.com
Joyce A. Martin	Contracting Officer’s Representative	301-458-4362	jcm9@cdc.gov
Michelle Osterman	eLearning Project Lead	301-458-4392	ibx6@cdc.gov

1 Rosenstock, I. M., Strecher, V. J., & Becker, M. H. (1988). Social learning theory and the Health Belief Model. Health Education Quarterly, 15(2), 175–183.

2 Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50, 179–211.