3_Supporting Statement B_NHGRI Short Course_final

Supporting Statement B For:

Surveys to Support an Evaluation of the National Human Genome Research Institute (NHGRI) Summer Workshop in Genomics (Short Course)

Updated December 14, 2015

Carla L. Easter

Education and Community Involvement Branch

National Human Genome Research Institute

Building 31, Room B1B55

Bethesda, MD 20892

Telephone: 301-594-1364

Fax: 301-480-5008

E-mail: easterc@mail.nih.gov

Table of Contents

B.1 Respondent Universe and Sampling Methods 1

B.2 Procedures for the Collection of Information 1

B.3 Methods to Maximize Response Rates and Deal with Nonresponse 2

B.4 Test of Procedures or Methods to be Undertaken 2

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 2

List of Attachments

Attachment 1: NHGRI Short Course Survey – Students and Faculty

Attachment 2: Background and Rationale for Short Course

Attachment 3: IRB Exemption Notice

Attachment 4: Invitation Letter to Short Course Students and Faculty

Attachment 5: Applicability of the Privacy Act

B. STATISTICAL METHODS

B.1 Respondent Universe and Sampling Methods

Intended respondents of the web-based surveys are faculty and students of the Short Course between 2004 and 2012. The sampling method for the survey is a population-based sample of all program participants. There will be no cohort-specific (or year-specific) analyses. There are a maximum of 314 participants between 2004 and 2012, with 187 faculty and 127 students.

Year	Number of Faculty	Number of Students	Total
2004	16	16	32
2005	18	18	36
2006	18	18	36
2007	20	13	33
2008	14	16	30
2009	18	17	35
2010	18	17	35
2011	36	0	36
2012	29	12	41
Total	187	127	314

Contact information (e.g., email address, phone number) is only available for 299 program participants based on a list provided to the evaluation team by NHGRI. The evaluation searched for updated contact information in the summer of 2015 and could not find current contact information for 77 potential respondents. While some of the previous contact information may be accurate, the likely pool of survey respondents is closer to 225. The anticipated response rate is approximately 50% given some respondents completed the program a decade ago.

B.2 Procedures for the Collection of Information

The ability to gather specific information through standardized web-based surveys (Attachment 1) from Short Course stakeholders is critical for the success of the proposed evaluation strategy. Specific survey questions associated with the medium- and long-term outcomes are presented in the table below.

Outcome Area	Survey Question
Continued participation in NIH-related activities (Medium-Term)	Please indicate the ways you have been involved with Short Course attendees, NIH or NHGRI since attending the program. (Select all that apply)
	Since the Short Course, have you been involved with Short Course attendees, NIH or NHGRI?
Dissemination of genomics beyond the classroom (professional related activities) (Medium-Term)	Beyond curriculum integration, in what other ways did you disseminate information from the Short Course to students? If you did not disseminate information beyond curriculum integration, please write “None.”
	Did the Short Course influence your research?
	Please describe how your experience at the Short Course influenced your research.
	To what degree did your experience at the Short Course influence your decision to engage in the following activities? Select "Not Applicable" if you did not engage in the activity listed.
Increased genomics knowledge by students at faculty’s institution (Medium-Term)	Please rate how much you think the following student-related factors influenced your efforts to transfer your knowledge to students. Select "Not Applicable" if the factor was not relevant to your situation.
New knowledge integrated into existing teaching materials (Medium-Term)	I was able to update my curriculum and teaching materials in the following content areas: (Select all that apply)
	I was able to update my genetics or genomics curriculum and teaching materials in the following ways: (Select all that apply)
Pursue a career related to genomics (broadly defined) (Medium-Term)	Does your current position involve teaching genetic or genomic material?
	Out of 100 percent, what best represents your current teaching/research ratio?
	Please describe how the Short Course influenced your decision to pursue new career or educational options.
	Please specify the type of instructor or professor position.
	Please specify trainee type.
	Please specify whether this is a tenure or non-tenure track position.
	Select the content areas you teach: (Select all that apply)
	To what degree did your experience at the Short Course influence your decision to pursue new career or educational options?
	To what degree did your experience at the Short Course influence your teaching to research ratio?
	What is your current primary Position or Occupation Title?
	Which of the following best describes your primary field of work?
Pursuit of coursework/learning opportunities related to genomics (Medium-Term)	Have you completed the credential or degree(s)?
	Out of 100 percent, what best represents your current teaching/research ratio?
	Please identify any credentials or degrees that you have received or pursued since attending the Short Course. (Select all that apply)
	What discipline(s) is your credential or degree associated with?
	What type of degree program are you enrolled in?
Updated curriculum at faculty institutions (Medium-Term)	How much time was required to update your curriculum and teaching materials after participating in the Short Course?
	Please select one of the three options below in response to the following statement: I was able to update my curriculum as a result of my participation in the Short Course.
Train and diversify the pipeline of genome professionals (Long-Term)	Has your Position or Occupation Title changed from [PIPE IN SELECTED CHOICE] since the time of your attendance in the Short Course?
	Please specify whether this is a tenure or non-tenure track position.
	What is your current primary Position or Occupation Title?

In order to maximize response rates, respondents will be initially contact by email and informed about plans to conduct an evaluation of the Short Course (Attachment 4). Program staff will monitor all emails that bounce back and identify other methods of contacting respondents for whom the email address is invalid. A second contact will be made by email that will invite the respondents to complete the survey by clicking on the link in the invitation letter, which will bring them to the on-line survey.

Beginning with study initiation and continuing through all phases of information collection, monitoring, and analysis, NHGRI and its evaluation contractor will take steps to ensure that the data collected are of the highest quality possible. This includes clearly articulating the goals of the program and evaluation to intended respondents in email communications and the introduction to the survey; conducting pilot tests on survey questions prior to administering the survey; and incorporating response validation within the survey (e.g., requesting responses when a question is skipped, enforcing validation checks on question types). Program staff will understand the purpose, background, objectives, and importance of the project, as well as their specific role and activities on the study.

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

To maximize response rates, respondents will be informed prior to the evaluation by email, and up to 3 follow-up attempts to contact non-responders will be made within six weeks of the dissemination of the survey. Communications with respondents will be concise and personalized. In addition, the survey was designed to be as short as possible while still eliciting information on the medium-term outcomes of interest. Response rates will be measured and recorded and once surveys have been completed, a non-response analysis will be conducted. The evaluation team will look for patterns of non-response (e.g., year of participation) and discuss any patterns that may emerge with NHGRI. Depending on the presence of patterns, and contextual information provided by NHGRI, the evaluation team will present data on non-response in its final evaluation report as a limitation of the analysis and speculate on how it may have under- or over-estimated the descriptive survey findings.

B.4 Test of Procedures or Methods to be Undertaken

The web-based survey, developed for the current evaluation after the feasibility study, underwent several iterations of development by the evaluation team to ensure questions were clear, concise, and aligned with key evaluation questions of interest. In addition, pilot tests were conducted with three non-evaluation team members to assess conceptual clarity and time burden as well as to test the web-based interface and skip logic. The time estimates for the survey averaged 13 minutes although we assume these may be slightly longer for program participants. Therefore, we present a conservative burden estimate of 30 minutes per survey.

Results from the pilot test revealed a few suggestions for increased clarity of responses (e.g., clarifying new degrees received since taking the Short Course). In addition, other minor changes were made based on feedback including formatting adjustments (font size for certain questions) and minor grammatical changes (e.g., word should be plural).

For quantitative data, internal validity will be checked as necessary for analysis (e.g., examine data for consistency of responses within a case). Descriptive and summary statistics will be calculated from survey responses to inform the evaluation questions. If warranted and appropriate, data may be cross-tabulated to determine if medium- and long-term outcomes differ between the student and faculty participants. Analysts will review and analyze the qualitative data by question (“Beyond curriculum integration, in what other ways did you disseminate information from the Short Course to students?”; “Please describe how your experience at the Short Course influenced your research.”; and “Please describe how the Short Course influenced your decision to pursue new career or educational options.”). The evaluation team will develop and apply a coding scheme to identify themes within the questions. These findings will be incorporated to provide additional contextual information for descriptive quantitative findings.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Dr. Julia Rollison

Evaluation Director, Ripple Effect Communications, Inc.

jrollison@rippleeffect.com; 800.277.5708 x725

Ph.D. in Research, Measurement and Evaluation

Dr. Erica Husser

Senior Analyst, Ripple Effect Communications, Inc.

ehusser@rippleeffect.com

Ph.D. in Human Development

iv