'09-05_ose_omb_ssb

Supporting Statement B for

Effectiveness of the NIH Curriculum Supplements Programs and Career Resources Study (OD/OSE)

December 4, 2008

Name: Dave Vannier, Ph.D. (OD/OSE)

Address: 6100 Executive Blvd, Suite 3E01

Telephone: (301) 496-8741

Fax: (301) 402-3034

Email: vannierd@od.nih.gov

B.1 Respondent Universe and Sampling Methods

Surveys of Customers of the NIH curriculum supplements and career resources

16,000 randomly-selected NIH curriculum supplement customers will be directed, via e-mail, to a 10-minute, Web-based survey that will measure implementation and satisfaction. This instrument will be very similar the survey used for the same purpose in the evaluation of the first three supplements (OMB No. 3145-0192). The survey will be sent once OMB approval is obtained.

Since 2004, 7,600 individuals have requested the Women Are Scientists videos. OSE will randomly survey 1,500 of these requestors, 20% of the respondent universe.

Teacher Workshop Evaluations

Since FY 2002, over 6,000 science teachers have attended a workshop on using the NIH curriculum supplements. 2,000 of these teachers will be sent a short follow-up survey after he/she attends a workshop. This e-mailed questionnaire will capture 1) how effective the workshop was in preparing the teacher to implement the lessons and 2) suggestions for further support in using the lessons.

In years two and three, two groups of 100 workshop attendees will be selected to answer a more in-depth survey. The selection criteria will reflect populations of interest to OSE, such as middle school teachers, pre-service teachers, urban teachers, and less-experienced teachers. Results from the short survey will likely identify sub-groups that are not being effectively trained during the workshops. These are of obvious interest to OSE and would likely be sent the longer survey

The data collected from the teacher workshop surveys will be subjected to very simple statistical analysis, such as determining means and standard deviations. The evaluation seeks to validate the overall effectiveness of the teacher workshops. It is unlikely that a more statistically rigorous approach will lead to greater insight.

B.2. Procedures for the Collection of Information

Data collection will be conducted electronically. Respondents will be contacted by e-mail and directed to a Web site to complete their respective survey. Data entered on line will go directly into a specific database for tabulation. Data from surveys returned by e-mail will be entered into the specific database by OSE staff.

These surveys seek to measure the effectiveness of OSE programs. A complicated statistical approach should not be necessary for us to meet our evaluation goals.

B.3 Methods to Maximize Response Rates and Deal With Non-response

Teacher Workshop Evaluations: 2,000 teachers will be surveyed for their assessment of the workshop they attended. Since these teachers have met the OSE staff, we anticipate that they will be likely to respond. The initial survey will be extremely concise to minimize the time burden on the participant. During years 2 and 3, 200 respondents from the initial survey will be asked complete a longer survey. We are optimistic that over 80% will respond, and we may offer a $25 honorarium for completing the longer survey. A final decision on using a payment incentive will be part of the justification that OSE submits when seeking OMB approval for the specific study.

B.4 Test of Procedures or Methods to be Undertaken

Prior to implementing any of the surveys proposed in this generic clearance, OSE will provide copies of the survey instrument, a description of the activity, and an estimate of the burden hours involved to OMB for specific approval.

During development, all surveys will be tested among federal employees for ease of completion. Once the surveys have been optimized in house, nine teachers will be randomly selected to test the each instruments as well. The Teacher Workshop Surveys are ongoing evaluations. Early results will influence the procedures and questions for future surveys.

Again, each proposed survey will be sent to OMB for approval prior to use.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Evaluations will rely exclusively on staff in the NIH Office of Science Education. Dave Vannier, Ph.D., will coordinate both projects and may be reached at (301) 496-8741 and vannierd@od.nih.gov.

- 3 -