The primary goal of the sampling design is to obtain a national random sample of NIH 2005 and 2006 grant recipients from publicly available information at each of these organizations. It is expected that these faculty will have graduate students in Ph.D, MD/Ph.D, and MD only programs, and most will be faculty members within natural science, engineering, and medical school departments of degree-granting institutions. There are approximately 14,000 NIH grantees per year. Our target population are grantees who are faculty of academic institutions or medical schools/medical centers who have primary responsibility for overseeing a doctoral student in the past 5 years. Based upon a review of the data from NIH grantees, approximately 75 percent of these grantees are faculty of academic institutions or medical schools/medical centers. A two-way stratification is proposed:
Faculty belonging to medical schools both stand-alone and those that are part of large academic institutions
Faculty belonging to all other academic institutions
The names, email addresses, and telephone numbers of 10,000 of these faculty members will be randomly selected within strata ( as indicated in Table 1). Since faculty members can be contacted directly using information from the sampling frame, no cost savings will be realized by clustering faculty members within institutions or any other clustering variable. Hence, individual faculty members can act as Primary Sampling Units (PSUs).
After data collection is complete, we will develop sampling weights to permit expansion of estimates to all faculty in the sampling frame. These sampling weights will be appropriately adjusted for nonresponse. Nonresponse adjustment is necessary to avoid bias in the estimates due to differential nonresponse. Total frequencies such as those for male and female faculty may also be poststratified to totals available in the frame, as deemed necessary after a review of the frequency distributions and finalization of data analysis. Analysis will be performed using software that accommodates the sampling design, such as SUDAAN, so that standard errors are estimated appropriately. The expected response rate using this sample design and the data collection procedures outlined below is 70 percent of those who are eligible because in the last five years they have had primary responsibility for overseeing a doctoral student.
To select a sample of faculty for the purposes of evaluating mentoring and advising of Ph.D students among those faculty, we will use a list of the NIH grantees in the past two years as a sampling frame. This will be used to identify strata from which the sample will be selected. The stratification variable identifying the type of grant the faculty member received can simply be created by merging the two lists and removing duplicates, awards to postdoctoral fellows, and awards to faculty in non-U.S. institutions. Using the Integrated Postsecondary Education Data System (IPEDS) Dataset Cutting Tool, provided by the National Center for Education Statistics (NCES), universities that contain either medical schools or veterinary schools will be distinguished from those that do not, for the purposes of stratification.
This study is focused on a variety of subdomains. A sample size of 10,000 was chosen to accommodate small subdomains, including, for example, faculty who are employed by medical schools that are not affiliated with a larger academic institution. According to the IPEDS Dataset Cutting Tool, there are only 53 such institutions. Previous studies of graduate students have shown about two-thirds report having a mentor. It seems conceivable that a similar proportion of faculty have mentored graduate students in the past 5 years. With a 66% eligibility expected (only those faculty with a recent graduate student in the past 5 years) along with an anticipated response rate of 70 percent, the original sample size is reduced to 4,620. Of the 1,863 institutions that grant graduate degrees, only 535 of which potentially have doctoral students, of which 53 is less than 10 percent. It is therefore conceivable that faculty from these institutions could constitute 10 percent of the sample. We may also be interested in differences in other subdomains such as between males and females in these schools which could result in a subsample of about 233.
b. Estimation procedure
The plan for the statistical analyses of the data, which is mainly limited to descriptive statistics, is presented in Section A. To summarize, this study attempts to get a snapshot view of the state of mentoring and advising in U.S. universities, using a variety of metrics. SUDAAN will be used to provide the standard error estimates to accommodate the sampling design.
c. Degree of accuracy for the purpose described in the justification
As explained in subsection (a) above, the large sample size is necessary to detect differences in study variables for small subdomains. We have shown how a subdomain with 233 respondents is conceivable for this study. Comparing two subdomains, with equal numbers in each stratum, for a comparison of proportions with 250 in each domain, with 80 percent power and a 5 percent level of significance, it would be possible to detect a difference of proportions of 0.145. This assumes a population size of 20,000.
The survey will be introduced by sending an email invitation to each faculty member who has been sampled (Appendix C). The literature suggests web surveys have a higher response rate when an email with the URL address is sent to the sample member. The email will introduce the study, stress its importance, review confidentiality, and provide a toll-free telephone number and an email address for the study’s help desk. The email will also include information about the web address of the survey (URL), and the user ID and password.
Following up on the initial invitation to participate in the survey, nonresponders will receive up to five email reminders, one every four days (Appendix C). As people complete the survey, the database used to track contacts will automatically delete them from the group scheduled to receive reminders. To address the possibility that spam filters may block the emails, the sender will be identified as the Office of Research Integrity. Emails will be sent on a staggered basis to reduce the possibility of some institutions receiving a large number of email invitations on one day.
Follow-up reminder telephone calls (Appendix C) will be made to sample members who have not responded after the email reminders. As needed to improve the response rate, sample members may also be offered the opportunity to complete the questionnaire with a telephone interviewer.
The data collection methods are designed to reach a targeted 70 percent response rate in a three-month period. This response rate projection is based on prior experience with similar data collection efforts.
The web questionnaire will be designed to maximize respondent usability, while incorporating web features to minimize the rate of missing data. The visual format will emphasize readability. To address the issue of different browsers and download speeds, we will test the web version thoroughly with several commonly used browsers. In addition, the web-based questionnaire will be thoroughly tested to verify proper navigation through the survey instrument. To facilitate response, easy-to-use links to help screens and completion instructions will be included. To ensure that respondents complete critical items, we will incorporate web functionalities, such as the ability to stop respondents from moving to another question without completing the critical item. For respondents’ convenience, the survey will be designed so that, if respondents cannot complete the survey at one time, their answers will be saved and they can access the survey later to complete it. Rigorous requirements will be in place to protect the security of the information over the internet. The project programmer will inspect the database to ensure that data are stored properly, secure on the web server, and within the firewalls.
This survey has a single data collection cycle.
To maximize the response rate, the email invitation with the URL and the easy-to-complete web questionnaire are designed to encourage participation. As faculty complete the survey, the database used to track contacts will automatically delete them from the group scheduled to receive email reminders. For those who do not respond, there will be follow-up email reminders and, as needed, follow-up reminder telephone calls to attain the projected response rate. Professional, experienced interviewers will make the follow-up calls; they will receive training on the background of the project, information on the sample members being contacted, and the survey instrument. During the final contact, we will offer sample members the opportunity to complete the questionnaire by telephone. We expect to achieve a 70 percent response rate using these methods.
To avoid bias in estimates, sampling weights will be adjusted for nonresponse. It is anticipated that it may not be possible to contact selected faculty, and if they are contacted, some may refuse to respond, or fail to respond for other reasons. Following standard practice, weight adjustments will occur in two steps, with the first adjustment for unlocated sample members, and the second adjustment for uncooperative sample members.
During the questionnaire development, nine in-person interviews were conducted to pilot test the questionnaire with NIH grantees who have had primary responsibility for overseeing at least one doctoral student’s research within the last five years. These faculty members were employed at several different academic institutions and in various departments. A protocol was developed for the pilot testing that included an assessment of the survey instrument’s overall clarity, wording of specific questions, faculty member’s understanding of the questions and terminology, and their suggestions for improving the survey. Careful testing provides a quality review on instrument wording, skip logic, transitions, and response burden to participants. With the pilot testing methodology, we monitored and debriefed participants to assess comprehension, clarity of instructions, question flow, and organization. The testing was used to provide an estimate of respondent burden for completing the questionnaire.
The following people were consulted on the statistical aspects of the study design:
Sandra Titus, Office of Research Integrity, 240-453-8437
Janice Ballou, Mathematica Policy Research, Inc., 609-750-4049
Arthur Bonito, RTI International, 919-541-6377
Laura Kalb, Mathematica Policy Research Inc., 617-301-8989
Julie Ladinsky, Mathematica Policy Research Inc., 609-936-2764
Eric Grau, Mathematica Policy Research Inc., 609-945-3330
Fran Macrina, Virginia Commonwealth University, 804-827-2262
Frank Potter, Mathematica Policy Research Inc., 609-936-2799
Brian Roff, Mathematica Policy Research Inc., 609-750-4041
This group consists of sampling statisticians and survey methodologists who have extensive experience in the design and implementation of surveys. There is also a subject matter expert on the team.
Arce, C., and W.H. Manning. “Minorities in Academic Careers: The Experience of Ford Foundation Fellows.” New York: Ford Foundation, 1984.
Blackwell, J. Mainstreaming Outsiders: The Production of Black Professionals. Second edition. Dix Hills, NY: General Hall, 1987.
Committee on Assessing Integrity in Research Environments, Board on Health Sciences Policy and Division of Earth and Life Studies, Institute of Medicine, National Research Council of the National Academies: “Integrity in Scientific Research: Creating and Environment That Promotes Responsible Conduct,” The National Academies Press, Washington, DC, 2002.
Council of Graduate Schools, Ph.D. Completion Project, “Ph.D. Completion and Attrition: Numbers, Leadership, and Next Steps,” Washington, DC, 2004.
Cronan-Hillix, T., L.K. Gensheimer, W.A. Cronan-Hillix, and W.S. Davidson. “Students’ Views of Mentors in Psychology Graduate Training.” Teaching of Psychology, vol. 13, 1986, pp. 123–127.
Guston, D.H. “Mentorship and the Research Training Experience.” In Responsible Science, Volume II: Background Papers and Resource Documents. Washington, DC: National Academy of Sciences, 1993. [www.nap.edu/openbook/0309047889/html/50.html]. Accessed June 11, 2006.
Haring-Hidore, M. “Mentoring as a Career Enhancement Strategy for Women.” Journal of Counseling and Development, vol. 66, 1985, pp. 147–148.
Hartnett, R.T. “Environment for Advanced Learning.” In Scholars in the Making: The Development of Graduate and Professional Students, edited by J. Katz and R.T. Hartnett. Cambridge, MA: Ballinger Publishing, 1976, pp. 49–84.
Langlais, P. “Ethics for the Next Generation.” Chronicle of Higher Education, January 13, 2006. [http://chronicle.com/weekly/v52/19/19b01.htm]. Accessed August 15, 2006.
Martinson, B., M. Anderson, and R. de Vries. “Scientists Behaving Badly,” Nature, 435, 737 738 (9 June 2005).
National Academy of Sciences. “Adviser, Teacher, Role Model, Friend: On Being a Mentor to Students in Science and Engineering.” Washington, DC: NAS, 1997. [www.nap.edu/readingroom/books/mentor/1.html]. Accessed June 11, 2006.
Nettles, M., and C. Millet. Three Magic Letters: Getting to Ph.D. Baltimore: Johns Hopkins Press, 2006.
Office of Research Integrity. “Survey of Research Integrity Measures Utilized in Biomedical Research Laboratories. Final Report.” [www.ori.hhs.gov/documents/research/ integrity_measures_final_report_11_07_03.pdg] Accessed July 17, 2006.
Office of Research Integrity. “Responsible Conduct in Research Mentoring.” [http://ori.hhs.gov/education/products/niu_mentorship/mentoring/relationship/relationship.html]. Accessed August 15, 2006.
“Ph.D. Completion Project.” [http://www.phdcompletion.org/ information]. Accessed June 11, 2006.
University of Toronto. “Graduate Supervision: Guidelines for Students, Faculty, and Administrators.” Toronto, Canada, 2002. [www.sgs.utorontoca/current/ supervision/guidelines.pdf]. Accessed on July 17, 2006.
Wright, David E., Sandra L. Titus, and Jered Cornelison. “Mentoring and Research Misconduct: An Analysis of Research Mentoring in Closed ORI Cases” under review 2007.
| File Type | application/msword |
| Author | DHHS |
| Last Modified By | DHHS |
| File Modified | 2008-03-27 |
| File Created | 2008-03-27 |