IMLS_LB21Eval_OMB_Part_B 12.17.12 Final

IMLS Laura Bush 21^st Century Librarian Grant Program Evaluation

Part B. Statistical Methods

General Background:

This evaluation is designed to inform the future directions of the Laura Bush 21^st Century Grant Program, including examining identifiers of past effective grants, a better understanding of funding needs, and best practices and lessons learned that can be considered for future programmatic emphasis. In addition, the study seeks to ascertain whether and how training in the field of library and information sciences (LIS) under the LB21 grant program has sustained benefits for LIS programs and partner organizations in addressing the critical needs of the LIS profession.

The evaluation addresses these intents through 25 research questions distributed across nine sets. (There originally was a tenth set, but it was eliminated in narrowing the scope of the investigation; the same is true for individual questions within Question Set 2 and Question Set 5.)

Question Set 1

1. What is the range of LIS educational and training opportunities that were offered by grantees under the auspices of LB21 program grants?

2. How many new educational and training programs were created by the program?

3. What are the placement outcomes of masters (doctoral) students?

Question Set 2

2-1. Among the sampled institutions, how many students received scholarship funds?

2-2. Were any parts of these scholarship programs sustained with university or private funds?

2-5. What were the important factors for success?

2-6. How effective were the various enhancements to the classroom activities that were provided by the grants (mentoring, internships, sponsored professional conference attendance, special student projects, etc.)?

Question Set 3

3-1. How many of the educational and training programs were sustained after the LB21 grant funds were expended?

3-2. What types of programs were sustained?

3-3. What resources, partnerships or collaborations were used to sustain these programs?

Question Set 4

4-1. Did these new scholarship or training programs have a substantial and lasting impact on the curriculum or administrative policies of the host program, school or institution?

4-2. If so, how were the curricula or administrative policies affected?

Question Set 5

5-1. What impact have these new programs had on the enrollment of master's students in nationally accredited graduate library programs? What impact have these LB-21 supported doctoral programs had on librarianship and the LIS field nationwide?

5-2. How have LIS programs impacted the number of students enrolled in doctoral programs?

5-5. For LB21 master's programs with library partners and/or internships as a program enhancement, did the employment opportunities/outcomes of program participants improve as a result of program participation?

Question Set 6

6-1. What substantive areas of the information science field are LB21 supported doctoral program students working in? What substantive areas of the information science field are LB21 supported early career faculty members working in?

6-2. Are these programs that will prepare faculty to teach master's students who will work in school, public, and academic libraries or prepare them to work as library administrators?

Question Set 7

7-1. What is the full range of "diversity" recruitment and educational activities that were created under the auspices of LB21 program grants?

7-2. What are the varied ways in which grant recipients have defined "diverse populations"?

7-3. Which of these programs were particularly effective in recruiting "diverse populations"

7-4. What were the important factors for success?

Question Set 8

8-1. What is (are) the most effective way(s) to track LB21 program participants over time?

8-2. What is the state of the art in terms of administrative data collection for tracking LB21 program participation among grantee institutions?

8-3. How can social media technologies be employed to identify and track past LB21 program participants?

Question Set 10

10. What has been the impact of the research funded through the LB21 program?

This is a qualitative program evaluation that usesusing a comparative case study approach. The initial unit of analysis consists of a grant project that was funded by the Laura Bush 21^st Century Librarian (LB21) Grant Program. LB21 grants included in this study were awarded in six grant program categories: (1) Master’s Student, (2) PhD Student, (3) Early Career Faculty, (4) Research, (5) Continuing Education, and (6) Institutional Capacity.

The characteristics of the projects funded by the program differ substantially across these grant program categories. The program guidelines across all grant program categories (except for research) asked grantees to address the need for greater diversity in the library profession. However, grantees varied widely within and across the five grant program categories in how they addressed diversity. The evaluation’s emphasis is on learning more about similarities and differences within and across grant program categories and their subsets where possible (particularly the five grant program categories in which diversity is an explicit theme). Thischoice resultsThis choice results in an evaluation design with two levels of cases for analysis: (1) grant program categories and (2) grant program subsets for five of the six grant program categories.

Exhibit 1. Levels of Analysis

Level 1 – Grant Program Category	Level 2 – Grant Program Category Subset
Masters Students	Masters Students Diversity
	Masters Students Non-Diversity
PhD Students	PhD Students Diversity
	PhD Students Non-Diversity
Continuing Education	Continuing Education Diversity
	Continuing Education Non-Diversity
Institutional Capacity	Institutional Capacity Diversity
	Institutional Capacity Non-Diversity
Early Career Faculty	Early Career Faculty Diversity
	Early Career Faculty Non-Diversity
Research

Major aims of the evaluation are to identify the lasting effects of the LB21 grant program and to identify best practices and innovations, developed in LB21 grant programs that can be more widely promoted to benefit and enrich the field of library and information science (LIS). A key indicator of both lasting effects and beneficial innovation is the continuation of programs or program elements by the grant-receiving organizations after LB21 grant funding was exhausted. In such cases, it is presumed that LB21 grant funds acted as “seed money” to enable the creation of programs or program elements that were so beneficial that the grant-receiving institution decided to sustain the innovation using other funds. Because the study seeks to identify exemplar outcomes of LB21 grants, and not to develop conclusions or make statements that characterize the grants as a whole or by category, statistical sampling methods are not used. Rather, cases were selected using stratified purposeful sampling, according to key characteristics and in sufficient numbers to achieve a balance between identification of as many exemplar outcomes as possible while simultaneously minimizing the burden on the universe of potential respondents.

The sources for data collection and analysis for this evaluation combine archived final project grant reports and interview data from grantees to construct a single base of evidence for each grant. This approach to data collection and analysis supports a qualitative comparative analysis that reduces respondent burden with rich enough information to enable comparative assessments within and across each program grant category and subset within a category. Additional comparisons will be made based on emerging patterns and where applicable.

The coding schema for the analysis is iterative, beginning with archival data analysis and proceeding to the interview data. It is to be regularly updated based on interview responses and ongoing internal conversations with the evaluation team in collaboration with IMLS.

Textual content analysis using a pattern matching analysis method will be applied in integrating coded narrative interview data with that for the archival data from final project grant reports. Archival data and interview data will be treated equally during the analysis with an independent field be used to identify the origin of the data to address specific patterns emerging from the blending of archival data with the interview data. Codes will be compared within and across program grant categories and their subsets. The approach will examine emerging themes and patterns as well as contrasts and outliers, primarily involving looking for answers to each of the IMLS-developed research questions for the program grant categories and subsets.

Initial and periodic tests of inter-rater reliability will be conducted to ensure consistency throughout the analysis process. This will be done by having coders code the same portion of text and comparing the codes for consistency. Any points of contention will be discussed and agreed upon. These tests are planned to occur approximately every five hours each week (about one hour each day) over the four weeks of time allotted for the individuals in the evaluation team assigned to coding and anlaysis.

1. Respondent Universe and Selection Methods to Be Used:

The universe of grant projects for this study consists of all LB21 grants awarded in fiscal years 2003 to 2007 for all grant project categories except the Early Career Faculty category. Since the Early Career Faculty category was added to the LB21 grant program in 2007, grants awarded in this category in fiscal years 2007 to 2009 are included in the study. These selection criteria yield a universe of 171 LB21 grants. Because this study draws on both archival data (primarily the final report submitted by the grant recipient) and data to be obtained via telephonic interview, only grants for which a final report exists are included in the study. This requirement eliminates one grant project for an effective universe of 170 grant projects.

The selected sampling strategy was developed based on the structure of the LB21 grant program. This program has six grant categories that address substantially distinct LIS training and recruitment needs. The categories are the following with total number of project grant awards in the time interval covered by the evaluation listed in parentheses:

1. LIS academic departments for support of masters’ students (82).

2. LIS academic departments for support of PhD students (21).

3. LIS academic departments and other organizations offering LIS continuing education (27).

4. LIS early career faculty support (13).

5. LIS academic departments for improved institutional capacity (17).

6. LIS faculty and other scholars’ research on LIS professional trends (10).

The emphasis on learning more about similarities and differences within and across grant program categories and their subsets where possible (particularly the five grant program categories in which diversity is an explicit theme) implies an evaluation design that incorporates different levels of analysis. This proposed evaluation approach operates at two levels: (1) grant program categories; and (2) grant program category sub-sets.

Exhibit 1. Levels of Analysis

Level 1 – Grant Program Category	Level 2 – Grant Program Category Subset
Masters Students	Masters Students Diversity
	Masters Students Non-Diversity
PhD Students	PhD Students Diversity
	PhD Students Non-Diversity
Continuing Education	Continuing Education Diversity
	Continuing Education Non-Diversity
Institutional Capacity	Institutional Capacity Diversity
	Institutional Capacity Non-Diversity
Early Career Faculty	Early Career Faculty Diversity
	Early Career Faculty Non-Diversity
Research

The evaluation adopts stratified purposeful sampling. This approach allows for illustrating characteristics enabling credible comparisons of all six grant program categories and program grant category subsets. In determining this sample, a proportional case size by category equal to 75 percent of the grants within each grant type was selected as this reasonably allows for adequate representation in all six program grant categories and to the grant category emphases in the five program grant categories where diversity is an explicit objective. This level reasonably allows for sufficient cases for within and across group comparative analysis while streamlining collection and reducing respondent burden.

The following approach was employed to identifying the grants to use within each grant type in doing the sampling. The first step was to identify the primary research goals. The primary goal of the research is to determine the approaches used by the more successful grant projects (best practices and lessons learned). Two secondary goals are to emphasize larger programs (since they have a greater monetary risk) and more recently employed practices (since these presumably will be more germane to the types of project grants that will be considered for future funding). To address these goals, cases were systematically removed from the universe of all projects comprised within each program grant categoryconsideration using this initial criterion: 1) Projects ranked with a value of 3 or below (using a 5-point ranking scale) based on an assessment of the project grants’ quality and richness from an overall analysis of each final project grant report. Afterwards, if projects still needed to be removed from any group, they were done so by following these two rules: 2) all else equal, larger grants were favored; and 2) all else equal, more recently awarded grants were favored.

Exhibit 2. Cases in Grant Program Categories and Grant Program Category Subsets

	Universe	Sample
Masters Grant Program Category: Diversity Subset Non-Diversity Subset	55 27	42 21
PhD Grant Program Category: Diversity Subset Non-Diversity Subset	8 13	6 10
Continuing Education Grant Program Category Diversity Subset Non-Diversity Subset	7 20	7 15
Institutional Capacity Grant Program Category Diversity Subset Non-Diversity Subset	7 10	6 8
Early Faculty Career Grant Program Category Diversity Subset Non-Diversity Subset	3 10	3 9
Research Grant Program Category	10	8

2. Procedures for Collection of Information:

This program evaluation does not employ statistical methods such as stratification, sampling strategies, or estimation techniques. In addition, data collection will not include periodic cycles and will instead be collected at one point in time.

The sources for data for this evaluation covering each project selected in the sample for this evaluation involve archived documents and telephone interviews. The two sources of archived data are final grant reports and project grant summaries which were posted on the IMLS website. Consent is not necessary for the use of such documents in this evaluation. There are no other archived documents used in this evaluation pertaining to past grant submissions.¹ The collection and analysis of archived final project reports preceded any data collection and analysis of interviews.

Contact information for telephone interview of grantees was initially validated by ICF International earlier this summer to expedite the final interview process. Copies of the telephone script and report of the most recent results of the endeavor as well as the script to be used for scheduling interviews are attached as appendices. Copies of the email notice, telephone script and report of the most recent results of the endeavor, and script to be used for scheduling interviews are attached as appendices.

Grantees of the selected cases will be contacted by IMLS to introduce the study before interviewing formally begins. This contact will be signed by the Director of IMLS to explain the research study and emphasize its importance and high-level interest in its outcomes. In addition to introducing the study and emphasizing the importance of participation, this communication will also introduce the study contractor who will be in direct contact with respondents for the data collection. This introduction will verify the legitimacy of the contractor’s subsequent contact. This formal letter will be the first and only official communication to the respondents from IMLS.

The study contractor, ICF International, will subsequently contact each respondent to further explain the research, describe confidentiality provisions, and schedule a convenient time for a telephonic interview of approximately one hour. If necessary, multiple attempts, using multiple modes of communication, will be made to schedule interviews. Communication efforts will be made by telephone calls and emails. Telephone and email scripts are included. Once an interview is scheduled, respondents will be sent a copy of the interview questions in advance of the interview so they can review the topic areas and refresh their minds about their grant programs. All interviewers have been trained in scheduling and conducting interviews. The training protocol is included.

. In order to minimize burden on respondents, verbal consent will be obtained during the phone interview after respondents are read their study rights. This is done to decrease burden on respondents by eliminating the need for an additional contact that would be required by using a written consent form that would need to be read, singed, and returned prior to the interview. Verification questions are included in the consent script to ensure the respondents understand their rights. There will be two individuals on the line to record verbal consent to these verification questions. The consent verification portion of the script is included in each of the interview protocols. Copies of the interview instruments are attached with the protocols to be followed for securing verbal consent. delineated in the beginning of each one.

Care will be taken to maximize the benefit of each interview and to avoid wasting the respondent’s time. The interview protocols will be tailored with questions relevant to the particular grant type. Prior to each interview, the interviewer will conduct a thorough review of data collected from a completed archival data analysis of grant files. This review will guide the interviewer in tailoring the interview to the unique characteristics and themes of the particular grant. Tailoring each interview in this way will make the best use of time and permit a richer dataset to be obtained by avoiding lines of inquiry that are already adequately addressed by the archival data analysis. Interviewer preparation and familiarity with the individual grant will enable on-the-fly adjustment of interview probes and follow-up questions to maximize depth and richness of data collected. With the archival data and interview protocol in hand, the interviewer will lead a guided conversation to bring forth essential information about the grant program’s activities and participants.

Interviews will be scheduled using the following procedure:

1. ICF will email interviewees, soliciting participation and requesting that they make contact to schedule their interviews.

2. If a response from the interviewee is not received within 2 business days, ICF will call them directly. If contact is not made, ICF will leave a voice message for the interviewee (or a message with someone else in the organization) requesting that the interviewee call back to schedule an interview.

3. Phone calls will be followed within 30-60 minutes by emails providing the same information, as many of the interviewees have indicated they are best reached through email. Emails are to be sent even if ICF is not able to leave a voice message.

4. This process of telephone calls followed by emails will be repeated two additional times, with a three day wait between contact attempts.

Templates of these notifications can be found at the end ofarea attached to this document.

3. Measures for Promoting a Maximum Response Rate & Addressing Non-Response:

In general, our strategy for promoting a maximum response rate depends on conveying an understanding of the importance of the research, promoting a sense of responsibility to the LIS field and professions, developing close relationships with the respondents, and employing multiple contacts and reminders as necessary. The primary threat to response rates is invalid contact information. To address this, ICF will be validating contact information and identifying gaps in information in advance of the study implementation. ICF will employ multiple methods to identify correct information including web searches and, if unable to obtain correct information via these less obtrusive methods, contact with University administrators.

4. Testing Procedures and Methods to be Undertaken:

There are no testing procedures used in this study. Information will be gathered using both archival research supplemented with data collected from phone interviews. Interviews will be guided by protocols containing questions, probes, and follow-up questions that have been specifically selected for each grant type, and in some cases theme within a grant type. The questions in each protocol were selected to address issues and provide information of greatest importance to the LB21 grant program and IMLS’s administration of the program. In general, the conversation resulting from each protocol will generate a more thorough picture of the program’s components, including overall goals, recruitment strategies, special activities, factors for success, and outcomes. A total of 11 different interview protocols will be employed in this research. This tailored approach will help ensure that the respondent is satisfied with the relevance of the interview and the value of the conversation in achieving important research goals.

5. Data Collection Contact:

Data will be collected for IMLS by an experienced team at ICF International, 9300 Lee Highway, Fairfax VA 22031. The team is led by:

Dr. Arnold Leonard — (703) 934-3604

Dr. Bradford Booth — (703) 934-3164

The IMLS data collection contact in Dr Matthew Birnbaum – (202) 653-4760.

The IMLS data collection contact is Dr. Matthew Birnbaum – (202) 653-4760.

Copies of the instruments used by the contracted evaluator in establishing telephone contact with the respondents and to be used in scheduling interviews are attached.

******************************************************************************

PHONE AND EMAIL NOTIFICATION TEMPLATES

******************************************************************************

Hello, my name is [FIRST NAME] [LAST NAME] and I am calling on behalf of IMLS [IF PROMPTED, the Institute of Museum and Library Services], about the Laura Bush 21 Grant Program [IF PROMPTED, the LB21 program]? Can I speak with [INSERT GRANTEE NAME]?

When grantee takes the line, or answering individual questions purpose of call:

[Reintroduce SELF as necessary] I’m calling because IMLS and its research contractor ICF International are hoping talk to you [or name GRANTEE] about an ongoing study of the LB21 Grant Program. You should have received a letter from IMLS in the mail recently, explaining the purpose of the evaluation program and letting you know that we would be calling to schedule a phone interview with you. Did you receive this letter?

[IF YES]

I’m pleased to hear that. We are interested in understanding your motivations for seeking the grant funds, how you used the grant funds, and any sustained benefits the grant program has provided. The interview will take no longer than an hour and will be scheduled at your convenience. When during the next few weeks would be a good time for you to complete the interview regarding your experiences with the LB21 program?

[IF NO]

I’m sorry to hear that you did not receive that. Let me take a moment to explain the project. IMLS is conducting an evaluation of the LB21 grant program for the purposes of identifying best practices and lessons learned for the future of the grant program. As part of the evaluation being conducted by their research contractor ICF International, we will be conducting phone interviews with past recipients of the grants to learn more about their experiences with the grant program. We are interested in understanding your motivations for seeking the grant funds, how you used the grant funds, and any sustained benefits the grant program has provided. The interview will take no longer than an hour and will be scheduled at your convenience. When between [three days from now] and [END OF DATA COLLECTION] would be a good time for you to complete the interview regarding your experiences with the LB21 program?

[IF YES]

Great! We will be conducting the interviews between [three days from now] and [END OF DATA COLLECTION]. When is a good time for you?

[MATCH DATE/TIMES PROVIDED TO INTERVIEWER AND RECORDER SCHEDULES TO SCHEDULE THE INTERVIEW]

Also, I just want to take a minute to make sure we have the correct contact info for you for the summer months.

We currently have the following contact information for you from IMLS. [READ CURRENT LIST AND UPDATE AS NECESSARY]

Field	Current List	Update/Summer Info
Title	TITLE	TITLE
Last Name	LAST NAME	LAST NAME
First Name	FIRST NAME	FIRST NAME
Institution	INSTITUTION	INSTITUTION
Telephone Number	PHONE	PHONE
Email Address	EMAIL	EMAIL

Thanks so much for your help today. You will be interviewed by [INTERVIEWER NAME]. She will send an email to you later today verifying the date and time of the interview. The email will also have a toll free phone number for you to call in for the interview and a passcode. If you would any questions in the meantime, you may contact me at [PHONE].

Thanks once again.

[END CALL]

[IF NO]

Ok. Is there anyone else who was familiar with the grant project who we may be able to contact for an interview? We are looking for someone who is very familiar with the grant project from its initiation to its completion and would be able to provide information on the motivations for seeking the grant funds, how you used the grant funds, and any sustained benefits the grant program has provided.

Thank you so much. Can you please spell [his/her] name for me? [RECORD NAME]

What is the best way is to reach [new contact name]?

[RECORD CONTACT INFORMATION; PROBE FOR ALL INFORMATION BELOW]

Title	TITLE
Last Name	LAST NAME
First Name	FIRST NAME
Institution	INSTITUTION
Telephone Number	PHONE
Email Address	EMAIL

Thanks so much for your help today. If you would like to verify this study or if you have any questions, you may contact [ENTER NAME] AT XXX.XXX.XXXX .

Thanks once again.

[END CALL]

IF SENT TO VOICEMAIL:

Hello [CONTACT NAME], my name is [FIRST NAME] [LAST NAME] and I am calling on behalf of the Institute of Museum and Library Services, about the Laura Bush 21 Grant Program. I’m calling because IMLS and its research contractor ICF International are hoping talk to you this summer about an ongoing study of the LB21 Grant Program and if it was helpful. We would like to schedule a phone interview with you for some time between [three days from now] and [END OF DATA COLLECTION]. Please give me a call back to schedule a time for your interview. My number is [PHONE]. If you are not the correct person to contact regarding this data collection effort, please let me know so that I may contact the appropriate person. Thank you.

(Note: Follow up with email below within 30-60 minutes.)

IF ASKED TO LEAVE A MESSAGE WITH SOMEONE ELSE:

My name is [FIRST NAME] [LAST NAME] and I am calling on behalf of the Institute of Museum and Library Services, about the Laura Bush 21 Grant Program. I’m calling because IMLS and its research contractor ICF International are hoping talk to [GRANTEE] this summer about an ongoing study of the LB21 Grant Program and if it was helpful. Could you leave him/her a message to call me back to discuss the possibility of scheduling a short phone interview? My name is [NAME] and my number is [PHONE]. Thank you.

(Note: Follow up with email below within 30-60 minutes.)

FOLLOW UP EMAIL:

Dear [INTERVIEWEE],

I am writing on behalf of the Institute of Museum and Library Services (IMLS), about the Laura Bush 21 Grant (LB21) Program. IMLS and its research contractor ICF International are hoping talk to you this summer about an ongoing study of the LB21 Grant Program. I left a voicemail for you earlier.

You should have received a letter from IMLS in the mail recently, explaining the purpose of the evaluation program and letting you know that we would be contacting you to schedule a phone interview. In case you did not receive that letter, let me take a moment to explain the project. IMLS is conducting an evaluation of the LB21 grant program for the purposes of identifying best practices and lessons learned for the future of the grant program.

As part of the evaluation being conducted by their research contractor ICF International, we will be conducting phone interviews with past recipients of the grants to learn more about their experiences with the grant program. We are interested in understanding your motivations for seeking the grant funds, how you used the grant funds, and any sustained benefits the grant program has provided. The interview will take no longer than an hour and will be scheduled at your convenience.

Please respond to this email, or call me at [PHONE] to schedule a time for your interview. We will be conducting the interviews over the next few weeks.

Thank you,

[NAME]

1 The contractor for this program grant evaluation also received a separate source of archived data that contains brief project grant summaries that IMLS publicized on its website for announcing project grant awards in the LB21 program. There is no information contained in each brief grant summary that is not contained in the final project grant report. The use of the grant summaries in the evaluation was consequently limited to the initial clarification of general research questions and their associated operational questions.

13