1875-0258 Att_Carol_White_PEP_Supporting_Statement_Part_B_12112012

American Institutes for Research®

Evaluation of the Carol M. White Physical Education Program -
Supporting Statement for Paperwork Reduction Act - Part B: Description of Statistical Methods

June 27, 2012

Revised December 11, 2012

Prepared By:

American Institutes for Research^®

1000 Thomas Jefferson St., NW

Washington, DC 20007

Prepared for:

U.S. Department of Education

Policy and Program Studies Service

Office of Planning, Evaluation, and Policy Development

Supporting Statement for Paperwork Reduction Act

B. Description of Statistical Methods

1. Universe and Respondent Selection

The surveys of the Carol M. White Physical Education Program (PEP) project administrators will be given to the entire universe (or population), rather than a sample, of the 77 PEP grantees that are FY 2010 cohort PEP grant recipients. This is because the size of the population is small enough (77 grantees) that sampling would not be appropriate. Furthermore, each of the PEP projects is somewhat unique and the U.S. Department of Education (ED) seeks a comprehensive picture of PEP activities. The Year 1 Survey achieved a 100% response rate.

The PEP evaluation’s case study design includes the recruitment of five grantees from the population of 77 (64 district-led projects and 13 CBO-led projects) FY 2010 PEP grant recipients. The American Institutes for Research (AIR) identified a purposive sample of 13 of the 77 grantees that proposed in their PEP applications to engage in body mass index (BMI) data collection and official partnerships. From the subset of 13 PEP projects, AIR recommended a set of five projects for the case study, along with four potential alternates (n=9), based on various project characteristics. The final five grantees will be selected based on discussions with the U.S. Department of Education (ED) and the availability and consent of the grantees. From each of the five PEP projects, telephone interviews will be conducted with the project directors, as well as a sample of up to 14 additional project personnel (such as school administrators (e.g., principals, vice principals), teachers (e.g., physical education [PE], health, nutrition), PE or health and wellness curricula coordinators, athletic directors, nutritionists, and food service staff) and personnel from their established partnerships, for a total of 75 case study participants/interviewees.

As it is unlikely a list of all project staff, personnel, and members of partner organizations exists, a list of potential interviewees will be requested from the project director and will be used as an initial contact list from which to implement a snowball sampling technique to identify a more comprehensive group from which to select the 14 interviewees. This sampling technique is often used when a population or viable sampling frame cannot be identified or obtained. In addition, it is a more rigorous method of identifying sources alternate to a list provided by the target of the evaluation, which may be unintentionally biased. Details regarding the identification of case study participants follow.

The pool of grantees awarded priority points for proposing both BMI data collection and establishing official partnerships included 53 district grantees and 10 CBO grantees. Self-report data from the Year 1 Survey provided another source of information to ensure that those grantees who proposed both priorities actually established the partnerships and implemented BMI data collection. Of the 63 PEP grantees awarded points for both competitive preference priorities, 56 (49 district-led and 7 CBO-led projects) reported they had established partnerships and had collected or had plans to collect BMI data.

To focus on the role of the grants in PEP projects’ BMI data collection and partnership efforts, the sample of grantees was restricted to those grantees who did not engage in these activities before the grant. The Year 1 Survey assessed the activities grantees conducted prior to and after receiving the PEP grants. Over half of the grantees (36), including all but one of the CBO grantees, established partnerships before receiving the PEP grant. A sample of grantees not engaging in such relationships prior to the grant emphasizes the focus on how PEP-supported partnerships function. Twenty-two of the grantees (17 district and 5 CBO) did not collect BMI data prior to the PEP grant.

Only 9 of the sample of 49 district grantees that were awarded points for both competitive preference priorities, and also reported in the Year 1 Survey that they had engaged in partnerships and BMI-related activities as part of their PEP project, indicated no prior partnerships and BMI data collection. With the additional exclusion of one of the nine district grantees, as it was a charter school and not representative of the district grantee population, the final sample of district-led PEP projects for the case study included eight grantees. None of the 7 CBO grantees that were awarded points for both priorities, and that also reported grant partnerships and BMI-related activities as part of their PEP project, met the criteria of no prior partnerships and no prior BMI data collection. All but one CBO grantee indicated prior collaborations with other community entities, and the single CBO grantee without prior partnerships reported collecting BMI data before receiving their PEP grant. However, the inclusion of one to two CBO grantees provides the opportunity to learn about potential differences between the grantee types that could be investigated in the future. Since none of the CBO-led projects met the criteria of no prior partnerships and no prior BMI data collection, the final sample of CBO-led PEP projects for the case study included the five CBO grantees that were awarded points for both competitive preference priorities, reported they had engaged in partnerships and BMI-related activities as part of their PEP project, and had not previously conducted BMI data collection, but had engaged in prior partnerships.

The final subset of 13 grantees includes: 8 district grantees not engaging in partnerships and BMI data collection prior to the PEP grant, and 5 CBO grantees that did not collect BMI data prior to the PEP grant. From the subset of 13 projects, the study team recommended a set of five projects for the case study, along with four potential alternates (n=9), based on project characteristics including: project locale, award amounts, number of reported BMI data collections, number of project partners, and reported level of partner involvement. The final five PEP projects will be based on the Department’s feedback regarding the recommended order, as well as information obtained during the recruitment phase (e.g., agreement to participate, accessibility to target case study project participants, and availability in concordance with the case study timeline).

2. Procedures for Collecting Information

The AIR study team will collect information about the design and implementation of the PEP through four survey instruments developed by the study team: Year 1 and Year 3 Program Design and Implementation (PDI) surveys for LEAs and Year 1 and Year 3 PDI surveys for CBOs.

The surveys will be administered on-line. Paper-and-pencil copies of the surveys will also be available upon request. Administrators of all 2010 PEP grantees will be contacted via email to take the survey. Given that PEP grantees made the commitment to participate in the evaluation in their funding applications, we expect all survey recipients to complete the surveys. In addition, administrators of all PEP grantees will receive a pre-notification email about the study emphasizing its importance and requesting their participation. Three to five emails over a period of two months (depending on response rates monitored on a weekly basis) will follow up with non-respondents to encourage participation.

For the case study component, the AIR study team will collect information about PEP projects’ BMI data collection and use as well as project partnerships through telephone interviews. The study team will email the interviewees the agenda and provide contact information for the staff conducting the interview approximately one week before each interview. Every effort will be made to minimize burden by establishing and maintaining clear communications with participants and remaining flexible to the changing demands of schedules and logistics.

Upon completion of the interviews, emails will be sent to all interviewees thanking them for their time, honesty, and insights, and once again it will be stressed that individual names and grantee-specific information will not be identified in any reports. In addition, interviewees will be informed that they may be contacted with clarification questions if necessary as the report is being written.

3. Methods to Maximize Response Rates

As mentioned in question B2, one method to maximize response rates for the surveys and case study interviews was to require grantees’ commitment to participating in the evaluation in their grant applications. In addition, ED publicized the study at the grantee kick-off meeting and will provide updates to grantees at all grantee meetings. The follow-up procedures for the surveys are discussed in question B2.

For the case study component, the study team expects a response rate of 100 percent for project director, project personnel, and partner personnel because all PEP grantees have agreed to participate in the study and they are all current grantees of the U.S. Department of Education.

In November 2012, the Director of PPSS sent an initial notification letter to the recommended five projects and four potential alternates (n=9), to inform them their project was selected as one of the few that may be included in the case study component of the PEP evaluation. In addition, the study team contacted all relevant school districts’ offices of research activities to procure approval to conduct the interviews and related activities.

The research team will follow-up with calls to each project director to provide more detailed explanations regarding the interview, including the duration and the proposed participants [i.e., project directors, school administrators (e.g., principal, vice principal), physical education (PE), health, and other project-related teachers, food service staff, and representatives of community partners (e.g., local education agencies, CBOs, state or local public health agencies)]. To encourage buy-in, the research team will inform grantees that key findings will be shared with them. Grantees will be notified of the purpose of the interview, ensured that the interviews are not related to any type of project audit or monitoring, and informed that results will not be presented by project. Grantees will also be told that their project will neither be penalized nor rewarded for their participation or related to any information provided during the interviews. However, information regarding agreement or refusal to participate will be provided to ED. If a grantee refuses to participate, the team will contact the next alternate project. This process will continue until five grantees commit to participate. In addition, if available, commitments will be obtained from two alternate grantees in case of attrition.

4. Tests of Procedures

The survey data collection instruments were developed, in part, based on a review of similar questionnaires used previously by ED and vetted by the evaluation’s Technical Work Group (TWG), a collection of experts in pertinent fields including prior grant recipients.

Usability of the website for the surveys was examined by AIR’s subcontractor, FirePig Partners, who specializes in online survey design and administration. Based on their recommendations and reviews by the study team, revisions were made as necessary to the design of the online surveys. The Year 1 Survey achieved a 100% response rate.

The case study design uses similar techniques employed by process evaluations and interview protocols used previously by ED. In addition, TWG members vetted the case study design and interview protocols.

5. Contacts for Statistical Aspects and Data Collection

The following individuals developed the study design, including statistical aspects and techniques for data collection:

• Tanya Taylor, Research Analyst, American Institutes for Research

• Wehmah Jones, Research Analyst, American Institutes for Research

• Herbert Baum, Research Analyst, Data Recognition Corporation

• Elham-Eid Alldredge, Research Analyst, Data Recognition Corporation

Notice of Trademark: “AMERICAN INSTITUTES FOR RESEARCH” and “AIR” are registered trademarks. All other brand, product, or company names are trademarks or registered trademarks of their respective owners.

1000 Thomas Jefferson Street, NW | Washington, DC 20007‑3835