LEA - Carol M. White Physical Education Program Survey

TO: Erica Lee, Program and Policy Studies Service (PPSS)

FROM: Tanya Taylor and Lauren Amos, American Institutes for Research (AIR), and Herbert Baum, Data Recognition Corporation (DRC)

RE: Final Case Study Protocols, Evaluation of the Carol M. White Physical Education Program (PEP)

DATE: July 27, 2012

This memorandum presents the final case study protocols for the qualitative data collection component of the Evaluation of the Carol M. White Physical Education Program (PEP). To address the evaluation questions specified in the Statement of Work (SOW) for the PEP evaluation, AIR proposed a two-component multi-method research plan that employed the use of (1) self-report surveys to obtain information from grantees about key design features of the PEP projects and their implementation process, and (2) case studies to garner more in-depth information about PEP projects, with a specific focus on those elements that changed most significantly with the Fiscal Year (FY) 2010 grantees.

The case study design (see Final Case Study Design Memo for the Evaluation of the Carol M. White Physical Education Program) focuses on PEP projects’ experiences related to the two new program competitive preference priorities: the collection and use of body mass index (BMI) measurements and the establishment of official partnerships. The case study research questions incorporate and expand on a subset of the SOW evaluation questions to garner illustrative data related to both partnerships and BMI data collection. Exhibit 1 lists the evaluation questions from the SOW and the related case study questions.

Exhibit 1. SOW and Related Case Study Evaluation Questions.

SOW Evaluation Questions	Related Case Study Questions
How many grantees develop partnerships with supporting community entities?	How did grantees establish partnerships? Does this vary by type of partner, and if so, how?
How do grantees work with partners?	What are the characteristics of partnerships that are unsuccessful? What are characteristics of partnerships that are maintained (i.e., that succeed)? Do grantees plan to sustain their projects after the grant funding period? If so, are there plans to sustain partnerships? How?
What roles do partners play?	What, if any, are the reported benefits of partnerships? What, if any, are the reported challenges of partnerships? Do benefits and/or challenges vary depending on type of partner, and if so, how?
How many grantees inform parents about the result of their child’s BMI assessment?	How are grantees collecting BMI data?
How do grantees use data, such as BMI and other performance measures, for improvement purposes?	How are grantees using BMI data? What, if any, are the reported challenges associated with BMI data collection? What, if any, are the reported benefits associated with BMI data collection?

Based on the revised SOW, the case study approach changed from conducting site visits to include conducting detailed telephone interviews with up to 15 key personnel from a sample of five PEP projects. Since PEP projects represent a diverse group of grantees, a brief screening survey will be administered to up to nine project directors from the small sample of case study sites in order to assess the appropriate key personnel to interview. Thus, AIR and DRC developed three semi-structured telephone interview protocols to address the following groups of interviewees.

• Project directors

• Project personnel, which may include but are not limited to: school administrators (e.g., principals, vice principals), teachers (e.g., physical education [PE], health, nutrition), PE or health and wellness curricula coordinators, athletic directors, nutritionists, and food service staff

• Key stakeholders at the community partners

Semi-structured interview protocols allow for the natural flow of information. The interviewer and respondent engage in a formal interview using the protocols, which contain a list of fundamental questions and topics to be covered during the conversation. The interviewer follows the protocol, but the semi-structured nature of the interviews and provided probes allow for a more free-flowing conversation and the ability to follow tangents (as appropriate) that provide important information that otherwise may have been missed.

The protocols were developed to address the focal case study questions in a format that allows for consistency in the information gathered across grantees and within respondent type, while permitting participants to provide detailed descriptions of their unique experiences related to BMI data collection and use, and the establishment and sustainability of official partnerships.

The telephone interview protocol for project directors concentrates on obtaining information describing how partnerships were established, project activities involving partners and their level of involvement, and the nature of the partner relationships, as well as how BMI data are being collected, reported, and used. Perceived challenges and benefits of each will also be gathered. The use of open-ended questions encourages detailed responses, and the probes were designed to both engage participants in a dialogue and address information provided in the Year 1 Survey (e.g., personnel involved with PEP projects, types of partnerships, benefits and challenges of partnerships, planned use of BMI data) in order to facilitate triangulation analyses (i.e., using the multi-method case study and survey data to assess the same evaluation questions). The additional two telephone interview protocols are developed to gather similar information, but from the differing perspective of each group, which will allow for consistency in type of information obtained across interviews for comparison analyses. The respective telephone protocols for each group follows.

Telephone Interview Protocol:

Project Directors

The protocol provides a list of planned key questions. The interview itself will be conversational with questions flowing from previous responses when possible. Note that elements of the interview, such as specific partnerships and plans for BMI data collection and use, will be informed by information previously obtained through materials provided by the grantees and the screening survey.

Introduction

Hello [INTERVIEWEE], my name is [NAME] and [NAME] is also on the phone. We are researchers from the American Institutes for Research (AIR) and Data Recognition Corporation (DRC) and as you know we are conducting an evaluation of the Carol M. White Physical Education Program (PEP) sponsored by the U.S. Department of Education. I want to thank you for taking the time to speak with me today. The purpose of the evaluation is to gain an understanding of the activities being conducted by PEP projects in order to offer insight into how the program may be enhanced to further promote healthy living among America’s youth.

For reference and to clarify the notes, I would like to record the session. You may request that the recording be stopped at any point during the interview. May I do so?

[AFTER RECEIVING PERMISSION, BEGIN RECORDING; IF NO, PROCEED WITHOUT]

[THE FOLLOWING PROVIDES THE INFORMATION FOR THE REQUIRED ELEMENTS OF INFORMED CONSENT INCLUDING: PURPOSE OF THE STUDY/INTERVIEW; PROCEDURES; CONFIDENTIALITY; VOLUNTARY PARTICIPATION; AND DOCUMENTATION OF CONSENT]

The purpose of this call is to obtain more in-depth information about PEP projects and to learn about the program. Specifically, in your role as Project Director (PD) of a PEP grant, we want to learn how the grant has affected project activities involving the collection and use of BMI data and the establishment of partnerships. We are also interested in obtaining a better understanding of what is working well, in addition to any challenges you may have encountered. In addition to project successes, we understand that things do not always go as planned, and in order to learn about all aspects of the project, we encourage you to be candid in your responses. This information will be used by the Department to improve the current program and help future programs with their start-up and implementation.

I will be conducting today’s interview and [NAME] will be taking notes. Participation in this interview as part of the evaluation of PEP is voluntary. You can decide not to participate or to discontinue your participation at any time without penalty. [You have agreed to this interview being recorded and may request that the recording be stopped at any point.] You also do not have to answer any questions you do not want to answer. Everything you say here will be kept confidential and no response will be attributed to any one grantee.

It’s useful to include quotes from interviews in evaluation reports. No names, job titles, locations, or specific PEP projects will be provided with any quotes, feedback, or comments you provide today. In addition, you may indicate at any time if you do not want specific feedback you provide to be quoted. Today’s session will take about one hour. I may occasionally interject with additional questions or to ask for clarification or detail, but otherwise this is your chance to speak freely about your experiences. Nothing you say will impact current or future PEP funding. You may contact us at any time with questions about the evaluation or this interview that we do not cover during this session.

Do you have any questions? Do you agree to participate in today’s interview?

[AFTER RECEIVING VERBAL INFORMED CONSENT, BEGIN INTERVIEW]

Interview

Background: (10 minutes)

1. Briefly describe your role as PD on this PEP grant.

Probes:

1. How long have you been the PD?

2. How did you become involved with this grant? (Probe for details related to prior roles or experiences with health-related programs)

3. Were you involved in the writing of the grant?

   1. If yes: Has this been helpful over the course of implementing the grant?

      1. If yes: How so?

   2. If no: Did this present any challenges over the course of implementing the grant?

      1. If yes: Please describe.

4. What are your responsibilities?

5. On which project activities do you spend most of your time?

2. Of those health-related (e.g., nutrition, fitness, physical well-being) needs your PEP project addresses, what would you say are the most pressing needs of the students you serve?

(Probe for reasons why these needs: e.g., has the project assessed this? If yes, in what way(s)?)

3. What do you see as the primary goals of your PEP project?

1. Have the goals been modified over the course of the PEP grant?

Partnerships: (10 minutes)

Now I’d like to learn more about the partner organizations that you work with as part of PEP, starting with how these partnerships were established and then moving into a description of the role that these partners play in implementing PEP.

4. I’d like to find out more about how those partnerships specific to the PEP grant were established and maintained. To the best of your knowledge, how did the partnership come about?

(Ask about each partner; If necessary provide examples: prior partnering relationship with the grantee and/or affiliates of the PEP project, cold call from a member of the grant writing team, paired via an external organization or contact; probe regarding how they know this information.)

Probes

1. What were the reasons for selecting each partner?

2. When was the partnership established? (i.e., before or after the grant was awarded)

3. Does the various partnership organizations’ goals/missions align with your PEP project’s goals?

   1. If yes: In what way?

4. Which, if any, of your partners were involved in the PEP grant application process?

   1. If involved: To the best of your knowledge, in what way were they involved. (If necessary provide examples: assessing needs, proposing/planning activities, writing sections)

   2. Has this involvement with the grant application been beneficial over the course of the grant?

      1. If yes: In what ways?

5. Have any of the partnerships ended?

Probes if yes

1. Which partnership(s)? Please describe the partner’s primary role/activities related to your PEP project.

2. What were the reasons for ending the partnership?

3. Was a new partner recruited to fulfill the old partner’s role?

6. Has your [school/district/organization] been involved in other partnering relationships related to youth health, physical fitness, physical education, or nutrition prior to this grant?

Probes if yes

1. Please briefly describe the nature of these partnerships.

2. Are these partnerships ongoing?

3. Have these partners played a role on the PEP grant?

   1. If yes: In what way?)

4. Have your other partnering experiences influenced your PEP partnerships?

(Probe for examples and details: lessons learned and applied, more resources for the PEP partnership, challenges with competing resources, perspectives, or priorities)

Partner activities: (20 minutes)

I’d also like to learn about the type of activities your partners have been involved with, as well as discuss the particular resources, services, or expertise they contribute.

For now, I want to discuss any activities (e.g., improving, or developing or creating new, policies, curricula, or in-school or outside-school programs; engaging in fitness or nutrition assessment; providing instruction, training, or professional development to students, school personnel, or other community entities’ staff; or establishing committees to address project goals) other than those related to body mass index (BMI) measurements since I will be asking about that a little later.

7. Which PEP project activities have your partners been involved with?

Probes

1. For each type of activity:

   1. Which partners are involved?

   2. Is [activity] ongoing? Or has it been completed?

   3. What’s the primary focus: physical education, nutrition, both, or another area?

   4. What is the primary goal of [the activity]?

   5. What specific resources, services, or expertise have partners provided?

   6. Do partners work with other PEP project staff in this task?

      1. If yes: Please tell me what other individuals are involved and a bit about their role.

   7. If applicable: Who was/is the target audience? (e.g., PEP project staff, teachers [area], students, parents)

      1. How many individuals would you say this [activity] has reached/included to date?

   8. How often would you say this [activity] or work related to this [activity] has taken place?

   9. What is the nature of the communications with the PEP partners?

1. How often do communications take place?

2. Have there been any challenges in communicating with the PEP partners? (Provide examples: insufficient amount of communication, difficulty coordinating meetings, lack of time, delays in response turnaround, lack of established communication channels, misunderstandings regarding information intending to communicate)

8. How would you describe the partner’s level of interaction in PEP project activities? Do you think the level of involvement has been about right? Or do you think partner involvement has either been more than necessary or less than optimal to accomplish the goals of the partnerships?

(Probe for examples supporting their response: In what way?)

9. Based on your experiences with your partners, describe what you consider to be primary benefits of the partnerships.

Probes

1. What, if any, are some specific examples of partnering benefits or strengths?

(Probe to identify examples: building knowledge base, capability of reaching more of the targeted population, additional resources or funding sources [such as…]; clarify the target of the benefit: self, students, parents, school or organization, community)

2. What, if any, are some specific examples of successes achieved through PEP project partnerships?

(Probe to identify success examples related to ability to reach audience(s), increased interest in programs/topics or activities)

3. Are there certain partnerships that have been more successful than others?

   1. If yes: Please describe.

10. What, if any, are some specific examples of obstacles or challenges that have been encountered in establishing partnerships and/or working with partners?

If yes: Please describe.

1. Which partnerships/aspects of the partnerships have been most challenging?

   1. To what do you attribute these challenges?

   2. Has this had an impact on the project?

      1. If yes: In what way?

   3. Have these challenges been addressed?

      1. If yes: In what way?

11. Now that your project has been underway for a while, is there value in involving the partners in additional ways?

If yes:

1. In what way(s)? (Probe to identify examples from specific project activities)

2. How would their participation in that activity enhance the project?

12. Once the PEP grant period has ended, are there plans for working with the partners and the partnership activities?

1. If yes:

1. With which partners?

2. In what capacity? (e.g., formal agreements established, informal discussions)

3. Will the partnership focus on the same activities implemented during the grant, new activities, or both?

   1. If new: What are the new/additional activities?

4. Please describe any plans for sustainable partnerships.

2. If not:

1. Please share some of the reasons why there are no plans to continue partnership activities.

2. Are there obstacles or challenges preventing the continued partnership? If yes: Please describe.

3. Have changes in the partners’ staff affected your relationships and/or partner sustainability? (e.g., level of interest or involvement)

1. If yes: Please explain.

13. Are there any additional partners you think would be beneficial to have involved with the PEP project?

If yes:

1. Who would these partners be? (Probe for specific name or description)

2. What need would they fill?

3. Why haven’t you recruited them for the project?

14. What, if anything, could the Department of Education’s PEP program office do to facilitate the experiences or sustainability of your partnerships? (e.g., technical assistance)

BMI measurements: (15 minutes)

15. Now I’d like to focus on your PEP project’s activities related to BMI measurements. Why did your project decide to collect BMI data?

16. Who is responsible for overseeing the BMI data collection? (Probe: In addition to the name obtain the position title)

17. I’d like to know about additional personnel and/or which of your partners are involved with the various BMI data collection processes and procedures.

For i-vi ask:

1. What, if any, specific resources, services, or expertise have your partners provided? (Probe for which partners)

2. Please list those involved and their roles.

(Probe: If names are provide ask for titles, e.g., physical education coordinator, program manager)

3. Are parents involved?

   1. If yes: In what way(s)?

BMI processes and procedures

1. performing assessments/measurements/data collection

2. developing guidelines and/or policies

3. storing data

4. informing parents

5. analyses

6. reporting

18. Has your PEP project offered and/or provided any training or professional development to project personnel related to BMI data collection and use of BMI measurements for this grant?

1. If yes: Please describe.

Probes

1. When did this occur?

2. Did you request participant feedback?

1. If yes: Which aspects were reported as most/least helpful?

3. Will your project be offering additional training?

1. If yes: Please describe.

2. If no: Do you plan to offer training or professional development opportunities in the future?

19. Let’s focus on the actual data collection process and procedures. What would you say has worked well in efforts related to BMI data collection?

(Probe: Ask them to be specific.)

1. Is your BMI data collected independently or as part of a broader fitness evaluation?

20. Have there been any challenges associated with BMI data collection?

1. If yes: Please describe.

1. Have these challenges impacted the ability to meet BMI-related project goals?

   1. If yes: How?

2. Have any of these challenges been addressed?

   1. If yes: How?

3. In what ways could the data collection process be improved?

21. Now let’s focus on the use of BMI data. How is your project currently using BMI data and what are the plans for BMI data use? Who is responsible for overseeing the use of BMI data? Who has access to the BMI data? (If necessary, provide examples: assess weight status of students across time, assess outcomes related to PEP project activities, guide physical activity or nutrition-related program development, inform policies, provide parents with information related to children’s BMI)

Probes

1. Is or will the BMI data be used in coordination with other data?

1. If yes: Please describe. (e.g., use in coordination with GPRA measures [which ones], other measures)

22. Are parents informed of their children’s BMI measures?

If yes: Please walk me through the reporting procedures.

1. In what way are parents informed? (Probe for information about the means by which parents are informed [e.g., letters, in-person conferences] as well as the type of information [e.g., BMI number, charts])

2. How were the reporting procedures developed?

3. Do you provide follow-up with parents of overweight or obese students (as indicated by relatively high BMIs)?

   1. If yes: Please describe.

4. Do you provide resource information to parents on where to go for help if their child has a relatively high BMI?

   1. If yes: Please describe.

5. Do you have any information regarding how parents/families are using this information?

   1. If yes: Please provide specific examples.

6. Have you received any complaints from parents regarding the information they receive?

23. Have there been any challenges associated with the use of BMI data?

If yes: Please describe.

Probes

1. Have these challenges impacted project goals?

   1. If yes: How?

2. Have any of these challenges been addressed?

   1. If yes: How?

24. What, if anything, could the PEP program office do to facilitate, support, or assist with BMI data collection? (e.g., technical assistance)

25. What are your overall impressions about the role of BMI data related to your PEP project?

1. Has BMI data collection enhanced or detracted from your project?

2. What are the students’ general views about the BMI data collection?

   1. Are they using the data?

      1. If yes: How?

Closing (5 minutes)

26. If you were speaking with a Project Director of a future PEP grant, what advice or lessons learned would you tell them?

27. Is there anything else you would like to share about your program, especially as it relates to the use of BMI data and partnerships?

28. Do you have any additional comments you would like to make about PEP as a whole?

Thank you again for your time and valuable feedback. Please contact me if you have any questions or if you think of anything else you’d like to share.

[PROVIDE CONTACT INFORMATION AGAIN]

Telephone Interview Protocol:

Project Personnel

The protocol provides a list of planned key questions. The interview itself will be conversational with questions flowing from previous responses when possible. Note that elements of the interview, such as specific partnerships and plans for BMI data collection and use, will be informed by information previously obtained through materials provided by the grantees and the screening survey. Personnel will include predetermined appropriate positions, such as school administrators (e.g., principals, vice principals), teachers (e.g., physical education [PE], health, nutrition), PE or health and wellness curricula coordinators, athletic directors, nutritionists, and food service staff.

Introduction

Hello [INTERVIEWEE], my name is [NAME] and [NAME] is also on the phone. We are researchers from the American Institutes for Research (AIR) and Data Recognition Corporation (DRC) and as you know we are conducting an evaluation of the Carol M. White Physical Education Program (PEP) sponsored by the U.S. Department of Education. I want to thank you for taking the time to speak with me today. The purpose of the evaluation is to gain an understanding of the activities being conducted by PEP projects in order to offer insight into how the program may be enhanced to further promote healthy living among America’s youth.

For reference and to clarify the notes, I would like to record the session. You may request that the recording be stopped at any point during the interview. May I do so?

[AFTER RECEIVING PERMISSION, BEGIN RECORDING; IF NO, PROCEED WITHOUT]

[THE FOLLOWING PROVIDES THE INFORMATION FOR THE REQUIRED ELEMENTS OF INFORMED CONSENT INCLUDING: PURPOSE OF THE STUDY/INTERVIEW; PROCEDURES; CONFIDENTIALITY; VOLUNTARY PARTICIPATION; AND DOCUMENTATION OF CONSENT]

The purpose of this call is to obtain more in-depth information about PEP projects and to learn about the program. Specifically, in your role as [POSITION] of a PEP grant, we want to learn how the grant has affected project activities involving the collection and use of BMI data and the establishment of partnerships. We are also interested in obtaining a better understanding of what is working well, in addition to any challenges you may have encountered. In addition to project successes, we understand that things do not always go as planned, and in order to learn about all aspects of the project, we encourage you to be candid in your responses. This information will be used by the Department to improve the current program and help future programs with their start-up and implementation.

I will be conducting today’s interview and [NAME] will be taking notes. Participation in this interview as part of the evaluation of PEP is voluntary. You can decide not to participate or to discontinue your participation at any time without penalty. [You have agreed to this interview being recorded and may request that the recording be stopped at any point.] You also do not have to answer any questions you do not want to answer. Everything you say here will be kept confidential and no response will be attributed to any one grantee.

It’s useful to include quotes from interviews in evaluation reports. No names, job titles, locations, or specific PEP projects will be provided with any quotes, feedback, or comments you provide today. In addition, you may indicate at any time if you do not want specific feedback you provide to be quoted. Today’s session will take about 40 minutes. I may occasionally interject with additional questions or to ask for clarification or detail, but otherwise this is your chance to speak freely about your experiences. Nothing you say will impact current or future PEP funding. You may contact us at any time with questions about the evaluation or this interview that we do not cover during this session.

Do you have any questions? Do you agree to participate in today’s interview?

[AFTER RECEIVING VERBAL INFORMED CONSENT, BEGIN INTERVIEW]

Interview

Background: (10 minutes)

1. I’d like to find out about your role on the project. Please briefly describe your current occupation and in what way you are involved with the PEP project.

Probes

1. How long have you been in this occupation with the district/school/organization?

2. How long have you been involved with the PEP project?

3. What are your responsibilities for the PEP project?

4. Have you held similar roles or had prior experiences in working on physical education/health-related grants?

   1. If yes: Please describe.

5. Were you involved in the PEP grant application process?

   1. If yes: Please describe in what way. (If necessary provide examples: assessing needs, proposing/planning activities, writing sections)

   2. Has this involvement with the grant application been beneficial over the course of the grant?

      1. If yes: In what ways?

2. Of those health-related (e.g., nutrition, fitness, physical well-being) needs the PEP project addresses, what would you say are the most pressing needs of the students being served?

(Probe for reasons why these needs: e.g., has the project assessed this, and if so, in what way(s)? Have these health-related problems changed over time?)

3. What do you see as the primary goals of this PEP project? (Probe: After pause, ask if there are any others?)

Activities: (5 minutes)

Now, I’d like to learn more about the type of activities you have been involved with related to physical fitness, nutrition, or other PEP project-related areas. These activities may include: improving, or developing or creating new, policies, curricula, or in-school or outside-school programs; engaging in fitness or nutrition assessment; providing instruction, training, or professional development to students, school personnel, or other community entities’ staff; or establishing committees to address project goals.

For now, I want to discuss any activities other than those related to body mass index (BMI) data collection since we will discuss that a little later.

4. Briefly describe the types of PEP activities you [and/or your staff] been involved with?

Probes:

1. What’s the primary focus of the activity: physical education, nutrition, both, or another area?

2. Who was/is the target audience? (e.g., other PEP project staff, teachers [area], students, parents, PEP partners)

   1. How many individuals would you say this [activity] has reached/included to date?

3. What are your perceptions of the impact of these activities on the behaviors and/or health of the youth in your community?

(If necessary, ask for specific examples/details: e.g., changes in teaching methods/pedagogical approach, changes in policies, changes in youth eating habits and food choices, increased activity in youth and/or families, increased teacher, youth, and/or parent knowledge of healthy eating and/or physical activities)

4. On which of these activities do you spend the most time?

Partnerships: (10 minutes)

5. Are any project partners involved in these activities? (Restate activities)

1. If yes, continue with probes for yes and the remainder of the section (i.e., questions 6-11).

2. If no: Have you had interactions with project partners outside of these activities?

   1. If yes, skip to #6

   2. If no, skip to #10

Probes if yes

1. Please tell me which partners are involved, which of the activities they are involved in, and a bit about their role (e.g., what resources, services, or areas of expertise does the partner provide).

2. To your knowledge, how did they become involved with these activities?

3. What is the nature of your or your staff’s communications with the PEP partners?

   1. How often do communications take place?

   2. Have there been any challenges in communicating with the PEP partners? (Provide examples: insufficient amount of communication, difficulty coordinating meetings, lack of time, delays in response turnaround, lack of established communication channels, misunderstandings regarding information intending to communicate)

   3. How would you describe the partner’s level of interaction in PEP project activities?

(Probe: Do you think the level of partner involvement has been about right, more than necessary or less than optimal to accomplish the goals of the partnership and the grant. You may need to ask this for each partner mentioned. If necessary ask for examples supporting their response.)

6. Based on your experiences with the PEP partners, describe what you consider to be primary benefits of the partnerships.

Probes

1. What, if any, are some specific examples of partnering benefits or strengths?

(Probe to identify examples: building knowledge base, capability of reaching more of the targeted population, additional resources or funding sources [such as…]; clarify the target of the benefit: self, students, parents, school or organization, community)

2. What, if any, are some specific examples of successes achieved through PEP project partnerships?

(Probe to identify success examples related to ability to reach audience(s), increased interest in programs/topics or activities)

3. Are there certain partnerships that have been more successful than others?

   1. If yes: Please describe.

7. What, if any, are some specific examples of obstacles or challenges that have been encountered in establishing partnerships and/or working with partners?

1. If yes: Please describe.

1. Which partnerships/aspects of the partnerships have been most challenging?

1. To what do you attribute these challenges?

2. Has this had an impact on the project?

1. If yes: In what way?

3. Have these challenges been addressed?

1. If yes: In what way?

1. Have changes in staff affected partner involvement?...sustainability?

   1. If yes: In what way?

8. Do you see value in involving the partners in additional ways?

If yes:

1. In what way(s)? (Probe to identify examples from specific project activities or activities not currently offered through the project)

Probes:

1. How would that strengthen the project?

2. What obstacles or challenges are there in involving the partner(s) in these activities?

9. To your knowledge what are the plans to continue the partnership activities after the PEP grant period has ended?

10. Are there any [additional] partners you think would be beneficial to have involved with the PEP project?

If yes:

1. Who would these partners be? (Name or describe.)

2. What need would they fill?

11. To your knowledge, has your [school/district/organization] been involved in other partnering relationships related to youth health, physical fitness, physical education, or nutrition prior to this grant?

Probes if yes

1. Please briefly describe the nature of these partnerships.

2. Are these partnerships ongoing?

3. If applicable, do you believe these other partnering experiences have had an influence on the PEP partnerships? (Probe for examples and details: lessons learned and applied, more resources for the PEP partnership, challenges with competing resources, perspectives, or priorities)

BMI measurements: (10 minutes)

12. Are you involved with any of the BMI data collection processes and procedures? (Provide examples: performing assessments/measurements/data collection, developing guidelines and/or policies, storing data, informing parents, analyses, reporting; probe for details related to each)

[Skip to closing section if not aware/involved with BMI measurements]

Probes if yes

1. Please describe the nature of your involvement.

2. What specific resources, services, or expertise have you provided related to BMI measurements? (If necessary, provide initial examples: performing measurements/data collection, developing guidelines and/or policies, storing data, informing parents, analyses, reporting; probe for details related to each; probe for details: e.g., procedures for taking measurements, how often is data collected, procedures for storing data, etc.)

3. What are the goals of the PEP project’s BMI data collection?

4. How long have you been involved in these efforts?

5. Have you received any training or professional development related to your project’s BMI data collection and use of BMI measurements?

   1. If yes: Please describe.

Probes

1. When did this occur?

2. Which aspects were most helpful?

3. Which aspects were least helpful?

4. Is there any additional training you would like to receive?

1. Have there been any challenges associated with data collection efforts?

   1. If yes: Please describe.

      1. Have these challenges impacted the ability to meet BMI-related project goals?

         1. If yes: How?

      2. Have any of these challenges been addressed?

         1. If yes: How?

2. What would you say has worked well in your/your staff’s efforts related to the PEP project’s BMI data collection?

3. In what ways could the data collection process be improved?

4. Is your BMI data collected independently or as part of a broader fitness evaluation?

13. To the best of your knowledge, how does the PEP project plan to use BMI data?

(If necessary, provide examples: assess weight status of students across time, assess outcomes related to PEP project activities, guide physical activity or nutrition-related program development, inform policies, provide parents with information related to children’s BMI, used in conjunction with GPRA measures.)

14. Are parents informed of their children’s BMI measures?

If yes:

1. In what ways are parents informed? Walk me through the reporting procedures

Probes

1. How often are results provided?

2. Is there follow-up with parents of students with relatively high BMIs?

1. Do you have any information regarding how parents/families are using this information?

   1. If yes: Please describe.

15. Are you aware of any challenges associated with the use of BMI data?

If yes: Please describe.

Probes

1. Have these challenges impacted project goals?

   1. If yes: How?

2. Have any of these challenges been addressed?

   1. If yes: How?

16. What are your overall impressions about the role of BMI data related to your PEP project?

1. Has BMI data collection enhanced or detracted from your project?

2. What are the students’ general views about the BMI data collection?

   1. Are they using the data?

      1. If yes: How?

Closing (5 minutes)

17. What advice or lessons learned would you share with a person in your position on a future PEP project?

18. Do you have any additional comments you would like to share about the PEP project, especially as it relates to the use of BMI data and partnerships, or about PEP as a whole?

Thank you again for your time and valuable feedback. Please contact me if you have any questions or if you think of anything else you’d like to share.

[PROVIDE CONTACT INFORMATION AGAIN]

Telephone Interview Protocol:

Community Partners

The protocol provides a list of planned key questions. The interview itself will be conversational with questions flowing from previous responses when possible. Note that elements of the interview, such as specific partnerships and plans for BMI data collection and use, will be informed by information previously obtained through materials provided by the grantees and the screening survey. Information about the range of community partner agencies and organizations at each site (e.g., local education agencies (LEA), colleges or universities, community-based organizations (CBO), food service or nutrition directors or boards, hospitals or clinics, external evaluation or monitoring agencies, park or recreational authorities, state or local public health departments or boards, other state or local government departments) will be obtained prior to the interviews as well.

Introduction

Hello [INTERVIEWEE], my name is [NAME] and [NAME] is also on the phone. We are researchers from the American Institutes for Research (AIR) and Data Recognition Corporation (DRC) and as you know we are conducting an evaluation of the Carol M. White Physical Education Program (PEP) sponsored by the U.S. Department of Education. I want to thank you for taking the time to speak with me today. The purpose of the evaluation is to gain an understanding of the activities being conducted by PEP projects in order to offer insight into how the program may be enhanced to further promote healthy living among America’s youth.

For reference and to clarify the notes, I would like to record the session. You may request that the recording be stopped at any point during the interview. May I do so?

[AFTER RECEIVING PERMISSION, BEGIN RECORDING; IF NO, PROCEED WITHOUT]

[THE FOLLOWING PROVIDES THE INFORMATION FOR THE REQUIRED ELEMENTS OF INFORMED CONSENT INCLUDING: PURPOSE OF THE STUDY/INTERVIEW; PROCEDURES; CONFIDENTIALITY; VOLUNTARY PARTICIPATION; AND DOCUMENTATION OF CONSENT]

The purpose of this call is to obtain more in-depth information about the PEP project and to learn about the program. Specifically, in your role as part of [AGENCY ORGANIZATION], a partner of [NAME PEP PROJECT], your feedback is important in helping us achieve our goal of learning more about how partnerships function, the various roles and activities of partners, as well as any involvement partners have with project’s BMI data collection, policies, and use. We are also interested in obtaining a better understanding of what is working well in the partnership and any challenges you may have encountered. In addition to project successes, we understand that things do not always go as planned, and in order to learn about all aspects of the project, we encourage you to be candid in your responses. This information will be used by the Department to improve the current program and help future programs with their start-up and implementation.

I will be conducting today’s interview and [NAME] will be taking notes. Participation in this interview as part of the evaluation of PEP is voluntary. You can decide not to participate or to discontinue your participation at any time without penalty. [You have agreed to this interview being recorded and may request that the recording be stopped at any point.] You also do not have to answer any questions you do not want to answer. Everything you say here will be kept confidential.

It’s useful to include quotes from interviews in evaluation reports. No names, job titles, locations, or specific PEP projects will be provided with any quotes, feedback, or comments you provide today. In addition, you may indicate at any time if you do not want specific feedback you provide to be quoted. Today’s session will take about 40 minutes. I may occasionally interject with additional questions or to ask for clarification or detail, but otherwise this is your chance to speak freely about your experiences. Nothing you say will impact current or future PEP funding. You may contact us at any time with questions about the evaluation or this interview that we do not cover during this session.

Do you have any questions? Do you agree to participate in today’s interview?

[AFTER RECEIVING VERBAL INFORMED CONSENT, BEGIN INTERVIEW]

Establishing partnerships: (5 minutes)

1. To the best of your knowledge, how did the partnership with [name of PEP grantee] come about? (If necessary provide examples: prior partnering relationship with the grantee and/or affiliates of the PEP project, cold call from a member of the grant writing team, paired via an external organization or contact; probe regarding how they know this information)

Probes

1. When was the partnership established?

2. Were you and/or your organization involved in the PEP grant application process?

1. If yes: Please describe in what way. (If necessary provide examples: assessing needs, proposing/planning activities, writing sections)

2. Has this involvement with the grant application been beneficial over the course of the grant?

3. If yes: In what ways?

3. How long have you personally been involved with the partnership?

4. What prompted this relationship?

2. Have you and/or your organization been involved in other youth health, physical activity, physical fitness, physical education, or nutrition related partnerships with other organizations (school districts, CBOs, or other) prior to this grant?

Probes if yes

1. Please describe the nature of these partnerships.

2. Are these partnerships ongoing?

3. Have your other partnering experiences influenced your partnership with [NAME PEP PROJECT]? (Probe for examples and details: lessons learned and applied, more resources for the PEP partnership, challenges with competing resources, perspectives, or priorities)

PEP project knowledge: (5 minutes)

3. What do you understand to be the primary focus and goals of [NAME PEP PROJECT]’s PEP grant?

(Probe: After they pause, ask whether there are any others)

4. How do you see your organization’s goals and/or mission aligning with the focus and goals you just noted?

Activities: (10 minutes)

I’d like to learn about the type of activities you have been involved with related to physical fitness, nutrition, or other PEP project-related areas, as well as discuss the particular resources, services, or expertise your organization contributes.

For now, I want to discuss any activities other than those related to body mass index (BMI) data collection since we will discuss that a little later.

5. Let’s start by talking about any PEP project activities that you or your organization have been or are directly involved with. These activities may include: improving, or developing or creating new, policies, curricula, or in-school or outside-school programs; engaging in fitness or nutrition assessment; providing instruction, training, or professional development to students, school personnel, or other community entities’ staff; establishing committees to address project goals; promotion; or funding.

To your knowledge, what types of activities do you engage in as a partner to [NAME PEP PROJECT]?

Probes

1. For each activity:

1. Is [activity] ongoing? Or has it been completed?

2. What’s the primary focus: physical education, nutrition, both, or another area?

3. What is the primary goal of [the activity]?

4. What specific resources, services, or expertise have you/your organization provided?

5. Have/does your organization work with others in this task?

1. If yes: Please tell me what other individuals or organizations are involved and a briefly describe their role.

6. If applicable: Who was/is the target audience? (e.g., PEP project staff, teachers [area], students, parents)

1. How many individuals would you say this [activity] has reached/included to date?

7. How often would you say this [activity] or work related to this [activity] has taken place?

8. What are your perceptions of the impact of [the activity] on the behaviors and/or health of the youth in your community?

If necessary provide probes

1. Do you think there is or will be an impact?

1. If yes: Please describe. (If necessary, ask for specific examples/details: e.g., changes in teaching methods/pedagogical approach, changes in policies, changes in youth eating habits and food choices, increased activity in youth and/or families, increased teacher, youth, and/or parent knowledge of healthy eating and/or physical activities)

2. If no: Why not?

1. Did you participate in any of these activities personally?

   1. If yes: What was the nature of your involvement?

6. Are there any additional activities you believe your organization could be doing to enhance [NAME PEP PROJECT]?

If yes: Please describe.

BMI measurements: (5 minutes)

7. Now I’d like to focus on any of your partnering activities with [NAME PEP PROJECT] that involve BMI measurements. Is your organization involved with them in collecting BMI data?

[Skip to closing section if not aware/involved with BMI measurements]

Probes if yes

1. Please describe the nature of your organization’s involvement.

2. What specific resources, services, or expertise have you/your organization provided related to BMI measurements? (Provide examples: performing measurements/data collection, developing guidelines and/or policies, storing data, informing parents, analyses, reporting; probe for details related to each)

3. What are the goals of [NAME PEP PROJECT]’s BMI data collection?

4. How long have you been involved in these efforts?

5. Have there been any challenges associated with these efforts?

   1. If yes: Please describe

   2. Have any of these challenges been addressed?

      1. If yes: How?

6. What would you say has worked well in your efforts related to the PEP project’s BMI data collection?

7. Is your BMI data collected independently or as part of a broader fitness evaluation?

Involvement: (5 minutes)

8. How would you describe your organization’s level of interaction in the partnering activities with [NAME PEP PROJECT]?

Probes

1. Do you think the level of involvement has been about right? Or do you think the partnering involvement was either more than necessary or less than optimal to accomplish the goals of the partnership? (Probe for examples supporting their response: In what way?)

   1. If not about right: Why do you think it was [less/more] than needed?

   2. How do you think the program would have benefitted from your involvement changing to the level you desired?

2. What is the nature of the communications with [NAME PEP PROJECT]?

1. Who is/are the primary contact(s)?

2. Are communications primarily via phone, email, in-person?

   1. Is this method effective?

      1. If not: What would you recommend?

3. How often do communications take place?

1. Is this amount of communication sufficient?

   1. If no: What would you recommend and why?

4. Have you/your organization experienced any difficulties with communications regarding the PEP grant and related activities? (Provide examples: difficulty coordinating meetings, lack of time, delays in response turnaround, lack of established communication channels, misunderstandings regarding information intending to communicate)

1. If yes: Please describe.

Benefits and challenges: (5 minutes)

9. Based on your current experience partnering on the [NAME PEP PROJECT], can you describe one or two primary benefits and/or challenges of the partnerships?

(Probe: Can you provide some specific examples of partnering benefits or strengths? If necessary, provide examples: building knowledge base, capability of reaching more of the targeted population, additional resources or funding sources [such as…])

1. How has this impacted your/your organization’s work on the project?

2. Are the challenges being addressed?

   1. If yes: How?

10. Can you provide specific examples of successes achieved as a result of partnering? (If necessary, ask about success examples related to ability to reach audience(s), increased interest in programs/topics or activities)

Closing (5 minutes)

11. What advice or lessons learned would you provide to an organization similar to yours that was considering partnering with a PEP project?

(Probe: When they pause ask if there is anything else?)

12. To your knowledge, are there plans to continue working as a partner with [NAME PEP PROJECT] after the PEP grant period has ended? (Probe for specifics about the plan.)

13. Do you have any additional comments you would like to make about PEP as a whole?

Thank you again for your time and valuable feedback. Please contact me if you have any questions or if you think of anything else you’d like to share.

[PROVIDE CONTACT INFORMATION AGAIN]

American Institutes for Research^® 32