0990-Evaluation National Bone
0990-Evaluation National Bone.doc
The National Bone Health Campaign Pilot Site Project
OMB: 0990-0337
Supporting Statement
for Paperwork Reduction Act Submissions
Section A
Evaluation of the Office on Women’s Health
National Bone Health Campaign
Best Bones Forever Pilot Site Project
Prepared for:
Office on Women’s Health
Prepared by:
Shattuck & Associates, Inc.
Supporting Statement for Paperwork Reduction Act Submissions
Evaluation on the Office on Women’s Health
National Bone Health Campaign
Best Bones Forever Pilot Site Project
This request is for a new approval of data collected as part of a new national bone health campaign pilot site project. The Office on Women’s Health (OWH) and its contractors are working to plan, implement, and evaluate a community-based pilot program as part of its efforts to increase bone-healthy behaviors through the National Bone Health Campaign (NBHC). The data collected in the evaluation of this community-based program will assess whether the program has its intended effect on participants in the pilot site projects. This assessment will help determine the feasibility of investing additional resources to expand the program and study it further. The primary data collection activities in the project revolve around evaluating a revised version of the Bodyworks program. The original Bodyworks program was evaluated under OMB Control No: 0990-0306.
A) Justification
1. Need and Legal Basis
In 2004, the first ever Surgeon General’s report on bone health and osteoporosis indicated that bone health diseases, such as osteoporosis, are a significant public health concern in the United States [Office of the Surgeon General (OSG), 2004]. Although healthy, strong bones are a vital aspect of health, millions of people suffer from bone disease and/or fractures, and in many cases these ailments are preventable. As a result, the Surgeon General called for a national action plan to improve Americans’ bone health.
The NBHC’s Best Bones Forever pilot program is a direct response to the Surgeon General’s report. This report emphasized the importance of early prevention (OSG, 2004) and of promoting bone-healthy behaviors during childhood and adolescence. Research indicates that ingesting recommended amounts of calcium and vitamin D as well as participating in bone-strengthening physical activities during childhood, adolescence, and early adulthood can help to prevent osteoporosis later in life (Chevalley, Rizzoli, & Bonjour, 2004; French, Fulkerson, & Story, 2000; National Institutes of Health, 2000; NOF, 2002; Schettler & Gustafson, 2004; OSG, 2004; Wang et al., 2003). Because females on average reach 90% of their skeletal mass by the age of 18 (NIAMS, 2005), and given that women are at higher risk for suffering from bone diseases (OSG, 2004), it is important to communicate these bone healthy behaviors to girls when they are developing their skeletal mass. Because of this and due to the recommendations of the NBHC’s Behavior Change Expert Panel, young females between the ages of 9 and 14 are the primary target audience of the NBHC Best Bones Forever Pilot Project.
Prior research on understanding girls’ eating and physical activity behaviors found parental influence to be important. Specifically, the family environment largely influences children’s success in achieving bone-healthy behaviors, for example, Borra et al. (2003) argued that parents must learn how to encourage their children’s eating and physical-activity habits and help their children maintain those habits. Likewise, parents must be empowered to address children’s physical activity and reduce negative nutritional influences within the family environment (Hart, Herriot, Bishop, & Truby, 2003). With regard to behavior change, family social support is a significant predictor of calcium intake and bone-strengthening physical activity among girls 8-11 years old (Ievers-Landis et al., 2003), and mothers’ reported physical activity and calcium supplementation has been related positively to these same behaviors in their children (Wizenberg, Oldenburg, Frendin, De Wit, & Jones, 2006). Given the important role parents play in developing, promoting, and encouraging bone-healthy behaviors in their children, they will be another important part of the NBHC pilot program efforts.
Each of these populations --young females ages 9-14, and their parents or caregivers-- will be targeted in the NBHC Best Bones Forever Pilot Program. This NBHC pilot program will include pilot projects in three sites around the country. Each site will coordinate comprehensive bone health messaging and programming in its community, building on research demonstrating that changes in awareness and behaviors are effectively achieved through multiple channels.
The project will be implemented using a Technical Assistance (TA) model where TA will be provided to each site project coordinator and to each coalition throughout the program to increase capacity, address concerns, and ensure that issues are resolved in a timely manner. The TA will be in areas such as coordinator support, coalition training, BodyWorks facilitator training and support, health education materials, program development, media and communications and evaluation.
The primary component that each coalition will plan, implement and (help) evaluate is the Revised BodyWorks program. BodyWorks is a federally funded obesity prevention toolkit and program designed to help the parents of adolescents make healthy food choices and become more physically active (OMB Control No: 0990-0306). The program focuses on parents as role models and provides them with hands-on tools to make small, specific behavior changes by improving family eating and activity habits to prevent obesity among their children and help them maintain a healthy weight.
The original parent curriculum has been adapted to include and highlight more activities about bone health and develop a companion program specifically for girls that will be educational, interactive, and engaging—making it fun to learn about eating healthy foods and being more physically active. For example the revised curriculum incorporates messages about the importance of calcium, vitamin D and bone-strengthening physical activity for bone health and about the critical bone building years (ages 9-18). The evaluation tools used for OMB Control No: 0990-0306 have been revised for this evaluation to reflect the adaptations described above.
Each pilot site will conduct four 10-session BodyWorks programs of 15 parents and 15-20 girls each. Each program will require a minimum of two trainers, one for parents and one for girls.
The focus of this OMB request is data collection with coalition members to assess the success and satisfaction with the capacity building component of the project. In addition, the focus is also on data collection to assess the effectiveness of the Revised BodyWorks program. These two data collection efforts are conducted under authority of Section 306 of the Public Health Service Act (42 USC 242k).
The information gathered by this data collection will provide needed insight into one intervention designed to promote bone strengthening behaviors during childhood and adolescence. Because bone strengthening behaviors during childhood, adolescence, and early adulthood can help prevent osteoporosis later in life the need to have evaluation information about interventions that strive to promote these behaviors is critical. With preliminary results showing positive effects from the original BodyWorks program, additional information supporting the success of this program is needed to further build the case for future implementation of this intervention. A more detailed listing of the project research questions is presented in Attachment 1.
2. Information Users
The data collected during the research activities associated with this pilot project will be used by the OWH to evaluate effectiveness of the pilot project and make recommendations on the feasibility of future national level expansion. In addition, it will provide important outcome data to show the pilot program’s impact on the primary audiences, girls ages 9-14 and their parents. While data has been collected on the original BodyWorks program (OMB Control No: 0990-0306) there has been no previous collection of information on the revised BodyWorks and there is no current collection.
Several research tasks will be undertaken in an effort to provide further support for the success of the original BodyWorks program; to assess the effectiveness of the revised BodyWorks program; and to explore the feasibility of expanding this community-based model to a broader level.
Coalition leaders, members and site coordinators will be asked to participate in pre and post intervention web-based surveys to assess the success of the TA provided to each coalition. These surveys will assess the coalition leaders’, members’ and site coordinators’ TA needs; their bone health knowledge, planning skills, implementation skills, and evaluation skills; and their satisfaction with the TA provided. The non-experimental direct analysis design (U.S. Office of Management and Budget, 2004) was chosen for this data collection effort given program resource limitations and because the main reasons for this analysis are to examine how the TA model facilitates the implementation of the pilot revised BodyWorks intervention and to provide information for program management purposes.
Implementing and evaluating the revised BodyWorks program is a main focus of this project. The BodyWorks evaluation will explore the effectiveness of the program by assessing outcomes such as knowledge gain, positive shifts in attitudes; changes in the levels of parent or caregiver and girls self-efficacy and skills supporting healthy eating and physical activity as well as
healthy behavior changes among participating adults and their daughters. The methodology for this evaluation will be to implement pre and post test self-administered questionnaires in a quasi-experimental design using a non-equivalent comparison group. In addition to the outcome evaluation, several process evaluation tasks will be undertaken to better understand the implementation of each session and the program overall. These process evaluation tasks include fidelity instruments filled out by trainers and session evaluations filled out by participants.
Each of these research tasks will provide data that will be used by the OWH to evaluate effectiveness of the pilot project. The study will provide important outcome data about the pilot program’s impact on the primary audiences, girls ages 9-14 and their parents or caregivers. This data will add to the evaluation evidence already collected on the BodyWorks program. Should positive evaluation data be found, this study will help build the case for further implementation of this intervention in additional populations.
Given the research design, limitations to the study are inevitable. For example, generalizability will be limited due to the non-representative nature of the program participants. That is, the participants in the program in each site won’t necessarily be representative of the entire US population of girls ages 9-14 and their caregivers. Therefore caution will be used in interpreting any and all of the results of this data collection.
Additionally there are inherent limitations in the use of a non-equivalent comparison group which include threats to internal and external validity. Several steps will be taken to limit these threats to validity. For example, in order to limit selection bias with respect to subjects, information about the demographic composition of the experimental groups will be used to recruit comparison group participants who are as similar as possible to the program participants. In addition, information about the experimental group’s motivation to change personal and family health habits will be used to ensure comparison group participants are as similar as possible to the program participants.
In an effort to limit differential attrition, effort will be made to reduce attrition within the comparison group. The relatively short time period between pre testing and post testing of the comparison group limits potential differential attrition. In addition, periodic contact will be made to each comparison group member to remind them of the project and encourage participation.
Historical threats will be addressed and limited in two ways. First, comparison group measurements will take place within the same 6-9 month time frame. Second, comparison group and experimental group members will be asked on pre and post test surveys about their experience with other related health education programs. Analyses can then be controlled for by this experience limiting the effect of differential exposure to these other programs.
3. Improved Information Technology
Where possible data will be collected using web-based surveys. For example, data collected to assess the TA needs and the success of the TA provided to each coalition will be collected using web-based surveys. Coalition leaders, members and site coordinators will access the web-based surveys through a hyperlink that will be prominently displayed in an introductory e-mail sent to each potential respondent at the beginning and the end of the pilot project contract time period. Each respondent will answer the survey questions and submit all responses electronically. Benefits of web-based surveys include reduced implementation costs, simplified questionnaire formatting, improved data quality, elimination of data entry, reduced processing costs and faster data collection (Witmer et al., 1999). In addition, submission of electronic data reduces the burden on respondents in that their data is submitted at the click of a button.
4. Duplication of Similar Information
Previous studies have collected information about bone health knowledge, attitudes, beliefs, and/or behaviors from children and parents; however, these surveys do not provide specific information about the effectiveness of NBHC materials and pilot project activities, which is the focus of this request.
A recent search for data about the NBHC target audiences was undertaken and the following sources were identified:
National Health and Nutrition Examination Survey (NHANES). NHANES is a program of studies developed to measure the health and nutritional status of U.S. adults and children. It combines interviews with physical examinations to obtain its database. The yearly survey assesses a nationally representative sample, and the information it collects is used to measure disease prevalence, risk factors for diseases, and to assess the association between nutritional status and health promotion and disease prevention.
Youth Media Campaign Longitudinal Survey (YMCLS). The YMCLS is a national survey of children and tweens ages 9-13 and their parents. Specifically, the survey gauges the physical activity-related beliefs, attitudes and behaviors of youth and their parents. It also measures youth exposure to the VERB campaign. The baseline YMCLS was administered in spring 2002 to more than 3,000 children in the targeted age range and their parents. The 2003 version of the survey was fielded April – June 2003, and the 2004 version in April – June 2004. Although this study provides useful data in the realm of physical activity, its target audience does not include girls over the age of 13.
Youth Risk Behavior Survey (YRBS). The YRBS, administered to middle school and high school students, is designed to collect data in six priority categories of health-risk behaviors among youth and young adults, specifically, behaviors that contribute to unintentional and intentional injuries; tobacco use; alcohol and other drug use; sexual behaviors that contribute to unintended pregnancy and STDs, including HIV infection; unhealthful dietary behaviors; and physical inactivity. Specific questions about milk, fruit, and vegetable consumption are included on the high school version of the survey but are not asked of middle school students.
National Longitudinal Study of Adolescent Health (Add Health). This study, which is administered through the University of North Carolina at Chapel Hill, is a study of the health-related behaviors of adolescents in grades 7 to 12 and their effects in young adulthood. The Add Health study collects data to explore three primary sources of adolescents’ differential health: different social environments, different health-related behaviors, and different strengths and vulnerabilities. Specifically, the school-based data assesses the effects of individual attributes of adolescents and attributes of their various environments on health and health-related behavior in areas such as diet, physical activity, health service use, morbidity, injury, violence, sexual behavior, contraception, sexually transmitted infections, pregnancy, suicidal intentions/thoughts, substance use/abuse, and runaway behavior. Data are collected also on height, weight, pubertal development, mental health status, and chronic and disabling conditions. The Add Health in-school questionnaire was administered to students in grades 7 to 12 in 80 high schools and 52 middle schools from September 1994 through August 1996 in two waves. Nationally representative data were collected from approximately 6,500 students. An in-home questionnaire was also administered to a core sample of students. A third wave was administered in 2001 and 2002 to Add Health respondents 18-26 years of age.
Simmons National Consumer Survey. The survey examines attitudes and lifestyle of children (6-11) and teens (12-17), with a primary focus on what drives purchasing decisions of youth in the United States.
The TRU Study. This biannual study, conducted by Teenage Research Unlimited, explores multiple aspects of teens’ lives, including their values and self-perceptions, how they spend their free time, teens as consumers and influencers of their parents’ purchases, and teen media. The study reaches 2,000 teens, 12-19 years old.
Original BodyWorks Evaluation. This evaluation (OMB Control No: 0990-0306) is providing valuable information about the original BodyWorks program however it does not provide any information on the revisions made to the BodyWorks program for this campaign. BodyWorks has been adapted to include and highlight more activities about bone health. In addition, a new companion program has been specifically designed for adolescent girls. This adolescent program is educational, interactive, and engaging—making it fun to learn about eating healthy foods and being more physically active. Messages about the importance of calcium, vitamin D and bone-strengthening physical activity for bone health and about the critical bone building years are incorporated in this new adolescent program.
Many of the aforementioned surveys collect data from youth similar in age to those targeted by the NBHC, however, they do not assess effectiveness, outcome of and satisfaction with this specific pilot project’s intervention and activities. In addition, some such as the YMCLS and Add Health data set, contain time-bound data and therefore cannot be used because the data collected do not represent girls who are currently ages 9 to 14. These other surveys, also do not provide information on all of the main bone health focus areas of this pilot project: bone-strengthening physical activity and calcium and vitamin D consumption. Also, the proposed methodology includes testing parent/daughter dyad participants and data from these other studies cannot deliver such a sample. The current OMB request was created because none of the above data sources provide the necessary information to evaluate each component of the NBHC pilot project.
5. Small Businesses
No small businesses will be involved in this study.
6. Less Frequent Collection
This is a one-time study that will begin and end during a 13-month project period. For the evaluation of the revised BodyWorks program, participants will be asked to complete a pre test questionnaire at the beginning of their BodyWorks training and a post test questionnaire during the last training session ten sessions later. In addition, comparison group participants will also be asked to complete pre and post test questionnaires. The pre and post test questionnaire methodology is very important to measuring the impacts of the program. Having BodyWorks participants completing the post test questionnaire at the last class will most likely enhance the ability to capture responses from the greatest number of respondents. It will also decrease the burden on respondents, since they will not have to respond once the training is complete.
In addition to the pre and post test questionnaires, participants will be asked to complete 10 session evaluations – one at the end of each session. The information provided by these very short assessments will reveal information about whether the activities in each curriculum are understandable, interesting and educational. This information will help modify and enhance the curricula for potential further implementation beyond the pilot. If an activity is not understandable, interesting and educational with the pilot participants, then the activity will be reviewed and adapted as needed for future inclusion in the curricula. Without these evaluations, there will be no way to know the participant’s satisfaction and experience with the curricula.
The trainers of both the parent/caregiver and girls curricula will be asked to fill out 10 fidelity instruments, one after each session. These fidelity instruments will provide valuable information about the extent to which the trainers are implementing the curricula as intended. If there is failure to implement the curricula as planned, there is the potential to conclude erroneously that the results of the evaluation can be attributed to the conceptual or methodological foundation of the intervention, rather than the fact it was not delivered as intended. Similarly, if the curricula are not implemented with fidelity, data suggesting that the intervention did not have the desired outcomes also must be questioned. Studying fidelity of implementation can explain why innovations succeed and fail and can allow for the identification of changes made to a program during implementation as they might affect outcomes (Dusenbury et al., 2003). Understanding how fidelity moderates the outcomes of the intervention can be crucial to guiding revisions to interventions for further implementation.
While less frequent data collection might reduce the burden on the revised BodyWorks trainers and participants, the session by session data collection will provide more current and useful information. In addition, the more frequent collections allow for shorter reference periods between reports, and this may reduce bias. By collecting most information from participants in a pre and post test format; but collecting short session specific data after each session we have tried to reach the goal to “strike a balance between the need for current information and the need to reduce public reporting burden” (Graham, 2006).
For the evaluation of the TA model, data will be collected before and after the pilot project contract period. The pre and post test questionnaire methodology is again very important to measuring changes in knowledge, attitudes, self-efficacy, behavior and behavioral intentions taking place during the pilot project contract period. Collection of this data is necessary to examine the usefulness and appeal of the TA model. There are no legal obstacles to reduce the burden.
7. Special Circumstances
Participants of the BodyWorks program will be asked to complete pre and post test questionnaires at the beginning and end of the 10 session program. In addition, participants will be asked to complete session evaluations at the end of each session. Similarly, trainers will be asked to fill out fidelity forms at the end of each session. Understanding the fidelity participant evaluation of the intervention is crucial to guiding revisions to the intervention for further implementation.
There are no other special circumstances applicable to this project. This request complies with the regulation.
8. Federal Register Notice/Outside Consultation
In accordance with the Paperwork Reduction Act of 1995, a 60-day Federal Register Notice was published in the Federal Register on December 1, 2008, vol. 73, No.231; pp. 72803 (see Attachment 2). There were no public comments.
9. Payment/Gift to Respondents
To encourage participation and to increase response rate parents/caregivers and girls who participate in the BodyWorks comparison group will receive an incentive (valued at ~ $75.00 for each adolescent/adult dyad). A small acknowledgement of time and trouble in the form of an incentive (valued at ~ $10.00) will be given to BodyWorks participants when they complete the pre and post test questionnaires. Evidence exists that incentives make a difference in response rates. For example, meta-analytic results reported by Church (1993) indicated that across 74 different surveys, both monetary and non-monetary rewards increased response rates (the average increase in response rates was 19.1% for monetary rewards and 7.9% for non-monetary rewards). Thus, providing an incentive to respondents to participate in a survey has been shown to be an effective method of increasing response rates.
10. Confidentiality
All survey respondents will be informed that their data will be kept private to the extent allowed by law. Parental consent will be collected for all participants under the age of 18. Participation assent will be collected for all parents/caregivers and girls. Participants will be informed that names will not be linked to any data and that results will be presented in aggregate. Participants will be informed that all hard copy data will be kept under lock and key and all electronic data will be protected by the use of passwords that only the principal investigator and project manager have assess to. Identifying information will be kept separate from data. When data is no longer needed it will be destroyed.
11. Sensitive Questions
No questions are considered to be sensitive to respondents; however, because girls ages 9-14 are one of the primary target groups, their responses will be monitored closely during initial testing to ensure that respondents do not interpret questions as sensitive.
12. Burden Estimate (Total Hours & Wages)
The maximum hour burden for all respondents to complete all instruments is estimated to be 755. The burden table below presents the hour burden by respondent type.
Type of Respondent |
Form Name |
No. of Respondents |
No. of Responses per Respondent |
Average Burden per Response (in hours) |
Total Burden (in hours) |
Parent/Caregiver participant in the Revised BodyWorks program |
Parent/Caregiver Pre test Questionnaire |
171 |
1 |
30/60 |
86 |
Parent/Caregiver Post test Questionnaire |
153 |
1 |
30/60 |
77 |
|
Parent/Caregiver Session Evaluation Forms (10 forms) |
153 |
10 |
3/60 |
77 |
|
Parent/Caregiver Revised BodyWorks program comparison group participant |
Parent/Caregiver Pre test Questionnaire |
63 |
1 |
30/60 |
32 |
Parent/Caregiver Post test Questionnaire |
50 |
1 |
30/60 |
25 |
|
Adolescent participant in the Revised BodyWorks program |
Adolescent Pretest Questionnaire |
228 |
1 |
30/60 |
114 |
Adolescent Post test Questionnaire |
204 |
1 |
30/60 |
102 |
|
Adolescent Session Evaluation Forms (10 forms) |
204 |
10 |
3/60 |
102 |
|
Adolescent Revised BodyWorks program comparison group participant |
Adolescent Pre test Questionnaire |
63 |
1 |
30/60 |
32 |
Adolescent Post test Questionnaire |
50 |
1 |
30/60 |
25 |
|
Trainers of the Revised BodyWorks program |
Facilitator Feedback Forms (10 forms) |
22 |
10 |
5/60 |
18 |
Coalition leaders, members, and site coordinators |
Coalition Pre test Survey |
86 |
1 |
20/60 |
29 |
Coalition Post test Survey |
72 |
1 |
30/60 |
36 |
|
Total Hours |
755 |
||||
12a. Estimated Annualized Burden Hours
12b. The total annual burden cost for the evaluation is estimated to be $6,604.40. The hourly wage estimates for all surveys were based on the Department of Labor, Bureau of Labor Statistics median weekly earnings for women 16 years and over who are full-time wage and salary workers. The following table shows how the total annual burden cost was calculated for the adult respondents.
Type of Respondent |
Form Name |
Total Burden (in hours) |
Hourly Wage Rate |
Total Respondent Costs |
Parent/Caregiver participant in the Revised BodyWorks program |
Parent/Caregiver Pre test Questionnaire |
85.5 |
$17.38 |
$1,494.68 |
Parent/Caregiver Post test Questionnaire |
76.5 |
$17.38 |
$1,338.26 |
|
Parent/Caregiver Session Evaluation Forms (10 forms) |
76.5 |
$17.38 |
$1,338.26 |
|
Parent/Caregiver Revised BodyWorks program comparison group participant |
Parent/Caregiver Pre test Questionnaire |
31.5 |
$17.38 |
$556.16 |
Parent/Caregiver Post test Questionnaire |
25 |
$17.38 |
$434.50 |
|
Adolescent participant in the Revised BodyWorks program |
Adolescent Pre test Questionnaire |
114 |
$0 |
$0.00 |
Adolescent Pre test Questionnaire |
102 |
$0 |
$0.00 |
|
Adolescent Session Evaluation Forms (10 forms) |
102 |
$0 |
$0.00 |
|
Adolescent Revised BodyWorks program comparison group participant |
Adolescent Pre test Questionnaire |
31.5 |
$0 |
$0.00 |
Adolescent Post test Questionnaire |
25 |
$0 |
$0.00 |
|
Trainers of the Revised BodyWorks program |
Facilitator Feedback Forms (10 forms) |
18.3 |
$17.38 |
$312.84 |
Coalition leaders, members, and site coordinators |
Coalition Pre test Survey |
28.7 |
$17.38 |
$504.02 |
Coalition Post test Survey |
36 |
$17.38 |
$625.68 |
|
Total Cost |
$6,604.40 |
|||
13. Capital Costs
There are no maintenance of capital costs to respondents.
14. Cost to Federal Government
OWH has awarded Hager Sharp an overall NBHC contract of $5,852,863 of which approximately 6% or $348,700 will be used to conduct and report on all components of this evaluation project from February 2008 to October 2011. Below are cost estimates from items 13 and 14 in a single table. The total amount needed was determined by an estimate of the number of labor hours needed times approximately $100 per hour. The estimated costs for travel, incentives, printing and postage are estimates. There are no start-up costs associated with this evaluation.
Note: Labor costs include development, analysis, and reporting costs
Cost Estimates for Proposed Evaluation
Description |
Estimated Cost |
Capital Costs |
|
Contractor Labor Costs |
$335,000 |
Development of Evaluation Plan – Data collection & analysis methods |
Included in labor |
Development of Survey Instruments |
Included in labor |
Development of OMB Supporting Statement |
Included in labor |
Implementation of Evaluation Plan |
Included in labor
|
Operations Costs/Data Collection |
|
Printing and Postage for surveys |
$200 |
Facilitation of Survey Implementation |
Included in labor |
Travel to site kick-off events (3) |
$5,500 |
Participant incentives |
$8,000 |
Data Analysis |
|
Survey Data Analysis (3 Surveys) |
Included in labor |
Data Reporting |
|
Survey Reports |
Included in labor |
Total Estimated Cost to Federal Government |
$348,700 |
15. Program or Burden Changes
This is a new data collection. All hours will be considered a program increase.
16. Publication and Tabulation Dates
The results of this data collection will be tabulated and summarized in final reports that will be submitted to OWH. These reports will be internal documents and are not intended for publication in academic literature or on the Internet. The reports will discuss the findings related to the impacts of the revised BodyWorks program, and the effectiveness of the TA model to build the coalition’s capacity to plan, implement and evaluate program efforts. A report will be generated for each individual program site as well as a combined three site report. Repeated measures analysis controlling for the cluster effects of group administration will be the primary analysis used for pre and post test data. In addition, descriptive analyses (i.e. frequencies, cross tabulations, and analysis of variance) will be used to analyze additional satisfaction and process evaluation data. More detailed analysis plans are presented in Attachment 3. The tables in Attachment 3 demonstrate the link between the project research questions, the evaluation forms, specific form questions and analysis plans.
The duration of the activities will span 23 months. The timetable for key activities is as follows:
Timeline |
Task |
4/2009 |
Pilot site selection |
5/2009 |
Coalition Pre test Survey data collection |
5/2009 |
Coalition Orientation |
7/2009 – 8/2009 |
Coalition On-Site Kick off meetings |
9/2009- 3/2010 |
BodyWorks Pre test data collection (intervention and comparison group) |
9/2009- 3/2010 |
BodyWorks session evaluation data collection |
9/2009- 3/2010 |
BodyWorks fidelity data collection |
12/2009- 5/2010 |
BodyWorks post test data collection (intervention and comparison group) |
9/2009 – 4/2010 |
Bone Health Outreach Survey data collection |
5/2010 |
Coalition Post test Survey data collection |
7/2010 |
Complete data entry |
8/2010 |
Begin data analysis |
3/2011 |
Complete reports |
17. Expiration Date
No approval to eliminate the expiration date of OMB approval is requested.
18. Certification Statement
There are no exceptions to the certification statement.
Works Cited
Borra, S. T., Kelly, L., Shirreffs, M. B., Neville, K., & Geiger, C. J. (2003). Developing health messages: Qualitative studies with children, parents, and teachers help identify communications opportunities for healthful lifestyles and the prevention of obesity. Journal of the American Dietetic Association, 103, 721-728.
Center, J., & Eisman, J. (1997). The epidemioloy and pathogenesis of osteoporosis. Bailliere's Clinical Endocrinology and Metabolism, 11, 23-62.
Chevalley, T., Rizzoli, R., & Bonjour, J. P. (2004, May). Calcium, exercise, vitamin D affect bone health in girls. Findings presented at the 2004 IOF World Congress on Osteoporosis, Rio de Janeiro, Brazil.
Church, A. H. (1993). Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opinion Quarterly, 57, 62-79.
Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research Theory and Practice, 18(2), 237-256.
French, S. A., Story, M., Fulkerson, J. A., Himes, J. H., Hannan, P., Neumark-Sztainer, D., et al. (2005). Increasing weight-bearing physical activity and calcium-rich foods to promote bone mass gains among 9-11 year old girls: Outcomes of the Cal-Girls study. International Journal of Behavioral Nutrition and Physical Activity, 2, 8.
Graham, JD (2006). Questions and Answers When Designing Surveys For Infomraiton Collections. Washington, DC: Office of Information and Regulatory Affairs Office of Management and Budget.
Hart, K. H., Herriot, A., Bishop, J. A., & Truby, H. (2003). Promoting healthy diet and exercise patterns amongst primary school children: A qualitative investigation of parental perspectives. Journal of Human Nutrition and Dietetics, 16(2), 89-96.
Ievers-Landis, C. E., Burant, C., Drotar, D., Morgan, L., Trapl, E. S., Colabianchi, N, et al. (2005). A randomized controlled trial for the primary prevention of osteoporosis among preadolescent girl scouts: 1-year outcomes of a behavioral program. Journal of Pediatric Psychology, 30(2), 155-165.
National Institutes of Health. (2002, March 27-29). Osteoporosis prevention, diagnosis, and therapy: NIH consensus statement. 17(1): 1-36.
National Institute of Arthritis and Musculoskeletal, and Skin Disorders. (2005). Osteoporosis: Peak Bone Mass in Women. (http://www.niams.nih.gov/Health_Info/Bone/Osteoporosis/bone_mass.asp)
National Osteoporosis Foundation (NOF). (2002). America’s bone health: The state of osteoporosis and low bone mass in our nation. Washington, DC: National Osteoporosis Foundation.
Office of the Surgeon General (OSG). (2004, October 14). Bone health and osteoporosis: A report of the Surgeon General 2004. Rockville, MD: Department of Health and Human Services.
Schettler, A. E., & Gustafson, E. M. (2004). Osteoporosis prevention starts in adolescence. Journal of the American Academy of Nurse Practitioners, 16(7), 274-82.
U.S. Office of Management and Budget (2004). What constitutes strong evidence of program effectiveness? (http://www.whitehouse.gov/omb/part/2004_program_eval.pdf)
Wang, M. C., Crawford, P. B., Hudes, M., Van Loan, M., Siemering, K., & Bachrach, L.
K. (2003). Diet in midpuberty and sedentary activity in prepuberty predict peak bone mass. American Journal of Clinical Nutrition, 77, 495-503.
Witmer, D. F., Colman, R. W. and Katzman, S. L. (1999) From paper-and-pencil to screen-and-keyboard: Toward a methodology for survey research on the internet. Doing internet research: Critical issues and methods for examining the net, Jones, S. ed., pp. 145–161. Sage, Thousand Oaks, CA.
Wizenberg, T. M., Oldenburg, B., Frendin, S., De Wit, L., & Jones, G. (2006). A mother-based intervention trial for osteoporosis prevention in children. Preventive Medicine, 42(1), 21-26.
NBHC Best Bones Forever Pilot Site Project
Research Questions
Process Evaluation Research Questions
General Pilot Program
Is this pilot program viable as a replicable bone-health education and obesity prevention program?
What aspects of the pilot program are crucial to its success?
What are the best practices employed by coalitions to successfully implement the BodyWorks program and/or bone health outreach activities?
What are the lessons learned from coalitions around implementing the BodyWorks program and/or the bone health outreach activities?
Do coalitions intend to continue any aspects of the program once the pilot intervention has ended?
To what extent does the technical assistance model help address the program planning and implementation challenges identified by the recently concluded evaluation of the original BodyWorks program?
What does the pilot program look like?
What are the characteristics of the…
Pilot sites (general populations, and previous experience with and exposure to obesity prevention, nutrition, and physical activity campaigns and programs).
Coalitions (roles, stakeholder group, prior experience).
Outreach/Education Activities (description of activities, target audience, number served,).
BodyWorks Programs (populations served, number served).
How do coalition members describe the process and function (team function, activity participation, strengths, successes, challenges, advice) of the…
Coalitions
Outreach/Education Activities
BodyWorks Programs
Specific Coalition Capacity Building
Do the coalitions have the capacity to communicate bone health messages to their communities?
To what extent are coalition members using the technical assistance services and resources designed to help them communicate bone health messages to their communities?
To what extent is there an improvement in bone health-related knowledge and attitudes among coalition members from the beginning to the end of the funding period?
To what extent do coalition members by the end of the funding period have the knowledge and attitudes necessary to support their coalition’s efforts to communicate bone health messages to the larger community?
How satisfied are coalition participants with the technical assistance support?
To what extent are coalition members aware of the technical assistance services and resources that are available to them?
What is the level of satisfaction among coalition participants with the technical assistance services and resources that they did receive?
Specific Outreach
How were the outreach activities planned and implemented?
How do the coalitions choose which outreach activities to plan and implement?
How do the strengths and/or limitations of the coalition influence the selection of outreach activities?
How do the strengths and/or limitations of the coalition influence the implementation of the outreach activities?
Specific BodyWorks
Was the BodyWorks program implemented as intended?
To what extent were the BodyWorks sessions implemented? (How many sessions were implemented? Were sessions implemented in full or in part?)
To what extent were sessions implemented as they were designed? (Were sessions modified? Was content added or left out?)
How well prepared were trainers to facilitate BodyWorks?
To what extent were facilitators confident in their ability to implement BodyWorks?
To what extent were facilitators satisfied with the support and resources they were given to implement BodyWorks?
What were participants’ experiences with Body Works?
How many sessions did parents/adolescents attend?
How did parents/adolescents use the BodyWorks toolkit materials?
To what extent were parents/adolescents satisfied with program content and materials?
To what extent were parents/adolescents satisfied with the program facilitator?
To what extent are certain sessions that are more useful/of greater interest to parents/adolescents than others?
Outcome Evaluation Questions
Did BodyWorks participants take steps toward improving general family eating and general exercise habits by increasing knowledge, attitude, self efficacy and overcoming barriers around nutrition and physical activity?
Did parents’/adolescents’ knowledge related to physical activity and nutrition increase during the BodyWorks program?
Did parents’/adolescents’ attitudes related to the importance of nutrition, physical activity, and overweight/obesity prevention improve during the Bodyworks program?
Did parents’/adolescents’ confidence in their ability to perform behaviors that support improved family eating and exercise habits increase during the BodyWorks program?
Did participants’ ability to overcome barriers related to physical activity improve during the BodyWorks program?
Do BodyWorks participants engage in healthier family eating and exercise habits as a result of the BodyWorks program?
Did parents/adolescents engage in healthier family eating and exercise habits as a result of the BodyWorks program?
Did parents and adolescents engage in healthier family eating and exercise habits together as a result of the BodyWorks program?
To what extent do the parents/teens intend to engage in healthier family eating and exercise habits one month after the BodyWorks program ends?
Can bone health messages be successfully integrated into a more general health and nutrition program?
Do BodyWorks participants take steps toward improving bone strengthening eating and exercise behaviors by increasing knowledge, attitudes, and self efficacy around behaviors such as getting daily recommended amounts of calcium, vitamin D, and bone strengthening physical activities?
Do BodyWorks participants engage in bone strengthening eating and exercise habits as a result of the BodyWorks program?
Do BodyWorks participants intend to engage in bone strengthening eating and exercise habits one month after the program as a result of the BodyWorks program?
Are improvements in parental knowledge, attitudes, self efficacy, and behaviors around bone health related to similar improvements in their adolescent children?
Are increases in parental knowledge, attitudes, and self efficacy about bone strengthening nutrition and physical activity related to similar increases in adolescents?
Is an increase in parental behaviors that support improved bone strengthening nutrition and physical activity related to similar behavior change in adolescents?
Is an increase in parental behavioral intent that supports improved bone strengthening nutrition and physical activity related to similar behavior intent in adolescents?
NBHC Best Bones Forever Pilot Site Project Matched Research Questions, Evaluation Forms, Form Questions and Analysis Plans
Process Evaluation Questions |
||
General Pilot Program |
||
|
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
Coalition Post Survey |
Q30 |
Univariate descriptive analyses |
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
Coalition Post Survey |
Q36 |
Univariate descriptive analyses |
|
||
|
||
Coalitions (roles, stakeholder group, prior experience, demographics). |
||
Evaluation Form |
Form Questions |
Analysis Plan |
Coalition Pre Survey |
Q2-Q10 |
Univariate descriptive analyses |
Coalition Post Survey |
||
BodyWorks Programs (populations served, number served). |
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Pre Test |
Q19-Q48 |
Univariate descriptive analyses |
BodyWorks Adolescent Pre Test |
Q19-Q26 |
|
BodyWorks Parent Post Test |
Q25-Q26 |
|
BodyWorks Adolescent Post Test |
||
|
||
Coalitions |
||
Evaluation Form |
Form Questions |
Analysis Plan |
Coalition Post Survey |
Q11, Q23-26, Q31-35 |
|
Outreach/Education Activities |
||
Evaluation Form |
Form Questions |
Analysis Plan |
Coalition Post Survey |
Q23 |
|
BodyWorks Programs |
||
Evaluation Form |
Form Questions |
Analysis Plan |
Coalition Post Survey |
Q23 |
|
Specific Coalition Capacity Building |
||
|
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
Coalition Pre Survey |
Q21-22 |
Univariate descriptive analyses |
Coalition Post Survey |
Q21-II; Q22-II |
Univariate descriptive analyses |
|
|||
Evaluation Form |
Form Questions |
Analysis Plan |
|
Coalition Pre Survey |
Q12-Q20 Knowledge Scale: Q12-Q17 Attitude Scale: Q18 Confidence Helping Girls Scale: Q19 Confidence Helping Parents Scale: Q20 |
NOTE: Matched Pre and Post Surveys only |
|
Coalition Post Survey |
|||
|
|||
Evaluation Form |
Form Questions |
Analysis Plan |
|
Coalition Post Survey |
Q12-Q20 Knowledge Scale: Q12-Q17 Attitude Scale: Q18 Confidence Helping Girls Scale: Q19 Confidence Helping Parents Scale: Q20 |
NOTE: Post Surveys only |
|
|
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
Coalition Post Survey |
Q21-I; Q22-I |
Univariate descriptive analyses |
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
Coalition Post Survey |
Q21-III; Q22-III Q27-29 |
|
Specific BodyWorks |
||
|
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
Parent BodyWorks Fidelity Forms |
QI; QII; QV Activity Completion Scale: QII-1 Achieving Learning Objectives Scale: QII-2 Engagement Scale: QII-3 I; II; V |
|
Adolescent BodyWorks Fidelity Forms |
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
Parent BodyWorks Fidelity Forms |
III; V |
|
Adolescent BodyWorks Fidelity Forms |
||
|
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
Parent BodyWorks Fidelity Forms |
IVa; V |
|
Adolescent BodyWorks Fidelity Forms |
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
Parent BodyWorks Fidelity Forms |
IVb |
|
Adolescent BodyWorks Fidelity Forms |
||
|
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Post Test |
Q18; Q19 |
Univariate descriptive analyses |
BodyWorks Adolescent Post Test |
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Post Test |
Q20-I; Q20-II |
Univariate descriptive analyses |
BodyWorks Adolescent Post Test |
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Post Test |
Q20-III; Q21; Q23-Q24 |
|
BodyWorks Adolescent Post Test |
||
Session Evaluations |
QI-1; QI-2; QI-3; QIII Activity Understandability Scale: QI-1 Activity Interest Scale: QI-2 New Knowledge Scale: QI-3 |
|
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Post Test |
Q22 |
|
BodyWorks Adolescent Post Test |
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
Session Evaluations |
QI; QII |
Multi-activity scale descriptive comparisons |
Fidelity Instrument |
QII-2-3 |
|
Outcome Evaluation Questions |
|
||
|
|||
|
|||
Evaluation Form |
Form Questions |
Analysis Plan |
|
BodyWorks Parent Pre Test |
Q7-Q11; Q13-Q16 Knowledge Scale: Q7- Q11; Q13-Q16 |
|
|
BodyWorks Adolescent Pre Test |
|||
BodyWorks Parent Post Test |
Q5-Q9; Q11-Q14 Knowledge Scale: Q5-Q9; Q11-Q14 |
||
BodyWorks Adolescent Post Test |
|||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Pre Test |
Q5 Caregiver Priority for Self Scale: Q5a Caregiver Priority for Daughter Scale: Q5b |
|
BodyWorks Adolescent Pre Test |
||
BodyWorks Adolescent Post Test |
Q3 Caregiver Priority for Self Scale: Q3a Caregiver Priority for Daughter Scale: Q3b |
|
BodyWorks Parent Post Test |
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Pre Test |
Q12 Self-Efficacy Scale: Q12a-i |
|
BodyWorks Adolescent Pre Test |
||
BodyWorks Parent Post Test |
Q10 Self-Efficacy Scale: Q10a-g(i) |
|
BodyWorks Adolescent Post Test |
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Pre Test |
Q6 Barriers Scale: Q6a-g |
|
BodyWorks Adolescent Pre Test |
||
BodyWorks Parent Post Test |
Q4 Barriers Scale: Q4a-g |
|
BodyWorks Adolescent Post Test |
||
|
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Pre Test |
Q17-Q18 Healthy Eating Scale: Q18a-h |
|
BodyWorks Adolescent Pre Test |
||
BodyWorks Parent Post Test |
Q15-Q16 Healthy Eating Scale: Q16a-h |
|
BodyWorks Adolescent Post Test |
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Pre Test |
Q17i; Q17k |
Repeated measures analyses with controls* |
BodyWorks Adolescent Pre Test |
||
BodyWorks Parent Post Test |
Q15i; Q15k |
|
BodyWorks Adolescent Post Test |
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Post Test |
Q17 Behavior Intent Scale: Q17a-i |
|
BodyWorks Adolescent Post Test |
||
|
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Pre Test |
Q7-Q8; Q14-Q16; Q5a; Q5f; Q12b; Q12h Knowledge Scale: Q7-Q8; Q14-Q16 |
|
BodyWorks Adolescent Pre Test |
||
BodyWorks Parent Post Test |
Q5- Q6; Q12-Q14; Q3a; Q3f; Q10b; Q10h Knowledge Scale: Q5- Q6; Q12-Q14 |
|
BodyWorks Adolescent Post Test |
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Pre Test |
Q17c; Q17l |
Repeated measures analyses with controls* |
BodyWorks Adolescent Pre Test |
||
BodyWorks Parent Post Test |
Q15c; Q15l |
|
BodyWorks Adolescent Post Test |
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Post Test |
Q17b; Q17c |
Univariate descriptive analysis |
BodyWorks Adolescent Post Test |
||
|
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Pre Test |
Q7-Q8; Q14-Q16; Q5a; Q5f; Q12b; Q12h Knowledge Scale: Q7-Q8; Q14-Q16 |
|
BodyWorks Adolescent Pre Test |
||
BodyWorks Parent Post Test |
Q5- Q6; Q12-Q14; Q3a; Q3f; Q10b; Q10h Knowledge Scale: Q5- Q6; Q12-Q14 |
|
BodyWorks Adolescent Post Test |
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Pre Test |
Q17c; Q17l |
|
BodyWorks Adolescent Pre Test |
||
BodyWorks Parent Post Test |
Q15c; Q15l |
|
BodyWorks Adolescent Post Test |
||
|
||
Evaluation Form |
Form Questions |
Analysis Plan |
BodyWorks Parent Post Test |
Q17b; Q17c |
|
BodyWorks Adolescent Post Test |
||
* Parent Pre controls: Location; Age (Q20); Race/Ethnicity (Q21-Q22); Other Program Exposure (Q26). Adolescent Pre controls: Location; Age (Q20); Race/Ethnicity (Q22-Q23); Other Program Exposure (Q24).
** Coalition Pre Survey controls: Location (Q6); Team Role (Q7).
Prepared by Shattuck & Associates, Inc. for Hager Sharp on behalf of the Office on Women’s Health
February 25, 2009
| File Type | application/msword |
| File Title | A) Justification |
| Author | Jennifer Boyle |
| Last Modified By | DHHS |
| File Modified | 2009-02-25 |
| File Created | 2009-02-25 |
© 2024 OMB.report | Privacy Policy