Evaluation Plan

Attachment D

The State Nutrition and Physical Activity Programs

to Prevent Obesity and Other Chronic Diseases

Evaluation Plan

The State Nutrition and Physical Activity Programs

to Prevent Obesity and Other Chronic Diseases

Evaluation Plan

In October 2000, the Division of Nutrition and Physical Activity (DNPA) at the Centers for Disease Control and Prevention (CDC) initiated the State Nutrition and Physical Activity Programs to Prevent Obesity and Other Chronic Diseases. This program was established to prevent and control obesity and other chronic diseases by supporting States in the development and implementation of nutrition and physical activity interventions, particularly through population-based strategies such as policy-level changes, environmental supports and the social marketing process. The goal of the program is to attain population-based behavior change in increased physical activity and better dietary habits; this leads to a reduction in the prevalence of obesity, and ultimately in a reduction in the prevalence of obesity-related chronic diseases.

The first cycle of funding was for fiscal year 2002 (FY 02). Six states were funded for a three-year period; the following year, 6 more states were funded (Cooperative Agreement number 00099). These states were required to:

• Develop a plan for preventing obesity for one of more populations.

• Develop appropriate internal and external partnerships to carry out the plan.

• Develop, conduct and evaluate small-scale nutrition and physical activity interventions targeting a population(s).

In FY 04, additional funding was allocated, and a total of 28 states were funded: 21 “capacity building” states and seven “basic implementation” states (Cooperative Agreement number 03022).

Responsibilities for Capacity Building states include:

• Develop a coordinated nutrition and physical activity program infrastructure.

• Collaborate and coordinate with state and local government and private partners, including members of the population throughout the planning process.

• Conduct a planning process that leads to a comprehensive nutrition and physical activity plan to prevent and control obesity and other chronic diseases, and start to implement the plan.

• Identify and assess data sources to define and monitor the burden of obesity.

• Implement and evaluate an intervention to prevent obesity and other chronic diseases.

• Evaluate progress and impact of the state plan and intervention projects.

Responsibilities for the Basic Implementation states include all those of the capacity building states and additionally:

• Expand the existing coordinated nutrition and physical activity program infrastructure. (Year 1)

• Implement the State comprehensive plan for nutrition and physical activity and review and update the plan periodically. Develop mini-grants and other mechanisms to support communities to adopt effective interventions. (Years 1-5)

• Expand partnerships with State Health Department units, the State Education Agency, other State agencies, local communities, and private partners to maximize impacts of the basic implementation program. (Years 1-5)

• Develop a new or apply an existing intervention and evaluate it to prevent obesity and other chronic diseases. (Years 1-5).

• Collaborate with partners on secondary prevention strategies. (Years 1-5)

• Develop resources and training materials to help other state and local projects adopt successful programs. (Years 4-5)

The plan for evaluating the State Nutrition and Physical Activity Programs to Prevent Obesity and Other Chronic Diseases is based on a logic model framework that graphically illustrates the chain of events, or if-then relationships, occurring in the program. This framework shows how inputs of CDC resources and investments are linked with CDC programmatic activities; how CDC’s funds and actions are linked with State program inputs; how State inputs of resources and investments are linked with State and community interventions; and how the actions of States are related to short-term outcomes, short-term outcomes to medium-term outcomes, and medium-term outcomes to long-term outcomes or ultimate program goals.

The evaluation of DNPA’s State Nutrition and Physical Activity Programs to Prevent Obesity and Other Chronic Diseases has been designed to focus on three primary areas: CDC training and technical assistance; State Plan development; and State and community level interventions. Within each of these primary evaluation areas, the plan identifies specific evaluation questions that have been chosen for study. For each evaluation question, one or more indicators or measures have been identified that will be used to answer the question. The plan also provides details of data sources. The allocation of resources, and hence, the execution of the evaluation plan, is dependent upon DNPA management.

The following discussion provides an overview of the major areas that are addressed in the evaluation plan; details are contained in the evaluation plan table.

1. CDC Training and Technical Assistance

In developing and implementing the State Nutrition and Physical Activity Programs to Prevent Obesity and Other Chronic Diseases, CDC has mobilized financial and human resources, established a programmatic infrastructure, engaged consultants and other technical resources, and synthesized evidence-based findings on nutrition and physical activity. The principal CDC actions resulting from these investments center on funding for State programs, and the delivering of training and technical assistance to the funded States. The evaluation of CDC’s activities will examine CDC’s resources and investments and determine how they were put together to create the State Nutrition and Physical Activity Program in order to define the context of the program. The evaluation will also look at the extent to which training and technical assistance was provided to the funded states, by both CDC staff and contracted consultants, so CDC can better develop training programs to meet the current and future needs of States and mobilize resources to respond to technical assistance needs that arise during State program implementation.

2. State Plan Development

One of the primary requirements of States receiving funding from CDC for the State Nutrition and Physical Activity Programs to Prevent Obesity and Other Chronic Diseases is the development of a State Plan that focuses on one or more populations. The State Plan is intended to include objectives for communications, behavior change, environmental support, policy-level change, surveillance, intervention strategies, partnerships and program resources. To develop the State Plan, states use their resources and investments, including funding from CDC and other sources, dedicated staff time and effort, involvement of partners and stakeholders and other inputs, to establish a state infrastructure for preventing and controlling obesity and other chronic diseases through nutrition and physical activity interventions. If States make these resource investments they are then able to conduct strategic planning activities to develop a State Plan, and identify one or more target populations for intervention. These planning activities include assessments of existing efforts in obesity and other chronic disease prevention and control, so that gaps in the current efforts, opportunities for increased change, and barriers to change for the population(s) are identified. This assessment also includes the identification of data sources that will be needed to develop, carry out, and evaluate the State Plan, and to further define and monitor the burden of obesity and other chronic diseases. During the state planning process, funded States will identify and involve internal and external partners and stakeholders based on the population(s), including members of the population(s), and participate in training, conferences and cooperative communication with national and state collaborators, including other funded states.

• The evaluation of DNPA’s State Nutrition and Physical Activity Programs to Prevent Obesity and Other Chronic Diseases will look at the extent to which states conducted strategic planning activities to develop their State Plan, the quality of the State Plans that were developed, and the extent to which states implemented the plans during the funding period of the CDC Cooperative Agreement.

The results of the evaluation of State Plan development will provide information that can help establish an understanding of current state capabilities to address the problem of obesity and other chronic diseases and the ability of states to develop a sustainable program. This evaluation can also assist DNPA in developing a national surveillance plan, coordinating nutrition and physical activity activities between state programs, directing national partnership planning and determining state training needs. The results can further be used to ensure that the needs of the population(s) are adequately addressed, and that the methods are effective, appropriate and scientifically sound.

3. State Interventions

A unique feature of DNPA’s State Nutrition and Physical Activity Programs to Prevent Obesity and Other Chronic Diseases is the requirement for states to develop, implement and evaluate small-scale (capacity building) or statewide (basic implementation) interventions that target one or more populations. The purpose of the small-scale interventions is to test the feasibility and effectiveness of nutrition and physical activity strategies to prevent or control obesity and other chronic diseases. It is anticipated that the statewide interventions that address nutrition and physical activity-related behaviors will foster behavior change by mobilizing levels of social structure, and striving to achieve overall balance in individual and environmental approaches.

The evaluation of State and community interventions will examine the steps taken to develop intervention protocols and methods, including the involvement of stakeholders and members of the target population(s). These interventions are expected to be undertaken within the context of measurable objectives that are specific, measurable, achievable, relevant and time-framed. The interventions are to be subjected to rigorous evaluation supported by collaborations with the Prevention Research Centers or other university affiliates or relevant experts, and CDC. The evaluation of DNPA’s State Nutrition and Physical Activity Programs to Prevent Obesity and Other Chronic Diseases places a priority on identifying measurable outcomes or impacts of these State and community interventions.

The results of the evaluation of State interventions are important to understand the contextual differences in the interventions, and to document methods and procedures for replication. The evaluation also will ensure that scientific integrity is applied to the development and implementation of the interventions. The evaluation will inform the public health community of promising approaches for the prevention of obesity and other chronic diseases, and will identify topic areas in which additional research is needed.

All item numbers in the Data Sources column refer to the current 2007 version of the progress monitoring report.

Evaluation Questions	Indicators	Data Sources
1. CDC Training and TA
What were CDC’s resources and investments and how were they put together to build and run the State Nutrition and Physical Activity Program?	Description of resources and investments and how they were put together to create the State Nutrition and Physical Activity Program	Internal program files and records
To what extent was training provided to funded states?
What training needs of state project staff were identified?	Description of needs self-identified by state staff in response to contractor training needs assessments Description of training needs identified by Project Officers and other DNPA staff based on analysis of requirements of the obesity prevention project	Results of contractor training needs assessments Project Officers’ and DNPA records Training committee records
What project-specific training was designed and presented to state staff to meet their training needs?	Description of training courses, content and learner objectives Number, titles, department and division of participating state program staff	Contractor training records Training event participant rosters I4, I4g, I4h
How effective or useful were the project-specific training events for state staff?	Participant evaluation of course content and satisfaction Six-month participant follow-up evaluation of practicality and use of course information	Results of participant course evaluations Results of six-month follow-up I13
What technical assistance (TA) was designed and delivered to states to meet identified needs?	Number, types, and content of TA activities designed and delivered by contractors Number, types, and content of TA activities designed and delivered by CDC Project Officers and other staff	Contractor TA logs Project Officers’ TA records
How effective or useful were the TA activities provided for state project staff by contractors?	State reports on satisfaction, including a list of reasons for use or disuse of contractor TA	State feedback on evaluation of contractor TA
How effective or useful were the TA activities provided for state project staff by CDC?	State reports on satisfaction, including a list of reasons for use or disuse of CDC TA	State feedback on evaluation of CDC TA
2. State Plan Development
What criteria did states use to identify one or more populations in the state based on need for prevention and control of obesity and other chronic diseases?	Description of criteria and data sources used to identify populations for intervention projects in the state Description of characteristics of this population	PL22,PO3, IM9,
What actions did states take to assess (a) existing efforts in obesity and other chronic disease prevention and control, (b) gaps in current efforts, (c) opportunities for increased service, and (d) barriers to service for the population?	Description of actions taken by states to assess a, b, c and d	IM36,, IM 26- IM 30
What data sources are states using to monitor the burden of obesity and other chronic diseases as well as to monitor trends in nutrition and physical activity as it relates to obesity and chronic diseases?	Description of surveillance systems states are currently using to monitor the burden of obesity and other chronic diseases. Description of how these data sources are being used as well as any reports generated from these analyses.	DS 31a, DS 31b, IM 4
A. With what internal and external partners did states collaborate and coordinate in developing the state plan? B. How successful (or not) were the collaborations with various partners (external and internal)?	Number and types of partners: Pre-existing internal partners (e.g., 5 A Day) New internal organizational partners Pre-existing external partnering organizations New external partnering organizations Examples of successful collaborations with internal and external partners in the development of the state plan Description of problems that were encountered working with these partners and how these problems were addressed.	C26, C29,C30, C32, C33, C34, C35, C36, C37,, PL3
What activities did states conduct to include and involve citizens of the population(s) in developing the state plan?	Description of activities that took place involving citizens of the population(s)	PL21
What activities to collaborate with Prevention Research Centers (PRCs) and other academic partners did states undertake to develop the state plan?	Number and description of activities with which PRCs were involved List of other academic partners and description of activities in which these partners were involved	C26
How well does the state plan incorporate evidence-based information on interventions?	Components of CDC Community Guides included in the plan Other evidence-based information on interventions and sources included in the plan Rating on quality indicators for incorporation of evidence-based information on interventions into the state plan	Written state plan PL12-PL17, PL23
How well does the state plan for nutrition and physical activity programs to prevent obesity and other chronic diseases relate to other state health department goals and activities?	Description of how plan activities support other health department activities Description of how other health department activities support the plan for nutrition and physical activity programs to prevent obesity and other chronic diseases Rating on quality indicators for effectiveness and appropriateness of integration of state plan with other state health department goals and activities	Written state plan PL12-17
How well were measurable objectives and outcomes to prevent and control obesity and other chronic diseases through nutrition and physical activity interventions included in the state plan?	Rating on quality indicators for effectiveness and appropriateness of measurable objectives and outcomes	Written state plan PL 26
How well did the state plan reflect established methods and procedures for implementing the state plan to achieve the measurable objectives and outcomes?	Rating on quality indicators for effectiveness and appropriateness of methods and procedures for implementing the state plan	Written state plan PL12-13
To what extent did states implement the state plan?
What state or community policies and/or legislations were initiated, modified or planned as a result of the state planning process?	Type and description of policies and/or legislations that were modified or initiated Type and description of policies and/or legislations that were planned but not initiated Descriptions of the reasons why specific policies and legislations were blocked or not implemented	PO3, PO5, P011, PO13
What state or community environmental changes were initiated, modified or planned as a result of the state planning process?	Type and description of community environmental supports that were modified or initiated Type and description of community environmental supports that were planned but not initiated Description of the reasons why specific environment supports were blocked or not implemented	PO19, PO21
In how many communities, and to what extent, did states implement the state plan for nutrition and physical activity?	Number of communities where state plan was implemented Description of communities’ activities related to the state plan for nutrition and physical activity	Implementation section especially IM 23 (a-h)
To what extent did states undertake data collection, analysis and reporting to evaluate the impact of the state plan on (a) obesity and other chronic diseases, (b) nutrition as it relates to obesity and other chronic diseases, and (c) physical activity as it relates to obesity and other chronic diseases by the end of the CDC funding period?	Number and description of data sources being used to monitor trends in a, b, and c. Description of how these data sources are being used to evaluate the impact of the state plan on a, b, and c, including any reports generated from analysis of these data.	DS311, DS31b, EV 18-23, EV 29
To what degree did states commit resources to continue their efforts towards sustaining physical activity and nutrition activities throughout the state?	Amount and sources of funding appropriated for or reallocated to sustain physical activity and nutrition activities throughout the state Number of FTEs and types of new positions or program components authorized and dedicated to sustain physical activity and nutrition activities throughout the state Number of FTEs and types of reallocated positions dedicated to sustain physical activity and nutrition activities throughout the state	R5,R6,R8,R9, R10, R12, C3-C24
3. Intervention Activities
To what extent did states develop interventions for the targeted population(s)?
3.1.1 What steps did states take to develop an intervention for the target population(s)?	Description of steps undertaken to develop the intervention, including a description of the audience selected and the methods for selecting this audience. Description of how these data sources are being used to monitor the intervention, including any generated reports from analysis of these data. Description of any new data sources developed to monitor the impact of the intervention	IM9
3.1.2 To what extent did representatives of the target population(s) participate in the development, implementation and evaluation of the intervention?	Description of steps taken to involve representatives of the target population(s) to participate in the development, implementation and evaluation of the intervention	IM98, EV29
3.1.3 What were the components or details of the intervention developed by the state?	Description of the strategies the intervention is using to change the behavior.	IM8, IM24-IM30, IM34(a-h), IM35, IM 36, IM102
3.1.4 What evidence-based information on interventions was incorporated into the interventions?	Components of CDC Community Guides included in the interventions Other evidence-based information on interventions and sources included in the interventions	IM35, IM36
3.1.5 To what extent did states implement social marketing strategies identified in the state plan?	Description of selected behavior. Description of any pretesting of intervention strategies or messages with selected audience (s). Description of how the intervention will address benefits and barriers to suggested behavior change, including any competition that may discourage behavior change.	IM8, IM39, IM52, IM71, IM98
What were the outcomes/impacts of the interventions?	Measured outcomes and results of statistical analyses	EV18,EV19, EV31, EV31, S2, S3
What barriers were encountered in implementing the intervention, and how successful were the solutions to overcome these barriers?	List of barriers, solutions undertaken and results	IM39, IM40, IM42, IM44, IM46, IM48, IM50, IM51a, IM51c, IM52, IM53, IM53, IM57, IM59, IM63, IM65, IM67, IM71, IM72, IM74, IM76, IM84, IM86, IM88, IM90, IM92, IM94, IM 98, IM99
To what extent did states disseminate the results of the intervention?	Number, type and description of dissemination activities	EV34
* To what extent was a mini-grant (or similar other) program implemented to make local projects possible?
* To what extent was training and resources made available to local projects?		EV15, EV30
* To what extent were secondary prevention efforts implemented and evaluated?		Pl 24, EV18, Ev19, Ev20

11