Logic Model and Literature Review

DFC National Evaluation: Systems, Measures, and Tools

Attachment 3:

Internal Logic Model, Measurement Mapping,

and Brief Literature Review

Internal Logic Model, Measurement Mapping, & Brief literature Review

 

 

Overview of Internal Logic Model, Measurement Maps, & Literature Review

This appendix presents and describes the various constructs and variables targeted by the DFC National Evaluation. First, we present the internal National Evaluation logic model, which was developed by the technical advisory group (TAG) and members of ICF’s National Evaluation team. Next, we present a series of tables that lists the exact items used to measure each variable and construct listed in the internal model (i.e., measurement maps). Finally, this appendix ends with a brief literature review. This review provides the empirical base for all selected constructs and variables, and provides a high-level description on how current and proposed variables are being measured in the evaluation and will be used in analyses. For more detailed literature reviews, please see some recent comprehensive reviews that have focused on coalition dynamics and effectiveness across a number of research areas (Ansari & Wells, 2006; Foster-Fishman et al., 2001; Granner & Sharpe, 2004; Israel, Schulz, Parker, & Becker, 1998; Roussos & Fawcett, 2000; Thomson, Perry, & Miller, 2009; Zakocs & Edwards, 2006).

Background

While the logic model working group was initially tasked with revising the National DFC logic model, it quickly became apparent that a much more specified logic model was needed to guide our work and ensure the collection of appropriate data. This internal model was designed to highlight the details that were not included in the external DFC National Evaluation logic model (see p. 15). For instance, the external model identifies the importance of coalition structure and processes as a major component of DFC and links this to the implementation of strategies and activities, which is then hypothesized to produce positive community and population-level outcomes. However, little to no detail regarding how we will measure these major components is provided. This was purposeful as the intent of the external model is to produce an overarching vision and road map on how DFC coalitions produce positive change in their communities. While such a top level perspective is needed for communicating with more than seven-hundred and fifty coalitions and their partners, the guiding internal National Evaluation logic model mandates much more specificity (see Figure 2 below).

Figure 2. Internal DFC National Evaluation Logic Model

CONTEXT	Coalition Structure and Processes	Strategies and Activities	COALITION CAPACITY	COALITION EFFECTIVENESS
Community Sense of community / Community social organization Community readiness for change ATOD needs ATOD prevention resources Related initiatives Socio-economic context Critical events Stability Coalition Organizational history Past collaboration involvement Funding history Past accomplishment Institutional / grassroots focus Critical Events	Member Competency Member coalition involvement Member relationships Skills and knowledge Structure Formalized structures Degree of centralization Membership breadth and diversity Leadership structure Institution / grassroots mix Workgroup/committee organization Processes Formalized procedures Task and role clarity Supports for member involvement Leadership style Communication Conflict resolution Shared decision making Information / expertise input and use Strategic planning framework (SPF) Use data to inform decisions External relations / resource access	Coalition Role Membership and Focus Direct implementation Advocacy Initiation / support Member contributions External partnering Programmatic Capacity Clear, realistic objectives Unique, innovative services Need driven Cultural competence Use of evidence-based practice Strategy / Activity Mix Information / Education Coordination Environmental policy / enforcement Environmental assets and opportunities Direct service programs Coalition & Community Outputs Coalition outputs Population targets/ penetration Volume (dose) Comprehensiveness	Coherence Shared vision Shared solutions Inclusiveness Broad involvement Commitment /Satisfaction Coalition Climate Open Trusting Values diversity Shared input Positive External Relations Non-member agencies Activity focus Access community resources Community role / recognition Capacity Building Effort Outreach / recruitment Access to Training / TA Build member, coalition, and programmatic capacity Continuous Improvement Data based & Systematic Explicit decision points (e.g., annual action plans) Responsive / adaptive	Community/ System Outcomes
				Norms & Awareness ATOD risk perception / acceptance Media Systems & Policy Change Collaboration Service integration Public / business /education policies Sustainable Accomplishments Systems Services
				Community Behavioral Outcomes
				Substance Use Prevalence Core use measures (revised) STOP Act measures specific to alcohol unchanged Contributors & Consequences Educational performance Health & wellness Crime Family Social Economic

The internal model is designed to fill this gap by breaking down each of the major DFC components or domains in the external model (i.e., context, coalition structures and processes, strategies and activities, coalition capacity, and coalition effectiveness) and lists the exact constructs and variables that will be used to assess these components. To provide an example, within the domain of context, the first column indicates that there are two overarching constructs: (1) community context, and (2) coalition context. We have selected eight variables to assess community context and six variables to assess coalition context. These variables were identified in the research and practice literature as critical to the success of community change initiatives targeting a multitude of community and population-level outcomes. For example, the variables for community context include:

1. Sense of community / Community social organization

2. Community readiness for change

3. ATOD needs

4. ATOD prevention resources

5. Related Initiatives

6. Socio-economic context

7. Critical events

8. Stability

Measurement Maps and Data Sources

While the internal model presents some more specificity, what is still needed is the exact item content and data source for each domain, construct, and variable in the internal model. This ensures that the National Evaluation is collecting item-level data that is needed to test the hypothesized relations in both the internal and external models. Thus, what follows is a series of measurement maps providing this linkage (see Tables 1-5).

For DFC, there are three primary sources for the DFC National Evaluation – the Coalition Online Management and Evaluation Tool (COMET), the Coalition Classification Tool (CCT), and core outcome data (core). COMET data is reported by DFC grantees every six months, except for STOP Act grantees, who collect data every three months for the activity/strategy implementation and proposed population served fields. Data are typically entered by the coalition chair and/or other leaders who are able to answer questions regarding planning and implementation activities, including budget-related and other administration information. The CCT is collected one time a year and represents a more typical coalition survey that asks members about their perceptions regarding coalition dynamics, functioning, outputs, and impacts. Core outcome data is submitted every other year via the COMET data system. Other data sources include process and outcome data that will be collected during annual case study site visits and publicly available data (e.g., Census data, UCR data, FARS data). Please see the following tables below, which provide linkage down to the item level. This is followed by a brief literature review which provides the empirical rationale for both the external and internal models and briefly describes both current and proposed variables, data source, and how they will be used in analyses.

Table 1. Variables and Items of Context

VARIABLE		ITEM		DATA SOURCE
Community Context
Sense of Community / Community Social Organization	Community Social Organization. New proposed variable that aligns with risk and protective framework.		CCT
	Assessment Section. Community Needs Assessment. Target community name.		COMET
	Assessment Section. Community Needs Assessment. Target geographic area (urban, suburban, rural).		COMET
	Assessment Section. Community Needs Assessment. Target specific geographic area.		COMET
	Assessment Section. Community Needs Assessment. Target zip codes.		COMET
	Assessment Section. Community Needs Assessment. Text (Please further describe the geographic area(s) selected above).		COMET
Community readiness for change	Community Readiness. New proposed scale to address an important and integral variable for community change efforts – readiness at the community level.		CCT
	Assessment Section. Assessment Progress. Accomplishment or barrier (new drop downs).		COMET
	Assessment Section. Assessment Progress. Descriptive Text.		COMET
	Assessment Section. Community Needs Assessment. Text (Assessment summary and key findings).		COMET
ATOD Needs	Assessment Section. Community Needs Assessment. Substance of issue in the community.		COMET
	Assessment Section. Risk and protective factor framework. ID of assets and risks.		COMET
	Assessment Section. Community Needs Assessment. Text (Assessment summary and key findings).		COMET
ATOD prevention resources	Readiness scales above address some of these resources (e.g., leadership, financial resources).		CCT
	Assessment Section. Community Needs Assessment. Text (Assessment summary and key findings).		COMET
	Assessment Section. Risk and protective factor framework. ID of assets and risks.		COMET
	Assessment Section. Risk and protective factor framework. New proposed item. “To what extent does your coalition use a risk/protective factor approach in your prevention efforts?”		COMET
	Capacity Section. Funding Streams. Dollar amount.		COMET
	Capacity Section. Funding Streams. Dollar amount by source.		COMET
Related initiatives	Assessment Section. Community Needs Assessment. Modified variable. “Approximately, how many groups, excluding your own organization, are working on the prevention of substance abuse in your community?”		COMET
	Assessment Section. Community Needs Assessment. Modified variable. “In the past 6 months, how many of these groups (excluding your own DFC partners/members) have you worked with to reduce substance abuse in your community through shared activities and events, such as co-sponsoring substance abuse prevention activities?”		COMET
SES Context	Assessment Section. Community Needs assessment. Text (Please further describe the geographic area(s) selected above).		COMET
	Assessment Section. Text (Assessment summary and key findings).		COMET
	Census. Poverty Rate.		Census
Critical Events	Assessment Section. Assessment Progress. Accomplishment or barrier (new drop downs). Descriptive Text.		COMET
	Capacity Section. Capacity Progress. Accomplishment or barrier (new drop downs). Descriptive Text.		COMET
	Planning Section. Planning Progress. Accomplishment or barrier (new drop downs). Descriptive Text.		COMET
	Implementation Section. Implementation Progress. Accomplishment or barrier (new drop downs). Descriptive Text.		COMET
	1Evaluation Section. Evaluation Progress. Accomplishment or barrier (new drop downs). Descriptive Text		COMET
	Protocol/Item to be developed for case study sites.		Case Study
Stability	Census. Rates of residential stability.		Census
Other	There will be text fields for each section in order to provide ample opportunity for coalitions that want to further provide more context and detail regarding their needs assessment process, activities, accomplishments, challenges, etc.		COMET CCT
Coalition Context
Organizational History	Assessment Section. Assessment Activity. Activity Name (new drop downs) – Descriptive text.		COMET
	All SPF Sections. Accomplishments/Barriers.		COMET
Past Collaborative Involvement	Case Study Sites. Social network analysis.		Case Study
	Case Study Sites. Qualitative Data. Collaborative representatives, neighborhood leaders, etc.		Case Study
Funding History & Current Additional Sources	Capacity Section. Funding Streams. [What is your coalition’s total annual operating budget (with a number of categories to list money amounts under, including DFC grant, in-kind contributions, etc.)].		COMET
Coalition Readiness for Change	Coalition Readiness. New proposed scale to address an important and integral variable for community change efforts – readiness at the coalition level.		COMET
Past Accomplishments	Assessment Section. Assessment Progress. Accomplishment or barrier (new drop downs). Descriptive text.		COMET
	All SPF Sections. Accomplishments.		COMET
	Protocol/Item to be developed for case study sites.		Case Study
Institutional / Grassroots Focus	Capacity Section. Coalition Membership.		COMET
	Protocol/Item to be developed for case study sites.		Case Study
Critical Events	Administration Section. Changes in Leadership.		COMET
	All SPF Sections. Accomplishments/Barriers.		COMET
Other	There will be text fields for each section in order to provide ample opportunity for coalitions that want to further provide more context and detail regarding their activities, accomplishments, challenges, etc.		COMET

Table 2. Variables and Items for Coalition Structure and Process

VARIABLE	ITEM	DATA SOURCE
Member Competency
Member Coalition Involvement	Capacity Section - New Proposed Item. “How many active members are there in your coalition from each of the 12 sectors (Active members means an individual has attended at least one meeting in the last 6 months).	COMET
	Capacity Section. New Proposed Item. In addition to the number of active members, another column asks to rate the average level of sector involvement from (1) Low to (5) High.	COMET
Member Relationships	Communication and Conflict Management. New proposed scale to address an important coalition dynamic.	CCT
	Social Network Analysis. Case Study Sites.	COMET
	Protocol/Item to be developed for case study sites (interviews, focus groups)	COMET
Skills & Knowledge	Assessment Section. Community Needs Assessment	COMET
VARIABLE	ITEM	DATA SOURCE
Coalition Structure
Formalized Structures	(2) Coalition Type - Which of the following best describes your coalition? Loosely organized to a highly formal arrangement.	CCT
	(3) Organizational Structure - How would you describe your coalition’s organizational status? Reports to a larger organization to fully independent	CCT
Degree of Centralization	(5a) We have a board or governing body that sets the direction of the Coalition? (A board or governing body is defined as a formal group that makes decisions for the coalition).	CCT
Membership Breadth & Diversity	Capacity Section - New proposed item “How many active members are there in your coalition from each of the 12 sectors. (Active members means an individual has attended at least one meeting in the last 6 months).	COMET
	Cultural Diversity Scale. Individual items can be use from this scale (e.g., 12a, 12e, 12f, 12g).	CCT
Leadership Structure	(2) Coalition Type - Which of the following best describes your coalition? (e.g., Loosely organized to a Highly Formal Arrangement).	COMET
	(4) Confidence in Task Completion Scale. Individual items, such as 4k.	CCT
Institution / Grassroots Mix	(6) Identification of Community Leaders Scale.	CCT
	(7) Community Leadership in Coalition Efforts Scale.	CCT
Workgroup / Committee Organization	(5b) – Formalized Procedures; Established subcommittees	CCT
VARIABLE	ITEM	DATA SOURCE
Coalition Processes
Formalized Procedures	(5b) Written Procedures – E.G., policy for leadership rotation, written expectations for member participation, leadership selection, written description for decision making) & Formalized Procedures – E.G., hold regularly scheduled meetings, written agenda, prepare and distribute minutes, organizational chart, established subcommittees, paid or in-kind staff.	CCT
	Planning Section. Logic Model.	COMET
	Planning Section. Strategic Plan.	COMET
	Planning Section. Action Plan(s).	COMET
Task & Role Clarity	Planning Section. Action Plan(s).	COMET
Supports for Member Involvement (Member Satisfaction)	Member Satisfaction Scale. Proposed new scale with three items.	CCT
Leadership Style	(6) Identification of Community Leaders Scale.	CCT
	(7) Community Leadership in Coalition Efforts Scale.	CCT
	Coalition Leadership Scale. New Proposed Scale.	CCT
	Committee Leadership Scale.	CCT
	Coalition Coordinator Leadership Scale.	CCT
Communication	(8) Collaborative Decision Making Scale.	COMET
Conflict Resolution	(9) Frequency of Conflicts. Propose deleting.	COMET
	(10) Cause of Conflicts. Propose deleting.	COMET
	(11) Overall Impact of Conflicts. Propose deleting.	COMET
Shared Decision Making	(8) Collaborative Decision Making Scale.	CCT
Information / Expertise Input & Use	Capacity Section. Assistance Needed by Coalition.	COMET
	Capacity Section. Proposed Assistance Provided by Coalition.	COMET
	Capacity Building Capacity Scale.	CCT
Strategic Planning Process (SPF Framework)	Planning Section. Strategic Plan.	COMET
	Planning Section. Action Plan.	COMET
	Planning Section. New Proposed Item. “To what extent does your coalition use the SPF framework?	COMET
	Planning Section. New Proposed Item “To what extent does your coalition incorporate other planning frameworks (e.g., CCT)?	COMET
Data Use (e.g., needs, feedback, evaluation)	(13) Performance Evaluation – Yes/No items that tap into program monitoring (system for monitoring and tracking coalition activities) and performance and general evaluation activities (collect and analyze data coalition activities and community indicators).	CCT
External Relations / Resource Access	(13) One specific item of yes/no. “Use relations with local Universities or others to get assistance on evaluating coalition efforts.”	CCT
	Protocol/Item to b developed for case study sites	Case Study

Table 3. Variables and Items for Strategies and Activities

VARIABLE		ITEM	DATA SOURCE
Coalition Role
Membership and Focus		Capacity Section. Coalition Membership.	COMET
		(17) Coalition Focus 1. Exclusively coordination to exclusively environmental.	CCT
		(22) Coalition Focus 2. Directly in the community or indirectly by building partner agencies’ capacity.	CCT
Direct Implementation		Planning Section. Strategies to achieve objective.	COMET
		Implementation Section. Implementation Activity.	COMET
		Implementation Section. Implementation Progress. Accomplishment/Barrier and text field.	COMET
		(22) Coalition Focus 2. Directly in the community or indirectly by building partner agencies’ capacity.	CCT
Advocacy		Planning Section. Strategies to achieve objective.	COMET
		Implementation Section. Implementation Activity.	COMET
		Implementation Section. Implementation Progress.	COMET
Initiation / Support		Planning Section. Strategies to achieve objective.	COMET
		Implementation Section. Implementation Activity.	COMET
		Implementation Section. Implementation Progress.	COMET
Member Contributions		Capacity Section. Basic Collaborative Activity. Is collaboration among members of your coalition (Increasing, Decreasing, Staying the Same)?	COMET
		Capacity Section. Basic Collaborative Activity. Rating of degree of involvement by sector.	COMET
External Partnering		Case Study. Social Network Analysis.	Case Study
Programmatic Capacity
Clear & Realistic Objectives		Planning Section. Logic Model.	COMET
		Planning Section. Strategic Plan.	COMET
		Planning Section. Action Plan(s).	COMET
		Planning Section. Objective(s). Consistent with other sites (dropdowns)?	COMET
Unique & Innovative Services		Planning Section. Action Plan(s).	COMET
		Protocol/Item to be developed for case study sites.	Case Study
		Case Study. Document Review.	Case Study
		Implementation Section. Implementation Progress. Accomplishment/Challenges and text field.	COMET
Need Driven		Assessment Section. Community needs assessment (all subsections).	COMET
Cultural Competence		(12) Cultural Diversity – Representative staff members, culturally appropriate materials, accessible meeting times, staff trained to be culturally sensitive in interactions with target population, etc..	CCT
		Case Study – Review of all Documents, including social marketing materials.	Case Study
Use of Evidence-Based Practices		Planning Section. Strategic Plan.	COMET
		Planning Section. Action Plan(s).	COMET
		Implementation Section. Activities.	COMET
		Protocol/Item to be developed for case study sites.	COMET
VARIABLE		ITEM	DATA SOURCE
Strategy/Activity Mix
Information / Education		(17) Coalition Focus 1 – exclusively helps coordinate programs and services, split between coordinating and environmental/ policy change, to exclusively environmental or policy change.	CCT
		Implementation Section. Strategy.	COMET
		Implementation Section. Implementation Activities.	COMET
		Implementation Section. Implementation Progress. Accomplishments/Barriers with text field.	COMET
Coordination		(17) Coalition Focus 1 - Exclusively helps coordinate programs and services, split between coordinating and environmental/ policy change, to exclusively environmental or policy change.	CCT
		Implementation Section. Strategy.	COMET
		Implementation Section. Activity.	COMET
		Implementation Section. Implementation Progress. Accomplishments/Barriers with text field.
Environmental Policy / Enforcement		(17) Coalition Focus 1 – Exclusively helps coordinate programs and services, split between coordinating and environmental/ policy change, to exclusively environmental or policy change.	CCT
		(21) Environmental Strategies – nine items assessing the degree to which the coalition believe in and have the ability to address environmental change strategies.	CCT
		Planning Section. Goals and Objectives. Strategies to achieve objective. (e.g., modifying/changing policies, physical design)	COMET
		Implementation Section. Strategy.	COMET
		Implementation Section. Activity.	COMET
		Implementation Section. Implementation Progress. Accomplishments/Barriers with text field.
Environmental Assets & Opportunities		Implementation Section. Strategy.	COMET
		Implementation Section. Activity.	COMET
		Implementation Section. Implementation Progress. Accomplishments/Barriers with text field.
Direct Service Programs		(22) Coalition Focus 2 – One item that asks the degree to which the coalition is engaged in directly in the community (coordinating programs and pursuing policy changes), indirectly in the community (building the capacity of other organizations), or a mix of both of these approaches.	CCT
		Implementation Section. Implementation Activity.	COMET
		Implementation Section. Implementation Progress. Accomplishments/Barriers with text field.	COMET
VARIABLE		ITEM	DATA SOURCE
Coalition and Community Outputs
Coalition Outputs	Planning Section. Objective (new list of dropdowns). To what extent has the objective been achieved (from Not at all to exceed our objective)?
	Implementation Section. Implementation Progress. Accomplishments.		COMET
	Case Study sites. Document Review.		Case Study
Population Targets / Penetration	Implementation Activity. New categories/columns which includes penetration.		COMET
	Case Study sites. Document Review.		Case Study
	Protocol/Item to be developed for case study sites.		Case Study
Volume /Dosage	Planning Section. Objective (new list of dropdowns). To what extent has the objective been achieved (from Not at all to exceed our objective)?		COMET
	Implementation Section. Implementation Activities. New categories/columns which includes dosage.		COMET
	Implementation Section. Implementation Accomplishments/Challenges.		COMET
Comprehensiveness	Planning Section. Goals and Objectives. Strategies to achieve objective.		COMET
	Implementation Section. Implementation Activities.		COMET
	Implementation Section. Implementation Accomplishments/Challenges.		COMET
	New Proposed Variable. Implementation Section. Implementation Strategy. Count total number of activities within each strategy (and collect related dosage information).		COMET

Table 4. Variables and Items for Coalition Capacity

VARIABLE	ITEM	DATA SOURCE
Coherence
Shared Vision	Confidence in Task Completion Scale. Provide direction and vision for the Coalition through its leadership	CCT
	Planning Section. Strategic Plan	COMET
	Planning Section. Logic Model	COMET
	Planning Section. Action Plan(s) – Years 2-5	COMET
Shared Solutions	(25) Synergy Scale	CCT
Inclusiveness	Shared decision making items can be used for this variable.	CCT
	Planning Section. Strategic Plan	COMET
	Planning Section. Action Plan(s) – Years 2-5	COMET
Broad Involvement	(8) Shared decision making	CCT
	Capacity Section. Coalition Membership.	COMET
Commitment / Satisfaction	(25) Synergy – One item specifically asks about commitment of partners.	CCT
	Capacity Section. Coalition Membership. Rate the average level of involvement for each represented sector. Yes/No for representation and a Likert scale from (1) Low to (5) High.	COMET
Coalition Climate
VARIABLE	ITEM	DATA SOURCE
Open	Protocol/Item to be developed for case study sites.	Case Study
Trusting	Social Network Analysis.	Case Study
	Capacity Section. Challenges and Accomplishments.	COMET
Value Diversity	(12) Cultural Diversity Scale.	CCT
Shared Input	Shared Decision Making Scale. Shared decision making items can be used for this variable.	CCT
Positive External Relations
VARIABLE	ITEM	DATA SOURCE
Non-Member Agencies	Capacity Section. Coalition Membership.	COMET
	Capacity Section. Challenges	COMET
Activity Focus	Capacity Section. Basic Collaborative Activity. This section now includes # people who attended event, along with a rating of degree of success of event (None to A Great Deal)	CCT
	Capacity Section. Assistance Provided by Coalitions. This new variable asks for a text description of any training and technical assistance provided by the coalition.	COMET
Access Community Resources	Capacity Section. Funding Stream. In the next 12 months do you expect your coalition’s funding level to (Increase, Decrease, Stay about the Same)	COMET
	Capacity Section. Funding Stream. Fundraising/private donations. Dollar Amount.	COMET
	Capacity Section. Funding Stream. Foundations/non-profit foundations. Dollar Amount.	COMET
	Capacity Section. Funding Stream. City/county government. Dollar Amount.	COMET
	Protocol/Item to be developed for case study sites.	Case Study
Community Role & Recognition	(26) Collective Efficacy Scale – One item specifically asks about this variable. “Our coalition has the support of other organizations and influential community leaders.”	CCT
Capacity Building Effort
VARIABLE	ITEM	DATA SOURCE
Outreach/ Recruitment	Capacity Section. Coalition Membership.	COMET
	Protocol/item to be developed for case study site.	Case Study
Access to Training & Technical Assistance (T?TA)	(24) Capacity Building Capacity – Ability to produce education al materials, resources to send members to workshops and conferences, information on research based prevention programs and strategies, etc.	CCT
	Capacity Section. Assistance needed by coalition. Listing of 15 core competencies for coalitions and they rate themselves on the degree to which they could benefit from training and technical assistance from Not at All to A Great Deal.	COMET
	Capacity Section. Assistance needed by coalition. Descriptive text field.	COMET
	Capacity Section. Assistance needed by coalition. Mode of TA	COMET
	Capacity Section. Assistance needed by coalition. Source of TA	COMET
Build Member , Coalition, and Programmatic Capacity	Capacity Section. Assistance needed by coalition. Listing of 15 core competencies for coalitions and they rate themselves on the degree to which they could benefit from training and technical assistance from Not at All to A Great Deal.	COMET
	Protocol/Item to be developed for case study site.	Case Study
VARIABLE	ITEM	DATA SOURCE
Continuous Improvement
Data-Based	(24) Capacity Building Capacity. Two of the items specifically reference research or evidence-based programs.	CCT
	(13) Performance Evaluation Scale.	CCT
	(19) Action Plan Activities Scale.	CCT
Systematic
Explicit Decision Points	Planning Section. Strategic Plan.	COMET
	Planning Section. Action Plan.	COMET
	(19) Action Plan Activities Scale. Specific items can be used from this scale (e.g., Created a realistic timeline for completing activities).	CCT
Responsive & Adaptive	(27) Coalition Self-Assessment Scale.	CCT

Table 5. Variables and Items for Coalition Effectiveness

VARIABLE	ITEM	DATA SOURCE
Norms & Awareness
ATOD Risk Perception / Awareness	Evaluation Section. Core Outcome Measure. Peer Disapproval.	CORE
	Evaluation Section. Core Outcome Measure. Perception of Risk/Harm.	CORE
	Assessment Section. Risk/Protective Factor. Trend is (Increasing, Decreasing, Staying the Same).	COMET
Media Other	Implementation Section. Accomplishments.	COMET
	Implementation Section. Challenges.	COMET
	Evaluation Section. Accomplishments.	COMET
	Evaluation Section. Challenges.	COMET
	Protocol/item to be developed for case study sites.	Case Study
VARIABLE	ITEM	DATA SOURCE
Systems & Policy Change
Collaboration	Organizational Impact Scale. This proposed scale has been included to better document and assess how the quality of collaboration matters in coalition efforts.	CCT
	Interorganizational Coordination Scale. Proposed new scale with four items	CCT
	Capacity Section. Basic Collaborative Activity. Is collaboration among members in your coalition (Increasing, Decreasing, Staying the Same)	COMET
Service Integration	Organizational Impact. Proposed new eleven item scale.	CCT
	Protocol/Item to be developed for case study sites.	Case Study
Policy Change – Public / Business / Education	Implementation Section. Strategy.	COMET
VARIABLE	ITEM	DATA SOURCE
Sustainable Accomplishments
Systems	(14) Coalition Sustainability Scale. This scale has been modified by including more items.	CCT
	(26) Perceived Effectiveness Scale. This is the collective efficacy scale re-named with additional proposed items to assess the degree and extent of successful coalition activities and outcomes.	CCT
	Planning Section. Planning Objectives. To what extent has the Objective been achieved? (From not at all to exceeded our objective).	COMET
	Implementation Section. Implementation Accomplishments/Challenges.	COMET
	Protocol/Item to be developed for case study sites.	Case Study
Services	Planning Section. Planning Objectives. To what extent has the Objective been achieved? (From not at all to exceeded our objective).	COMET
	Implementation Section. Implementation Accomplishments/Challenges.	COMET
	Protocol/item to be developed for case study sites.	Case Study
VARIABLE	ITEM	DATA SOURCE
Substance Use Prevalence
Core Measures - Revised	Evaluation Section. Core Outcome Section. Average Age of Onset. Alcohol, Tobacco, Marijuana, Prescription Drugs GPRA measures specific to alcohol remain unchanged for coalitions receiving STOP Act funds	CORE
	Evaluation Section. Core Outcome Section. Past 30-Day Use. Alcohol, Tobacco, Marijuana, Prescription Drugs GPRA measures specific to alcohol remain unchanged for coalitions receiving STOP Act funds	CORE
Contributors & Consequences
Educational Performance	Local Data if Available.	Local
	Archival Data if Available.	Educational District
Health & Wellness	Local Data if Available.	Local
	Community Readiness and Capacity. New proposed scale that assesses a community’s readiness and capacity to implement a community-based substance use prevention program (DFC).	CCT
	Community Social Organization. Proposed new variable that assesses the degree of interaction with neighbors, attachment to the neighborhood, and includes a perceived safety in the neighborhood scale.	CCT
Crime	Local Data if Available.	Local
	Archival Data if Available.	UCR; FARS
	Community Social Organization. Proposed new variable that assesses the degree of interaction with neighbors, attachment to the neighborhood, and includes a perceived safety in the neighborhood scale.	CCT
Family	Local Data if Available.	Local
Social	Local Data if Available.	Local
	Archival Data if Available.	Census; Other
Economic	Local Data if Available.	Local
	Archival Data if Available.	Census; Other

LITERATURE REVIEW

As highlighted throughout this appendix, this literature review is intended to provide empirical support and rationale for the re-design of the National DFC Evaluation Logic Model and the development of our internal model, which guided decisions for which variables to keep in COMET and other data sources, which to delete, what to add, and what additional analyses are needed to enhance our understanding of how coalitions can create community and population-level changes. The literature review is organized around the major components of the National DFC Evaluation logic model and includes: (1) context, (2) coalition structures and processes, (3) strategies and activities, (4) coalition capacity, and (5) coalition effectiveness. It provides a high level review of the State of current research for each major component and associated constructs and variables. We then briefly explain on how we conceptualize using it in our evaluation of DFC, including possible additional variables and analyses.

Context

Community

The local community context and history are important contextual factors that must be captured to adequately understand why each coalition choose to select specific objectives and strategies, address specific subpopulations of youth, and how they involved youth and community members and leaders in their efforts. In fact, we believe the community context impacts all facets of coalition operations as reflected in our external logic model. For our internal model, we broadened the importance of contextual variables to also include the context of the coalition, which includes organizational and collaborative history, coalition readiness for change, and type of change approach (institutional or grassroots focus).

In terms of community context, constructs included in the internal model include community social organization, community readiness for change, strengths/weaknesses of the community ATOD prevention resource system, number of related initiatives, socioeconomic context, critical events, and community stability. Since the prior evaluation did not include many of these contextual variables, we have selected a number of new measures to include in the evaluation. For instance, one aspect of the community context that has received extensive attention and empirical support in differentiating successful community-based efforts from unsuccessful ones is the concept of community readiness for change (Arthur & Blitz, 2000; Bowen, Kinne, & Urban, 1997). Community readiness has been defined as “the relative level of acceptance of a program, action, or other form of decision-making activity that is locality-based” (Edwards, Oeting, & Littlethunder, 1997). There are many different conceptualizations of community readiness for change and there is also substantial conceptual and measurement overlap with related constructs, such as community capacity for change. Given the state of the research, it is important for researchers to task themselves “capacity and readiness for what and where?” (Foster-Fishman, Cantillon, Van Egeren, & Pierce, 2006, p. 3,). Thus, given DFC’s main goal is to reduce substance use among youth, we created a community readiness and capacity for change scale specifically for this initiative². This scale queries coalition members about their community’s knowledge of and support for DFC goals, leadership, resources, and history of collaboration to address community concerns and issues.

Measures that assess community identify and community social organization have also been found to be important contextual variables that can help facilitate or impede the implementation of community-based programs and services designed to promote community health. Over the last couple of decades, research has re-established that community matters in terms of a number of outcomes, from educational success to violence and drug use (Bursik & Grasmick, 1993; Leventhal & Brooks-Gunn, 2000; Sampson, Morenoff, Gannon-Rowley, 2002. Given the overarching importance of community and its influence on every major component in our external model, one of the main goals of the evaluation is to increase the measurement and understanding of how community contextual conditions can impact the degree of implementation of community policies and practices targeting substance use reduction. For instance, in line with the protective and risk factor approach (Hawkins & Catalano, & Miller, 1992), we propose using a measure of community social organization that assesses the general safety of the neighborhood and the degree to which community members feel they are connected to their neighbors and are able to act and create change in the community (Cantillon, 2006).

Research on neighborhood effects has established that the degree of poverty within a community, particularly concentrated poverty, and the amount of residential turnover or stability has important and long-standing impacts on community rates of violence and drug use (Colder, Mott, Levy, & Flay, 2000; Sampson & Groves, 1989; Sampson, Raudenbush, & Earls, 1997). For our evaluation, we plan on using census data, along with some quantitative scales to create a neighborhood typology and assess whether DFC activities are more successful in some types of neighborhoods compared to others, or if different neighborhood types utilize different community change strategies. These types of analyses will provide a much deeper understanding of the influence of the neighborhood environment and context compared to the current urban, suburban, and rural typology. We also plan to use qualitative data collected from neighborhood leaders at our case study sites to augment this quantitative analysis to obtain a richer perspective on how concentrated poverty and neighborhood instability impact the implementation of DFC.

Community readiness and social organization or sense of community are broad indicators of a community’s infrastructure and capacity for creating positive change. More specifically, communities need to have the resources (ATOD prevention resources) to successfully implement a comprehensive and community-wide initiative designed to reduce substance use and promote community well-being (Wandersman, 2009). This includes the financial resources to implement evidence-based practices, reimburse key participants, and access to high-quality coordinated training and technical assistance resources to support effective implementation practice. In fact, research has demonstrated that many community-based programs fail to be adequately implemented because they are lacking this infrastructure, which has been also been labeled the prevention support system (PSS) (Saul et al., 2008). In a related manner, the existence of prior or ongoing initiatives facilitates the development and implementation of community-based interventions. Such initiatives often force communities to address the typical challenges encountered when designing and implementing social change programs and initiatives, such as the amount of time it takes to produce noticeable change and degree of collaboration required to improve the delivery of appropriate services and improve youth outcomes.

Coalition

Since coalitions are comprised of a subset of community members, many of whom are community leaders or leaders of community-based agencies, we also created a coalition readiness for change scale, which includes the same items in community readiness but also asks about prior positive collaboration among partner agencies, available financial resources, and availability of formal and informal supports and services. Our prior experience has supported differentiating broader community readiness and capacity from the coalition itself, as they often differ, and such measures provide an understanding of the various levels involved in comprehensive community change initiatives (Kreger, Brindis, Manuel, & Sassoubre, 2007; Levy et al., 2005).

As mentioned under community context, another important variable in differentiating coalitions’ ability to create positive community change is simply their past experience in collaborative efforts, including key organizations or agencies’ history or being involved in these activities and efforts. We intend to gauge how prior collaborative work impact DFC through qualitative data obtained during site visits aas well as a social network analysis, which will include an item specific to prior collaborative work and relations. Also, while each DFC community receives the same amount of grant funding, some DFC communities are able to broker additional funding from foundations, local governments, or obtain in-kind funding from key participating agencies and organizations. Finally, coalitions can differ depending upon the degree to which their constituents are from State and typical neighborhood agencies compared to those that are formed by neighborhood residents and described as grassroots in nature. It is important to understand these differences as they often lead to different goals, objectives, and strategies for change (Chavis, 2001).

Coalition Structures and Processes

Member Competency

Coalitions work to produce community change by connecting neighborhood residents, organizations, and institutions together in a manner that their collective efforts are much greater than individuals or organizations working in isolation. In other words, a coalition’s effectiveness is heavily dependent upon individuals’ and organizations’ skills, knowledge, assets, and resources and coalition members are regarded as its primary asset (Butterfoss et al., 1993). While this capacity can be enhanced through education, experience, coaching, and training and technical assistance (Florin, Mitchell, Stevenson, & Klein, 2000; Foster-Fishman et al., 2001), it nonetheless represents a coalition’s baseline or starting point in terms of its ability to foster and create community and systems change. Initial assessment of members’ skills and knowledge is often part of the needs assessment process, along with a broader assessment of the larger community. In fact, a comprehensive needs assessment process often increases members’ skills and knowledge as they learn firsthand about prevalence of youth drug use and ATOD needs and prevention resources. It is also the first step in the strategic planning framework (SPF) process, which is the community planning process used by all DFC coalitions to plan, implement, and evaluate their activities and efforts toward substance use reduction.

As the research and practice literature emphasize, participating on a coalition often means dealing with conflict among members and organizations due to differing priorities and perspectives. A healthy coalition requires that individual members make compromises for the benefit of the coalition’s overarching vision, mission and goals. In fact, one of the major review articles used to inform our selection of constructs and variables identified active member involvement as one of six factors that were associated with coalition effectiveness (Zakocs & Edwards, 2006). Past experience with working on coalitions, degree of member involvement, and positive member relationships provide individuals with experience in negotiating these difficult decisions and processes for the greater good. Currently, COMET queries coalition leaders to report on the number of active members in the coalition by the required twelve sectors. We propose adding another column and simply asking the coalition leader to report on the degree of active involvement by sector. This simple addition to COMET provides a much more accurate assessment of involvement, not just member attendance at meetings. The addition of this item also allows for the possibility of assessing a typology of active member involvement. In other words, is there a right mix of players or do certain sectors need to be actively involved for healthy coalition functioning, and is this associated with impacts on systems and community change? Given the focus on at least twelve sectors, this addition would allow us to investigate if sectoral representation of all sectors is needed, how much, and at what levels; or, if other dynamics like leadership are the key to effective coalition functioning.

While nobody denies the importance of member relationships in coalition functioning, this variable is difficult to operationalize and measure, outside of more in-depth qualitative and case study methodologies. We plan on identifying the importance of member relationships by conducting social network analyses in each of our 36 case study sites over the course of the initiative. The utilization of this innovative methodology in coalition research has led to the identification of the relations required to bring about community change (Nowell, 2009). This analysis also provides us a means to delve greater into quality of collaboration, a measure often assessed by counts of MOUs and other methodologies that do not truly assess this important variable. For instance, we plan on asking members to rate each other on (1) communication frequency, (2) shared philosophy, (3) responsiveness to concerns, (4) legitimacy, (5) trust in follow-through, and analyze whether higher degrees of true collaboration are associated with higher coalition functioning and positive community changes. Also, as mentioned in the prior section, given the importance of past relationships and participation in related initiatives, we plan on including this as a question in the network analysis to better understand how community and coalition context affect member relationships.

Coalition Structure

An important component of any coalition to act effectively in the community is having an array of diverse stakeholders that represent all constituencies and members of the community. In fact, to ensure this adequate representation, DFC requires participation from twelve community sectors, including education, law enforcement, youth, and community leaders among others. This requirement is strongly supported by the literature as membership breadth and diversity provides members with access to a wide array of resources to build collaborative capacity and enhances credibility with the local community (Butterfoss et al., 1993; Florin et al., 1993). Recruitment of a diverse membership base is one of the first significant tasks coalitions encounter, and, often requires considerable attention and resources to maintain and sustain throughout the lifecourse of coalition. This is particularly the case with youth, the target population of coalition efforts, and their engagement and sustainability in efforts is truly needed to bring about community change. Tracking the diversity of membership can also assist in identification of possible technical assistance and training needs to ensure coalition membership is representative of, and credible with, the targeted community.

As coalitions grow and mature, they often institute a number of formalized structures and processes to increase their functioning and capacity for community change (Zakocs & Edwards, 2006). This column or component of our internal model reflects this increasing capacity and formalization. In terms of coalition structure, this includes the implementation of a leadership structure and often a number of workgroups/committees and other formalized structures to be responsible for completing work outside of coalition meetings. This formalization is often guided by the completion of strategic and action plans that specify goals, objectives, activities and coalition members responsible for ensuring the successful completion of coalition work. Implementation of such governance and procedural structures has also been cited in collaborative literature reviews as one of the main factors in coalition effectiveness (Roussos & Fawcett, 2000; Zakocs & Edwards, 2006). The degree of formalization also often depends on the local community context and history of collaborative efforts in the community. For instance, while coalitions with formalized structures have been associated with improved functioning and outcomes, some communities and coalition types (grassroots versus institutional or agency-based) may feel this degree of formalization is unwarranted and possibly even an unhealthy approach to community change efforts. It is these types of community and contextual elements that must be understood when implementing, and evaluating, community change efforts.

Coalition Processes

As highlighted in reviews of the field, collaborative structure and process variables are the most heavily researched and understood components of coalition effectiveness. Reviews have indicated the importance of establishing effective coalition processes, such as: (1) having a clear vision and mission, (2) leadership, (3) role clarity, (4) group cohesion, and (5) formal governance procedures among others (Israel, Schulz, Parker, & Becker, 1998; Roussos & Fawcett, 2000). However, at the same time, one of the most valid criticisms of coalition research is the degree to which collaborative processes and dynamics (shared vision, leadership, conflict management, shared decision making) are tied to coalition functioning rather than the actual linking these mechanisms to community changes (Zakocs & Edwards, 2006).

Our current measurement and analysis plan focuses heavily on better understanding these variables by creating overarching metrics that move beyond analysis of individual processes and tying these larger constructs to community and systems changes. For instance, Allen (2005) created an overarching climate construct that included (1) effective conflict resolution, (2) presence of a shared mission, (3) shared decision making, and (4) task oriented and inclusive leadership. Additionally, given the large number of variables, aggregation to higher-level constructs is the only meaningful way to analyze data in a parsimonious manner that can actually lead to implications for policy and practice for DFC. Finally, it should be noted that where we identified important missing structure and process variables (e.g., leadership), we have included them in our instruments, in particular the CCT.

An integral component under coalition processes includes the utilization of the strategic planning framework (SPF), which focuses coalition activities into five distinct components (i.e., assessment, capacity, planning, implementation, and evaluation), while also acknowledging the importance of sustainability planning and cultural competence in each phase or framework component. As part of our evaluation, we plan on attempting to better understanding how the SPF is linked to coalition functioning and community change. This is one of the reasons we recommend to continue to include data entry into COMET based on the components of the SPF, despite some evidence from our qualitative analysis of dropdown boxes that coalition members did not completely understand the differences between all components of the framework. This finding from our qualitative analysis is also the reason we propose highlighting definitions to a greater extent and providing examples of what constitutes SPF activities prior to data entry. This will increase the quality of data collected and also serve as a continuous education process for the SFP framework and essential coalition processes. We feel an evidence-informed community change framework is needed to educate coalition members regarding the iterative processes of community and systems change. Meanwhile, the role of evaluation is to attempt to unpack how SPF processes lead to change. For instance, we can utilize an interrupted time-series design to examine the effects of each SPF component (e.g., assessment) on the type and scope of community changes (e.g., increased awareness of ATOD use). This type of innovative analysis has been recently used to better understand the strategic planning process and how differential outcomes are achieved depending upon community and coalition context, and degree and intensity of community planning (Watson-Thompson, Fawcett, & Schulz, 2008).

Strategies and Activities

The strategies and activities column is perhaps the most straightforward of the internal model as it focuses on the explicit strategies and activities implemented in the community to increase collaboration and to decrease youth substance use. While there has been an exponential increase in evidence-based “clearinghouses” across a number of fields in recent years (e.g., SAMHSA’s National Registry of Evidence-Based Programs and Practices, DOE’s What Works Clearinghouse, OJJDP’s model programs), broader dissemination and implementation of these certified programs and practices often do not meet with the same success as initial results, which were often established in a small number of settings and with much higher levels of funding. In fact, effective implementation of evidence-based programs and practices has been limited across all fields, in particular community-based programs and services (Arthur et al., 2010; Durlak & Dupre, 2008; Julian, Ross, & Partridge, 2008).

One of the main goals of ICF’s evaluation is to strengthen attribution between process (i.e., what coalitions do and how they do it) and outcomes (i.e., results of their efforts). This is the reason such effort went into revisiting and revising the National Evaluation logic model, and perhaps more importantly, development of a specified internal model that required an item-by-item intensive review of every data source and variable. Another major related goal of the National Evaluation is to deconstruct DFC programs and activities to identify best practices – what works, why it works, and in what situations it works. In essence, while there has been tremendous growth in identifying effective prevention programs, there needs to be an equal amount of effort in identifying how to effectively implement such programs across a diverse array of community settings. We intend to fill this void by focusing on documenting the different types of strategies and dosage of activities required to result in community-level changes in youth substance use rates. By building the evidence base between process and outcomes, we intend to provide communities with the information needed to bridge science to effective community-based practice.

Coalition Role

As previously discussed under coalition context, coalitions often have more of an institutional or agency-based focus or are more grassroots in nature and led by local community leaders or those most affected by the issue the coalition is addressing. The different types of coalition members and make-up are important because each brings a different perspective toward the best way to focus coalition efforts toward reducing substance use (Chavis, 2001). DFC purposely combines an agency and grassroots approach by requiring the membership of twelve sectors, including a mix of agencies and institutions (schools, law enforcement) and local parents, youth, and youth-serving organizations. While DFC coalitions often engage in a number of roles in their planning, implementation, and evaluation of substance use reduction efforts, they generally focus more intensely on some roles than others. For instance, some coalitions focus more on coordinating community-based efforts of prevention programs or services while others focus more on environmental or policy change. Other roles identified in the internal model include direct implementation, advocacy, initiation/support, member contributions, and external partnering. Currently, the CCT collects information pertaining to coalition focus and roles and this data can be used to assess if certain types of coalition roles work more effectively than others, or employ a different mix and strategies and activities. We also plan to utilize the case study component to delve deeper into the relationship between membership and coalition focus, role, selection of strategies and activities, and what relationship, if any, this has to community changes and outcomes.

Programmatic Capacity and Strategy/Activity Mix

Programmatic capacity generally refers to the processes and outcomes of the assessment, capacity building, and planning stages; that is, the development high-quality strategic and action plans that outline the key goals, objectives, and activities to achieve identified objectives in a realistic timeframe. If coalitions adequately plan and prepare for the development and implementation of evidence-based programs and practices, they will be much more effective in this stage. For instance, the major reviews of coalition research have demonstrated the importance of action planning, establishing clear and realistic objectives, as well as infusing cultural competence in all prevention and intervention activities (Roussos & Fawcett, 2000; Zakocs & Edwards, 2006).

DFC coalitions are guided in their efforts by seven overarching strategies to achieve community change, which include: (1) providing information, (2) enhancing skills, (3) providing support, (4) modifying/changing policies, (5) changing consequences, (6) enhancing access/reducing barriers, and (7) physical design³. Another broad category, strengthening coalitions, captures how the coalition develops its own capacity and capability to accomplish the seven community change strategies. These change strategies operate along a continuum and target varying levels of intervention from more general information sharing and education to targeting changes at higher levels, such as changing consequences (e.g., increasing compliance and enforcement of tobacco sales), changes in physical designs of communities (e.g., outlet density, lighting), and changes to policies and practices at multiple levels (city, county, State). Given the multi-causal nature of ATOD use and related problem behaviors, community coalitions need to utilize a mix of multilevel strategies in order to effectively combat the various influences and pressures that can negatively affect our youth (Butterfoss & Francisco, 2004; Foster-Fishman & Behrens, 2007; Kreger, Brindis, Manuel, & Sassoubre, 2007). Thus, the type(s) and intensity or dosage of prevention and intervention activities are important mediators between effective planning and positive community outcomes (Beebe, Harrison, Sharma, & Hedger, 2001).

These strategies are currently assessed in the COMET database system and we intend on keeping all categories but modifying data entry to decrease burden and increase the collection of information related to strategy mix and activity type, scope and dosage. As discussed throughout the main text, the current COMET system is extremely burdensome because of the extensive linking involved, all of which should be already captured in strategic and action plans. For instance, when adding a new objective, COMET requires linking this objective to: (1) strategies to achieve objectives, (2) percent coalition efforts and resources into achieving objective, (3) date objective established, (4) link objective to targeted risk factor, (5) link objective to targeted protective factor, (6) link objective to DFC core outcome measure, (7) link objective to targeted substance, (8) link objective to targeted grade, (9) link objective to targeted gender, (10) targeted date for achieving objective, (11) rating to what extent objective was achieved. Additional details are requested at the activity level, such as status, date started, and date complete for each activity further increasing burden. We propose de-linking this extensive chain and focusing more on what is important: number of activities (by strategy), who is involved, at what level, total number of coalition hours involved, scope of activity (# served), dosage of activity, and impact of activity. Please see Appendix C for more details.

Coalition and Community Outputs

We have included coalition and community outputs in this column to emphasize the importance of celebrating the small wins that occur while coalitions are working to change larger community and macrolevel variables. Our experience has taught us the importance of taking seriously the documentation of all activities and other community outputs, and even graphically displaying the number of activities and other accomplishments the coalition has engaged in over time. This is because one of the biggest challenges people cite when working on coalitions to address larger social issues is simply time – the amount of time it takes to produce visible change at the community and individual levels⁴. In our prior work, while logic modeling helps visualize the bigger picture, it is also helpful to break down longer-term changes by creating activity models, which visually display, at a more concrete and discrete level, how each activity builds upon another and may ultimately produce community and population-level change. For DFC, examples could be producing activity models around each of the core outcomes, or for each of the identified risk and protective factors. Of course, this variable also includes more traditionally conceived outputs such as the quality of the strategic and action plan, social marketing materials, development of curricula, etc. Particularly for coalitions with local evaluators, documenting these activities and showing members all the work they are doing will keep them motivated and participating over the long-term change process involved in coalition work.

Coalition Capacity

Coherence and Coalition Climate

The coalition capacity column of our internal logic model reflects the need to build on the most important assets of coalitions – the voluntary membership of individuals, organizations, institutions, agencies, and community members. The five constructs of coalition capacity include: (1) coherence, (2) coalition climate, (3) positive external relations, (4) capacity building effort, and (5) continuous improvement. First, while logic models are almost always displayed in a linear fashion, it should be recognized the often considerable overlap of stages, which are represented by columns in our model (Florin, Mitchell, Stevenson, & Klein, 2000). For instance, there is considerable overlap between coalition structures and processes and coalition capacity. This is due to the fact that this column identifies how individual member capacity and coalition capacity, which includes most of the structures and processes already discussed, need to be developed to increase the effectiveness of coalition activities and efforts in the community. In essence, the coalition structures and processes column describes where communities are at the beginning or baseline of their efforts, while the coalition capacity column attempts to capture what has been accomplished in building on the coalition’s initial functioning.

The importance of establishing a shared vision and mission for coalitions has been identified by the majority of collaborative research, particularly by research syntheses and reviews (e.g., Foster-Fishman et al., 2001). Indeed, in terms of strategic planning, one of the major first steps is to use needs assessment data to inform the development of a shared vision and mission of the coalition. Our own prior research on community-based coalitions has indicated that a shared vision keeps members committed during challenging or difficult times and also facilitates the integration of new members. This is vital given the inevitably of member turnover during the course of a community change initiative. As discussed, in order to be an effective change agent, coalitions need broad and diverse membership from all community sectors. This not only establishes credibility but access to a larger number of resources and perspectives. Unifying these different perspectives is required and the first step toward a coordinated group effort is buy-in to the overarching goals and vision of the coalition.

Commitment by coalition members is also vital for successful community change efforts. Research suggests that targeted community level changes often take 3-10 years to become detectable; broader population-level change can take even longer (Roussos & Fawcett, 2000). Thus, commitment and satisfaction are vital to engage and sustain members in such long-term change efforts. Perhaps even more vital is assessing commitment and satisfaction during coalition efforts so that adjustments can be made and technical assistance sought if there is a drop in this important variable – before members “vote with their feet” by leaving the coalition. Thus, we have proposed including a short measure of member satisfaction in the CCT. Inclusiveness is an indicator of coherence and represents how inclusive and diverse coalition membership is throughout the course of coalition change efforts, which has been identified as a major factor explaining coalition effectiveness (Zakocs & Edwards, 2006). For DFC, this includes substantial community representation, in particular youth, and the rationale for us including an extra item regarding youth on the pre-existing cultural diversity scale. We also plan on combining a number of coalition climate variables (e.g., trusting, shared input) and assessing how such a metalevel construct is associated with coalition functioning and effectiveness.

Positive External Relations

For a coalition-driven community change effort, environmental changes will not occur unless members and organizations establish credibility and enjoy consistently positive relations with the larger community, including the target population (e.g., youth) (Berg, Coman, & Schensul, 2009). This is particularly important at the beginning of community change initiatives, and research has suggested the importance in establishing “small or quick wins” (i.e., small scale projects that produce quick and visible results) to help promote community-buy in and jump-start larger activities that will involve greater access to community resources (Deacon, Foster-Fishman, Mahaffey, & Archer, 2009).

Despite the fact that researchers, and, especially practitioners recognize the importance of external partnerships, little attention has been paid to this variable in evaluation efforts, except possibly for case studies. While we plan on developing protocols to capture this in the case study component (e.g., community leader interviews, community focus groups), we also have included an interorganizational coordination measure in the CCT. This variable asked members to rate how coalition efforts have increased coordination of their efforts and how organizations now view themselves as a larger system addressing the issue of drug use in the community instead of separate silos addressing specific domains (e.g., enforcement, treatment). Currently, there are a number of areas within COMET that obtain data on the number of activities in the community, and we have proposed including more ratings of whom in the community was targeted by each activity (scope), the number of people who participated (penetration), and the degree of success and impact of the activity. This type of data can accurately gauge coalition activity focus and participation of non-coalition member agencies, and assess if this is associated with differential outputs and outcomes.

Capacity Building Effort & Continuous Improvement

In a major effort and review the collaborative literature, Foster-Fishman et al. (2001) synthesized the core competencies and processes of coalitions into an overarching capacity building framework. According to the framework, there are four distinct levels of capacity to build for sustainable community change efforts – member capacity, relational capacity, organizational capacity, and programmatic capacity. Member capacity includes the core skills, knowledge, and attitudes that individual members bring with them to the table. For DFC, this would be an individual’s awareness and knowledge of substance use in the targeted community, availability of ATOD in the community, and consequences of ATOD use. This level also includes an individual’s or organization’s ability to work collaboratively with members of the community and different organizations, ability to create and guide and implement evidence-based programs for substance youth prevention, and the ability to build an effective coalition infrastructure for change.

Relational capacity involves the development of a positive internal working climate and positive external climate. As previously discussed, a positive coalition climate includes the development of an open and honest environment where people feel free to share information and input and conflict is handled in a transparent and fair manner. Such a positive climate facilitates the development of common understanding of community assets and problems, promotes the development of a shared vision, mission, and solutions. The importance of comprehensive planning and establishment of a shared vision to describe a coalition’s overarching goals is vital to keeping members focused on the ultimate goals of their collaborative activities and efforts (Bartunek et al., 19996; Roussos & Fawcett, 2000). While coalitions need active involvement by its key members during meetings and other coalition work, participation by non-member agencies is critical if community-based activities and interventions are going to successful (Roussos & Fawcett, 2000). In fact, the involvement of all community sectors and members is so critical to coalition success that we created a separate construct entitled Positive External Relations in our internal model as discussed above.

Organizational capacity includes coalition processes that are more formalized in nature, such as detailed action plans, establishment of clear roles and responsibility, strong leadership, and effective internal and external communication. Many of these variables and constructs were highlighted earlier under coalition processes, procedures, and structures and this column recognizes the need to build on these constructs. The importance of leadership in producing an internal and external climate cannot be overstated as research has indicated its importance to coalition effectiveness (Allen, 2005; Zakocs & Edwards, 2006). While community leadership was well measured by the CCT, we noticed that coalition leadership measures were lacking. Given the central importance of leadership, we have included three short measures that assess the multiple levels of leadership common in coalition efforts (i.e., coalition leadership, committee leadership, and coordinator leadership). The leadership scales were originally adapted from Butterfoss’s (1998) coalition effectiveness self-inventory tool and have been used to study coalitions that work to reduce and prevent domestic violence (Allen, 2005). It should also be noted that there are individual leadership items spread out in other CCT scales, such as confidence in task completion that can be used to augment these proposed scales or create a larger leadership metric. Finally, also discussed earlier under strategies and activities, programmatic capacity involves clear and realistic programmatic objectives, ecologically valid prevention and intervention strategies, and the utilization of data to inform programming and any necessary adaptations to achieve desired results.

In sum, research has demonstrated the importance of building capacity at all these levels, and how such capacity building efforts can lead to more effective coalitions and community change (Florin, Mitchell, Stevenson, 1993; Florin, Mitchell, Stevenson, & Klein, 2000; McMillan et al., 1995). In fact, in a thorough review of the literature, many of the indicators of capacity (e.g., shared vision, leadership, resources, technical assistance and support) were found to be linked to enhanced partnerships and improved population-level health outcomes (Roussos & Fawcett, 2000). Thus, capacity building efforts must be continuous and must target multiple levels (member, coalition, organizational, and programmatic) if coalition efforts and activities are going to successfully address and community conditions and outcomes.

Coalition Effectiveness

Community & System Outcomes

The coalition effectiveness column represents the major outputs, impacts, and outcomes of coalition efforts to increase collaboration and decrease youth substance use. These outcomes are listed separately at both the system level and at the behavioral level to visually display the multiple levels at which DFC coalitions operates and intends to achieve outcomes. The fact that DFC targets multiple levels (e.g., community, school, individual) to reduce youth substance use is in accordance with current prevention science research that suggests multilevel interventions are not only more successful, but also more sustainable (Trickett & Schensul, 2009; Weeks et al., 2009).

DFC utilizes a risk and protective factor targeted approach in their effort to reduce substance use. This approach helps stakeholders understand that it is integral to target and build assets of youth, as well as, target the more traditional risk factors. As the research literature highlights, negative behaviors such as drug and alcohol use are usually not found in isolation, but are bundled with other negative behaviors such as poor educational outcomes and delinquency (Catalano et al., 2002; Coulton, Korbin, SU, & Chow, 1995). This approach also encourages communities to collect data (i.e., assessment) to find out current drug use problems and then tailor prevention activities to the local community context of risk and protection (Hawkins et al., 2008). This is extremely important as every community is unique and even evidence-based programs need to be adaptable and tailored to community conditions if they are going to be successful outside of well-funded demonstration grants and trials. It is also in line with research on asset-based community development and environmental strategies for change, which indicate one of the best ways to mobilize the community is to focus on assets and not solely on problems (McKnight, J.L. (1996). Given the research findings that have indicated positive impacts on precursors to and delay of negative youth behaviors, particularly when provided with training and technical assistance (Feinberg et al., 2002; Hawkins et al., 2008; Manger, Hawkins, Haggerty, & Catalano, 1992), we suggest continuing to use these measures (and framework). However, we have suggested an enhancement to the on-line system (COMET) that goes beyond a check-all approach. For instance, with needs assessment data, even if communities are targeting a host of risk and protective factors, they are certainly targeting some more than others. This is why we suggest, at a minimum, ranking in order the top five protective and top five risk factors. In this manner, analyses can group communities by these targeted factors and correlate what strategies, activities, and dosage are effective in reducing risk or promoting protection, and ultimately, reducing rates of youth substance use. Currently, there is little discoverable information with a check-all approach to data entry.

Increasing collaboration is one of the main goals of DFC, yet past research has not included enough measures that truly evaluated this overarching goal of the program. In response, we have included a number of measures to better address this goal, and, moreover, to understand the link between collaborative efforts and community changes and outcomes (e.g., decreased youth substance use rates). As previously discussed, one of ways we address this gap is to utilize social network analysis at our case study sites. This innovative methodology has been increasingly used to understand coalition functioning and how it is associated with various degrees of collaboration (Friedman et al., 2007; Nowell, 2009). Social network analysis will also allow us to study whether, for instance, the density of networks among collaborative members leads to increased coordination of programs and implementation of environmental strategies that target the various levels of risk and protective factors (individual, family, school, and community). Another measure we propose including is interorganizational coordination. This measure assesses the degree to which coalition members are increasingly working together to address youth substance use and has been found to be an outcome of high functioning coalitions (Allen, 2005). Another proposed measure, organizational impact, assesses how coalition efforts are creating changes within the various child-serving organizations and agencies. That is, this measure asks members to report on how participating in coalition activities has assisted them in changing policy, procedure, and practice within their own organizations. Taken together, these measures go well beyond typical assessments of collaboration, and will provide DFC with information on how the overall quality of collaboration matters.

In terms of community behavioral outcomes, we have not made any changes to the Core outcomes to ensure comparability with historical data. We do realize that some of these indicators may change or additional indicators included, depending on what public use data can be gathered that assesses population-level outcomes beyond substance abuse. As proposed in our logic models, there are likely to be impacts on these behaviors as broad-based community change initiatives are theorized to have an impact on correlated behaviors and outcomes. An emphasis on this type of data also helps DFC communities realize that the goals of program are much broader than reducing drug use and also includes the prevention of other problem behaviors, as well as the promotion of community health and well-being. It is with this latter category that we believe DFC may have, as of yet, unrealized impacts outside of anecdotal information. For example, in addition to being a measure of context, we also look at community social organization and community readiness and capacity for change as potential indicators of community health. That is, do DFC activities, over time, help increase the social organization, cohesion, and health of communities? This is serious area to assess since social organization and related variables (sense of community, social capital, informal social control, and collective efficacy) have been found to be associated with lower rates of violence, delinquency, and drug use (Beyers, Bates, Pettit, & Dodge, 2003; Brooks-Gunn, Duncan, Klebanov, & Sealand, 1993; Elliott et al., 1996; Sampson, Raudenbush, & Earls, 1997). Or, as discussed under context, does the initial level of community readiness and capacity influence the degree of implementation of DFC and lead to better outcomes. That is, do those communities who start out higher in terms of readiness and capacity end up with better outcomes than their less “ready” peers? To close, with the proposed enhancements to the data system, proposed new variables, metrics, and analyses, we believe we can help move prevention science forward to better understanding what are the key strategies and activities to implementing effective community-based drug prevention initiatives.

1 It should be noted that accomplishment or barrier section can be used across many topics; however, currently it is text only. In the future, we have proposed adding dropdowns based on historical DFC data (and still keep a text field) and this will allow for more quantitative and nuanced analyses.

2 ICF created and used a community readiness scale for a child welfare-led community initiative designed to improve safety, permanency, and well-being. This scale was created after a thorough review of the research on readiness and capacity for change. This scale proved to be successful in differentiating communities ready for comprehensive change compared to their peers, and also allowed us to track readiness over the course of the initiative.

3 These strategies were developed by CADCA’s National Coalition Institute and have been slightly adapted over time during DFC program operations.

4 Please see ICF’s “Systems and Organizational Change Resulting from the Implementation of Systems of Care” report for the Children’s Bureau.

Office of National Drug Control Policy Page 0