Office of Management and Budget
Clearance Package Supporting Statement A: Justification
Drug Free Communities Support Program National Evaluation
Supported by:
Executive Office of the President
Office of Administration
Office of National Drug Control Policy
750 17th Street, NW
Washington, DC 20006-4607
Shannon Weatherly
(202) 395.6774 (phone)
(202) 395-6711 (fax)
August 2011
Table of Contents
A. JUSTIFICATION 1
A.1. Circumstances Making the Collection of Information Necessary 1
A.2. Purpose, Requested Revisions/Additions, and Use of the Information 2
A.3. Use of Information Technology to Reduce Burden 7
A.4. Efforts to Identify Duplication 9
A.5. Impact on Small Businesses or Other Small Entities 9
A.6. Consequences of Less Frequent Data Collection 9
A.7. Special Circumstances Influencing Collection 10
A.8. Federal Register Notice and Consultation Outside the Agency 10
A.9. Explanation of Payment or Gift to Respondents .11
A.10. Assurance of Confidentiality Provided to Respondents 12
A.11. Justification for Sensitive Questions 12
A.12. Estimates of Hour Burden Including Annualized Hourly Costs 12
A.13. Estimate of Other Total Annual Cost Burden to Respondents or Record
Keepers 15
A.14. Annualized Cost to the Federal Government 15
A.15. Explanation for Program Changes or Adjustments 15
A.16. Time Schedule, Analysis Plans, and Publication 15
A.17. Reason(s) Display of OMB Expiration Date is Inappropriate 20
A.18. Exceptions to Certification Submissions 20
LIST OF ATTACHMENTS:
Attachment 1. Drug Free Communities Re-Authorizing Legislation
Attachment 2. Drug Free Communities Support Program National Evaluation Plan
Attachment 3. Internal Logic Model, Measurement Mapping, and Brief Literature Review
Attachment 4. Drug Free Communities National Evaluation: Systems, Measures, and Tools
Attachment 5. COMET Data Collection Plan
Attachment 6. Alignment of DFC Core Measures to NOMS
Attachment 7. CCT Data Collection Plan
Attachment 8. COMET and CCT Proposed Item Revision Mock Up
Attachment 9. Sample Uses of the National Evaluation Data
Attachment 10. Case Study Interview Protocols
Attachment 11. Case Study Social Network Survey
Attachment 12. Federal Register Notice
Attachment 13. List of Current Technical Advisory Group (TAG) Members
Attachment 14. Coalition Online Management and Evaluation Tool (COMET) Data Systems: Findings from Focus Groups and Social Media Web Sites
A. Justification
A.1. Circumstances Making the Collection of Information Necessary
The current package represents a request for revisions to a previously approved collection pertaining to the Drug Free Communities Support Program (DFC). DFC was created by the Drug Free Communities Act of 1997 (Public Law 105-20), reauthorized through the Drug Free Communities Reauthorization Act of 2001 (Public Law 107-82) and reauthorized again through the Office of National Drug Control Policy Reauthorization Act of 2006 (Public Law 109-469). The latest reauthorization (see Attachment 1) extended the program for an additional five years until 2012. The DFC authorizing statute (21 USC §1521–1535) provides that community anti-drug coalitions can receive Federal grant funds and that the amount of each DFC grant award shall not exceed $125,000 annually. This revision package is also intended to cover the Substance Abuse and Mental Health Services Administration’s (SAMHSA) Sober Truth on Preventing Underage Drinking (STOP) Act Program, which funds current and former DFC grantees. The STOP Act Program is being evaluated based on the same data being collected for the Office of National Drug Control Policy’s (ONDCP) DFC Program and, therefore, any changes to the current data collection will impact these grantees. STOP Act grants are authorized under the Public Health Service (PHS) Act (42 U.S.C. 290bb-25b), Section 519B.
As the lead agency for setting US drug control policy and strategy, ONDCP provides funding through the DFC Program to build community capacity to prevent substance abuse among our nation’s youth. ONDCP directs the DFC Program in partnership with the Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Prevention. DFC has two primary goals. The first is to reduce youth substance abuse. The second is to support community anti-drug coalitions by establishing, strengthening, and fostering collaboration among public and private non-profit agencies, as well as Federal, State, local, and tribal governments to prevent and reduce substance abuse. An important objective of the DFC Program is to assist community coalitions in becoming self-sufficient. Currently, there are 718 DFC grantees that will be impacted by the revisions requested with this submission.
The purpose of the STOP Act program is to prevent and reduce alcohol use among youth in communities throughout the United States. The program was created to strengthen collaboration among communities, the Federal government, and state, local, and tribal governments; to enhance intergovernmental cooperation and coordination on the issue of alcohol use among youth; to serve as a catalyst for increased citizen participation and greater collaboration among all sectors and organizations of a community that first demonstrates a long-term commitment to reducing alcohol use among youth; and to disseminate to communities timely information regarding state-of-the-art practices and initiatives that have proven to be effective in preventing and reducing alcohol use among youth. There are currently 101 STOP Act Program grantees impacted by this submission.
Under reauthorization legislation (21 § USC 1702), Congress mandated a National Evaluation be undertaken to determine the effectiveness of the DFC Program in meeting its objectives. ONDCP has already managed the initial five-year National Evaluation (2004–2009). In 2009, a new DFC National Evaluation contract was awarded to build on the prior evaluation and resulting knowledge to continue to assess whether, how, and to what extent the DFC program has contributed to preventing and/or reducing substance use among young people (12–17 years of age) in the US. As in the previous National Evaluation, the current evaluation will make use of a single data collection tool to gather information. The Coalition Online Management and Evaluation Tool (COMET) is the current OMB-approved web-based performance system that is used by both DFC and STOP Act grantees to submit their required semi-annual progress reports, including the four current core measures (age of onset, past 30-day use, perception of risk or harm, and perception of disapproval of use by parents for tobacco, alcohol, and marijuana), and houses the Coalition Classification Tool (CCT), designed to capture information on coalition performance or characteristics with regard to four functional areas: (1) coalition development and management, (2) coordination of prevention programs/services, (3) environmental strategies, and (4) intermediary of community support organizations. The current request pertains to revisions to COMET and the CCT that have been determined necessary to reduce burden on grantees, increase the quality of the data, and facilitate the monitoring and tracking of grantee progress.
Additionally, the new evaluation includes a case study component intended to document coalition practices. Specifically, the case studies provide the opportunity to:
Conduct interviews with coalition leadership and determine how successful coalitions achieve positive outcomes in diverse coalition settings.
Conduct interviews and surveys with coalition partners from a number of agencies to determine effective practices in developing healthy collaborative relationships that contribute to coalition effectiveness.
Elicit the insights and observations of respondents to generate hypotheses for investigation in statistical analyses of COMET and CCT data, and to determine whether the results of our evaluation are corroborated by the experiences of “front line” staff.
Collect illustrative examples of findings embedded in the larger analysis of DFC data.
The information from the case studies will be shared with DFC grantees to illustrate not only what works, but also how it works. It is estimated that nine DFC grantees will be selected each year to highlight in the case studies (see Section B. Collections of Information Employing Statistical Methods for site sample selection and data analysis). Clearly, the results from a small sample of purposively selected cases will be illustrative of application in specific contexts, not representative of the full population of grantee experience.
A.2. Purpose, Requested Revisions/Additions, and Use of the Information
Purpose of Collection
The overall goal of the DFC National Evaluation is to assess the DFC Program’s effectiveness in preventing and reducing youth substance use. Two primary objectives of the evaluation are to: (1) support an effective grant monitoring mechanism that provides the Federal government with the expertise, system, functions, and products to collect, analyze, and report data collectively, and (2) regularly monitor and measure data in order to demonstrate the progress of the DFC program and its grantees. Within these broad objectives, the evaluation addresses a series of specific questions which are presented in Section A.16, Time Schedule, Analysis Plans, and Publication, and in Attachment 2, Drug Free Communities Support Program National Evaluation Plan.
With the data provided through COMET and the CCT, the evaluation can examine both direct and indirect relationships between measures of DFC Program’s effectiveness and changes in substance use outcomes in DFC-funded communities. First, the strategies, initiatives, and activities of DFC coalitions are examined to determine whether they have a direct relationship to substance abuse outcome measures of interest, such as the proportion of youth who report using tobacco in the last 30 days. Second, the indirect relationship between the DFC program and enhancing the capacity of grantee coalitions to influence change in the community is assessed by evaluating a number of scales captured through the Coalition Classification Tool (CCT). These scales include coherence, coalition climate, positive external relations, capacity building effort, and continuous improvement. Additionally, the case studies described previously will provide more in-depth qualitative information regarding what works, why, and under what conditions. This latter information is intended to assist coalitions with revising/refining their programs and with the replication of effective practices.
In order to demonstrate these connections, it is first necessary to strengthen process measures, add new measures of coalition operations, strengthen outcomes for analysis, and collect qualitative data from a sample of grantees. The revisions/additions described below are all intended to achieve these goals.
Requested Revisions
As part of the current five-year evaluation, ICF International (ICF), the new evaluation contractor, was asked to review existing systems, measures, and tools available for the evaluation and to make recommendations for revisions or changes. ICF has engaged in due diligence to assess the current data collection processes for the DFC National Evaluation. These approaches include: stakeholder meetings; focus groups with SAMHSA managers and project officers; DFC grantee feedback via a social media website and focus groups; item-by-item review and analysis of COMET and the CCT; content analysis of COMET open-ended questions; a review of the core measure data to ensure compliance with currently accepted National Outcomes Measures (NOMS); and input from the Technical Advisory Group (TAG; see Section A.8. for more discussion). In doing so, specific recommendations for revising and enhancing COMET, the core measures, and CCT were identified. Additionally, ICF worked with ONDCP, SAMHSA, and the TAG to develop a revised logic model for the DFC Program based on a review of current research and expert input. Additionally, ICF examined both COMET and the CCT to ensure alignment of the logic model with the data being required of the DFC grantees (see Attachment 3, Internal Logic Model, Measurement Mapping, and Brief Literature Review). These recommendations have been translated into the revisions being requested under this submission and highlighted below and in Attachment 4, Drug Free Communities National Evaluation: Systems, Measures, and Tools).
The requested revisions to COMET and the CCT represent system enhancements and item/content revisions and are intended to significantly reduce grantee reporting burden. In addition, these requested revisions will allow for better reporting of process, output, and outcome data. The revisions are highlighted below.
COMET Revisions (Semi-Annual Progress Report, including Core Measures)
It was determined that COMET imposes a large amount of data entry burden on respondents. The two primary reasons for this unanticipated burden and proposed revisions to address this problem are as follows:
The specificity of data collection for every activity conducted and the explicit linking of activities to objectives is time-consuming and results in extensive burden and repetition, while providing little programmatic information that is not already captured in grantee-developed strategic and action plans. Additionally, there is a gap in dosage data reported in the planning and implementation sections of COMET. The de-linking of objectives to strategies, activities, core outcomes, targeted substance(s), grade, and gender is requested.
Many COMET questions are open-ended, which results in substantial response burden and extensive post-data collection coding by the data managers. Based on an analysis of historical COMET data and policy relevance, items and response choices have been developed for select questions. This will provide more standardized data that better lends itself to analysis.
In addition to these changes in format, COMET has been revised to capture immediate outcomes (i.e., outputs) of coalition operations. We believe that DFC grantees will derive more utility from this framework, which will in turn improve the quality of data we ultimately receive. The specific deletions, modifications, and additions of items to COMET are depicted in Attachment 5, COMET Data Collection Plan, including the rationale for each revision. Proposed revisions to the core measures include: (1) the addition of prescription drugs as a core substance and (2) improved adherence to SAMHSA’s National Outcome Measures (NOMS), which includes the addition of a peer disapproval measure, the modification of the perception of risk of alcohol to focus on binge drinking, and the removal of age of first use as a core measure (see Attachment 6: Alignment of DFC Core Measures to the NOMS). Other system enhancements to COMET that are requested to reduce burden include: (1) a restructuring of the system to reflect DFC’s revised logic model, and (2) provision of “mouse-overs” to provide examples of acceptable entries under each element. This will ensure that data will be entered more accurately and with less duplication.
One final revision involves the CCT and affects the reporting of data. Historically, a single member of the coalition has completed the CCT. However, this is not considered in the research community to be an effective method, as a single respondent reporting on a group or organization can lead to substantial bias. This is particularly the case if the leader of the group or coalition is completing the instrument. An important revision that will improve the reliability of the data and increase the credibility of the evaluation is requesting that two coalition members complete the CCT rather than a single individual. While this will add to the overall burden estimate for this data collection, based on scientific practices the benefit of the additional respondents far outweighs the increased burden for this particular tool, which is collected only once a year. Below is also a detailed description of how ICF suggests that the CCT be redesigned.
CCT Revisions
Two primary objectives of the DFC National Evaluation Plan focus on (1) strengthening the measurement of process data, and (2) developing new measures for coalition operations. To meet these objectives, in addition to analyzing COMET, a thorough review of CCT data was conducted. The reliability of existing CCT scales and individual item correlations was examined. In general, ICF found that the CCT scales had acceptable reliability levels and most items were sufficiently correlated with their respective scales. However, it was determined that 143 items were answered by the coalition leader regarding their coalition’s functioning and dynamics, and only 24 of these items are needed to comprise the typology assessment for the evaluation. As a result, these 24 items have been designated as core questions in the CCT moving forward. This allows for the tracking of coalition development across time and preserves meaningful historical data. Some other items have been identified for deletion or modification. Additionally, a review of the literature revealed data on indicators of the quality of collaboration and associated outputs and outcomes that needed to be incorporated into the CCT. Specifically, measures of quality of coalition processes (e.g., leadership) outputs and outcomes (e.g., inter-organizational coordination, perceived effectiveness), and local community context (e.g., community readiness for change, community social organization) have been identified for inclusion. Attachment 7, CCT Data Collection Plan identifies these revisions, including a rationale for each change. Together, these revisions are intended to yield higher quality process data on day-to-day coalition activities. Other format revisions that are intended to reduce burden and improve data quality include changing the layout of the CCT by placing all typology questions together (rather than organizing by functional areas) and arranging all Likert-type items together, instead of by section, to allow for faster completion of the survey, even with additional questions.
While the specific revisions to COMET and CCT are highlighted above and presented in detail in the designated attachments, a mock up of the tools with the requested revisions is provided in Attachment 8, COMET and CCT Proposed Item Revision Mock Up.
Also included as part of the overall evaluation of DFC are GPRA measures provided by the National Coalition Institute to ONDCP. The Institute has, over time, determined that new measures should be captured in order to continue to determine the effectiveness the Institute has in developing coalitions. To this end, the following modified measures are being requested: number of technical assistance instances provided, number of coalitions that receive Institute training or technical assistance related to coalition data collection, evaluator or outcome measurement, number of coalitions that participate in Institute-sponsored distance learning sessions, and the percentage of new DFC grantees that attend the National Coalition Academy that request additional Institute training in the second year of their grant. Because these data are collected as part of the Institute’s standard practices (training/distance learning registrations and attendance records, TA tracking), no new or additional burden is being placed on the Institute as a result of these modifications.
Use of the Revised Information
To date, the data collected through COMET and the CCT (and the Institute GPRA measures) have been widely used by ONDCP, SAMHSA and the previous and current national evaluator to monitor and assess progress, inform training and technical assistance delivery, and evaluate the effectiveness of DFC. Additionally, the data and results of the evaluation have been shared with the grantees to help inform coalition operations and programming. The data have been widely used by ONDCP to demonstrate to Congress, grantees, and other stakeholders the progress and impact of DFC (see Attachment 9, Sample Uses of National Evaluation Data).
Moving forward, the data will continue to be used to prepare Status Reports and will be used in the analysis for the current DFC National Evaluation (see Section A.16 for more detail on the Analysis Plan).
Requested Additions
Case Study Interviews and Social Network Survey
The new data collection needed to support the case study component of the DFC National Evaluation will be limited to a sample of nine DFC grantees per year. The site visit sample is selected through consideration of several criteria, including (a) outcome success [and consistency] as indicated in data from COMET and the CCT; (b) recognition for or demonstration of innovative or exemplary capacity building or strategy development; (c) relevance to priority practices or issues identified by the evaluation team; and (d) representation of different settings, populations, and presenting problems in coalition communities. The selection process will be completed each year of the DFC National Evaluation (i.e., to identify nine new coalitions to highlight through the case studies). Annual site visits will be conducted to each case study location to conduct interviews and administer the Social Network Survey as described previously. All coalition chairs will be interviewed and administered the survey. Additionally, the DFC staff member identified as having primary oversight of the grant will be interviewed. Finally, a sample of 12 coalition members from each site will be selected to participate in interviews and to complete the survey. One member of each of the 12 key sectors represented on the DFC coalitions will be selected. (For more detail on site and respondent selection, see B.1. Respondent Universe and Sampling.)
The interviews will obtain information regarding current practices; specifically, the case studies are designed to gather data on the implementation of effective practices which can be replicated by DFC grantees (see Attachment 10, Case Study Interview Protocols). This will include information related to: (a) how identified effective practices achieve positive outcomes, (b) how partners have worked together to maximize the impact of those practices, (c) what additional data is available to substantiate the evidence behind a given practice, (d) the cost of replicating a given practice, including start-up and maintenance costs, (e) how practices were implemented, and what could have been done to make the implementation process go more smoothly, (f) among what populations/settings the practice has the greatest effect, and (g) any other special considerations in replicating a practice (e.g., finding funding, sustainability issues, etc). The same process will be followed for each year of the National Evaluation with a different set of nine grantees. Additionally, the coalition chair and all coalition members will be asked to complete a short Social Network Survey in order to better understand, assess, and evaluate the impact of the coalition. The survey will include items on five key factors that influence the quality of collaboration across coalitions: (1) communication frequency, (2) responsiveness to concerns, (3) trust in follow through, (4) legitimacy, and (5) shared philosophy. These variables have been identified in the literature and through previous assessments as key elements to high levels of collaboration (see Attachment 11, Case Study Social Network Survey).
Use of the New Information
The new data collected as part of the case studies will be used to investigate the relationship between coalition processes and outcomes. Substance abuse prevention strategies – including environmental approaches – are notoriously difficult to attribute to positive outcomes since we are essentially modeling a non-event. The presence of numerous exogenous factors limits our ability to quantify outcomes with certainty; we also need qualitative data to truly understand what is happening and why. The case study data will be used to strengthen attribution of findings, and to obtain important information regarding key considerations in the replication of effective practices. Strong measurement of setting, design, and implementation characteristics is crucial to maximizing the learning opportunities that can come out of the evaluation.
Additionally, data will be used to conduct a social network analysis to determine how partners work together and develop a deeper understanding of intra- and inter-organizational relationships (i.e., depth and quality of relationships). More specifically, social network analysis methods will be used to study the interactions between each coalition agency/organization (i.e., to determine which organizations interact more and the nature of those interactions) and to study network characteristics, such as centrality, clustering of the most highly interacting players, and gaps in interactions. An index of collaboration will be constructed to indicate strength of collaboration for any one agency or organization, allowing for exploration of the relationship between collaboration and numbers of participants in strategies, types of strategies, and community outcomes.
In addition to providing hypotheses for further exploration, and providing examples of application of practices that are supported in the larger data set, the evaluation team will use case study information to proactively provide give-backs in the form of Practice Briefs and Policy Briefs (which summarize effective practices for policymakers). Improving participative involvement and interaction with grantees has been a major objective of the evaluation team, and improving the flow of information back to grantees will contribute to this process. It will help ensure that lessons learned from the evaluation are being shared as soon as possible.
A.3. Use of Information Technology and Burden Reduction
A web-based performance measurement system referred to as the Coalition Online Management and Evaluation Tool (COMET) is being used to collect all data for the DFC Program. This system, described further in Section B.2, is software-independent, requiring only that users have a web browser and access to the Internet. Users are able to enter data freely, closing and opening the system as they wish, with data being saved and maintained by the system. For example, a user can begin entering data on a computer at work, close the application, and return to the application on their home computer.
COMET is an integrated grant management and evaluation information system consisting of the following components: (1) process data collection, which is collected once every six months, (2) the Coalition Classification Tool (CCT), which is collected once a year, and (3) core outcome data, which is collected once every two years. Information collected for the purpose of managing DFC grants is also used in the evaluation.
Compared with the paper reports originally submitted by grantees, use of COMET as the data collection mechanism for the previous evaluation significantly reduced the reporting burden on respondents. However, subsequent review of the system and input from users identified some challenges that need to be addressed. Thus, the revisions requested with this submission are intended to ensure that the use of COMET does, in fact, provide the type of efficiencies expected with online reporting as explained in the previous OMB submission and summarized below:
Integrated data collection. Once respondents input information into COMET, information can then be propagated to appropriate fields throughout the system.
SAMHSA’s Prevention Platform (a web-based tool for coalitions: https://www.preventionplatform.SAMHSA.gov ) has undergone significant re-development since the original OMB package was submitted in 2005. With re-development complete, COMET and the Prevention Platform have been integrated, such that grantees can easily access the COMET system via the Prevention Platform. This eliminates the burden of having to keep track of a separate entry portal in order to gain entry to COMET.
Integrating the data collection activities required by the evaluation and those needed for grants management reduces the overall burden on grantees by providing a single source of information. This integrated system represents a forum where Government Project Officers and grantees can view identical information in real time and proactively answer questions of and about grantees.
The majority of coalitions has Internet access and is able to access and utilize COMET. However, for coalitions without Internet access, the Government Project Officer is able to generate a blank “report” version to capture the progress reporting and evaluation information in hard copy and then, enter the information into the system on behalf of the coalition.
Reducing reporting burden for grants management. Currently, DFC grantees are required to submit semi-annual reports into COMET. Originally, grantees were required to submit a paper report to their Government Project Officer, which was time consuming and required on a quarterly basis. COMET captures and retains semi-annual report information, which allows coalitions to enter or edit information and report accomplishments throughout the year. This spreads the reporting burden over a longer period. Additionally, much of the information is retained in the system and propagated from reporting period to reporting period, thus eliminating the need to re-enter information that has not changed during a given reporting period. Finally, because the information can be captured throughout the year (i.e., proactively) by coalitions and immediately shared with Government Project Officers as needed, the official reporting requirements for grant monitoring and tracking were changed to a semi-annual rather than a quarterly schedule (STOP Act grantees, however, will be expected to report data on a quarterly basis).
A.4. Efforts to Identify Duplication and Use of Similar Information
The revisions requested to COMET and the CCT, in particular the addition of variables, are based on a review of current research and the need to incorporate new and appropriate measures to evaluate the effectiveness of the DFC Program in reducing youth substance use. Additionally, additions to the core measures reported in COMET were necessary to incorporate prescription drug use and peer disapproval. ICF has determined that most grantees are already collecting these measures; therefore, we do not expect that the addition of these core measures/substances will pose significant additional burden. Moreover, by dropping age of first use, which is the most computationally-intensive core measure, we believe that net burden will be neutral to slightly lower for the majority of DFC grantees.
A.5. Impact on Small Businesses or Other Small Entities
Data captured in COMET, including the core measures and the CCT are collected from funded DFC coalitions, some of which may be small entities, as defined by OMB. Compared with early paper-based reporting requirements, COMET streamlines access to and submission of coalition data, thus reducing the paperwork burden on these small entities (see Section A.3 for a full discussion of use of information technology and burden reduction). COMET includes detailed instructions and a “Help” function in the event that technical difficulties are encountered. A Helpline number is also provided should respondents need additional assistance in using the system or completing the instruments. Finally, as part of the DFC National Evaluation, technical assistance, including the “Ask an Evaluator” webinar series is available to all grantees with special emphasis placed on data collection and reporting. Therefore, no significant impact on small entities is expected.
A.6. Consequences of Collecting the Information Less Frequently
The proposed data collection supports multiple purposes as described in Section A.2, including grant progress reporting, support of developmental progress, training and technical assistance, and evaluation.
Grantee progress report data submitted in COMET are collected to assess a coalition’s performance and to ensure coalitions receive the technical assistance they need to meet their goals and objectives. Current DFC Program policies require that these data be collected semi-annually, a requirement clearly spelled out in the Terms and Conditions of the grant award and in the request for applications.
While grantee progress report data is captured semi-annually, the CCT is completed annually. These two components of COMET cannot be completed less frequently without adversely affecting the quality and reliability of evaluation data, and without compromising effective grants management. Anecdotal evidence indicates that coalition capabilities and capacities can develop and expand substantially over the period of a year. Administering the CCT less frequently risks missing these developmentally important changes. Similarly, data from the Semi-Annual Progress Report cannot be collected less frequently without impairing the capacity to assess the effectiveness of the DFC Program as a whole.
A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
This information collection is consistent with the provisions 5 CFR 1320.5(d)(2). Specifically:
Grantees are not required to report information to the agency more often than quarterly. For the DFC Program, progress reports are due semi-annually and the CCT is completed annually. For STOP Act grantees, progress reports are due quarterly.
Grantees are not required to prepare a written response to the collection of information in fewer than 30 days.
Grantees are not required to submit more than one original and two copies of any document. For the DFC Program, grantee reports are completed online and not via hard copy.
Grantees are not required to retain records for more than 3 years.
All information collection has been designed to produce valid and reliable results that can be generalized to the universe of the study.
Statistical data classification will not occur in the absence of review and approval by OMB.
Information collection will not be conducted in a manner that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use.
Grantees are not required to submit proprietary trade secrets or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.
A.8. Federal Register Notice and Consultation Outside the Agency
The notice required in 5 CFR 1320.8(d) was published in the Federal Register on May 2, 2011 (FR vol. 76, no.84, pp. 24537-24538 (see Attachment 12: Federal Register Notice).
The DFC Support Program National Evaluation Plan was initially reviewed by the Technical Advisory Group (TAG) on April 22, 2010 (see Attachment 13 for a list of current TAG members). Subsequently, the work of the logic model working group and input from stakeholders, including grantees, has informed the development of the final National Evaluation Plan submitted to ONDCP on December 27, 2010. Central to the development of the evaluation plan were the recommended revisions to COMET and the CCT. These revisions, as stated previously, were informed by input from representatives from ONDCP, SAMHSA, KIT Solutions (the contractor managing COMET), the TAG, and DFC grantees. Specifically, 20 representatives from SAMHSA (13 project officers and 7 managers) participated in two single-profession focus groups in June 2010. In July 2010, focus groups were facilitated by ICF with DFC grantees at the CADCA Mid-Year Training Institute to discuss data entry concerns related to COMET. Finally, in August 2010, 384 responses were received during a two-week period through the social media website1 established by ICF to obtain real-time responses from DFC grantees about the COMET data system, including information about their experiences with COMET, suggestions for specific improvements, and to capture their vision of the ideal data system for DFC data collection. The results of these meetings, focus groups, and social media website feedback collection are provided in Attachment 14, Coalition Online Management and Evaluation Tool (COMET) Data Systems Assessment: Findings from Focus Groups and Social Media Web Sites.
A.9. Explanation of Any Payment or Gift to Respondents
No payments are made to respondents.
A.10. Assurance of Confidentiality Provided to Respondents
The information to be collected pertains solely to DFC coalitions (e.g., their characteristics, activities, functions, and community-level outcomes). No outcome data for any individual persons (e.g., youth who participate in DFC coalition activities) are sought. Names and contact information for individual persons in their official capacities as coalition leaders or representatives of organizations comprising the coalition are captured. This information is routinely provided in the course of grant application and administration. All personal contact information is treated in a confidential manner. No narratives gathered as part of the information collection activities will be attributed to a specific individual in any reports. Below we describe the handling and reporting procedures employed in order to maintain the privacy of individuals who provide data in their capacities as coalition representatives.
Every DFC coalition participating in the National Evaluation is assigned a unique identification number by the ICF evaluation team. This ID number is used to monitor the DFC coalition’s status throughout the evaluation.
Access to identifying information is limited to the ICF evaluation team
Coding documents and computer files of survey data refer to DFC coalitions by their ID numbers only. No name or institutional identifiers other than ID numbers appear on computer forms.
Individual databases and computer files are protected by passwords or other techniques to restrict access to staff involved in data analysis.
No data used in the DFC National Evaluation will be reported in any form that can be traced back to individual DFC coalitions. For example, cell sizes of less than 10 will not be reported to further protect respondents from identification.
Coalitions are not asked to provide individual-level outcome information or any outcome information for subgroups that could be used to identify responses of individuals.
Upon approval of the revisions to COMET and CCT, ICF will seek and receive clearance for the protection of human subjects from their IRB in order to comply with 45 CFR 46.
A.11. Justification for Sensitive Questions
No questions are asked that are of a sensitive nature.
A.12. Estimates of Hour Burden Including Annualized Hourly Costs
Based on input from the DFC grantees during the focus groups and social media website feedback collection, it was determined that it was taking many grantees longer than originally estimated with the previous OMB submission to enter data into COMET. As a result of this feedback, the above revisions -- in particular, the layout and format of questions -- are being requested to reduce this burden. While a pilot of the revisions to COMET (and CCT) has not been conducted to date, based on the rationale for the revisions and the expected benefits of the system enhancements, in particular to the usability of the system, it is expected that time spent in any semi-annual report period devoted to COMET reporting will be approximately five hours. To help ensure minimum reporting burden on the grantees, ongoing technical assistance is available through KIT Solutions, ICF, and Government Project Officers to prepare grantees for the changes and to address problems or issues in real-time.
The revisions to the CCT, both in format and content, are not expected to impact the previous burden estimate for this tool. The improvement in format and layout, as well as the balance in adding new items, modifying/condensing existing items, and deleting others is expected to result in a CCT that will take approximately one hour to complete. However, the addition of 1 more respondent (coalition member) for each grantee while improving the reliability of the data will increase the overall burden for this instrument.
The burden estimates for COMET, the CCT, and the Case Study Interviews and Best Practices Survey are presented in the table below.
Exhibit 1: Estimates of Hour Burden
Type of Respondents |
Number of Respondents |
Frequency of Response |
Average Time per Response (in hours) |
Total Annual Burden (in hours) |
Instrument – COMET (Semi-Annual Progress Report) |
||||
DFC Grantee Program Directors (non-STOP Act) |
628 |
2 |
5 |
6,280 |
DFC Grantee Program Directors with STOP Act Grants |
90 |
4 |
5 |
1,800 |
STOP Act Grantee Program Directors* |
11 |
4 |
5 |
220 |
Instrument – CCT (Coalition Classification Tool) |
||||
DFC Grantee Program Directors and Coalition Members |
1,436** |
1 |
1 |
1,436 |
Case Studies – Interviews |
||||
DFC Coalition Sector Members*** |
108 |
1 |
1.5 |
162 |
DFC Coalition Chair/Program Director |
9 |
1 |
2 |
18 |
Case Studies – Social Network Survey |
||||
DFC Coalition Sector Members**** |
216 |
1 |
.25 |
54 |
DFC Coalition Chair/Program Director |
9 |
1 |
.25 |
2.25 |
Total |
9,972 |
|||
* STOP Act grantees that were prior DFC grantees
** Includes 2 representatives from each coalition completing the tool annually
*** 12 members of the 9 case study coalitions will be interviewed
**** All coalition members will complete the Social Network Survey (estimate 24 members per case study coalition)
Exhibit 2: Annualized Cost to Respondents
Type of Respondents |
Number of Respondents |
Frequency of Response X Average Time Per Response |
Hourly Wage Rate*** |
Annualized Cost to Respondents |
Instrument – Semi-Annual Progress Report (COMET) |
||||
DFC Grantee Program Directors (non-STOP Act) |
628 |
10 |
$20.55 |
$129,054 |
DFC Grantee Program Directors with STOP Act Grants |
90 |
20 |
$20.55 |
$36,990 |
STOP Act Grantee Program Directors* |
11 |
20 |
$20.55 |
$4,521 |
Instrument: Coalition Classification Tool (CCT) |
||||
DFC Grantee Program Directors and Coalition Members |
1,436** |
1 |
$20.55 |
$29,510 |
Case Studies – Interview Protocols |
||||
DFC Coalition Sector Members**** |
108 |
1.5 |
$20.55 |
$3,329 |
DFC Coalition Chair |
9 |
2 |
$20.55 |
$270 |
Case Studies – Social Network Survey |
||||
DFC Coalition Sector Members***** |
216 |
.25 |
$20.55 |
$1,110 |
DFC Coalition Chair/Project Director |
9 |
.25 |
$20.55 |
$46 |
Total |
$204,830 |
|||
* STOP Act grantees that were prior DFC grantees
** Includes 2 representatives from each coalition completing the tool annually
*** The hourly wage represents the average hourly wage for community and social services occupations reported by the Bureau of Labor Statistics for the National Occupational Employment and Wages, May 2009
**** 12 members of the 9 case study coalitions will be interviewed and surveyed
***** All coalition members will complete the Social Network Survey (estimate 24 members per case study coalition)
A.13. Estimate of Other Total Annual Cost Burden to Respondents or Record Keepers
There are no capital/start-up or operational/maintenance of services costs to the respondents associated with this evaluation.
A.14. Annualized Cost to the Federal Government
The annualized contract cost for development/revision of the data collection system and instruments, data collection, data processing, analysis, and reporting is $950,000. In addition, one Federal employee will be involved for approximately 30% of his/her time over the five years of the project. Annual costs to the government for Federal staff to oversee and support this project are $50,000 for each year, resulting in a total annualized cost to the Federal government of approximately $1,000,000.
A.15. Explanation for Program Changes or Adjustments
COMET and the CCT are both existing tools used to collect required data from the DFC grantees. The revisions requiring OMB approval are based on input from users of the system (DFC grantees) and the data currently reported (ONDCP, SAMHSA, national evaluator). Based on this input, new burden estimates are necessary that more accurately reflect the effort involved in reporting required data using the system and tools. These revisions are necessary to reduce actual burden on grantees, ensure better data quality, and ultimately to support the DFC National Evaluation.
A.16. Time Schedule, Analysis Plans and Publication
Time Schedule
The data collection for the evaluation will occur throughout the five-year period of the contract. A specific time schedule is provided in the table below and has been and will continue to be dependent on the evaluation team’s access to COMET data following each of the semi-annual report cycles. Case study data will be collected once a year with each of the nine identified “best practice” DFC grantees.
Exhibit 3: Evaluation Time Schedule
Activity |
Time Schedule |
Letters sent to respondents informing of changes to COMET and CCT |
2 weeks after OMB approval |
Revised Semi-Annual Progress Reporting (COMET) |
Ongoing across the five years (semi-annually) |
Revised Coalition Classification Tool |
Annually for five years |
New Case Study Interview Protocols and Surveys |
Annually for five years |
Data analysis: Includes classification, data quality assessment, and cross-sectional analyses |
Ongoing for five years; conducted with current and historical data and will continue with revised data |
Submit Annual Report |
Annually across the five years |
Conduct longitudinal data analysis |
Ongoing across the five years |
Submit final report |
6–8 months prior to end of evaluation |
Publish findings on DFC National Evaluation |
As outlined in the contract between ONDCP and ICF |
Analysis Plans and Publication
For two decades, communities have expanded efforts to address social problems through collective action. Based on the belief that new financial support enables a locality to assemble stakeholders; assess needs; enhance and strengthen the community’s prevention service infrastructure; improve immediate outcomes; and reduce levels of substance use, DFC-funded coalitions have been able to implement strategies that have been supported by prior research.2 Research also shows that effective coalitions are holistic and comprehensive; flexible and responsive; build a sense of community; and provide a vehicle for community empowerment.3 Yet, there remain many challenges to evaluating them. Specific interventions vary from coalition to coalition, and the context within which interventions are implemented is dynamic. As a result, conventional evaluation models involving comparison sites are difficult to implement.4
The proposed analyses in the current DFC National Evaluation Plan allow for the expansion of previous analysis to include a far greater range of hypotheses concerning the coalition characteristics that contribute to stronger outputs, stronger coalition outcomes, and ultimately, stronger community outcomes. These exploratory and confirmatory analyses are designed to take advantage of the evaluation opportunities provided by the large sample and comprehensive process and outcome data collected by the National Evaluation. The addition of qualitative information collected in site visits will contribute to development of hypotheses for full sample testing, the interpretation of quantitative findings, and the provision of clear examples and guides for application of evidence-informed practice. There are four major features of our analytic approach:
First, the current approach to the evaluation will systematically deconstruct encompassing measures (e.g., maturation stages) currently abundant in the coalition evaluation literature into more specific constructs that are directly applicable to practice. For example, measurement will focus on discrete components of coalition structure and function such as membership, organizational structure, decision making, communication, collaboration strategies, intervention strategies and the use of data. These are strategies and functions that coalitions must perform, and that define their capacity. This will provide measures of multiple coalition characteristics that may differentiate coalitions, may be important to producing effective coalitions, and may operate differently across different settings and in different coalition systems. Exploratory clustering and latent structure analysis techniques will be used to ground and specify constructs from the literature, and to identify measures that emerge from the experience of DFC coalitions.
Second, the evaluation will use a natural variation approach. This approach examines the naturally occurring differences or variations in coalitions’ organization, function, procedures and management strategies, with the intent to provide concrete, evidence-informed lessons on how to construct effective coalitions in diverse settings. The natural variation approach has specific implications for data measurement and analysis:
The approach is based in recognition that coalitions operate in diverse communities with different problem and capacity contexts; have different membership, structures, and procedures; and face multiple options for prevention strategy. Our natural variation approach uses logic models to organize and guide measures of multiple relevant characteristics of coalition context and process, intervention design, program implementation, and community and population outcomes (see Attachment 3, Internal Logic Model, Measurement Mapping, and Brief Literature Review).
The large array of organized measures potentially, relevant contributors to, and constraints on coalition success supports exploratory and confirmatory analyses of the factors that are associated with coalition success across (or within) differences in community context, or for different outcome objectives. The large number of DFC coalitions provides a strong opportunity to learn from their documented experience, and will provide findings with strong external validity. These are strong advantages in the development of evidence-based practices.
Third, the evaluation uses a multi-method approach to data collection, measurement and analysis. Data collection includes program records and observations (COMET); key informant self-report and assessment (CCT); field research including interviews, focus groups and record review (site visits); and secondary sources (community context, YRBS comparisons). Analysis of these multiple sources includes mixed-method measurement (e.g., categorical coding of records or comments, inductive identification of programs clusters or types, constructed variables using multiple data sources) that grounds concepts in coalition experience. Analyses may also mix approaches, such as testing the strength of the relation of specific process and outcome variables across diverse sites, testing differences between identifiable types of coalition strategy (e.g., institutional or grass-roots focus) or intervention strategy; or identifying practices most suited to particular community contexts (e.g., differences in urbanicity). The multi-method approach will allow for different “sub-studies” within the larger DFC National Evaluation and will provide opportunities for building strong guides for application of evaluation findings.
Fourth, the statistical analyses of COMET and CCT will be enhanced by using qualitative information from purposively selected site visits. For example, sites may be selected to explore how coalitions that were successful in achieving broad, active collaboration accomplished this in diverse community settings. These analyses may document that sites in small communities succeeded with collaboration strategies different than those in larger communities. This difference is only illustrative in the small number of sites being visited, but can suggest hypotheses that can be confirmed or refuted across the full sample of DFC coalitions. On the other hand, the site-visit documentation of practices found to be promising in the larger study analyses can improve understanding of how to implement them in real world settings.
The natural variation, multi-method approach proposed for the National Evaluation also has strong advantages in compensating for the limitations of individual methods and design elements in the study. For example:
The relatively small number of site visits, and the purposive selection of coalitions, means that findings from site visits are not systematically representative of the full population of coalitions. By carefully coordinating the content (e.g., site visit protocols reflect the constructs in the internal logic model upon which COMET and CCT measures are mapped), the site visit information can be linked to the representative findings from COMET and CCT through (a) generating hypotheses to be tested, or (b) providing examples of how findings from COMET and CCT analyses apply in sites.
The natural variation approach to measurement uses multiple-item latent structures, constructed variables, and re-categorized response items to build multiple-item measures of logic model constructs that will reduce the impact of measurement error that is to some extent inevitable in the COMET and CCT components that depend on self-report of organizational processes and accomplishments. (For a complete discussion of data collection and processing actions taken to minimize data error, see B2 and B3 below.)
The DFC coalition sample is a census of DFC grantees, is large, and is therefore not a probability sample of the larger population of community coalitions. It follows that statistical significance tests are at best a rule of thumb criteria for estimating the likelihood that findings may be attributable to method error or insensitivity. The natural variation approach provides alternative information for estimating confidence that findings are not attributable to method error or insensitivity by identifying patterns of strong descriptive findings that are consistent with program hypotheses derived from the logic model and exploratory investigation. To support this logic the evaluation team uses change score effect sizes, analysis of the consistency of relations across multiple measures and contexts, and assessments of the relation of the descriptive magnitude of associations or contrasts against the background of inter-relations in the data. Where feasible, findings are stated in ways that support the assessment of face valid performance metrics (e.g., numbers of persons benefitting, potential cost benefits, potential reduction of negative consequences).
In summary, the evaluation approach is grounded in an awareness of the measurement and analysis opportunities and limitations inherent to the sample and data collection environment. While data, measurement, and analysis limitations will be fully acknowledged in reporting findings, the overall design fully utilizes alternative opportunities in the DFC data and design to mitigate and compensate for these limitations.
In summary, by better understanding the DFC Program and its mechanisms for contributing to positive change, the National Evaluation can deliver an effective, efficient, and sensitive set of analyses that will meet the needs of the program at the highest level, while also advancing prevention science.
The table below provides a summary of select stakeholder groups (column 1), selected examples of research questions relevant to each stakeholder group (column 2), the products through which findings and lessons can be communicated to each group in a useful way (column 3), and a preliminary identification of analysis methods that the evaluation will support (column 4). A more detailed description of the evaluation plan and products anticipated from the evaluation are provided in Attachment 2.
Exhibit 4. Key Evaluation Questions, Products, and Analytic Methods Relevant
for Each Stakeholder Group
Stakeholder |
Key Research Questions |
Products |
Methods |
|
ONDCP and SAMHSA |
|
|
|
|
Leaders of Coalitions |
|
|
|
|
Schools |
|
|
|
|
Local Governments |
|
|
|
|
Law Enforcement |
|
|
|
|
Social Service Agencies |
|
|
|
|
Judicial Agencies |
|
|
|
|
* A new core measure (peer disapproval) and new core substance (prescription drugs) are part of the requested revisions to COMET. Age of first use will be dropped as a core measure.
A.17. Reason(s) Display of OMB Expiration Date is Inappropriate
No exemption from displaying the expiration date is requested.
A.18. Exceptions to Certification for Paperwork Reduction Act Submissions
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.
1 A memo entitled, “Social Media, Web-Based Interactive Technologies, and the Paperwork Reduction Act” from OMB dated April 7, 2010 provides for the use of social media for obtaining such feedback as described here. The evaluation team established the social media website in response to this memo, and the data collection proceeded in concert with the guidance established in the document. The social media website was found to be a highly efficient mode of obtaining a significant amount of feedback in a short time period.
2 Brounstein, P. & Zweig, J. (1999). Understanding Substance Abuse Prevention Toward the 21st Century: A Primer on Effective Programs. Washington, DC: U.S. Department of Health and Human Services.
3 Wolf, T. (2001). Community Coalition Building–Contemporary Practice and Research: Introduction. American Journal of Community Psychology, 29(2), 165-172.
4 Gruenewald, P.J. (1997). Analysis Approaches to Community Evaluation. Evaluation Review, 21(2), 209-230.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Joanne and Shukri Abed |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |