SUPPORTING STATEMENT
Part B
Evaluation of the Second Decade Project Community Planning Guide
August 23, 2016
Office of the Assistant Secretary for Health (OASH)
U.S. Department of Health and Human Services
B. Collection of Information Employing Statistical Methods 3
1. Respondent universe and sampling methods 3
2. Information Collection Procedures 3
The Office of the Assistant Secretary for Health (OASH) of the U.S. Department of Health and Human Services (HHS) is conducting an evaluation of their Second Decade Project Community Planning Guide (“the Guide). Evaluation participants are community leaders, coalition members, and secondary stakeholders (adolescent health experts and state/local health department officials) identified by a point of contact in each of the five communities piloting the Guide. Therefore, sampling methods are not applicable. An estimated 430 respondents will be asked to participate.
Information collection procedures for the Evaluation are described below.
Step 1: Obtain 1) a list of coalition members and community leaders from points of contact in each of the five pilot communities and 2) a list of secondary stakeholders from OASH to determine the appropriate respondents.
Step 2: Confirm contact information for respondents.
Step 3: Disseminate a letter from OASH (Attachment A) that describes the Evaluation sponsorship and purpose, explains how respondents were identified, invites participation in the appropriate data collection activity (i.e., the Community Leader Interview, the Coalition Member Focus Group, the Coalition Assessment Survey, or the Secondary Stakeholder Survey) and encourages cooperation when ICF staff make contact to arrange a telephone interview or focus group, or to provide the online survey link.
Step 4: Schedule interviews and focus groups. This may involve multiple phone calls and emails; up to 4 attempts for each data collection activity are planned.
Step 5: Conduct interviews and focus groups. Interviewers and focus group facilitators will record each interview and focus group with the consent of respondents.
Step 6: Disseminate via email a link to the two Web-based surveys (Coalition Assessment Survey and Secondary Stakeholder Survey) to the appropriate respondents. The Web-based surveys will be open for 30 days. We will send weekly reminder emails to participants for up to 4 weeks.
However, each pilot community organization will receive a $500 stipend at the beginning and at the end of data collection, for a total of $1000. The stipend is provided to the pilot community for participating in the in-depth review of the Guide; and for time spent identifying and working with community members on the pilot project. From our experience, communities are more likely to participate in evaluation activities if provided a stipend.
Handling the data
All qualitative data collected will be managed and analyzed using ATLAS.ti 7. Community Leader Interviews and Coalition Member Focus Groups transcripts will be imported into ATLAS.ti 7. A codebook will be developed for each data collection activity. Codebooks will be developed according to the themes of each to help ensure coded data will always be directly related to a specific evaluation question or questions. Additional codes will be added to the codebook, included codes related to cross-cutting themes and other patterns that emerge as part of the data collection and coding process. The codebook will include also code definitions, and inclusion and/or exclusion criteria.
Three ICF team members will code transcripts. Interrater reliability will be established by assigning all coders the same transcription to code, and then comparing both the decisions on which codes were selected for different segments of text and where each coder drew the boundary of the text segment. An interrater agreement rate above 80% is required before coders are allowed to code transcripts independently, and an interrater test will be conducted for 10% of all transcripts. After all transcripts are coded, the evaluation team will be able to create reports of all text segments related to each evaluation question, and by site. Data queries will be run primarily through Boolean searches.
A Microsoft Access database will be used to centrally store quantitative data, including progress tracking documentation. This data management system will include a data dictionary detailing the field names, data types, and data sources. Consistent site identifiers will be utilized in the database relational tables to allow for synthesis and reporting across evaluation process and data collection tools. Quantitative survey data (e.g. Coalition Assessment and Secondary Stakeholder Surveys) collected through a secured, online web portal (Survey Monkey) will be downloaded and imported into the Microsoft Access database. The data file will be maintained in a secured server file to maintain file integrity and data security.
The quantitative data analysis will be conducted with SPSS version 22.0 to describe data trends across the pilot communities. Descriptive statistics, including frequencies, percentages, measures of central tendency (means, medians, and modes), and measures of dispersion (range, standard deviation), will assess variation across sites regarding coalition attitudes, coalition adolescent health activities, and community context (secondary data). Data will be standardized (e.g. per capita estimates, per 10,000 residents rates) to allow comparisons across pilot sites. Bivariate analyses (e.g. chi-square, analysis of variance) will be employed to examine the relationships between coalition characteristics, community contexts, and evaluation outcomes (i.e. plan comprehensiveness, plan quality).
Using a convergent mixed-method framework, quantitative and qualitative data will be integrated after their initial analysis to improve interpretation of the results. Qualitative data theme frequency and proportion, particularly a priori themes related to utility, quality, and appropriateness of the Guide, will be aligned with quantitative survey and secondary data. For example, analysis of community coalitions will utilize both existing data from document reviews and coalition perspectives from survey and focus groups to identify alignment in community plans. The convergence or divergence of trends across the data will be explored.
Comparative analysis of integrated data will occur with a primary team member identifying trends, outliers, and key thematic variance in the datasets. The analytic team will review the findings to determine consensus. Integrated reporting may include joint displays, data matrices presenting quantitative and qualitative findings by major themes, highlighting data trends.
Participation in this evaluation is voluntary, and potential respondents may decline to participate. However, as communities were selected by Regional Health Authorities from OASH to pilot the Guide and participate in the evaluation, there will be a strong incentive to participate. Likewise, the points of contact in each community will help encourage participation among coalition members and community leaders.
The letter from OASH described in Step 3 of the Information Collection Procedures above will encourage participation.
Interviewers and focus group facilitators will follow up via multiple routes (e.g., telephone and email) to reach respondents who are initially unresponsive. They will accommodate difficult schedules by offering to conduct interviews and focus groups outside of regular business hours. Focus groups will be conducted using Adobe Connect software, allowing participants to conveniently participate from their home or office. Survey participants will receive a reminder email to prompt their completion of the survey.
To minimize burden on respondents and make the process more efficient for them, respondents will be given a list of interview topics (Attachments C and E) in advance of the interview and focus groups. This will enable them to gather information, and assess and identify any questions they might have, prior to the interview.
ICF has conducted numerous evaluations similar to this proposed OASH evaluation and is using well established standard best practices for data collection. We will be conducting a pretest of our protocols with a sample of 6 respondents to ensure our methods will collect meaningful information in an efficient matter.
ICF has full responsibility for the development of the overall statistical design, and assumes oversight responsibility for data collection, monitoring and analysis. The individuals responsible for overseeing data collection and analysis are:
Gingi Pica, MPH
ICF Macro, Inc.
40 Wall Street, Fl. 34
New York, NY 10005
(212) 941-5555
Phyllis Ottley, PhD
ICF Macro, Inc.
3 Corporate Square, NE
Suite 370
Atlanta, GA 30329
(404) 321-3211
Donoria Evans, MPH
ICF Macro, Inc.
3 Corporate Square, NE
Suite 370
Atlanta, GA 30329
(404) 321-3211
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | SUPPORTING STATEMENT |
| Author | wcarroll |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |